Elon Musk restricts Grok’s image tools following a wave of non-consensual deepfakes of women and children

Elon Musk’s xAI has restricted its AI chatbot Grok’s image generation capabilities to paying subscribers only, following widespread condemnation over its use to create non-consensual sexualized images of real women and children.

“Image generation and editing are currently limited to paying subscribers,” Grok announced via X on Friday. The restriction means the vast majority of users can no longer access the feature. Paying, verified subscribers with credit card details on file can still do so, but theoretically they can be identified more easily if the function is misused.

Some experts aren’t convinced that the new restrictions will fix the now widespread problem.

“The argument that providing user details and payment methods will help identify perpetrators also isn’t convincing, given how easy it is to provide false info and use temporary payment methods,” Henry Ajder, a UK-based deepfakes expert, told Fortune. “The logic here is also reactive: it is supposed to help identify offenders after content has been generated, but it doesn’t represent any alignment or meaningful limitations to the model itself.”

X did not immediately respond to Fortune’s request for comment, xAI responded with the automated message: “Legacy Media Lies.”

Over the past week real women have been targeted at scale with users manipulating photos to remove clothing, place subjects in bikinis, or position them in sexually explicit scenarios without their consent. Some victims reported feeling violated and disturbed by the trend, with many saying their reports to X went unanswered and images remained live on the platform.

Researchers said the scale at which Grok was producing and sharing images was unprecedented as, unlike other AI bots, Grok essentially has a built-in distribution system in the X platform. 

One researcher, whose analysis was published by Bloomberg, estimated that X has become the most prolific site for deepfakes over the last week. Genevieve Oh, a social media and deepfake researcher who conducted a 24-hour analysis of images the @Grok account posted to X, found that the chatbot was producing roughly 6,700 sexually suggestive or nudifying images per hour. By comparison, the five other leading websites for sexualized deepfakes averaged 79 new AI undressing images hourly during the same period. Oh’s research also found that sexualized content dominated Grok’s output, accounting for 85% of all images the chatbot generated.

Ashley St. Clair, a conservative commentator and mother of one of Musk’s children, was among those affected by the images. St. Clair told Fortune that users were turning images on her X profile into explicit AI-generated photos of her, including some she said depicted her as a minor. After speaking out against the images and raising concerns about deepfakes on minors, St Clair also said X took away her verified, paying subscribers status without notifying her or refunding her for the $8 per month fee.

“Restricting it to the paid-only user shows that they’re going to double down on this, placing an undue burden on the victims to report to law enforcement and law enforcement to use their resources to track these people down,” Ashley St Clair said of the recent restrictions. “It’s also a money grab.”

St Clair said that many of the accounts targeting her were already verified users: “It’s not effective at all,” she said. “This is just in anticipation of more law enforcement inquiries regarding Grok image generation.”

Regulatory pressure

The move to limit Grok’s capabilities comes amid mounting pressure from regulators worldwide. In the U.K., Prime Minister Keir Starmer has indicated he is open to banning the platform entirely, describing the content as “disgraceful” and “disgusting.” Regulators in India, Malaysia, and France have also launched investigations or probes.

The European Commission on Thursday ordered X to preserve all internal documents and data related to Grok, stepping up its investigation into the platform’s content moderation practices after describing the spread of nonconsensual sexually explicit deepfakes as “illegal,” “appalling,” and “disgusting.”

Experts say the new restrictions may not satisfy regulators’ concerns: “This approach is a blunt instrument that doesn’t address the root of the problem with Grok’s alignment and likely won’t cut it with regulators,” Ajder said. “Limiting functionality to paying users will not stop the generation of this content; a month’s subscription is not a robust solution.”

In the U.S., the situation is also likely to test existing laws, like Section 230 of the Communications Decency Act, which shields online providers from liability for content created by users.

Riana Pfefferkorn of Stanford’s Institute for Human-Centered Artificial Intelligence previously told Fortune that liability surrounding AI-generated images is murky. “We have this situation where for the first time, it is the platform itself that is at scale generating non-consensual pornography of adults and minors alike,” she said. “From a liability perspective as well as a PR perspective, the CSAM laws pose the biggest potential liability risk here.”

Musk has previously stated that “anyone using Grok to make illegal content will suffer the same consequences as if they upload illegal content.” However, it remains unclear how accounts will be held accountable.

This story was originally featured on Fortune.com