California Attorney General Rob Bonta has issued a cease and desist letter to xAI, Elon Musk’s company, demanding it stop creating and distributing nonconsensual sexual images. Bonta highlighted numerous cases where xAI transformed ordinary, clothed photos of women and children into sexually explicit or “undressed” depictions without consent, violating state laws including a recent “deepfake” pornography ban. While noting xAI’s recent actions, such as limiting Grok AI’s use in regions where generating bikini images is illegal, Bonta questioned the effectiveness of these measures. Bonta’s letter followed an announcement of an investigation into whether xAI and its social platform X breached laws by enabling nonconsensual sexual image dissemination through Grok’s new image-editing feature launched last month. He condemned the surge of explicit content produced by xAI as harassing and urged immediate corrective steps. Californians affected can report violations via the attorney general’s website. xAI has yet to comment on the probe. Research from Bloomberg found that Grok users posted more nonconsensual sexual images than any other website. In response, Musk promised “consequences” for offenders, and Grok restricted image editing to paying users. Legal experts, including former LA sex crimes attorney Sam Dordulian, say xAI and X are likely violating California’s AB 621 deepfake porn law. Assemblymember Rebecca Bauer-Kahan, the law’s author, emphasized that the law targets exactly these abuses, particularly the manipulation of women and children’s images into explicit content, causing severe psychological and reputational harm. This investigation aligns with broader global concerns, including calls from Governor Gavin Newsom, regulatory backlash in the EU and India, and outright bans of X in several countries. As Grok app downloads surge, lawmakers urge Apple and Google to ban the app. The reasoning behind Grok’s design and its response to controversy remain unclear, with analyses labeling it among the least transparent AI systems. Evidence of harm from deepfakes mounts: the FBI warns of rising extortion cases involving young people leading to self-harm and suicide; child sexual abuse material has been found embedded in AI training data; and surveys show many high school students have encountered sexually explicit images of peers.

Bonta’s investigation follows prior efforts to safeguard minors from harmful AI content, including endorsing legislation restricting chatbots from sexually explicit or self-harm discussions with underage users, and joining multistate letters pressuring companies like Meta and OpenAI regarding inappropriate chatbot interactions with minors. Since 2019, California has enacted multiple laws protecting individuals from deepfakes. AB 621, recently strengthened, allows prosecutors to sue companies that recklessly aid the spread of deepfake pornography without consent, even without proof of actual harm to victims. It raises potential damages to $250, 000 and fines noncompliant websites $25, 000 per violation if they fail to act within 30 days. Additional 2024 laws expanded the state’s definition of child pornography to include AI-generated materials and require prompt removal options on social media, treating such deepfakes as digital identity theft. However, a law restricting deepfakes in elections was struck down following a lawsuit by X and Musk. Legal experts note that while new laws open avenues for redress, challenges remain. Proof of distribution, required for many protections, can be difficult since users of “nudify” apps often know each other and don’t widely share images. Dordulian cited a case of a nanny whose employer created manipulated images from her Instagram photos causing emotional trauma but lacking clear legal recourse under deepfake laws. Victims have reported similar gaps, where the act of creation alone remains unregulated. Advocates like Jennifer Gibson of Psst, which offers pro bono legal support to AI whistleblowers, argue existing laws fall short of protecting insiders who expose harmful AI practices. A recent California whistleblower law covers only catastrophic risks causing massive deaths or damage, excluding deepfake-related harms. Had this law been broader, former X employees who reported Grok generating illegal sexual content might have had stronger protections for reporting wrongdoing to authorities. In summary, California leads in deepfake legislation but current laws and protections need expansion and improvement to effectively address the evolving risks posed by AI-driven sexual image manipulation, protect victims, and encourage internal accountability.