From Face-Swap Novelty to Nonconsensual Abuse: What San Francisco Is Challenging
AI nudify apps are mobile tools that use face-swapping and image-generation models to create realistic, nonconsensual nude or sexually explicit images of real people, often by placing their faces onto pornographic bodies, turning what looks like harmless photo editing software into a mechanism for sexual abuse and harassment at scale. On Thursday, San Francisco city attorney David Chiu sent cease-and-desist letters to Apple and Google demanding removal of 13 such face-swap apps from their stores, eight from the Apple App Store and five from Google Play, because they enable AI-generated nonconsensual nude images. This is not a symbolic warning; the letters accuse both companies of “aiding and abetting” illegal deepfake pornography by hosting the apps and taking a cut of in-app payments. The message is blunt: nudify apps removal is now a legal obligation, not just a moderation choice.
App Store Moderation Has Failed Women and Girls
Chiu’s move is a direct indictment of how app store moderation has handled AI deepfake enforcement so far. These programs are marketed as playful face-swapping tools, yet investigators found they can paste people’s faces onto explicit images or generate fake nude content that overwhelmingly targets women and girls. “Generating non-consensual intimate images is illegal, harmful, and completely unacceptable,” Chiu said, arguing that Apple and Google have likely made millions of dollars in fees from nudification apps that amount to sexual abuse infrastructure. Both platforms already claim to prohibit pornography, abuse, and harassment, yet researchers keep finding AI sexual-image generators in their stores—including some rated suitable for children. When an ecosystem contains around 100 nudify-capable apps generating about USD 120 million (approx. RM552 million) in revenue and 480 million downloads, we are past the point of isolated oversight errors; we are seeing a systematic failure to treat nonconsensual deepfakes as a predictable, preventable harm.
Why Enforcement at the Distribution Layer Matters
San Francisco’s legal strategy is to shift responsibility upstream, from shady app developers to the platforms that distribute and monetize them. The city’s letters argue that California law prohibits supporting services that create deepfake pornography and that Apple and Google are aiding and abetting the sale of explicit deepfake images by hosting nudify apps and processing payments. This is the crucial pivot: AI deepfake enforcement is no longer framed only as content moderation on social networks, but as a duty at the app distribution level. The city has given both companies 28 days to explain how they will comply, warning that civil enforcement could follow with penalties of at least USD 25,000 (approx. RM115,000) per violation if they fail to respond. The demands go further, urging them to cut payment-processing ties and introduce recurring reviews to stop rebranded replacement apps from slipping back in. In short, San Francisco is saying that if you run the store, you own the risks of what you sell.
Apple and Google’s Reactive Response Shows the Limits of Current Rules
Under pressure, both companies point to policies that, on paper, already ban this behavior. Google says its store does not allow apps containing sexual content, that it has removed the five Android apps flagged by Chiu’s office, suspended “hundreds” of violating apps, and even restricted search terms like “nudify” on Google Play. Apple stresses that developers are responsible for their content, noting that it has removed three of the flagged apps and is terminating their developer accounts while forcing four others to fix policy violations or face removal. Yet both admit they previously allowed these tools through review, despite rules against pornography and abusive sexual content. That is the uncomfortable truth: app store moderation remains mostly reactive, dependent on researchers, journalists, and now prosecutors to flag harms after the damage is done. When studies find that 70% of tested face-swap apps allow nude face pasting without safeguards, claiming that guidelines alone protect users rings hollow.
The Future of Platform Accountability in the Age of AI Deepfakes
San Francisco’s campaign fits into a wider legal push to treat deepfake nudification as a structural problem rather than isolated bad actors. The same city attorney has already sued 16 popular AI undressing websites that attracted 200 million visits in half a year, while federal authorities recently seized two sites accused of publishing hundreds of thousands of deepfake nude images of famous women. This trend suggests that app stores and payment processors will increasingly be seen as central enforcement chokepoints. Chiu has warned that his office will consider further legal action if Apple and Google do not become more proactive. For the platforms, the choice is stark: either invest in serious, recurring AI deepfake enforcement—especially around nudify apps removal—or accept being treated as co-responsible for the sexual abuse they enable. For everyone else, the outcome will signal whether large tech companies can still hide behind policy language, or whether the law will finally force them to design distribution systems that put consent and safety ahead of growth.
