San Francisco’s Legal Shock To The App Store System
San Francisco’s cease-and-desist orders against AI “nudify” apps are a legal attempt to force Apple and Google to stop distributing and profiting from tools that turn ordinary photos into nonconsensual nude images, exposing a deeper failure of app store moderation and AI abuse prevention that leaves women and girls especially vulnerable to image-based sexual exploitation. On Thursday, city attorney David Chiu ordered the nudify apps removal of 13 services—eight on Apple’s App Store and five on Google Play—that use artificial intelligence to generate fake intimate images without consent. He gave both tech giants 28 days to comply before facing potential civil penalties under a recent California law aimed at non-consensual deepfake pornography. This is not a routine enforcement action; it is a direct challenge to the idea that app stores can stay neutral while taking a cut from AI-powered abuse tools.
The Human Cost: Nonconsensual Images AI And Sexual Abuse
Behind the legal language is a brutal reality: these apps use AI to strip clothes from photos, creating fabricated nude or sexually suggestive images that can be used to harass, exploit, and intimidate victims, especially women and girls. The companies earn fees through in-app payments, meaning they share in revenue from this abuse. “Generating non-consensual intimate images is illegal, harmful, and completely unacceptable,” city attorney David Chiu said. Researchers have repeatedly found nudify apps in mainstream app stores, some even rated as suitable for children, despite existing bans on pornography, abuse, and harassment. That gap between policy and practice is the real scandal. For ordinary users, the threat is no longer limited to shady websites; it is embedded in the same app ecosystems they trust for banking, messaging, and games, turning app stores into distribution channels for image-based sexual abuse.
App Store Moderation Is Failing AI Abuse Prevention
Apple and Google both claim to prohibit pornographic and abusive content and say they have removed dozens of nudify and deepfake apps when researchers or journalists flag them. Google notes it has already deleted “hundreds” of apps with nudifying features for policy violations, including the five Android apps cited by Chiu. Yet San Francisco’s move rests on a blunt judgment: existing safeguards are not preventing harmful apps from repeatedly slipping through review and reappearing in different guises. The core failure is reactive app store moderation. Policies look strict on paper, but enforcement depends on external reporting and slow case-by-case takedowns. Simply publishing acceptable-use rules no longer convinces regulators when illegal, nonconsensual images AI services keep resurfacing. By threatening deepfake enforcement against the platforms themselves, the city is insisting that Apple and Google act as active gatekeepers, not passive marketplaces.
From Former App Store Gatekeeper To AI Likeness Governance
The industry already has a clue about how serious this problem is: one of the architects of Apple’s app store rules now spends his time building systems to govern AI likeness use. As Senior Director of App Store Review from 2009 to 2016, Phillip Shoemaker grew the review team from four to more than 300 people and wrote many of the guidelines that determined what could be published. After leaving, he discovered that likeness itself—distinct from legal identity—had no governance infrastructure. His new venture, PersonaShield, helps creators, influencers, and public figures monitor unauthorized AI likeness use, automate takedowns, and earn revenue from licensed AI-generated images. That shift from app review to AI likeness governance shows a recognition that traditional moderation tools are not enough. The problem is no longer a few bad apps; it is an ecosystem where anyone’s face can be weaponized at scale.
What Deepfake Enforcement Should Demand From Platforms Next
San Francisco has given Apple and Google 28 days to remediate the nudify apps removal before considering further legal action. The letters warn that continued violations could trigger significant civil penalties under California law and may shape broader debates about platform responsibility for harmful AI services. Enterprises that build image-generation systems, developer APIs, and cloud AI products can no longer assume that only the developer’s intent matters; regulators are signaling that distribution and monetization channels share accountability. For victims, these steps will not erase nonconsensual images already created or completely deter determined abusers. But they mark an important shift: app stores are being told that nudify apps are not edge cases, they are foreseeable abuse. If platforms want to keep their gatekeeper role, they must invest in proactive AI abuse prevention and tight deepfake enforcement—before the next scandal forces lawmakers to do it for them.
