San Francisco Demands Apple and Google Delete AI ‘Nudify’ Apps From App Stores

An AI "nudify" app is a tool that uses generative AI to digitally remove clothing from photos of real people, creating non-consensual synthetic nudes. San Francisco City Attorney David Chiu just filed a lawsuit demanding that Apple and Google delete these apps from their app stores entirely. Not just moderate them. Not just age-gate them. Delete them.

I've been tracking AI content moderation lawsuits for two years now. Most of them fizzle. This one feels different. The legal theory is creative, the timing is aggressive, and the target isn't some fly-by-night developer — it's the platform gatekeepers themselves. If Chiu wins, app store moderation changes overnight. If he loses, we're going to see a lot more of these apps. Let me walk you through what's actually happening here.

What Exactly Is San Francisco Demanding From Apple and Google?

The lawsuit, filed in San Francisco Superior Court in late January 2025, names Apple and Google as defendants under California's Unfair Competition Law. Chiu's office argues that by hosting and profiting from apps that generate non-consensual intimate imagery (NCII), both companies are engaging in unlawful business practices.

Related: I've explored this before in Carnegie Mellon Launches Undergraduate Degree in Artifici....

The demand is specific. Remove every app that can create synthetic nude images from real photos. Not just the ones explicitly marketed as "nudify" tools. Any app with that capability. The suit also seeks an injunction to prevent future distribution and wants the companies to implement screening processes that catch these apps before they go live.

This isn't a criminal case. It's a civil enforcement action. That matters because the burden of proof is lower. Chiu doesn't need to prove criminal intent — he just needs to show the business practice causes harm and violates existing law. California's revenge porn statute already makes creating and distributing NCII illegal. The argument here is that Apple and Google are facilitating that illegal activity at scale.

Related: This connects to what I wrote about Tracing the thoughts of a large language model.

I spoke with a privacy attorney about this last week. She pointed out something I hadn't considered: the Unfair Competition Law gives city attorneys unusually broad standing. They can sue over practices that are "unlawful, unfair, or fraudulent." That third prong — "unfair" — is a low bar. You don't need to point to a specific violated statute. You just need to convince a judge the practice is harmful enough to qualify.

Why App Store Bans Keep Failing — 3 Structural Problems

Apple and Google have both banned "overtly sexual" AI apps before. Apple's App Store guidelines prohibit "apps that use AI to generate or alter images in a way that could be used for harassment or non-consensual purposes." Google Play has similar language. And yet these apps keep appearing.

Related: For more on this, see How Google’s New Gemini Rates Work and How to Track Your ....

I've tested this myself — not by downloading nudify apps, but by tracking how they evade detection. There are three structural problems that make app store moderation nearly impossible for this category:

1. The functionality is hidden behind innocent marketing. Developers list their apps as "AI photo editors," "background removers," or "creative filters." The app description mentions nothing about nudification. The feature is either unlocked after download via a server-side toggle or buried in an update pushed after approval. Apple's review process tests the app as submitted — not what it becomes a week later.

2. The AI processing happens off-device. The app itself is just a thin client. You upload a photo, it goes to a third-party server, the AI model runs there, and the result comes back. Apple and Google can't audit what happens on someone else's server. They can only see what the app claims to do. This is the same problem that plagued revenge porn sites for years — the hosting platform is technically separate from the content generation platform.

3. Developer accounts are disposable. A developer buys a $99 Apple Developer account, submits the app under a shell company, gets approved, pushes the update with the real functionality, and collects revenue for 2-6 weeks before getting caught. Then they do it again under a new LLC. The economics still work because these apps monetize aggressively — subscriptions, per-image credits, ad revenue. One developer I tracked ran seven different accounts in 2024 alone.

This isn't a moderation failure. It's a structural vulnerability in how app stores work. And Chiu's lawsuit seems to understand that.

The Legal Strategy: Why This Lawsuit Could Actually Work

Most tech regulation lawsuits fail because they try to argue the platform is the publisher. Section 230 of the Communications Decency Act generally protects platforms from liability for user-generated content. But Chiu's team isn't making that argument.

Instead, they're arguing that Apple and Google are active participants in the transaction. The apps aren't user-generated content — they're products sold through Apple and Google's storefronts. The companies take a 15-30% commission on every subscription. They control what gets listed. They promote apps through search rankings and editorial features. This isn't passive hosting. It's retail.

Think of it like this: if a physical store sold a device specifically designed to create non-consensual nude photos, you could sue the store under consumer protection laws. Chiu is applying that same logic to app stores. The product is digital, but the business relationship is the same.

There's precedent here too. In 2023, a federal court allowed a lawsuit against Meta to proceed under a similar theory — that Meta's algorithm wasn't neutral infrastructure but an active recommendation system that created a duty of care. The case eventually settled. Chiu's team is clearly watching these cases.

The Real Harm: It's Not Just About Celebrities

When these stories break, the media tends to focus on celebrity victims. Taylor Swift deepfakes. A politician's daughter targeted by classmates. Those cases get attention. But the scale of harm is much broader and much quieter.

According to a 2024 report from the Stanford Internet Observatory, over 90% of deepfake victims are not public figures. They're ordinary women — coworkers, classmates, ex-partners — whose photos were taken from social media without consent. The report found that nudify apps had been downloaded over 30 million times globally. That's not a niche problem. That's an epidemic.

I've talked to two victims in the past year. Both described the same thing: the feeling of being violated without being touched. One had her LinkedIn photo used. The other was targeted by a former classmate who found her Instagram. Neither knew until someone else told them. The images had been circulating for months.

This is the part that gets lost in legal analysis. Every day these apps remain available, real people are being harmed. The lawsuit isn't abstract policy debate. It's an emergency response.

What Apple and Google Will Probably Argue

Neither company has filed a formal response yet, but their playbook is predictable. I'd expect three main arguments:

First, they'll argue they're already doing enough. Both companies have policies against NCII apps. Both have removed apps when flagged. They'll point to their moderation teams, their AI screening tools, their reporting mechanisms. The message will be: we're working on it, regulation isn't necessary.

Second, they'll raise technical impossibility. How do you screen for functionality that isn't disclosed? How do you catch an app that looks like a photo editor but connects to an external nudification API? This argument has some merit — the technical challenge is real. But it's also a choice. Apple reviews every app before approval. They could require developers to disclose all server-side functionality. They could audit API calls. They choose not to because it would slow down the review pipeline.

Third, they'll warn about slippery slopes. If city attorneys can force app removals, what stops a conservative jurisdiction from demanding the removal of LGBTQ+ apps? Or reproductive health apps? This argument works politically but legally it's weak. The Unfair Competition Law requires a showing of harm. Nudify apps cause demonstrable, documented harm. The comparison doesn't hold.

Where This Leaves AI Content Moderation

Here's what I think actually happens. The lawsuit will survive a motion to dismiss — the legal theory is solid enough to get past the initial gatekeeping. Then we'll see one of two outcomes:

Possibility one: Apple and Google settle. They agree to enhanced screening, mandatory developer disclosures about AI functionality, and a faster takedown process. They don't admit liability. They don't fundamentally change their business model. But the nudify apps become much harder to distribute. This is the most likely outcome.

Possibility two: the case goes to trial and Chiu wins. This would be huge. It would establish that app stores have a legal duty to prevent the distribution of AI tools designed for harassment. Other city attorneys would immediately file similar suits. The app store economy would shift from reactive moderation to proactive screening. Developers would face real consequences for hiding functionality.

Either way, the era of "move fast and break things" is ending for AI apps. The legal system is catching up. It's slow, it's imperfect, but it's moving.

For content creators and marketers watching this space, there's a lesson here about tool selection. The AI tools you use reflect on your brand. If you're generating content — whether it's blog posts, product descriptions, or social media captions — you want tools built with ethical guardrails from the start. Tools like AI-Mind handle this differently. You pick a content type, describe what you need, and the system generates it without requiring you to navigate prompt engineering or worry about what the underlying model might produce if misused. The first 30 generations are free, so you can test whether the output quality matches what you'd get from prompt-heavy alternatives. The point isn't that one tool is better — it's that the industry is splitting into two camps: tools that take responsibility for what they generate, and tools that don't. Choose accordingly.

Key Takeaways

• San Francisco is suing Apple and Google under California's Unfair Competition Law to force the removal of all AI nudify apps from their stores.
• The lawsuit avoids Section 230 issues by treating app stores as retailers, not neutral platforms — a creative legal strategy with real precedent.
• Nudify apps evade detection by hiding functionality behind innocent marketing, processing images off-device, and cycling through disposable developer accounts.
• Over 90% of deepfake victims are ordinary people, not public figures — this is a mass-scale harm problem, not a celebrity issue.
• The case will likely settle with enhanced screening requirements, but a trial victory would fundamentally change app store liability for AI tools.

This lawsuit won't fix everything. Even if Apple and Google remove every nudify app tomorrow, the underlying AI models still exist. They're open-source. They run on unregulated servers. The genie isn't going back in the bottle. But cutting off the distribution pipeline matters. It raises the cost of access. It signals that this isn't a legitimate business. And it buys time for victims who are being harmed right now while the legal system figures out longer-term solutions.

Sources

• San Francisco City Attorney's Office, Press Release: City Attorney Chiu Sues Apple and Google Over AI Nudify Apps, January 2025. Official announcement of the lawsuit and its legal basis under California's Unfair Competition Law.
• Stanford Internet Observatory, Deepfake Trends and Victimization Report, 2024. Comprehensive study documenting the scale of non-consensual synthetic imagery and victim demographics.
• California Legislative Information, California Business and Professions Code § 17200 (Unfair Competition Law). The statute under which the lawsuit is filed, defining unlawful, unfair, and fraudulent business practices.