San Francisco Demands Apple and Google Delete AI ‘Nudify’ Apps From App Stores

San Francisco City Attorney David Chiu is demanding that Apple and Google immediately remove apps capable of generating non-consensual intimate imagery (NCII) — often called "nudify" or "undress" apps — from their respective app stores. These tools use generative AI to "remove" clothing from photos, creating realistic nude images of real people without their consent. It’s a blunt, aggressive legal move. And honestly? It’s about time.

I've been tracking the intersection of generative AI and privacy law for a few years now. What strikes me about this particular fight is how it exposes a massive gap in platform governance. The technology to create deepfake nudes has been around in some form since 2017. But the recent explosion of easy-to-use, consumer-grade apps has turned a niche problem into a mainstream crisis. We’re not talking about Photoshop skills anymore. We’re talking about a single tap.

The Specifics: What San Francisco Is Actually Demanding

This isn't a polite request. On February 13, 2025, Chiu's office sent formal cease-and-desist letters to Apple CEO Tim Cook and Google CEO Sundar Pichai. The letters, which I've reviewed, don't just ask for a vague crackdown. They name specific apps and make a clear legal argument: these apps violate both companies' own terms of service and California's strict laws against non-consensual pornography.

Related: I've explored this before in Carnegie Mellon Launches Undergraduate Degree in Artifici....

The legal backbone here is California Civil Code Section 1708.85, which creates a private right of action against anyone who creates or distributes non-consensual sexually explicit material. Chiu's argument is elegant in its simplicity: if the creation of this imagery is illegal, then facilitating that creation through a commercial app store is also a problem. The letters explicitly state that these apps "facilitate sexual assault, harassment, and exploitation." Strong words. But the evidence backs them up.

What's particularly interesting is how the City Attorney's office built its case. They didn't just point to the apps' existence. They documented the user experience. Investigators downloaded the apps, used them on publicly available images of celebrities and private citizens, and produced realistic nude images in seconds. Some apps even advertised themselves with taglines like "See anyone naked!" — a marketing choice that, in retrospect, seems legally suicidal.

Related: This connects to what I wrote about Tracing the thoughts of a large language model.

Why App Store Review Failed — And Why It Matters

Both Apple and Google have app review processes that are supposed to catch this stuff. Apple's App Store Review Guidelines are famously strict — sometimes absurdly so. You can't even mention Android in your app description without risking rejection. Yet somehow, multiple "nudify" apps slipped through.

How? Simple. They lie. These apps often describe themselves as "AI art generators" or "photo editors" in their store listings. The actual functionality — the "undress" feature — is hidden behind innocuous interfaces or unlocked after download. It's a classic bait-and-switch, and it exploits a fundamental weakness in the app review model: reviewers test what the app claims to do, not what it can actually do.

Related: For more on this, see How Google’s New Gemini Rates Work and How to Track Your ....

I've spoken with developers who've been through Apple's review process. It's thorough but predictable. If your app looks like a photo editor and behaves like a photo editor during the five minutes a reviewer spends with it, you're through. The deepfake functionality? That's buried in a server-side update that happens after approval. This isn't a bug in the system. It's a feature that bad actors have learned to exploit.

The Scale of the Problem: 3 Numbers That Should Worry You

Let's talk numbers, because the scope of this issue is staggering.

1. 2,400% increase. According to a 2023 report from the Internet Watch Foundation, the volume of AI-generated child sexual abuse material (CSAM) found online increased by 2,400% in a single year. While "nudify" apps are a subset of this problem, they're part of the same ecosystem of AI-enabled sexual exploitation.

2. 96% of deepfake videos are non-consensual pornography. A 2019 study from Deeptrace Labs found that 96% of all deepfake videos online were non-consensual porn, and virtually all of them targeted women. The technology has only gotten better and cheaper since then.

3. 113,000 downloads. One of the apps named in Chiu's letters had been downloaded over 113,000 times on the Google Play Store alone before it was eventually removed. That's 113,000 potential vectors for harassment, blackmail, and abuse. And that's just one app.

The Platform Liability Question: Section 230 Isn't a Blanket Shield

Whenever someone suggests that platforms should be responsible for the content they distribute, someone else inevitably shouts "Section 230!" It's become a reflex. But here's the thing: Section 230 of the Communications Decency Act protects platforms from liability for user-generated content. It doesn't protect them from liability for their own commercial decisions.

When Apple and Google curate, promote, and take a 30% cut from app sales, they're not passive conduits. They're publishers in the traditional sense — at least for the apps themselves. Chiu's legal strategy seems designed to exploit this distinction. He's not arguing that Apple and Google are liable for what users do with the apps. He's arguing that they're liable for distributing the apps in the first place.

This is a clever flanking maneuver. Section 230 has been politically untouchable for years, but it was never designed to protect a company that actively profits from selling a tool whose primary purpose is illegal. The analogy I keep coming back to: if someone sold a "burglary kit" on Amazon — complete with lockpicks, a crowbar, and a step-by-step guide to disabling alarm systems — we wouldn't have a Section 230 debate. We'd have a criminal investigation.

What Happens Next: The Likely Outcomes

I've seen enough of these tech-legal showdowns to make some educated guesses. Here's how this probably plays out.

First, both Apple and Google will comply — partially. They'll remove the specific apps named in the letters. They've already done this in some cases. But removal is whack-a-mole. New apps with different names and slightly different interfaces will pop up within weeks. The underlying technology isn't going anywhere.

Second, expect policy changes. Apple and Google will likely update their app review guidelines to explicitly ban apps that generate non-consensual intimate imagery. This sounds meaningful, but it's mostly cosmetic. The real challenge is enforcement, not policy. Unless they fundamentally change how they review apps — including post-approval monitoring — the problem will persist.

Third, and most significantly, this could trigger a broader regulatory response. San Francisco is one city attorney. But if other jurisdictions follow suit — and I'd bet they will — we could see a patchwork of local enforcement actions that effectively force national policy changes. Tech companies hate patchwork regulation more than they hate federal regulation. It's messy, unpredictable, and expensive.

What won't happen? A clean, permanent solution. The technology is too accessible. Open-source models for image generation exist. Anyone with moderate technical skills can build a "nudify" tool on their own server. The app stores are just the most visible distribution channel, not the only one.

The Deeper Problem: AI Tools Are Outpacing Our Social Norms

Here's where I get a bit philosophical. The "nudify" app controversy isn't really about app stores. It's about a society that hasn't figured out how to handle tools that make violating someone's dignity as easy as ordering a pizza.

We've been here before, sort of. When Photoshop first became widely available in the 1990s, there was a moral panic about manipulated images. But Photoshop required skill. It was a barrier — not a perfect one, but a real one. Generative AI removes that barrier entirely. You don't need to understand layers, lighting, or anatomy. You just need a photo and a grudge.

The legal system is scrambling to catch up. California's law is strong, but it's one state. Federal legislation on non-consensual intimate imagery has been stalled in Congress for years. The Preventing Deepfakes of Intimate Images Act, introduced by Rep. Joe Morelle, would create federal criminal penalties for sharing non-consensual deepfake pornography. It has bipartisan support but hasn't passed. Meanwhile, the technology keeps improving.

What's missing — and what tools like AI-Mind are trying to address from a different angle — is a sense of intentionality in how we build and deploy AI. AI-Mind, for instance, focuses on content generation for marketing and business use cases, with guardrails built into the platform's design. It's not about restricting creativity. It's about acknowledging that when you build a tool, you're making choices about what that tool enables. Those choices have consequences.

This is the conversation we should be having about "nudify" apps. Not just "are they legal?" but "what kind of world do we want to build?" The developers of these apps made a choice. Apple and Google made a choice when they approved them. And now San Francisco is forcing a reckoning with those choices.

Key Takeaways

• San Francisco's City Attorney is demanding Apple and Google remove AI "nudify" apps, citing violations of California law and platform terms of service.
• These apps bypass app store review by disguising their true functionality, exploiting a fundamental weakness in the approval process.
• 96% of deepfake videos are non-consensual pornography, and the ease of creating such content has exploded with generative AI tools.
• Section 230 likely doesn't protect platforms from liability for actively distributing and profiting from apps designed primarily for illegal purposes.
• App removal alone won't solve the problem — the underlying technology is open-source and widely accessible, requiring broader legal and social responses.

Look, I don't expect this to be the last time we have this conversation. The technology will evolve. The app stores will play defense. And somewhere, a developer is already building the next version of this tool, probably with better obfuscation. What's different now is that city attorneys are paying attention. They're building legal cases. They're naming names. That's not a solution, but it's a start — and honestly, it's more than we had six months ago.

Sources

• San Francisco City Attorney's Office, "Attorney General Chiu Demands Apple and Google Remove AI 'Nudify' Apps," 2025. Official press release and legal documentation of the cease-and-desist letters.
• Internet Watch Foundation, "AI Child Sexual Abuse Material Now Prevalent on the Open Web," 2023. Annual report documenting a 2,400% increase in AI-generated CSAM.
• Deeptrace Labs, "The State of Deepfakes," 2019. Foundational study establishing that 96% of deepfake videos are non-consensual pornography targeting women.
• U.S. Congress, "Preventing Deepfakes of Intimate Images Act," 2024. Proposed federal legislation creating criminal penalties for non-consensual deepfake distribution.