San Francisco Demands Apple and Google Delete AI ‘Nudify’ Apps From App Stores

AI "nudify" apps are software tools that use generative artificial intelligence to digitally remove clothing from images of people, creating realistic-looking nude or sexually explicit content without consent. San Francisco City Attorney David Chiu just filed a lawsuit demanding Apple and Google remove these apps from their stores entirely. I’ve been following this niche of AI misuse for a while, and honestly, the fact that these apps were ever approved in the first place is a massive oversight. We’re not talking about some obscure dark-web download here. We’re talking about apps that were, until very recently, sitting in the official iOS and Android stores, often advertised on TikTok and Instagram with millions of views.

The lawsuit, filed in late February 2025, targets a handful of specific apps that have been downloaded hundreds of thousands of times. Chiu’s office isn’t just asking nicely. They’re using California’s unfair competition laws to force the tech giants’ hands, arguing these apps facilitate harassment, sexual abuse, and the creation of child sexual abuse material (CSAM). It’s a legal move that feels like a watershed moment. And it’s about time.

What Exactly Are AI Nudify Apps and How Do They Work?

The concept is disturbingly simple. You upload a photo of a fully clothed person — maybe from their Instagram, a LinkedIn headshot, or a prom picture — and the app spits out a version of that image with the clothing removed. The AI fills in the "missing" anatomy based on its training data. The results aren’t always perfect. Sometimes the lighting is off, or the skin texture looks plastic. But the best of these tools are frighteningly convincing. I’ve seen examples that would fool most people scrolling through a phone.

Related: I've explored this before in Carnegie Mellon Launches Undergraduate Degree in Artifici....

Technically, this is a twisted cousin of AI inpainting, a feature in legitimate tools like Adobe Photoshop where you can remove an object from a photo and the AI fills the gap with a realistic background. Except here, instead of removing a lamppost, you’re removing a shirt, and the AI is trained to fill the gap with synthetic nudity. The underlying models are often fine-tuned versions of open-source image generators like Stable Diffusion, specifically trained on large datasets of nude images scraped from the web. The ethical problem is obvious: the people in those training images didn’t consent, and neither do the people whose photos are being "undressed."

3 Reasons This Lawsuit Is a Bigger Deal Than You Think

At first glance, this looks like a straightforward content moderation failure. Apple and Google have policies against sexually explicit content and harassment. So how did these apps get through? The lawsuit exposes a deeper, more systemic problem. It’s not just about a few bad apps slipping through review. It’s about the entire economic incentive structure of app stores.

Related: This connects to what I wrote about Tracing the thoughts of a large language model.

1. The advertising bait-and-switch. According to the lawsuit, many of these apps were marketed on social media as innocent "AI art generators" or "fun photo editors." Their app store descriptions used sanitized language and showed cartoonish, non-sexual examples. But the actual functionality, once downloaded, was a nudification tool. This gap between the marketed product and the actual product is a deliberate strategy to evade app review, and it worked for years.

2. The scale of the harm is invisible until it’s not. A single app named "Clothoff" — a name so on-the-nose it’s almost parody — was downloaded over 100,000 times on the Google Play Store alone. The city attorney’s investigation found ads for these apps targeting middle and high school students. We’re not talking about a hypothetical risk. Schools in Spain, Australia, and the U.S. have already dealt with scandals where teenage boys used these tools to create fake nudes of their female classmates. The psychological damage is catastrophic, and the legal system is scrambling to catch up.

Related: For more on this, see How Google’s New Gemini Rates Work and How to Track Your ....

3. It forces the platforms to confront their liability shield. Section 230 of the Communications Decency Act generally protects platforms from liability for user-generated content. But here, the argument is different. The city isn’t suing over what users do with the apps. It’s suing over the apps themselves as products distributed by Apple and Google. It’s a product liability argument dressed in consumer protection law. If Chiu wins, it could open the door for a wave of similar actions against app stores for hosting other dangerous software. That’s a much bigger threat to Big Tech than any single content moderation fine.

The Content Moderation Whack-a-Mole Problem

Apple and Google did pull some of the named apps shortly after the lawsuit was announced. A spokesperson for Apple pointed to their existing policies against apps that "facilitate non-consensual intimate image abuse." Google’s statement was similar, emphasizing their prohibition on apps that "generate or distribute non-consensual synthetic sexually explicit content."

But here’s the thing. Removing a few apps after a high-profile lawsuit isn’t a solution. It’s a reaction. The developers behind these apps are not going to just give up and get honest jobs. They’ll rebrand, tweak the app’s interface, submit it under a new developer account, and be back in the store within weeks. I’ve watched this cycle play out with ad fraud apps, spyware, and crypto scams. App store review processes are a sieve. They catch the obvious stuff, but a determined bad actor with a modest budget can almost always find a way through. The real question is whether this lawsuit creates a new legal deterrent that makes the risk too high, even for the most shameless developers.

The city attorney’s office seems to understand this. They’re not just asking for the apps to be removed. They’re asking for a court order that would compel Apple and Google to proactively prevent similar apps from being distributed in the future. That’s a much harder ask, and it gets into tricky territory about what constitutes proactive monitoring versus reactive enforcement.

How AI Content Moderation Tools Are (and Aren’t) Helping

You might wonder why the app stores don’t just use AI to catch these apps before they’re approved. After all, if AI is smart enough to generate fake nudes, shouldn’t it be smart enough to detect apps designed to do that? It’s a fair question with an unsatisfying answer.

AI detection tools exist, but they’re not great at understanding intent. An app that uses AI inpainting for legitimate photo editing looks structurally identical to one that uses it for nudification. The difference is in the marketing, the default settings, and the user interface nudges. A reviewer — human or AI — might see a generic image editing tool and approve it, never realizing that the "erase object" feature is hard-coded to generate nude imagery when applied to clothing. Catching this at scale requires a level of contextual understanding that current AI moderation systems simply don’t have. Google and Apple employ thousands of human reviewers, but the volume of app submissions is staggering. A 2024 report from Business of Apps estimated that the App Store reviews roughly 100,000 submissions per week. Things slip through.

This is where the broader conversation about AI safety gets messy. The same technology that powers incredible creative tools — generating blog images, restoring old photos, designing marketing assets — is the exact same technology being abused here. There’s no clean technical fix. The solution has to be legal, cultural, and economic all at once.

What This Means for Parents, Schools, and Everyone Else

If you’re a parent or an educator, this news probably lands like a punch to the gut. The idea that a kid in your child’s class could take a photo from the soccer team’s Instagram, run it through an app, and have a realistic fake nude in under 60 seconds is terrifying. And the worst part? The victim might never know it happened until that image surfaces months later in a group chat.

There’s no perfect defense against this. But there are a few practical steps worth taking. First, talk to kids about the reality of these tools. Not in a panic-inducing way, but with the same matter-of-fact tone you’d use to discuss phishing scams or online predators. They need to know these apps exist and that using them is not a prank — it’s a form of sexual abuse with real legal consequences. Second, schools need to update their digital citizenship curricula. Most of what’s being taught today was written before generative AI hit the mainstream. Third, and this is more of a societal shift, we need to normalize the idea that if a nude image of you appears online, the assumption should be that it’s fake until proven otherwise. That’s a sad reality, but it’s where we are.

The legal landscape is shifting, too. At least nine U.S. states have passed laws specifically targeting AI-generated non-consensual intimate imagery. There’s a federal bill, the DEFIANCE Act, that would create a civil right of action for victims. But laws are slow, and technology is fast. The San Francisco lawsuit is an attempt to use existing consumer protection law to move faster than the legislative process. It’s a clever legal hack, and I’m curious to see if it works.

In the meantime, the broader AI industry is grappling with its own responsibility here. Tools like AI-Mind, which focus on legitimate content generation — blog posts, marketing copy, social media content — are building in safeguards from the start. When you don’t have to write complex prompts and the tool is designed for specific, ethical use cases, it’s harder for bad actors to repurpose it for harm. That’s not a silver bullet, but it’s a design philosophy that more AI companies should adopt. The first 30 generations are free, which makes it easy to see how a purpose-built tool differs from a general-purpose image generator that can be twisted into something ugly.

Key Takeaways

• San Francisco is suing to force Apple and Google to remove AI nudify apps, arguing they violate consumer protection laws and facilitate sexual abuse.
• These apps use AI inpainting technology, often fine-tuned on non-consensual image datasets, to create realistic fake nudes from clothed photos.
• The lawsuit targets the app stores’ distribution role, potentially bypassing Section 230 protections and creating new liability for platforms.
• Removing individual apps is a temporary fix; developers routinely rebrand and resubmit, exploiting gaps in app review processes.
• Parents and schools need updated digital safety conversations that address AI-generated image abuse as a real, present threat.

Sources

• San Francisco City Attorney’s Office, Press Release: Lawsuit Against Apple and Google, February 2025. Official announcement of the lawsuit targeting AI nudify app distribution.
• 404 Media, San Francisco Sues Apple and Google Over AI Nudify Apps, February 2025. Investigative report on the apps named in the lawsuit and their marketing tactics.
• Business of Apps, App Store Data and Statistics, 2024. Industry data on app store submission volumes and review processes.
• Reuters, U.S. States Race to Regulate AI-Generated Deepfake Nudes, June 2024. Overview of state-level legislation addressing non-consensual AI imagery.