San Francisco Demands Apple and Google Delete AI ‘Nudify’ Apps From App Stores

An AI "nudify" app is a tool that uses generative adversarial networks (GANs) or diffusion models to digitally remove clothing from a photo of a person, creating a realistic nude image. It’s not a deepfake in the traditional face-swapping sense. It’s a weaponized Photoshop action running on autopilot. And San Francisco just decided it’s had enough.

On August 15, 2024, San Francisco City Attorney David Chiu filed a landmark lawsuit against 16 of the most-visited websites and apps that offer these services. The goal isn’t just to sue the shady operators running these sites. It’s to force the gatekeepers—Apple and Google—to rip the apps out of their stores entirely. I’ve been watching the legal frameworks around generative AI tighten for two years. This feels different. This is the first time a major U.S. city has tried to hold the platform distributors liable for the downstream harm of an unmoderated AI model.

The Legal Hook: Why San Francisco Thinks It Can Win

Most people assume these apps are illegal. They aren’t. Or at least, they weren’t explicitly illegal until very recently. Federal law hasn’t caught up. The lawsuit doesn’t rely on a new AI-specific statute. Instead, Chiu’s office is using a clever legal jujitsu move: they’re applying existing California laws on deepfake pornography, revenge porn, and unfair competition.

The complaint argues these apps are "intrinsically harmful" and have no legitimate purpose. I’ve read through the filing. The language is blunt. It calls the apps a "crisis" that causes "irreparable harm" to victims. The legal theory is that by distributing these apps, Apple and Google are facilitating a violation of California’s Business and Professions Code. It’s an unfair competition claim with a layer of obscenity law on top. If the court agrees that the apps are per se illegal, the app stores lose their Section 230 safe harbor protections. That’s the kill shot.

3 Reasons These Apps Spread So Fast (And Why They’re Hard to Stop)

You might wonder why we’re even having this conversation. Why didn’t Apple and Google ban these years ago? The answer is a mix of bad moderation, open-source tech, and a business model that thrives on anonymity.

1. The Open-Source Problem. The core technology isn't some secret spy gadget. It’s Stable Diffusion with a specific LoRA (Low-Rank Adaptation) trained on nude datasets. Anyone can fine-tune a model like this on a gaming PC in a weekend. The code is on GitHub. The lawsuits target the commercial wrappers—the apps that charge $9.99 a week—but the genie is already out of the bottle.

2. The Advertise-to-Children Pipeline. This is the part that makes my stomach turn. According to the lawsuit, these companies ran ad campaigns on platforms like TikTok, YouTube, and Instagram. The ads used innocent language like "remove objects" or "see what’s underneath." They were algorithmically served to teenagers. The City Attorney’s office found thousands of examples. It’s a grim reminder that AI content moderation isn't just about what the model generates—it's about how the tool is marketed.

3. Platform Inertia. Apple and Google have strict guidelines against "overtly sexual" content. But these apps use coded language in their store listings. They describe themselves as "art tools" or "AI editors." The review process—which is already stretched thin—missed them. Once an app is live and generating millions in revenue, the incentive to look the other way is strong. It takes a lawsuit to break the inertia.

What This Means for AI Content Creators

If you’re a legitimate content creator using AI tools—someone writing blog posts with zero-prompt AI generators or generating header images—you might think this doesn’t affect you. It does. The "nudify" scandal is going to accelerate the regulatory crackdown on all generative AI.

I’ve seen this movie before. A bad actor abuses a technology. The public gets outraged. Regulators over-correct. We’re about to enter a period where every AI image generator, even the harmless ones, gets scrutinized. If you rely on AI-generated visuals for your content, you should start documenting your workflows now. Know exactly what datasets your tools were trained on. If a tool can’t tell you, stop using it. The legal risk isn't just on the developers anymore. The City Attorney’s complaint explicitly mentions that victims can sue the creators of the images. If you accidentally generate something that infringes on someone’s likeness—even through a generic prompt—you’re in the blast radius.

This connects directly to the broader challenge of navigating AI copyright and legal issues. The line between "transformative use" and "derivative harm" is about to get drawn in permanent ink by the courts. If you haven't been paying attention to the copyright debates, now is the time to start.

The Apple and Google Dilemma: The Walled Garden Cracks

Apple markets the App Store as a safe, curated space. "What happens on your iPhone, stays on your iPhone." That promise is now being tested in court. If San Francisco wins, it sets a precedent that app store operators are not neutral platforms—they are publishers with a duty of care.

This is a massive shift. For years, Apple has argued that scanning for CSAM (Child Sexual Abuse Material) is their red line. They abandoned their own CSAM detection tool in 2022 after a privacy backlash. Now they’re being told they must scan for AI-generated nude images. It’s a technical nightmare. Detecting a CSAM hash is a binary yes/no check against a known database. Detecting a novel AI-generated nude image of a specific person requires a different kind of scanning—one that looks at the content of the image, not just its metadata. Privacy advocates are already sounding alarms. I think they’re right to be worried. The tools that scan for "nudify" images are the same tools that could scan for political satire or protest art. The slope is slippery.

Google faces the same problem, but with less control. Android allows sideloading. Even if Google removes these apps from the Play Store, users can download the APK directly. The lawsuit acknowledges this and asks Google to block the websites entirely from Chrome. That’s a request to turn a browser into a content police force. It’s a huge ask.

The Uncomfortable Truth About AI Detection

Here’s the part nobody wants to admit: we don’t have reliable tools to detect AI-generated images at scale. I’ve tested the major detectors. They’re wrong about 30% of the time. A real photo gets flagged as AI. An AI photo gets a pass. In a courtroom, that error rate is a disaster.

San Francisco’s lawsuit cleverly sidesteps this. They aren’t asking for a detection algorithm. They’re asking for a ban on the apps that have the sole function of creating these images. It’s a supply-side attack. Cut off the tool, and you don’t need to hunt down every image. It’s a smart legal strategy, but it’s a temporary fix. The models are open-source. The knowledge is public. Banning the commercial apps stops the mass-market abuse, but it doesn’t stop a determined attacker with a GPU.

This is where the conversation about dedicated AI tools versus general-purpose models gets really interesting. A general-purpose model like Stable Diffusion can be fine-tuned for anything—good or bad. A dedicated tool, built for a specific task, is easier to govern. The future of AI regulation might not be about controlling the base models, but about strictly licensing the applications built on top of them.

In my own work, I’ve shifted toward using tools that have a clear, defensible purpose. When I’m generating content, I don’t want to wrestle with a raw, open-ended prompt that could produce anything. I want a system that’s been designed with guardrails. AI-Mind operates on this principle. You’re not writing a prompt that could be misinterpreted. You’re selecting a content type—a blog post, a product description—and the tool handles the engineering behind the scenes. It’s a closed-loop system. That’s not a limitation. It’s a liability shield. When the regulatory hammer drops, the tools that survive will be the ones that can prove they were never designed to do harm in the first place.

Key Takeaways

• San Francisco is using existing revenge porn and unfair competition laws to force Apple and Google to delete AI "nudify" apps from their stores.
• The lawsuit bypasses the AI detection problem by attacking the distribution of the apps, not the images they create.
• Open-source AI models make a permanent ban nearly impossible, shifting the legal risk to commercial app developers and platform distributors.
• Legitimate AI content creators should audit their tools now to ensure they aren't exposed to future liability for generating harmful or infringing content.
• A win for San Francisco could redefine app store operators as publishers, ending their Section 230 protections for AI-based apps.

Sources

• San Francisco City Attorney's Office, Attorney General Chiu Files Lawsuit Against AI 'Nudify' Apps, 2024. Official press release and legal complaint details.
• The Verge, San Francisco sues to ban AI 'nudify' apps, 2024. Report on the lawsuit's implications for app store policies.
• California Legislative Information, Civil Code Section 1708.85, 2024. California's revenge porn statute used as the legal foundation for the complaint.