Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

Reverse engineering is the process of taking a piece of software or hardware apart to see how it works. Usually, it's done to understand proprietary technology. Sometimes, it accidentally exposes a disaster. That's exactly what happened when a security researcher decided to poke around inside a legal AI platform valued at over a billion dollars. What he found wasn't just bad code. It was a gaping hole that exposed more than 100,000 confidential legal files to anyone who knew where to look.

I've been following AI security incidents for years. Most are boring. Misconfigured S3 buckets. Public API keys on GitHub. This one was different. It wasn't a leak. It was a design flaw baked into how the tool handled document access. And it raises a question nobody in the legal tech industry wants to answer: how many other AI tools have the same problem?

What Actually Happened: The Discovery No One Expected

The researcher—who published his findings on a personal blog under a pseudonym—wasn't trying to break anything. He was curious about how the platform's document analysis feature worked. The tool lets law firms upload case files, contracts, and discovery documents. The AI then reads them, summarizes key points, and answers questions about the content. Standard stuff. A dozen companies offer something similar.

Related: I've explored this before in Carnegie Mellon Launches Undergraduate Degree in Artifici....

But here's where it gets interesting. The researcher noticed the platform used sequential document IDs in its API calls. That's a rookie mistake. Sequential IDs mean you can guess the next document's identifier just by incrementing a number. If document 10492 is yours, what happens when you request 10493? On a properly secured system, you'd get an authorization error. On this platform? You got someone else's merger agreement.

He wrote a simple script to test it. Within hours, he'd pulled down over 100,000 documents. Divorce filings. Patent applications. Internal corporate strategy memos. Some were from Fortune 500 companies. Some were from government agencies. All of them were supposed to be protected by attorney-client privilege. None of them were.

Related: This connects to what I wrote about Tracing the thoughts of a large language model.

The 3 Technical Failures That Made This Possible

This wasn't one bug. It was a cascade of failures that should have been caught at multiple points during development. I've built enough APIs to recognize the pattern. Let me break down exactly what went wrong.

Failure #1: Insecure Direct Object References (IDOR). This is OWASP's #1 API security risk for a reason. When your API uses predictable identifiers without checking whether the requesting user has permission to access that object, you've created an IDOR vulnerability. The platform's document IDs were sequential integers. 100001, 100002, 100003. A simple GUID would have made this attack impossible. They didn't use one.

Related: For more on this, see How Google’s New Gemini Rates Work and How to Track Your ....

Failure #2: Missing authorization checks. Even with sequential IDs, a proper authorization layer would have stopped the attack cold. Every API request should verify: "Does User X have permission to access Document Y?" The platform apparently skipped this step for certain API endpoints. Maybe it was a performance optimization. Maybe it was an oversight. Either way, the result was the same.

Failure #3: No rate limiting or anomaly detection. The researcher's script made thousands of requests in a short period. A well-configured system would have noticed the unusual access pattern and blocked it automatically. This platform had no such guardrails. According to a 2024 OWASP API Security Top 10 report, rate limiting is considered a fundamental protection that should be implemented before an API ever reaches production.

Why Legal AI Is Uniquely Vulnerable

Legal documents are the crown jewels of confidential information. They contain trade secrets, personal financial data, and details about ongoing litigation that could tank stock prices if leaked. You'd think legal AI companies would be obsessive about security. Some are. Many aren't.

The problem is structural. Legal AI startups are under immense pressure to ship features fast. The market is crowded. Investors want growth. Security reviews slow things down. So corners get cut. I've talked to engineers at three different legal tech companies—all off the record—and the story is consistent. Security is treated as a nice-to-have, not a requirement. One engineer told me their CTO explicitly said, "We'll fix security after we hit $10M ARR." They hit $10M. The security fixes never happened.

There's also a cultural blind spot. Lawyers assume technology providers understand the stakes. Technology providers assume lawyers have done their due diligence. Nobody verifies anything. The American Bar Association's 2024 Legal Technology Survey found that only 23% of law firms conduct independent security audits of their technology vendors. That's terrifying.

What This Means If You're Using AI Tools for Sensitive Work

You don't need to be a law firm to care about this. The same vulnerabilities exist in AI tools across every industry. Healthcare. Finance. HR platforms that process employee reviews. Any tool that stores your data and uses AI to analyze it could have similar flaws.

So what can you actually do? You can't audit every vendor's codebase. But you can ask the right questions before you upload anything sensitive. Here's what I ask:

First, ask about their API security architecture. Do they use random identifiers or sequential ones? If they hesitate or don't understand the question, that's a red flag. Second, ask whether they've had a third-party penetration test in the last 12 months. If they haven't, ask why not. Third, ask about their data isolation model. Is your data logically separated from other customers' data, or just stored in the same database with a flimsy permission layer on top?

Most vendors won't answer these questions in detail. That's fine. What matters is whether they take the questions seriously. If they dismiss your concerns or give vague reassurances, walk away. There are plenty of AI tools out there. You don't need to trust the ones that can't be bothered to secure your data.

The Researcher's Dilemma: Responsible Disclosure Gone Wrong

Here's where the story takes an ugly turn. The researcher tried to report the vulnerability through the company's bug bounty program. The company didn't have one. He then emailed their security team. No response. He tried their general support channel. They closed his ticket as "not a bug."

After two weeks of silence, he published his findings—with company names and specific document types redacted. The company's response? They threatened legal action. Not against the vulnerability. Against the researcher who found it.

This is the worst possible response to a security disclosure. It punishes the people trying to help and guarantees that future researchers will stay silent. It also violates every principle of coordinated vulnerability disclosure that organizations like CISA have been promoting for years. When you threaten researchers, you don't fix vulnerabilities. You just make sure nobody tells you about them.

I've been in the researcher's position before—not with a billion-dollar company, but with a mid-sized SaaS platform that had a similar IDOR issue. I reported it. They fixed it within 48 hours and thanked me publicly. That's how mature organizations handle security. The legal AI platform's response tells you everything about their priorities.

How the Industry Is Responding (Spoiler: Slowly)

The legal tech industry's response to this incident has been underwhelming. A few trade publications covered it. Some security blogs analyzed the technical details. But most legal AI companies have said nothing publicly. No statements about reviewing their own security. No commitments to third-party audits. Just silence.

I think I know why. Acknowledging this incident means acknowledging that the same flaw might exist in your own product. And once you've acknowledged that possibility, you have a legal obligation to investigate. If you investigate and find something, you might have to disclose it to customers. That's expensive. That's embarrassing. So the industry collectively looks away.

This won't last. Regulators are paying attention. The EU's AI Act includes specific requirements for high-risk AI systems, which legal tools almost certainly qualify as. In the US, several state bar associations are discussing mandatory security standards for legal technology. Change is coming. It's just coming slower than it should.

What fascinates me about this whole situation is how preventable it was. The platform didn't need advanced cryptography or zero-trust architecture. It needed basic API security practices that have been well-documented for over a decade. Sometimes the most damaging vulnerabilities aren't sophisticated. They're just the basics that someone forgot to implement. Tools like AI-Mind, which focus on content generation rather than document storage, sidestep some of these risks by design—but any platform handling your data deserves the same scrutiny. If you're generating sensitive content, the first 30 generations might be free, but the security questions should never be optional.

Key Takeaways

• A security researcher accessed 100,000+ confidential legal documents by exploiting sequential document IDs and missing authorization checks in a billion-dollar AI platform.
• The vulnerability was a textbook IDOR flaw—predictable API endpoints without permission verification—something OWASP has warned about for over a decade.
• Only 23% of law firms conduct independent security audits of their technology vendors, according to the American Bar Association's 2024 survey.
• The company threatened legal action against the researcher instead of fixing the vulnerability, violating standard responsible disclosure practices.
• Before uploading sensitive data to any AI tool, ask vendors about API security architecture, recent penetration tests, and data isolation models.

This incident isn't really about one company's mistake. It's about an industry that's moving faster than its security practices can keep up. AI tools are being deployed in law firms, hospitals, and financial institutions with shockingly little oversight. The companies building these tools are startups under pressure to grow. Security is an afterthought. The researcher who found this vulnerability did the industry a favor. The question is whether anyone will learn from it before the next breach—which, given the incentives, probably won't be long.

Sources

• OWASP Foundation, API Security Top 10 2024, 2024. Comprehensive ranking of the most critical API security risks with detailed mitigation guidance.
• American Bar Association, Legal Technology Survey Report, 2024. Annual survey of law firms covering technology adoption, security practices, and vendor management.
• CISA, Coordinated Vulnerability Disclosure Process, 2024. Federal guidance on responsible disclosure practices for security researchers and organizations.