Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files

A reverse engineering effort against a billion-dollar legal AI tool reportedly exposed more than 100,000 confidential files. The mechanism is the part that matters: the tool wasn't broken into through some exotic exploit. Its data handling was simply never designed to keep client documents separate from everything else the system touched.

If you work anywhere near legal, medical, or financial documents, the useful question isn't "was that company careless?" It's whether the same class of failure exists in the AI tools your team already uses. Most of the time, nobody has actually checked. That's the scenario this piece walks through: you're the person who has to figure out what your AI stack is really doing with sensitive files, without a security team and without breaking your workflow.

What "reverse engineering" actually means here

Reverse engineering an AI product usually means probing its API, watching what it sends over the network, and testing how it responds to inputs it wasn't designed for. You don't need source code. You need curiosity and a proxy.

In a legal AI context, the interesting surfaces are the ones that touch documents: upload endpoints, retrieval layers, and the logging that happens in between. A tool that ingests a contract and answers questions about it has to store that contract somewhere, chunk it, embed it, and probably cache the result. Every one of those steps is a place where "your file" and "someone else's file" can blur if the isolation isn't deliberate.

The reported exposure of 100k+ confidential files points at exactly that kind of blur — not a single dramatic hack, but a structural gap that made bulk access possible once someone looked closely.

The conventional approach, and where it breaks

The standard playbook for a team adopting an AI document tool looks like this: legal reviews the terms of service, IT confirms SSO works, someone signs off, and the tool goes live. That process catches maybe half the risk.

What it misses is behavioral. Terms of service describe intent. They don't tell you whether the retrieval index is shared across tenants, whether embeddings persist after you delete a document, or whether the vendor's own staff can query the corpus. Those are implementation details, and implementation details are where breaches live.

The real constraint here isn't money or tooling. It's that verifying data isolation requires someone to actually test the system, and almost nobody does. A law firm with 40 attorneys isn't going to run penetration tests on a SaaS product. They'll trust the SOC 2 report and move on. That's a rational decision, and it's also the exact gap the reported breach fell into.

How to audit an AI tool without a security team

You can do a meaningful audit in an afternoon. It won't be exhaustive, but it will catch the failure modes that actually show up in the news.

• Test deletion. Upload a document, ask a question about it, then delete it. Wait a day. Ask the same question again. If the system still answers, your data isn't gone.
• Watch the network. Run the tool through a proxy like mitmproxy or Burp Suite. You're looking for document content leaving your machine in unexpected places — analytics endpoints, third-party logging services, anywhere that isn't the vendor's own API.
• Probe the boundaries. Create two accounts under the same organization and see whether one can retrieve the other's documents through a direct API call. Tenant isolation failures are boring and extremely common.
• Read the retention policy, then test it. Retention policies are claims. The deletion test above is the evidence.

None of this requires a security background. It requires patience and a willingness to be annoying to the vendor's support team.

A worked example: the shared embedding index

Here's the failure mode that shows up most often, and it's worth walking through concretely because it's invisible from the outside.

Say you're evaluating a legal research assistant. You upload a 30-page merger agreement and ask it to summarize the indemnification clauses. The tool chunks your document into, say, 200-token segments, converts each into a vector embedding, and stores those vectors in a database. When you ask a question, it embeds your query, finds the nearest vectors, and feeds those chunks to the language model.

Now: if that vector database is keyed only by document ID and not by tenant ID, and document IDs are sequential or predictable, a second customer can retrieve your chunks by guessing IDs. No exploit needed. Just a loop and a few thousand requests.

That's the kind of thing reverse engineering surfaces. It's not glamorous. It's a missing WHERE clause. And it's why "we use encryption at rest" doesn't answer the question — encryption protects the disk, not the query logic.

Where AI tools genuinely fail at this

Be honest about the limits. AI document tools are worse at data isolation than traditional SaaS for three reasons.

First, the data is unstructured. A CRM has fields you can permission individually. A document is a blob, and the moment it's chunked and embedded, the original permission structure is gone unless someone rebuilds it.

Second, retrieval is fuzzy. Traditional databases return exact matches. Vector search returns "close enough," which means a permission bug doesn't throw an error — it quietly returns the wrong document with high confidence.

Third, the tooling is young. Most legal AI vendors are startups that shipped fast. That's not a moral failing, but it does mean their multi-tenancy architecture has had less time to be stress-tested than, say, Salesforce's.

If you're picking tools, this is where an internal database of 360 AI tools with recorded pricing and capability snapshots — verified as recently as late September 2026 — becomes genuinely useful. Not because it tells you which tool is secure, but because it tells you which ones have been around long enough to have a track record worth checking.

What to do before the next contract

Run the deletion test on every AI tool that touches client documents. If a tool fails it, that's your answer, regardless of what the terms of service say.

Ask vendors one specific question: "Is the retrieval index shared across customers, and how is it keyed?" A vendor that can't answer in plain language hasn't thought about it. That's not a red flag you can ignore.

And keep the blast radius small. If your AI workflow can operate on redacted documents, redact them. If it can work on summaries instead of full contracts, use summaries. The reported breach exposed 100k+ files because the system had access to 100k+ files. Fewer files in the system is a security control that costs nothing.

Key Takeaways

• Reverse engineering AI tools usually reveals missing isolation logic, not exotic exploits.
• Test deletion, watch network traffic, and probe tenant boundaries before trusting any document AI.
• Vector search fails silently — a permission bug returns the wrong document confidently, not with an error.
• Terms of service describe intent; only behavioral testing shows what a system actually does.
• Reducing what you upload is the cheapest security control available.

The lesson from a 100k-file exposure isn't that legal AI is unsafe. It's that "we reviewed the contract" is not a security posture. The teams that avoid this class of failure are the ones willing to spend an afternoon being suspicious — uploading a test document, deleting it, and checking whether it's really gone. That's a low bar, and it's higher than most organizations currently clear.

If you're generating internal documentation about your AI workflows — model cards, eval notes, audit trails — a tool like AI-Mind can handle the drafting from a plain description, which saves the prompt-writing overhead when you're documenting the same process across a dozen tools. The documentation habit matters more than the tool, though. Write down what you tested and what you found.

Sources

• AI Tool Database, Internally verified pricing and capability snapshot, 2026. Covers 360 AI tools, most recently verified 2026-09-24.