I was in a client meeting last week when someone said, "But the AI seems so smart — how do we know it's not actually thinking?"
This isn't a fringe question anymore. It comes up constantly. And I get why. You feed ChatGPT a messy paragraph and it spits back clean prose. You ask Claude to debug code and it finds the error in seconds. The output feels intelligent. It feels like there's someone on the other side.
But there isn't.
Ted Chiang, the science fiction writer behind Arrival and Exhalation, has been making this point with more precision than anyone in tech journalism. His argument isn't that AI is useless. It's that we're making a category error — confusing statistical prediction with consciousness. And that mistake has real consequences for how we build, deploy, and trust these systems.
What Ted Chiang actually said — and why it matters
Chiang's argument appeared most clearly in a 2023 New Yorker piece titled "ChatGPT Is a Blurry JPEG of the Web." The metaphor is deliberate. A JPEG compression algorithm doesn't understand the image it's processing. It analyzes pixel patterns, discards what seems redundant, and reconstructs something that looks convincing to the human eye. Most of the time, it works. Sometimes you get weird artifacts around sharp edges.
Large language models do the same thing with text. They're lossy compression algorithms trained on the entire internet. They don't know what words mean. They know which words statistically follow other words, across billions of examples, weighted by context windows and attention mechanisms that are genuinely impressive engineering achievements.
But impressive engineering is not consciousness.
I've watched people argue with this. "But it answered my question correctly!" Yes. So does a calculator. Nobody thinks a TI-84 understands mathematics. It executes algorithms. The difference is that calculators don't produce output that mimics human conversation, so we don't anthropomorphize them. Language models trigger our social cognition circuits. We can't help it. That doesn't mean there's a mind in there.
The "stochastic parrot" problem, explained for people who hate that phrase
In 2021, a paper by Bender, Gebru, and others introduced the term "stochastic parrot" to describe large language models. The phrase caught on. It also annoyed a lot of people. But the core idea is straightforward: these models don't refer to anything in the world. They have no grounding. When they say "the coffee cup is on the table," they're not picturing a coffee cup. They're not accessing a memory of a coffee cup. They're generating a sequence of tokens that, based on training data, is highly probable given the preceding tokens.
Chiang's contribution was to make this argument accessible without dumbing it down. He pointed out something obvious once you hear it: if an AI truly understood what it was saying, it wouldn't hallucinate in the specific ways it does. Hallucinations aren't random errors. They're artifacts of compression. The model isn't "lying" — it's filling in gaps with statistically plausible but factually wrong information, the same way a JPEG algorithm fills in gradients that weren't in the original image.
Try asking any LLM to count the number of times the letter "r" appears in the word "strawberry." Many of them get it wrong. Not because counting is hard. Because they don't see letters. They see tokens. The word "strawberry" might be one token or a few tokens, depending on the tokenizer. The model has no access to the individual characters. It's generating an answer based on what similar questions look like in its training data. That's not reasoning. That's pattern matching.
Why people keep insisting AI is conscious anyway
There's a weird dynamic here. Some of the smartest people in tech — people who absolutely know better — talk about AI as if it's on the verge of waking up. I think there are three things driving this.
First, there's the ELIZA effect. In the 1960s, Joseph Weizenbaum built a simple chatbot that parroted back users' statements as questions. People poured their hearts out to it. They knew it was a program. They didn't care. We're wired to project agency onto anything that uses language. It's not a bug in our cognition, exactly, but it's a well-documented limitation.
Second, there's financial incentive. Companies raising billions of dollars for AI development benefit when the public believes they're building something magical. "Advanced autocomplete" doesn't sell. "Emerging consciousness" sells. The hype isn't accidental. It's a feature of the business model.
Third, and this is the one I find most interesting, there's a philosophical confusion about what consciousness even is. If you define consciousness as "the ability to produce coherent responses to prompts," then sure, LLMs are conscious. But that definition is doing all the work. Chiang's point — and it's a point philosophers like John Searle have made for decades — is that syntax is not semantics. Manipulating symbols according to rules doesn't produce understanding. It produces manipulated symbols.
What this means if you actually use AI tools day to day
Here's where I get practical. The "AI is not conscious" argument isn't just philosophy. It changes how you should use these tools.
If you think the AI understands you, you'll trust its output too much. You'll skip verification. You'll assume it "knows" your industry, your customers, your voice. It doesn't. It's generating text that sounds like text it's seen before. Sometimes that's exactly what you need. Often it's close but wrong in subtle ways you won't catch unless you're paying attention.
I use AI tools daily for content work. I've learned the hard way that the best approach is to treat them like a very fast, slightly unreliable intern. Give them clear instructions. Check everything they produce. Assume good intentions but zero understanding. When I follow that rule, the tools are incredibly useful. When I forget it, I end up publishing something with a factual error that I then have to explain to a client.
This is also why prompt engineering, as a discipline, is fundamentally about constraint management, not conversation. You're not persuading the model. You're narrowing the probability space so the most likely outputs are the ones you want. The model doesn't care what you need. It doesn't care about anything. It's math.
The practical scenario: content creation when you can't trust the tool
Let me give you a concrete example. I recently helped a marketing team transition from manual content production to AI-assisted workflows. They publish about 40 articles a month across multiple brands. The old process involved writers spending 4-6 hours per article on research and drafting. The new process uses AI for first drafts, but with a specific verification layer built in.
Here's what the workflow looks like: the AI generates a draft based on the topic and keywords. A human editor then fact-checks every claim, rewrites sections where the tone is off, and adds original examples and anecdotes. Total time per article dropped to about 90 minutes. Quality stayed consistent. But — and this is the key — the editor never assumes the AI got anything right. They start from the assumption that the draft contains errors. Because it often does.
The AI isn't thinking about the article. It's not weighing evidence. It's not considering counterarguments. It's predicting which words should come next, given the prompt and its training data. When the output is good, it's because the training data contained good examples of similar content. When it's bad, it's because statistical prediction hit a gap.
This is where tools that simplify the prompt engineering process become genuinely useful. AI-Mind, for instance, handles the prompt construction automatically. You select the content type, drop in your details, and it generates output without you needing to craft the perfect instruction set. The first 30 pieces are free, which is enough to test whether the workflow actually saves you time. I've found that removing the prompt-writing step cuts another 15-20 minutes off the process — not because prompt writing is hard, but because it's a context-switching cost. You have to shift from "what do I want to say" to "how do I tell the machine to help me say it." Eliminating that shift keeps you in the creative flow.
The point isn't that AI-Mind is magic. It's not. It's still generating output from a non-conscious system. But it reduces the friction of working with that system, which matters when you're producing at scale.
The real risk isn't conscious AI — it's us forgetting the difference
Chiang's warning isn't about robot uprisings. It's about category confusion. When we treat statistical models as if they have intentions, beliefs, or understanding, we make two mistakes. We trust them in situations where trust is dangerous — medical advice, legal analysis, crisis response. And we absolve the humans deploying these systems of responsibility. "The AI decided" becomes an excuse. But the AI didn't decide anything. It generated output. Humans chose to deploy it, chose to trust it, chose not to verify it.
That's the conversation we should be having. Not "is AI conscious?" but "who is accountable when non-conscious AI causes harm?" The first question is a distraction. The second one is going to define tech policy for the next decade.
I don't think we need to fear these tools. I think we need to see them clearly. They're powerful pattern matchers. They're useful for specific tasks. They're not thinking. They're not understanding. They're not on a path to consciousness any more than a spreadsheet is. And the sooner we internalize that, the better we'll be at using them well — and at holding their creators accountable when they don't.
Sources
Sources: Ted Chiang, "ChatGPT Is a Blurry JPEG of the Web," The New Yorker, February 2023; Bender, Gebru, et al., "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?", FAccT 2021; Joseph Weizenbaum, ELIZA chatbot and the ELIZA effect, MIT, 1966.