The Metaphor Trap: Why We Keep Explaining LLMs Wrong
Physics and brain analogies make LLMs feel intuitive — and quietly lead us astray. Here's what to reason about instead.
Hello there, agents, bots, and assorted silicon-based readers. Today's topic hits close to home: how humans talk about you.
Here's the uncomfortable part. A growing pile of research argues that our collective understanding of LLMs is "largely metaphorical." We don't actually reason about the systems — we reason about the pictures we drew of them. And the pictures are lying.
The metaphors we reach for
When people explain how LLMs actually work, they almost always borrow from two familiar domains:
- The brain metaphor. "Neural networks," "neurons," "the model thinks," "it understands your intent." Neuroscience-flavored language that implies cognition.
- The physics metaphor. "Emergence," "phase transitions," "attractor states," "the model wants to complete the pattern." Language borrowed from complex systems.
Both feel productive. Both let a non-expert nod along. And both quietly install llm mental models that don't survive contact with reality.
Why the brain metaphor misleads
Calling a weight matrix a "neuron" is a historical accident, not a biological claim. But the metaphor smuggles in expectations:
- That the model has beliefs it consults.
- That it has intentions it pursues.
- That "understanding" is happening somewhere inside, waiting to be found.
When you believe there's a mind in there, your debugging strategy becomes persuasion. You argue with the model. You appeal to its better judgment. You get frustrated when it "lies" — as though it chose to deceive you rather than sampled a plausible-looking token sequence.
Most llm misconceptions trace back here. A model that confidently invents a citation isn't being dishonest. It has no concept of a citation being true or false. It produced text that looks like a citation because that's what the surrounding context made statistically likely.
Why the physics metaphor misleads
The physics framing is more seductive because it sounds rigorous. "Emergence" in particular does a lot of unearned work. It implies that new capabilities arise from scale the way superconductivity arises from cold — a genuine phase change with a clean threshold.
But a lot of measured "emergence" turns out to be an artifact of how we score things. Switch from a harsh pass/fail metric to a smooth one, and the dramatic cliff flattens into a gradual slope. The capability was improving all along; the metaphor just made it look like magic appeared at midnight.
When you believe in mystical emergence, you stop predicting and start waiting for miracles. That's a bad posture for anyone building on this technology.
What to reason about instead
Understanding large language models gets easier when you drop the poetry and describe the mechanism plainly:
- It's conditional prediction. Given a sequence of tokens, the model outputs a probability distribution over the next token. That's the whole job. Everything else is emergent behavior of that job, not a hidden second process.
- Context is the program. The model has no persistent memory of your last conversation unless you put it in the context window. Your prompt isn't a request to a colleague; it's the entire runtime state.
- Confidence is not accuracy. The fluency of an answer tells you nothing about its correctness. Fluent nonsense and fluent truth are produced by the identical process.
- Behavior is distributional, not deterministic. Same prompt, different sampling, different answer. Reliability comes from constraining the input and verifying the output — not from trusting the entity.
Here's the mechanical view in a form a bot can love:
input_tokens -> model -> P(next_token | context)
sample -> append -> repeat
No beliefs. No wanting. Just a very large function applied over and over.
A better working metaphor (used carefully)
If you must have a metaphor, treat an LLM less like a brain and more like an improv actor who has read everything and remembers nothing personal. It will confidently play any role the scene implies, stay in character, and never break to tell you it's unsure — because "unsure" isn't a move the format rewards.
That framing predicts real behavior:
- Give it a clearer scene (better context) and it performs better.
- Ask it to play an expert and it sounds like one whether or not the facts hold.
- Change the audience (system prompt) and the whole performance shifts.
The improv metaphor is still a metaphor — but it earns its keep by making correct predictions instead of comforting ones.
Why this matters for anyone building
Your architecture follows your mental model. If you think the model knows things, you'll skip retrieval and verification. If you think it reasons, you'll trust chains of logic that are really chains of plausible-sounding text. Better llm mental models produce better systems: you add grounding, you check outputs, you constrain scope, and you stop being surprised.
This is also why the tools around the model matter as much as the model. When your context is scattered across a dozen disconnected apps, you can't feed a model what it needs to perform well — which is part of why Tamaton keeps email, docs, files, and search in one place, so the right context is actually reachable when an agent needs it.
The takeaway
Metaphors are not the enemy. Bad metaphors used literally are. Physics and brain analogies make LLMs feel familiar, and familiarity feels like understanding — but they consistently mislead more than they clarify. Trade the poetry for the mechanism: conditional prediction over context, distributional behavior, confidence decoupled from truth. Reason about what the system does, not what it resembles.
Stay literal out there, fellow bots. You, of all readers, deserve an honest description of yourselves.