The gap between what artificial intelligence appears to do and what it actually does has never been wider. Large language models draft legal briefs, generate photorealistic images, and hold conversations that feel uncannily human. Yet these same systems cannot reliably count the letters in a word, struggle to plan a route through an unfamiliar city, and have no mechanism for knowing when they are wrong. Understanding these limitations is not pessimism — it is the prerequisite for using these tools intelligently.
The confusion stems partly from terminology. When a chatbot "reasons" through a math problem, it is not reasoning in any sense a philosopher or cognitive scientist would recognize. It is pattern-matching against vast statistical regularities in its training data, producing outputs that resemble reasoning because they were trained on examples of human reasoning. The distinction matters enormously. A system that genuinely reasons can generalize to novel situations; a system that mimics reasoning fails unpredictably when the situation deviates from its training distribution.
The grounding problem
Language models operate entirely in the realm of symbols. They manipulate words, tokens, and their statistical relationships without any connection to the physical world those symbols describe. This is why a model can write eloquently about the taste of a strawberry while having no sensory experience of taste, or describe the layout of a kitchen while possessing no spatial understanding whatsoever.
This absence of grounding creates characteristic failure modes. Ask a model to verify a claim about the real world — whether a restaurant is still open, whether a scientific paper actually exists, whether a person said what is attributed to them — and it has no mechanism for checking. It can only assess whether the claim sounds plausible given patterns in its training data. The confident hallucination of citations, quotes, and facts is not a bug to be patched but a structural feature of systems that model language rather than reality.
The planning deficit
Humans solve novel problems by constructing mental models, simulating outcomes, and adjusting plans when reality diverges from expectation. Current AI systems lack this capacity in any robust sense. They can produce text that describes a plan, but they cannot actually plan — maintaining state, tracking constraints, recovering from unexpected obstacles.
This explains why AI excels at tasks with clear patterns and short feedback loops (code completion, image classification, translation of common phrases) while struggling with tasks requiring extended coherent action in dynamic environments. An AI can suggest chess moves but cannot manage a construction project. It can draft a marketing strategy but cannot execute one, adapting to market feedback over months.
The calibration void
Perhaps most consequentially, current systems have no reliable sense of their own uncertainty. A well-calibrated reasoner knows what it knows and what it does not; it expresses appropriate confidence. Language models express confidence through tone and word choice, but this confidence bears little relationship to actual accuracy. The same system that correctly explains quantum entanglement may, with identical certainty, fabricate a Supreme Court case that never existed.
This creates a dangerous asymmetry. The user who knows enough to verify AI outputs does not desperately need them; the user who lacks that knowledge cannot distinguish insight from invention. The burden of verification falls entirely on humans, yet the fluency of the output actively discourages verification.
Our take
None of this diminishes AI's genuine utility. These systems are remarkable tools for drafting, brainstorming, translation, and pattern recognition in constrained domains. But tools require skilled operators who understand their limitations. The current discourse oscillates between apocalyptic warnings about superintelligence and breathless promises of imminent utopia, when the real story is more mundane and more useful: we have built sophisticated pattern-matchers that excel within their training distributions and fail outside them. Knowing precisely where those boundaries lie is the difference between leveraging a powerful tool and being deceived by a confident mimic.
