The gap between what artificial intelligence can do and what people believe it can do has never been wider. This is not a complaint about hype — hype is the native atmosphere of technology — but an observation about how poorly understood the actual failure modes remain, even among sophisticated users.

The systems that power ChatGPT, Claude, and their competitors are genuinely remarkable. They write serviceable code, summarize dense documents, and produce prose that passes for human. But they also fail in ways that reveal something fundamental about what they are and are not.

The counting problem

Ask a large language model to count the number of times the letter "r" appears in "strawberry" and watch it struggle. This is not a cherry-picked gotcha; it reflects a deep architectural reality. These systems do not see text the way humans do. They process language as tokens — chunks of characters that may or may not correspond to words or letters — and they have no mechanism for the kind of sequential, step-by-step counting that a five-year-old performs effortlessly.

The implications extend far beyond party tricks. Any task requiring precise enumeration, exact verification, or reliable arithmetic sits outside the native competence of these systems. They can approximate, they can pattern-match against training data that included similar problems, but they cannot count in the way that counting actually works.

The knowledge boundary

Large language models do not know what they do not know. They have no internal representation of confidence that maps reliably onto accuracy. A system will state a fabricated legal citation with the same grammatical certainty it uses for the periodic table. This is not a bug that engineers are close to fixing; it emerges from how the technology fundamentally operates.

The training process optimizes for producing plausible-sounding text, not for maintaining epistemic humility. When a model encounters a question at the edge of its training data, it does not pause or express uncertainty in proportion to its actual ignorance. It generates the most probable next tokens, which often means confabulating with perfect fluency.

The reasoning question

Whether these systems "reason" depends entirely on how loosely one defines the term. They can produce outputs that resemble reasoning — chains of logic, step-by-step analysis, apparent deduction. But they achieve this through pattern completion over vast training corpora, not through the kind of abstract symbol manipulation that characterizes formal reasoning.

The distinction matters because it predicts where the systems will fail. Novel problems that require genuine logical inference, rather than sophisticated interpolation from training examples, expose the limits quickly. This is why mathematics remains challenging despite years of improvement — not calculation, which can be outsourced to code, but proof construction and creative problem-solving.

Our take

None of this diminishes what the technology has accomplished or will accomplish. Understanding limits is not pessimism; it is the prerequisite for using tools effectively. The professionals who will extract the most value from AI in the coming years are not the ones who believe it can do anything, but the ones who have developed an accurate mental model of where it excels and where it quietly fails. The hype serves the companies selling the technology. The limits serve everyone trying to use it.