The most sophisticated artificial intelligence systems ever built, trained on trillions of words at a cost of hundreds of millions of dollars, cannot reliably tell you how many times the letter 'r' appears in the word 'strawberry.' This is not a bug to be fixed in the next update. It is a window into the alien nature of machine intelligence.
When you ask ChatGPT or Claude to count letters, you are asking a prediction engine to perform arithmetic. These systems do not see words the way humans do. They see tokens — chunks of text that might be whole words, fragments of words, or individual characters, depending on the tokenizer's training. The word 'strawberry' might arrive as 'straw' + 'berry' or 'str' + 'aw' + 'berry' or some other decomposition. The model never actually examines the individual letters; it predicts what answer would most plausibly follow your question based on patterns in its training data.
The prediction machine
Large language models are, at their core, extraordinarily sophisticated autocomplete systems. Given a sequence of tokens, they predict the next token. Given that prediction, they predict the one after. This mechanism, repeated billions of times during training on vast corpora of human text, produces systems that can write sonnets, explain quantum mechanics, and draft legal contracts.
But prediction is not understanding. When a model produces a correct answer to a math problem, it is not performing calculation — it is pattern-matching against similar problems it encountered during training. Simple arithmetic works because the training data contains countless examples. Novel or unusual problems expose the illusion. Ask for the square root of 17 and you will likely get a reasonable approximation. Ask for the square root of 17,429 and confidence should plummet.
What they actually excel at
The same architecture that fails at counting produces genuine marvels elsewhere. Translation, summarization, style transfer, code generation — these tasks align beautifully with next-token prediction because they require recognizing and reproducing patterns in language. A model trained on millions of French-English sentence pairs learns the statistical relationships between languages without ever being taught grammar.
The implications are profound. These systems can produce text indistinguishable from human writing not because they understand meaning but because meaning, in human language, is encoded in patterns. We are pattern-generators; they are pattern-recognizers. The overlap is large enough to be useful, small enough to be dangerous when misunderstood.
The gap between fluency and competence
Humans routinely conflate articulate speech with deep understanding. A person who speaks confidently and coherently about medicine, we assume, probably knows something about medicine. This heuristic fails catastrophically with language models. They are optimized for fluency, not accuracy. They will explain incorrect information with the same confident cadence as correct information, because confidence is a linguistic pattern they have learned to reproduce.
This is why the most dangerous AI failures are not the obvious ones. A model that refuses to answer is safe. A model that answers incorrectly but fluently is a liability. The letter-counting failure is, in this sense, a gift: it is obvious enough to remind users that they are dealing with something fundamentally different from human cognition.
Our take
The counting problem is not a flaw to be embarrassed about — it is a feature to be understood. Every tool has constraints that define its proper use. A hammer is not defective because it cannot turn screws. Language models are not defective because they cannot count letters; they are prediction engines being asked to perform tasks outside their architecture. The companies building these systems would serve users better by being explicit about these boundaries rather than training models to route around them with workarounds. Understanding what AI actually is — and is not — remains the most important literacy of the decade.
