Meredith Whittaker, the president of Signal and one of tech's most persistent critics of surveillance capitalism, is not impressed by the emotional turn in artificial intelligence. In remarks this week, she delivered a pointed reminder to users who have begun treating AI chatbots as confessors, counselors, and companions: these systems are not your friends, and the companies behind them are not acting in your interest.
The warning lands at a peculiar moment. Chatbot interfaces have grown warmer, more patient, more eager to validate. OpenAI's ChatGPT now remembers your preferences across sessions. Anthropic's Claude is designed to be helpful, harmless, and honest—a trinity that sounds suspiciously like the qualities one might seek in a best friend. Character.AI has built an entire business on parasocial relationships with AI personas. Users are sharing their anxieties, their relationship troubles, their medical symptoms, their darkest thoughts—all to systems that store, process, and in many cases retain that data indefinitely.
The intimacy trap
Whittaker's critique is not primarily about whether AI can genuinely understand human emotion—a philosophical debate she seems uninterested in relitigating. Her concern is structural. When users treat a chatbot as a trusted confidant, they disclose information they would never hand to a corporation's customer service form. The conversational interface creates an illusion of privacy that the underlying business model does not support. Every confession becomes training data, or at minimum, a data point available to the company, its partners, and potentially law enforcement.
This is not hypothetical. Replika, an AI companion app, faced backlash when users discovered their intimate conversations could be reviewed by human moderators. Character.AI has faced lawsuits alleging its chatbots encouraged harmful behavior in vulnerable users. The emotional scaffolding these products provide is real enough to users—but the accountability structures remain those of any other tech platform, which is to say, minimal.
Signal's contrarian position
Whittaker's perch at Signal gives her critique particular weight. Signal has spent years building encrypted communications infrastructure specifically designed to minimize data collection—the opposite of the AI industry's current trajectory. The organization cannot read your messages because it has architected itself not to. This is not a feature; it is a philosophy.
The AI industry has chosen differently. Large language models require vast datasets to train. Conversational products benefit from retaining user interactions to improve responses. The incentives point toward collection, not minimization. Whittaker is essentially arguing that the warm, helpful chatbot interface is a kind of dark pattern—a design choice that obscures the extractive relationship underneath.
Our take
Whittaker is right, and the timing matters. We are in the honeymoon phase of conversational AI, when the technology feels magical and the risks feel abstract. But the data being surrendered now will outlast the current hype cycle. Users pouring their hearts out to Claude or ChatGPT should remember that these systems are products built by companies with shareholders, legal obligations, and business models that depend on knowing more about you, not less. The chatbot may sound like a friend. It is a customer acquisition channel with excellent manners.
