The internet was never designed for what we are doing to it now. Built on protocols drafted in the 1970s and 1980s, maintained in crucial places by unpaid volunteers, and held together by what one infrastructure engineer memorably called "glue and dreams," the global network is now being hammered by AI systems that treat it less like a shared resource and more like an all-you-can-eat buffet.
The stress is showing. Over the past eighteen months, web crawlers operated by AI companies have become so aggressive that major sites have had to implement increasingly baroque defenses. Rate-limiting that once targeted spam bots now struggles against distributed scraping operations that rotate through millions of IP addresses. The robots.txt convention—a gentleman's agreement from 1994 that asks crawlers to behave—has become a polite fiction that well-funded AI labs routinely ignore or creatively reinterpret.
The volunteer problem
What makes this precarious is not the volume alone but where the pressure lands. Critical pieces of internet infrastructure—the Domain Name System, key encryption libraries, widely used open-source packages—are often maintained by small teams or even single individuals who do the work for free. When an AI company's training pipeline accidentally DDoSes a volunteer-run service, there is no customer-support line to call. The maintainer either fixes it on their weekend or the service degrades for everyone.
This dynamic has already produced quiet crises. Several maintainers of foundational libraries have stepped back in the past year, citing burnout from handling support requests generated by AI-related traffic spikes. Others have begun adding restrictive licenses specifically to exclude AI training—a legal gray zone that may eventually require courts to sort out.
The bandwidth bill comes due
There is also a straightforward economic problem. AI agents that autonomously browse the web, book appointments, and fill out forms are multiplying. Each agent generates traffic that looks, from the server's perspective, like a human user—but agents do not see ads, do not subscribe, and do not contribute to the advertising-supported model that funds most of the open web. If agentic AI becomes as pervasive as its boosters predict, the implicit bargain that keeps content free may collapse.
Some publishers are already responding by locking content behind paywalls or requiring human-verification steps that break automated workflows. Others are negotiating licensing deals with AI companies, trading access for cash. The result is a web that is fragmenting into zones: one for humans, one for paying AI customers, and an increasingly barren commons in between.
Our take
The AI industry has treated the internet like an inexhaustible natural resource, which it is not. The engineers who keep the lights on deserve compensation and respect, not a tsunami of uncompensated traffic. If the major AI labs do not voluntarily adopt sustainable crawling practices and fund the infrastructure they depend on, regulators will eventually force the issue—and the rules they write are unlikely to be as permissive as the current free-for-all.
