The county historical society has two volunteers, a flatbed scanner, and forty years of a weekly paper that stopped printing in 1998. Digitising it took them six years. The archive went up behind a plain search box, and for a while the logs read like a small reading room: a genealogist in Perth, a retired teacher checking an obituary, a burst of school projects every March. Last winter the hosting bill tripled and the reading room emptied. Almost none of the new traffic was a person.
Sitting in the root of that site, as it sits in the root of nearly every site you have visited, is a plain text file of four lines. For thirty years those lines were the closest thing the open web had to a shared rule about who may read what. Nothing enforced them, which was the point, and is also why they are coming apart.
Four lines at the root of a site
The mechanism is almost comically simple. Add a slash and the word robots.txt to any domain name and you will usually get a short list of instructions written for software: a line naming which automated visitor is being addressed, then lines listing the paths it is asked to stay out of. A crawler fetches that file first, reads it, and then decides what to do.
Decides. That verb is the whole story. The file cannot block a request, hide a page, or slow anything down, and a crawler that ignores it gets the same pages at the same speed as one that reads it carefully. Compliance was a courtesy, extended by companies who found it cheaper to be trusted than to be blocked.
Most people publishing anything online rely on this arrangement without knowing the file exists. The parish council posting planning notices, the historical society with its scanner: both are protected, to whatever extent they are protected at all, by a habit.
A convention from 1994, written down in 2022
Martijn Koster defined the convention in 1994, in the years when one badly written crawler could knock a university server over. It stayed as it was for an extraordinarily long time. No standards body owned it, no registry recorded who honoured it, and search engines simply documented their own behaviour for everyone else to copy.
Only in September 2022 did it become a formal internet standard. RFC 9309, written by Koster with Gary Illyes, Henner Zeller and Lizzi Sassman, set down the syntax and what a crawler should do when the file cannot be fetched. Twenty-eight years of practice became a document.
The standard is admirably honest about its own limits. It states plainly that these rules are, in its words, "not a form of access authorization". The people who wrote it were not pretending to have built a lock. They were writing down a request, and saying so.
What fourteen thousand domains did in a single year
Requests hold until the value of ignoring them rises, and training data changed that arithmetic. In July 2024 the Data Provenance Initiative, a research group led by Shayne Longpre, published Consent in Crisis: The Rapid Decline of the AI Data Commons, the first large longitudinal audit of consent signals across the web. The team examined 14,000 domains sitting underneath three of the corpora that models are built from.
Between April 2023 and April 2024, something close to a mass withdrawal happened. Roughly 5 percent of all tokens in the C4 corpus became fully restricted by robots.txt, and among the sources that are most actively maintained and most heavily used, the figure was above 28 percent. Terms of service told an even starker story: around 45 percent of C4 was restricted by site terms.
The paper also found the two signals frequently disagree, because a site's text file was written for search engines in 2011 and its terms were written by a lawyer in 2023. Neither was designed for the question now being asked of them.
The day a network changed the default
Once a norm stops holding, somebody builds a gate. On 1 July 2025, Cloudflare, which sits in front of a large share of the world's websites, announced that it would begin blocking artificial intelligence crawlers by default for new domains rather than waiting for each site owner to opt in, and that it was building a market where crawling could be paid for rather than merely requested.
The reasoning in the company's own announcement was economic, not moral. Search sent readers back to the sites it indexed, and those readers paid the bills. Answers do not. By Cloudflare's own count, getting a visitor back from OpenAI is roughly 750 times harder than it was from the Google of a decade ago.
Whatever one makes of the numbers, the structural change is real. Permission is moving out of a file the publisher controls and into infrastructure the publisher rents.
What the historical society actually gained
A voluntary convention had one enormous virtue: it cost nothing to join. The two volunteers with the scanner had the same instrument available as a national newspaper. It was weak for everybody, equally.
Enforcement at the network layer is stronger and it is not equal. It belongs to whoever your site sits behind, on terms you did not negotiate, and it protects the small archive only for as long as that suits the company providing it. A rule anyone could use is turning into a service some people buy.
There is a second cost, easy to miss in the annoyance at a bandwidth bill. The same file that keeps a training crawler out keeps out the archivers who preserve the public record, and the search engines that are still how most people find a local obituary. Blocking everything takes one line, and can remove a small archive from the web's memory as thoroughly as letting the domain lapse.
Reading the file on your own street
This is one of the few pieces of internet plumbing an ordinary reader can inspect directly.
- Take any site you rely on, add /robots.txt to the end of the domain, and read what comes back. It is a public statement about who that publisher is willing to be read by.
- Do the same for your council, library, and local news site. Look for whether the whole site is closed to everything, which is usually an accident rather than a decision.
- Remember that every line is a request. Anything you need kept private wants a password, not a polite note to software.
- If you run a small site, read your server logs before closing the doors. Naming the agents costing you money is better aimed than shutting out every machine.
- Ask whether the Internet Archive can still reach the pages you would be sad to lose. A site excluded from everything will not be there when the hosting stops.
The historical society did something unglamorous and sensible. They named the three agents responsible for most of the load, left the rest of the internet alone, and put a line on the front page asking anyone using the archive in bulk to email first. Two people did. One sent corrections to forty-one scanned obituaries, which no crawler has ever done.