llms.txt: what 137,000 sites' server logs actually show
The file has been sold for eighteen months as the way into AI answers. Then somebody read the logs.
An entire optimisation practice grew up around a single text file — scoring tools, validators, vendor packages, conference talks — before anyone checked whether its intended readers had arrived. In June 2026 somebody finally checked. The answer is not that llms.txt is worthless. It is that almost everyone has been wrong about who reads it, and that mistake changes what you should do about it.
The object in question
What llms.txt is
A single Markdown file at the root of a website. It says what the site is and links to the content the owner considers most important, on the theory that a language model or an agent can read it and orient itself without crawling everything. Jeremy Howard, co-founder of Answer.AI, proposed it in September 2024; a version 2 of the specification landed on 10 August 2026, two days before this piece was published.
It was a sensible engineering proposal. The AI-visibility framing arrived later, bolted on by the SEO industry as adoption spread, on the assumption that AI platforms would eventually reward sites that had one.
Two things it is routinely confused with, and isn't. It is not the practice of publishing Markdown copies of every page on your site — that is a separate tactic with separate problems, covered in our complete guide to the .md file format. And despite the filename, it is not a robots.txt-style directive. It permits nothing, blocks nothing, controls nothing. It is a suggestion left on a doorstep.
The official position
What Google says about llms.txt
Google published its first guide to optimising for generative AI features in Search in May 2026, and added a subsection specifically about llms.txt the following month, in a part of the document given over to mythbusting.
The position is not subtle. The guide states that you don't need to create new machine-readable files, AI text files, markup, or Markdown to appear in Google Search including its generative AI capabilities, as Google Search itself doesn't use them
. On whether to keep a file you already have, it says that doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them
.
Then, days later, the Chrome team shipped an llms.txt check inside Lighthouse's experimental Agentic Browsing audits, on the reasoning that without such a file agents may spend longer crawling a site to work out its structure.
This reads as a contradiction and isn't. Search and the agent stack are two different products with two different needs, and Google is answering for both separately. When John Mueller was pressed on it, his framing was that llms.txt is not done for search — that it is a temporary crutch, perhaps to save some tokens
for AI coding tools parsing developer documentation, rather than something a general website needs to think about.
Whether llms.txt "works" is the wrong question. The right one is which machine you are writing it for. The answer determines everything, and it is almost never the machine people think.
The evidence
What the server logs show
Until recently the entire llms.txt debate ran on single-site anecdotes. On 15 June 2026 Ahrefs published the first study large enough to settle it: server logs and bot analytics for 137,210 domains, covering every request to /llms.txt paths across that population during May 2026.
Most of these files have never been touched
The method matters, because it closes the escape hatches. Ahrefs checked each domain root for a file returning HTTP 200, confirmed each one was genuinely Markdown rather than an HTML page, and screened titles and content for error signals to rule out soft 404s and phantom files. The population being measured is real files that really resolve — not a count of sites that once configured something.
Of roughly 38,000 valid files, about 1,100 received any traffic at all. Not much traffic — any. The other 97% sat there for a full month untouched by a bot, a crawler, or a human being. Of the requests that did land, 96% came from bots.
The finding that reframes the rest came from looking at the failures. Ahrefs examined every request to an /llms.txt path that returned a 404, and found the AI-bot share of those requests was zero — 98% of them were humans, SEOs typing competitors' URLs into a browser to check. Where valid files drew almost entirely bot traffic, missing files drew almost entirely people.
Agents fetch llms.txt when a link, an index, or an instruction tells them it exists. Publishing one does not put you on anybody's radar. It does not announce itself. If nothing points at it, nothing arrives.
The readership
Who actually reads llms.txt
This is the part nobody in marketing has seen, and it is the most useful table in the study. Ahrefs classified every user agent that fetched an llms.txt file into twelve categories. Here they are by share of requests.
The largest reader is the SEO industry checking whether you have one
Read the top of that list again. The single largest consumer of llms.txt files is the SEO audit industry, checking whether sites have one. Add the GEO/AEO scoring tools, the dedicated llms.txt validators and the research crawlers, and roughly 12% of all traffic to these files is the industry studying itself — a supply chain of tools auditing a standard before its intended readers turned up.
Now read the bottom. AI retrieval bots — the crawlers that fetch pages to compose the answers everyone is optimising for — account for 1.1%. Slackbot, the link-preview service that fires when someone pastes a URL into a chat window, fetched more llms.txt files than PerplexityBot did.
Combine all four AI categories and you reach 19.5%, which sounds respectable until you notice how it splits. The largest AI reader by a wide margin is not a search bot at all. It is the agent layer: coding agents and the infrastructure built to serve them, at 10.5%. GPTBot led the training crawlers at 4.51%; but among the rest, the busiest single AI fetcher of llms.txt files was Claude-Code — outpacing every AI retrieval bot and every AI assistant in the dataset.
There is one more entry worth stopping on. The largest single research crawler in the dataset identifies itself as prompt-injection-survey. Somebody is systematically studying llms.txt files as a place to put instructions that agents are built to ingest and trust — and they arrived before the legitimate readers did. That is not an attack and should not be read as one; it is security researchers turning up early, which is roughly the best available outcome. But it tells you what kind of object this file really is: a trust boundary, not a marketing asset.
A fetch is not a read. Every number here is a ceiling on consumption, not a measure of it. A bot requesting the file tells you nothing about whether anything acted on what was inside — which means the real influence of llms.txt is somewhere at or below these figures, never above them.
The comparison everyone gets wrong
llms.txt vs robots.txt vs sitemap.xml
Most of the confusion in this space traces back to one framing: the pitch that llms.txt is "robots.txt for the AI era". It isn't, and the differences are the whole argument.
Only one of them controls anything
| File | Status | What it does | Who honours it |
|---|---|---|---|
| robots.txt | Formal standard, decades old | Controls crawler access. Directive. | Every major crawler, as a matter of policy |
| sitemap.xml | Supported convention | Aids discovery of URLs you already publish | Search engines, explicitly |
| llms.txt | Community proposal, no standards body | Describes and links. Controls nothing. | No AI platform has committed to reading it |
It is documentation, published at a conventional address. Its value is entirely a function of whether the specific machines you care about have been told to go and read it.
The honest yes
The one case where it pays off
All of which makes it sound like the answer is no. For most sites it is. But the evidence points to a real, narrow yes, and it deserves to be stated as precisely as the no.
The machines demonstrably reading these files are coding agents and agentic infrastructure. So the file earns its place when one of these is true:
- Your buyers research vendors through coding agents or agentic browsers. If someone evaluating you is doing it inside a terminal or an agentic browser rather than a search box, the file is in front of a reader who exists.
- You publish developer or API documentation. This is the use case the format was effectively built for, and the one Google's own people describe as legitimate.
- Your site exposes actions an agent might take — booking, quoting, configuring, ordering. An agent arriving to do something benefits from a map in a way an agent arriving to summarise something does not.
Notice what all three have in common. This is a business-to-agent decision, not a search decision. If your reason for shipping llms.txt is that you want to be cited more often in ChatGPT or appear in AI Overviews, the evidence says you are solving the wrong problem with the wrong file, and Google has told you so in writing.
It is also worth naming what would change this answer, because a verdict that can't be falsified isn't worth much. If agents end up mediating retrieval — an agent fetching and reading on a person's behalf, rather than a retrieval bot pulling pages into an index — then the layer that already reads llms.txt becomes the layer that determines visibility. The file would start mattering through the agent, not through the crawler. That is a plausible future and not a current fact, and the honest way to hold it is as a reason to watch your own logs rather than a reason to buy anything today.
The decision
Should you ship one?
Five questions. No score, no grade — one of four answers, based on the conditions above. Nothing is sent anywhere; the logic runs in your browser.
Should you ship an llms.txt?
Answer all five. The verdict appears underneath.
Answer all five to see it.
If you're shipping it
How to write one properly
Only read this if you landed on ship it. Four rules, each of which comes out of the evidence rather than out of anyone's opinion.
1. Route agents to it. Publishing is half the job.
Nothing probes for a file it hasn't been told about. Link it from your HTML, reference it in your documentation, and mention it anywhere an agent receives instructions about your site. An unlinked llms.txt is the single most common way to end up in the 97%.
2. Keep it descriptive. Nothing instruction-shaped.
Links and short descriptions of what's behind them. The moment a context file starts issuing instructions to whatever reads it, you have built an injection surface rather than a map — and agents are designed to trust this file in a way they don't trust other input.
3. Only link to resources you control.
A third-party page you linked in March can say something else in September, and no one on your side will know. Same-origin, or a domain you own.
4. Treat it as code, not as content.
Version-control it, restrict who can edit it, alert on unauthorised changes, and review anything your platform generates on your behalf. A stale or altered file misleads every agent that reads it, silently.
Here is our own, in full — short, linked, descriptive, with nothing an attacker would find worth rewriting:
# auxfirst
> Agentic experience design. We design how AI agents
> and the people accountable for them work together.
## Practice
- [What agentic user experience is](/agentic-user-experience.html): the discipline, defined
- [The AUX heuristics](/heuristics.html): the working principles
## Reference
- [MD files explained](/news/md-file-markdown-guide.html): the .md format in full
- [The agentic commerce stack](/agentic-commerce-stack.html): protocols, feeds, receipts
## Contact
- [Start a conversation](/index.html#contact)If you want to see what your own file looks like rendered — or check that a file someone handed you is what it claims to be — you can open it in our .md viewer. It runs in the browser and nothing is uploaded, which matters for exactly the category of file this article is about.
The alternative
What to do instead
If the file isn't the lever, the obvious question is what is. Three answers, in the order they pay off.
Structure the pages themselves. Google's own guidance is unusually direct: its generative features are grounded in the ordinary Search index, which means the work that makes a page retrievable is the work that has always made a page retrievable. The guide explicitly tells site owners to stop chunking content for AI, stop rewriting for AI systems, and stop overfocusing on structured data as an AI requirement. There is no separate AI channel to optimise for.
Write context files for the agents that genuinely read them. The agent layer is where the fetch data actually points, and llms.txt is the least interesting file in that category. AGENTS.md governs how work gets done in a workspace. A brand.md governs what the output is allowed to sound like. Both are read by systems you operate, which means you can verify they work instead of hoping. Our note on why .md files became the context format covers why that shape won.
Then design for the agents that arrive anyway. Whether or not you ship a context file, agents are already reading your site, and most organisations have never looked at what those agents can and cannot do once they get there. That is the question behind the Agent Operability Audit, and — on the commercial surface — behind the agentic commerce stack.
The uncomfortable summary: an entire optimisation practice grew up around a file, complete with scoring tools, validators, vendor packages and conference talks, before anyone checked whether the readers had arrived. Mueller called it a temporary crutch. What the logs suggest is that the crutch got a supply chain before it got a patient.
Read next & further reading
Method · Every figure in this piece comes from the Ahrefs study of 137,210 domains and describes request activity during May 2026; they will age, and they are ceilings on consumption rather than measures of it. Google's wording is quoted from its published guide. We verified both primary sources directly rather than relying on secondary coverage. Where the evidence supports only a narrow conclusion, we have stated the narrow one.