# llms.txt: What 137,000 Sites' Server Logs Actually Show

> Canonical: https://auxfirst.com/news/llms-txt-evidence.html
> Markdown representation of this article, for agents and LLM crawlers.

97% of llms.txt files got zero requests in May 2026. What the logs show, who actually reads the file, and the one case where shipping it pays off.

By Emil Krzemiński, founder of auxfirst. Published 12 August 2026.

An entire optimisation practice grew up around a single text file — scoring tools, validators, vendor packages, conference talks — before anyone checked whether its intended readers had arrived. In June 2026 somebody finally checked. The answer is not that llms.txt is worthless. It is that almost everyone has been wrong about *who reads it*, and that mistake changes what you should do about it.

## What llms.txt is

A single Markdown file at the root of a website. It says what the site is and links to the content the owner considers most important, on the theory that a language model or an agent can read it and orient itself without crawling everything. Jeremy Howard, co-founder of Answer.AI, proposed it in September 2024; a version 2 of the specification landed on 10 August 2026, two days before this piece was published.

It was a sensible engineering proposal. The AI-visibility framing arrived later, bolted on by the SEO industry as adoption spread, on the assumption that AI platforms would eventually reward sites that had one.

Two things it is routinely confused with, and isn't. It is **not** the practice of publishing Markdown copies of every page on your site — that is a separate tactic with separate problems. And despite the filename, it is **not** a robots.txt-style directive. It permits nothing, blocks nothing, controls nothing. It is a suggestion left on a doorstep.

## What Google says about llms.txt

Google published its first guide to optimising for generative AI features in Search in May 2026, and added a subsection specifically about llms.txt the following month, in a part of the document given over to mythbusting.

The position is not subtle. The guide states that you don't need to create new machine-readable files, AI text files, markup, or Markdown to appear in Google Search including its generative AI capabilities, "as Google Search itself doesn't use them." On whether to keep a file you already have, it says that doing so "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

Then, days later, the Chrome team shipped an llms.txt check inside Lighthouse's experimental Agentic Browsing audits, on the reasoning that without such a file agents may spend longer crawling a site to work out its structure.

This reads as a contradiction and isn't. Search and the agent stack are two different products with two different needs, and Google is answering for both separately. When John Mueller was pressed on it, his framing was that llms.txt is not done for search — that it is a temporary "crutch, perhaps to save some tokens" for AI coding tools parsing developer documentation, rather than something a general website needs to think about.

> **The distinction that runs through this whole piece.** Whether llms.txt "works" is the wrong question. The right one is **which machine you are writing it for**. The answer determines everything, and it is almost never the machine people think.

## What the server logs show

Until recently the entire llms.txt debate ran on single-site anecdotes. On 15 June 2026 Ahrefs published the first study large enough to settle it: server logs and bot analytics for 137,210 domains, covering every request to `/llms.txt` paths across that population during May 2026.

| Stage | Count | Share |
|---|---|---|
| Domains analysed | 137,210 | — |
| Publish a valid llms.txt | 38,360 | 28% of domains |
| Fetched even once in May | ~1,100 | 3% of those files |

Ahrefs note that their analytics customers skew technical and SEO-aware, so 28% adoption is an upper bound rather than a picture of the open web.

The method matters, because it closes the escape hatches. Ahrefs checked each domain root for a file returning HTTP 200, confirmed each one was genuinely Markdown rather than an HTML page, and screened titles and content for error signals to rule out soft 404s and phantom files. The population being measured is real files that really resolve — not a count of sites that once configured something.

Of roughly 38,000 valid files, about 1,100 received any traffic at all. Not much traffic — *any*. The other 97% sat there for a full month untouched by a bot, a crawler, or a human being. Of the requests that did land, 96% came from bots.

The finding that reframes the rest came from looking at the failures. Ahrefs examined every request to an `/llms.txt` path that returned a 404, and found the AI-bot share of those requests was zero — 98% of them were humans, SEOs typing competitors' URLs into a browser to check. Where valid files drew almost entirely bot traffic, missing files drew almost entirely people.

**No AI system goes looking for a file that isn't there.** Agents fetch llms.txt when a link, an index, or an instruction tells them it exists. Publishing one does not put you on anybody's radar. It does not announce itself. If nothing points at it, nothing arrives.

## Who actually reads llms.txt

Ahrefs classified every user agent that fetched an llms.txt file into twelve categories. Here they are by share of requests. Categories sum to 96%; the remaining 4% was human traffic.

| Category | Share of requests | AI? |
|---|---|---|
| SEO audit tools | 21.7% | |
| Other & unidentified | 14.9% | |
| General web crawlers | 13.1% | |
| Tech profiling tools | 11.6% | |
| AI agents & agentic infrastructure | 10.5% | AI |
| GEO / AEO scoring tools | 5.8% | |
| AI training crawlers | 5.3% | AI |
| llms.txt discoverability bots | 3.6% | |
| Service & social bots | 2.9% | |
| Research bots | 2.7% | |
| AI assistants | 2.5% | AI |
| AI retrieval bots | 1.1% | AI |

Read the top of that list again. The single largest consumer of llms.txt files is the SEO audit industry, checking whether sites have one. Add the GEO/AEO scoring tools, the dedicated llms.txt validators and the research crawlers, and roughly 12% of all traffic to these files is the industry studying itself — a supply chain of tools auditing a standard before its intended readers turned up.

Now read the bottom. **AI retrieval bots — the crawlers that fetch pages to compose the answers everyone is optimising for — account for 1.1%.** Slackbot, the link-preview service that fires when someone pastes a URL into a chat window, fetched more llms.txt files than PerplexityBot did.

Combine all four AI categories and you reach 19.5%, which sounds respectable until you notice how it splits. The largest AI reader by a wide margin is not a search bot at all. It is the agent layer: coding agents and the infrastructure built to serve them, at 10.5%. GPTBot led the training crawlers at 4.51%; but among the rest, the busiest single AI fetcher of llms.txt files was Claude-Code — outpacing every AI retrieval bot and every AI assistant in the dataset.

> The file has a readership. It is developers' agents, reading developers' documentation.

There is one more entry worth stopping on. The largest single research crawler in the dataset identifies itself as `prompt-injection-survey`. Somebody is systematically studying llms.txt files as a place to put instructions that agents are built to ingest and trust — and they arrived before the legitimate readers did. That is not an attack and should not be read as one; it is security researchers turning up early, which is roughly the best available outcome. But it tells you what kind of object this file really is: a trust boundary, not a marketing asset.

One caveat the coverage keeps dropping: a fetch is not a read. Every number here is a ceiling on consumption, not a measure of it. A bot requesting the file tells you nothing about whether anything acted on what was inside.

## llms.txt vs robots.txt vs sitemap.xml

Most of the confusion in this space traces back to one framing: the pitch that llms.txt is "robots.txt for the AI era". It isn't, and the differences are the whole argument.

| File | Status | What it does | Who honours it |
|---|---|---|---|
| robots.txt | Formal standard, decades old | Controls crawler access. Directive. | Every major crawler, as a matter of policy |
| sitemap.xml | Supported convention | Aids discovery of URLs you already publish | Search engines, explicitly |
| llms.txt | Community proposal, no standards body | Describes and links. Controls nothing. | No AI platform has committed to reading it |

The practical consequence: you cannot use llms.txt to keep anything out, you cannot use it to get anything in, and there is no compliance body to appeal to if a platform ignores it. It is documentation, published at a conventional address. Its value is entirely a function of whether the specific machines you care about have been told to go and read it.

## The one case where it pays off

All of which makes it sound like the answer is no. For most sites it is. But the evidence points to a real, narrow yes, and it deserves to be stated as precisely as the no.

The machines demonstrably reading these files are coding agents and agentic infrastructure. So the file earns its place when one of these is true:

- **Your buyers research vendors through coding agents or agentic browsers.** If someone evaluating you is doing it inside a terminal or an agentic browser rather than a search box, the file is in front of a reader who exists.
- **You publish developer or API documentation.** This is the use case the format was effectively built for, and the one Google's own people describe as legitimate.
- **Your site exposes actions an agent might take** — booking, quoting, configuring, ordering. An agent arriving to *do* something benefits from a map in a way an agent arriving to summarise something does not.

Notice what all three have in common. This is a business-to-agent decision, not a search decision. If your reason for shipping llms.txt is that you want to be cited more often in ChatGPT or appear in AI Overviews, the evidence says you are solving the wrong problem with the wrong file, and Google has told you so in writing.

It is also worth naming what would change this answer, because a verdict that can't be falsified isn't worth much. If agents end up mediating retrieval — an agent fetching and reading on a person's behalf, rather than a retrieval bot pulling pages into an index — then the layer that already reads llms.txt becomes the layer that determines visibility. The file would start mattering through the agent, not through the crawler. That is a plausible future and not a current fact, and the honest way to hold it is as a reason to watch your own logs rather than a reason to buy anything today.

## Should you ship one?

The interactive version of this decision is on the HTML page. The logic:

1. Do your buyers research vendors like you using coding agents or agentic browsers?
2. Do you publish developer or API documentation?
3. Does your site expose actions an agent could take — booking, quoting, configuring, ordering?
4. Does your CMS or site platform generate the file automatically?
5. Is there a named person who will keep it current and notice if it changes?

If any of 1–3 is yes **and** 5 is yes: **ship it, and route agents to it.** You have a readership and someone to keep the file honest, which puts you in a small minority.

If any of 1–3 is yes, 5 is no, and 4 is yes: **let the platform generate it, then leave it alone.** Review it once, make sure it links only to pages you control, and spend the hour you saved on the pages themselves.

If any of 1–3 is yes, 5 is no, and 4 is no: **skip it — an unowned file is worse than none.** Agents are built to trust this file, so a stale or quietly altered one misleads every agent that reads it and no one on your side finds out.

If none of 1–3 applies: **skip it, unless your platform already makes one.** No demonstrated readership, and no route to Google. The base rate for a file like yours is a 97% chance nothing ever fetches it.

## How to write one properly

Only if you landed on *ship it*. Four rules, each drawn from the evidence rather than from opinion.

**1. Route agents to it. Publishing is half the job.** Nothing probes for a file it hasn't been told about. Link it from your HTML, reference it in your documentation, and mention it anywhere an agent receives instructions about your site. An unlinked llms.txt is the single most common way to end up in the 97%.

**2. Keep it descriptive. Nothing instruction-shaped.** Links and short descriptions of what's behind them. The moment a context file starts issuing instructions to whatever reads it, you have built an injection surface rather than a map — and agents are designed to trust this file in a way they don't trust other input.

**3. Only link to resources you control.** A third-party page you linked in March can say something else in September, and no one on your side will know. Same-origin, or a domain you own.

**4. Treat it as code, not as content.** Version-control it, restrict who can edit it, alert on unauthorised changes, and review anything your platform generates on your behalf. A stale or altered file misleads every agent that reads it, silently.

Here is ours, in full:

```
# auxfirst

> Agentic experience design. We design how AI agents
> and the people accountable for them work together.

## Practice
- [What agentic user experience is](/agentic-user-experience.html): the discipline, defined
- [The AUX heuristics](/heuristics.html): the working principles

## Reference
- [MD files explained](/news/md-file-markdown-guide.html): the .md format in full
- [The agentic commerce stack](/agentic-commerce-stack.html): protocols, feeds, receipts

## Contact
- [Start a conversation](/index.html#contact)
```

## What to do instead

**Structure the pages themselves.** Google's own guidance is unusually direct: its generative features are grounded in the ordinary Search index, which means the work that makes a page retrievable is the work that has always made a page retrievable. The guide explicitly tells site owners to stop chunking content for AI, stop rewriting for AI systems, and stop overfocusing on structured data as an AI requirement. There is no separate AI channel to optimise for.

**Write context files for the agents that genuinely read them.** The agent layer is where the fetch data actually points, and llms.txt is the least interesting file in that category. `AGENTS.md` governs how work gets done in a workspace. A `brand.md` governs what the output is allowed to sound like. Both are read by systems you operate, which means you can verify they work instead of hoping.

**Then design for the agents that arrive anyway.** Whether or not you ship a context file, agents are already reading your site, and most organisations have never looked at what those agents can and cannot do once they get there.

The uncomfortable summary: an entire optimisation practice grew up around a file, complete with scoring tools, validators, vendor packages and conference talks, before anyone checked whether the readers had arrived. Mueller called it a temporary crutch. What the logs suggest is that the crutch got a supply chain before it got a patient.

## Read next

- [MD files explained — the complete .md file format guide](https://auxfirst.com/news/md-file-markdown-guide.html)
- [Markdown for AI — why .md files became the context format](https://auxfirst.com/news/markdown-for-ai.html)
- [The .md viewer — open and export any Markdown file, locally](https://auxfirst.com/tools/md-viewer.html)
- [The Agent Operability Audit](https://auxfirst.com/agent-operability-audit.html)
- [What Is Agentic User Experience (AUX)?](https://auxfirst.com/agentic-user-experience.html)
- [The 10 AUX heuristics](https://auxfirst.com/heuristics.html)

## Sources

- Louise Linehan and Xibeijia Guan, "We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read", Ahrefs, 15 June 2026 — https://ahrefs.com/blog/llmstxt-study/
- Google Search Central, "Optimizing your website for generative AI features on Google Search" — https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
- Jeremy Howard, the /llms.txt proposal (September 2024; v2, August 2026) — https://llmstxt.org/

Method: every figure in this piece comes from the Ahrefs study of 137,210 domains and describes request activity during May 2026; they will age, and they are ceilings on consumption rather than measures of it. Google's wording is quoted from its published guide. Both primary sources were verified directly rather than via secondary coverage.

---

Emil Krzemiński is the founder of auxfirst (https://auxfirst.com/), the agentic experience design agency — helping product, developer and business teams design AI systems that remember, adapt, and earn the right to act.