llms.txt Takes Ten Minutes to Publish and Is Almost Impossible to Measure. Here's the Trap in Every Report You've Read.
llms.txt is the easiest thing in the AI-visibility toolkit to sell. A single markdown file at the root of your domain, ten minutes to publish, a WordPress plugin will generate it for you, and about 5,400 people a month search for it in the US. Every agency deck has a slide about it.
It is also, on most sites, completely unmeasurable — and I can show that with logs instead of arguing it.
I run server-side logging on two live home-services sites in different markets, and the full per-agent and per-day tables are published as an open dataset. Last week of August into September: 6,676 requests from AI crawlers and assistants, fourteen distinct agents. Requests for llms.txt across both sites: zero.
That looks like a headline. It isn't, and understanding why is worth more than the headline would have been.
What is llms.txt supposed to do?
The pitch: a plain-text file at yourdomain.com/llms.txt giving a language model a clean, curated map of your site — who you are, what you sell, which pages matter — in markdown, with no navigation, no cookie banner, no scripts. robots.txt for meaning rather than permission.
It's a sensible idea. Models handle clean text better than a page wrapped in three layers of theme markup, and handing a machine an authoritative summary of your own business beats letting it assemble one from whatever it scrapes.
What it is not is a standard. No engine has committed to reading it. So the honest question isn't "should I publish one" — it costs almost nothing — but "can I tell whether anything read it?"
Did any AI agent actually fetch it?
In my logs, no. Here's the sample, two independent sites, every row a request from a known AI user agent:
| Site A (appliance repair, multi-location) | Site B (home services, different market) | |
|---|---|---|
| Window | Aug 25 – Sep 1 (8 days) | Aug 28 – Sep 1 (5 days) |
| Logged AI requests | 3,498 | 3,178 |
| Distinct agents | 14 | 10 |
| Live answer fetches | 425 | 104 |
Requests for llms.txt |
0 | 0 |
Live answer fetches are the ones that matter: a page pulled while an assistant composes a reply for a real person, as opposed to background crawling or training collection. On site A that's ChatGPT 256, Claude 129, Perplexity 40. The distinction, and why no third-party tool reports it, is the whole reason I built the logger.
Two sites, two markets, 6,676 requests, not one for llms.txt.
I sat on that number instead of publishing it, because a zero in a log has two possible explanations that lead to opposite advice.
Why can't a CMS-level logger see llms.txt at all?
Because of where the logger runs. Mine is a WordPress plugin: it fires when PHP executes. It sees every request WordPress handles — and nothing else.
Now look at what's physically on disk:
| Path | Exists as a file? | Who serves it | Requests logged, site A |
|---|---|---|---|
/robots.txt |
No | generated by WordPress on request | 221 |
/sitemap_index.xml |
No | generated by the SEO plugin on request | 16 |
/llms.txt |
Yes — a real 79 KB file | the web server, directly | 0 |
There it is. robots.txt and the sitemap don't exist as files — WordPress builds them when asked, PHP runs, the logger fires, and I see the requests. llms.txt is a genuine static file sitting on disk, so the web server hands it over without ever starting PHP. My logger isn't merely unlikely to catch those requests. It is structurally incapable of catching a single one.
So the zero measures my instrument, not the internet. Every agent in that table could have downloaded llms.txt every day of the window and the log would look identical. The same is true on site B: static file, PHP-level logger, guaranteed zero.
This isn't a quirk of my setup — it's the default. The popular WordPress plugins write llms.txt as a static file, and nearly every bot-tracking plugin hooks into PHP. Which means most people who tell you "we checked, AI never requests llms.txt" are reading their own blind spot. So are the ones showing you a suspiciously tidy dashboard.
I could have published the zero. Great contrarian headline, technically true, and wrong. The rule that stops it is the one I sell: before a number becomes a claim, prove the instrument could have recorded the opposite result. Mine couldn't.
What do the logs prove, then?
They're useless for llms.txt and genuinely useful for everything else. Most-requested URLs, same windows:
| Site A | Requests | Site B | Requests |
|---|---|---|---|
/robots.txt |
221 | /robots.txt |
112 |
| A "most reliable brands" data page | 37 | /sitemap_index.xml |
39 |
| Homepage | 35 | A brand failure-rate data page | 24 |
| A statistics page from the client's own job data | 34 | A "most reliable brands" data page | 17 |
| A brand-specific failure article | 22 | A city service page | 10 |
Three findings I'd stand behind, each of them replicated across two unrelated sites:
robots.txt is the single most-requested URL on both sites — on site A by a factor of six over the next URL. Not a page. Not the homepage. The permission file. Whatever happens with newer formats, the machines are demonstrably hammering the old ones right now. If your robots.txt accidentally disallows an AI agent, that is not a subtle problem, and on site B the sitemap is the second most-requested URL, so a stale sitemap is a live one too.
Data pages outrank service pages for machine attention. On site A the two most-requested content pages are both built from the client's own service records. On site B the top two are both /resources/ data pages, ahead of every location page on the domain. Those same pages are the ones that attracted the most referring domains. Original numbers get pulled by machines and cited by humans — one mechanism, two payoffs.
The agent mix is site-specific, so a generic benchmark is worthless. On site A, OpenAI's search crawler made 536 requests and GPTBot 59. On site B those flip: GPTBot 691, search crawler 100. Same week, same operator, opposite ratios. Anyone quoting you an industry-average AI crawl profile is quoting someone else's site.
One more, quieter: Meta's crawler is the largest single agent on both sites — 1,132 and 1,096 requests — and every one of those is background crawling, not answering anybody. Blend crawl and live fetches into one "AI traffic" number, as most reports do, and Meta drowns out the signal completely. That split, and why click-based analytics misses it entirely, is the subject of the post on citations versus orders.
How would you measure llms.txt properly?
One level below a plugin. Requests for a static file are only visible where the file is actually served:
- The web-server access log — Apache, Nginx, LiteSpeed. Every request for every file, PHP or not. On shared hosting it's often in the control panel; get raw log access before you promise anyone a report.
- The CDN or edge log — Cloudflare, Fastly. If something sits in front of your origin it may be answering these requests without your server ever hearing about it: a second blind spot stacked on the first.
- Force it through the application. Delete the static file and serve
/llms.txtfrom a route your own code owns. Then your existing logging just works, at the cost of a little overhead per request. On a site where the answer matters, this is the one I'd pick.
Option three generalises into a rule worth keeping: if a thing has to be measured, don't let the web server answer for it. Static files are fast and invisible. Choose which of the two you need, per file.
Should you publish llms.txt at all?
Yes — and hold it loosely.
Ten minutes, no downside, and the worst case is a file nobody requests. Write it yourself rather than letting a plugin dump your sitemap into it: an authoritative summary of your business in your own words is worth more to a model than a list of 400 URLs. Keep it current, because a stale one is worse than none.
What you shouldn't do is let anyone sell you llms.txt as a measured win. Someone quoting fetch numbers from a CMS plugin is quoting an artifact; someone quoting no numbers is quoting a hunch. The honest position today: cheap, plausible, unproven, and invisible on most sites unless you go down to the access log.
Meanwhile the same logs point at something that is measured, twice, on two unrelated domains: AI agents are pulling robots.txt more than any page you own, and reaching for data pages over service pages. That's where I'd put the ten minutes.
The gap between what's easy to publish and what's possible to verify runs through this entire field — what really separates GEO, AEO and SEO is mostly which claims each can back with a number. A channel you can't measure is a channel somebody else gets to describe for you.
Final caveat, since I'm making a point of them: these windows are five and eight days. Two sites, one operator, one week. That's a probe, not a study — and be suspicious of anyone whose AI-visibility numbers arrive with no window attached at all.
I build the measurement layer for AI visibility — server-side logging that separates a background crawl from a page pulled live to answer a real person, and attribution that carries that visit through to the invoice. The Citation Tracker page describes exactly what it records and what it cannot prove; engagement types and prices are public on pricing. If you want to know what AI is actually doing on your site, let's talk.
Related: