What AI agents actually do
on a live website.
Every published number about AI crawler behaviour comes from either a vendor sampling prompts from the outside or a survey. This comes from the inside: a server-side log on two live sites that records each request from a known AI agent and separates background crawling from a page pulled to answer a real person. Free to cite, with the windows attached and the limits stated.
Requests by agent and purpose.
Site A: appliance repair, multi-metro US, Aug 25 – Sep 1, 2026 (8 days), 3,498 requests, 14 distinct agents. Site B: home services, different market, Aug 28 – Sep 1, 2026 (5 days), 3,178 requests, 10 distinct agents. Highlighted rows are live answer fetches.
| Agent | Purpose | Site A | Site B |
|---|---|---|---|
| meta-externalagent | crawl | 1,150 | 1,188 |
| Amazonbot | crawl | 624 | 490 |
| OAI-SearchBot | search index | 541 | 101 |
| ChatGPT-User | live answer fetch | 256 | 53 |
| Google-Other | crawl | 233 | 134 |
| PerplexityBot | index | 206 | 11 |
| Claude-User | live answer fetch | 129 | 51 |
| Bytespider | crawl | 116 | 64 |
| ClaudeBot | index | 97 | 394 |
| GPTBot | training corpus | 60 | 692 |
| Perplexity-User | live answer fetch | 40 | — |
| Claude-SearchBot | search index | 25 | — |
| Google-Extended | training corpus | 19 | — |
| Gemini | live answer fetch | 2 | — |
Two things worth pulling out. meta-externalagent is the largest single agent on both sites and every one of those requests is background collection, so any blended “AI traffic” number is mostly Meta. And the OpenAI ratio inverts between the two sites: on A the search crawler made 541 requests against GPTBot's 60, on B it is GPTBot 692 against 101. Same operator, same week, opposite mix — which is why an industry-average AI crawl profile is not a useful thing to quote.
Per day, site A.
| Day | Crawl | Live answer fetches | Note |
|---|---|---|---|
| Aug 25 | 235 | 18 | partial day, logger installed 12:19 |
| Aug 26 | 533 | 101 | |
| Aug 27 | 537 | 68 | |
| Aug 28 | 380 | 20 | |
| Aug 29 | 192 | 13 | lowest live day |
| Aug 30 | 438 | 99 | |
| Aug 31 | 489 | 58 | |
| Sep 1 | 269 | 48 | partial day, cut at 19:56 |
Live fetches swing from 13 to 101 inside one week — a factor of 7.8 — while crawl volume moves far less. Anyone reporting a week-over-week percentage change on this metric is reporting noise, and anyone reporting a single day is reporting almost nothing.
Which URLs got pulled into answers.
Site A, live answer fetches only, crawlers excluded. Page descriptions rather than URLs, because the site belongs to a client.
| Page | Type | Live fetches |
|---|---|---|
| Brand reliability data page | data | 35 |
| /robots.txt | infrastructure | 34 |
| Statistics page built from the operator's own job records | data | 32 |
| Homepage | other | 29 |
| Component failure article for one brand | data | 20 |
| City-level repair cost page | data | 14 |
| Second city-level repair cost page | data | 12 |
| Food safety question page | data | 7 |
These eight URLs carry roughly 44% of all live fetches on the site. Six of the eight are pages built around a factual claim with a number attached. Not one is a service page or a location page, and those make up the large majority of the 1,703 URLs that were crawled. The full analysis →
Cite this data.
- Across 6,676 logged AI agent requests on two live sites, 529 were live answer fetches — 7.9%.
- On site A, AI agents touched 1,703 distinct URLs in 8 days and pulled 207 of them into live answers: 12.2%.
- Of those 207 URLs, 168 received exactly one live fetch, 31 received two to five, and 8 received six or more.
- Daily live answer fetches on site A ranged from 13 to 101 — a 7.8x swing inside one week.
- /robots.txt was the single most-requested URL on both sites.
- On site A, OpenAI's search crawler made 541 requests against GPTBot's 60. On site B the ratio inverted: GPTBot 692, search crawler 101.
- meta-externalagent was the largest single agent on both sites — 1,150 and 1,188 requests — and none of it was answering a person.
Ihor Odariuk, “AI Crawler Data: 6,676 Requests Measured”, server-side logs from two live home-services websites, August–September 2026. https://odariuk.com/ai-crawler-data
Generated from the same numbers rendered above, so a downloaded file cannot drift from the page. Each file carries its window and the caveats in the header rows.
Free to use with attribution. If you need a cut that is not here — one agent isolated, a longer window, a breakdown by page type — ask and I will either run it or say why the data will not support it.
How this was measured.
Where the logger sits
Server-side on every request, ahead of the page cache. This matters: a full-page cache serves the second and third fetch of the same URL without reaching PHP, so a counter placed behind the cache reports a fraction of the traffic and the owner concludes the channel is small.
How requests are classified
By user-agent, into four purposes: background crawling, training corpus collection, search-index building, and a live fetch made while composing an answer for a person. The last group is the one no third-party tool reports, because it exists only in your own log.
Definitions used on this page
A request is one logged hit from a matched agent. A live answer fetch is a request from ChatGPT-User, Claude-User, Perplexity-User or Gemini. Distinct URLs counts unique paths including query-free duplicates. Percentages are of logged AI requests, never of total site traffic.
Windows, stated everywhere
Site A Aug 25 – Sep 1, 2026 (8 days); site B Aug 28 – Sep 1, 2026 (5 days). Both start when the logger was installed, and the first and last days are partial. No figure on this page is presented without its window, because a metric quoted without one is how two true numbers end up looking like a contradiction.
What this dataset cannot tell you
- A fetch is not a citation. The agent pulled the page; whether it was quoted in the answer is a separate question that only prompt-sampling can answer.
- Classification is by user-agent string, which is a claim rather than evidence. On site B, requests for /wp-config.php.bak and /wp/.env arrived carrying assistant user agents — somebody forging the header. The volume is a rounding error against 529, but the honest word for these counts is "claimed".
- Static files are invisible to this instrument. The logger runs inside PHP, so a file served directly off disk — llms.txt on both these sites — cannot be counted at all. Zero requests for it in this dataset means nothing.
- Two sites in one industry, in windows of eight and five days. This is a probe, not a study, and the agent mix inverted between the two sites badly enough that no industry average should be drawn from it.
The instrument is described in full, including the part that broke, on the Citation Tracker page. If you want the same measurement on your own site, that is the audit.
Questions about the data.
A crawl is background collection: the agent is building an index or a training corpus and no human is waiting. A live answer fetch is a request an assistant makes while composing a reply for a person who has just asked something — it carries a different user agent, such as ChatGPT-User rather than GPTBot. In this dataset 92.1% of AI requests were crawling and 7.9% were live fetches, and reports that blend the two are dominated by whichever crawler happened to be busiest.
On site A, 1,703 URLs were crawled and 207 were pulled live. The URLs that got pulled were overwhelmingly pages built around a factual claim with a number attached — brand reliability, failure rates, repair costs. Service and location pages made up most of what was crawled and almost none of what was consulted, because a page whose purpose is to convert a visitor is not a source for a claim.
Far more concentrated than the headline suggests. Of the 207 URLs on site A that received any live fetch, 168 received exactly one. Only eight URLs received six or more, and those eight account for roughly 44% of all live fetches on the site. A business reading "207 pages get pulled into AI answers" would badly overestimate its own position.
Yes. Attribution and a link back is all that is asked, and there is a ready-made source line in the section above. If you want a cut that is not published here — a specific agent, a longer window, or the same measurement on your own site — ask and I will either run it or tell you why the data cannot support it.
The logger runs continuously; this page is refreshed when the window is long enough to change a conclusion rather than on a fixed schedule. Every figure on the page carries its window, so a stale table is visible as stale rather than passing as current.
Want these numbers for your own site?
The logger takes an afternoon to install and about a week of traffic before the live-fetch list means anything. After that you stop guessing whether AI touches your site.