AI Crawled 1,703 URLs and Pulled 207 Into Answers. The Gap Is the Whole Job.
Every AI visibility conversation starts in the wrong place: are the bots crawling me? They are. Crawling is cheap, and AI companies do it at industrial scale.
The question that decides whether any of it matters is narrower. Out of everything a crawler took, how much does an assistant actually reach for when a real person asks something?
I can answer that from server logs on two live sites, because the request that an assistant makes while composing an answer carries a different user agent than the background crawler. Eight days on one site, five on the other:
| Site A (appliance repair) | Site B (home services) | |
|---|---|---|
| Distinct URLs touched by AI agents | 1,703 | 1,290 |
| Distinct URLs pulled live to answer someone | 207 | 53 |
| Share | 12.2% | 4.2% |
The underlying per-agent and per-day tables are published as an open dataset. Eighty-eight percent of what got crawled on site A was never consulted for an answer. On site B, ninety-six percent. That gap is the actual work, and no third-party tool can show it to you because it only exists in your own log.
What do the pages that get pulled into answers have in common?
Here are the most-fetched URLs from the live-answer traffic on both sites, crawlers excluded:
| Site A — live fetches | Count | Site B — live fetches | Count |
|---|---|---|---|
/robots.txt |
33 | A brand failure-rate data page | 20 |
| A "most reliable brands" data page | 33 | A "most reliable brands" data page | 15 |
| A statistics page from the client's job records | 31 | /robots.txt |
5 |
| Homepage | 28 | A dishwasher repair cost page | 5 |
| A brand-specific compressor failure article | 18 | A smart appliance failure statistics page | 2 |
| A city-level repair cost page | 14 | A city repair cost blog post | 2 |
The pattern replicates across two unrelated domains, and it is not subtle: the pages assistants pull live are pages made of numbers.
On site A the top two content URLs are both built from the client's own service data — brand reliability and a statistics page assembled from tens of thousands of real jobs. On site B the top two are both from a /resources/ section of data pages, ahead of every single service and location page on the domain. Cost pages with actual figures show up on both.
What's conspicuously missing from both lists: service pages, location pages, "about us", anything written to convert rather than to inform. Those made up the overwhelming majority of the 1,703 and 1,290 URLs crawled. They were fetched, indexed, and then never reached for.
Three properties the fetched pages share:
They answer a question that has a factual answer. "How long do refrigerators last", "which brands fail most", "what does this repair cost". An assistant composing an answer needs a source for a claim, and a page whose whole purpose is a claim with a number is the cheapest source to use.
The number is specific and attributed. Not "appliances often fail early" but a figure with a base attached — a percentage across a named number of service visits. That's a sentence an assistant can lift intact, which is also why those pages accumulate more referring domains than anything else on the domain. Same property, two payoffs.
They're self-contained. The claim doesn't depend on the paragraph above it or a chart three screens down.
The service pages fail all three. That doesn't mean they're bad pages — they exist to convert the visitor who already decided. It means expecting them to earn citations is a category error, and it explains why so many businesses conclude AI visibility doesn't work for them: they built entirely the wrong kind of page and then measured for citations.
Does this mean location and service pages are wasted?
No, and the log is not the right instrument for judging them.
Service and location pages get found through classic search by people ready to buy. Their job is a phone call, not a citation. What the data says is that they are not the assets that get you into AI answers, and if your entire site is service and location pages, your citable surface is close to zero regardless of how well those pages are written.
The practical shape that follows: a layer of data pages built from whatever proprietary information the business already has, each linking down into the money pages. The data page earns the citation and the link; the money page takes the visitor. On the client site above, that statistics page is simultaneously the most-fetched content URL by assistants and the biggest single source of referring domains. One asset, two channels.
If you have no proprietary data, you have job records, invoice amounts, call logs, warranty claims, parts orders. That is data. Most service businesses are sitting on a dataset nobody in their industry has published, and are instead publishing the same "5 signs your dishwasher is broken" article as everyone else.
What was hiding in the live-fetch log on the second site?
This is the part I did not expect, and it changes how you should read anyone's AI traffic report.
Sorting site B's live-answer fetches, a few URLs near the bottom were not content at all:
/wp-config.php.old
/wp-config.php.bak
/wp/.env
/z9x8c7v6b5-debug-trigger-...
Those are requests for credential files, arriving with a user agent claiming to be an AI assistant fetching a page for a person. No assistant asks for wp-config.php.bak. Someone is forging the user-agent string — either a vulnerability scanner using an AI agent name because it's usually allowed through, or something deliberately probing under cover.
Two consequences worth carrying:
The user-agent string is a claim, not evidence. Anybody can send any user agent. If your AI-visibility number comes from user-agent matching alone — mine does, and I'm saying it about my own instrument — some fraction of it is not AI. Verification means checking the request's IP against the ranges the AI companies publish, or a reverse DNS lookup. My logger doesn't do that yet; until it does, the honest word for these numbers is "claimed".
A permissive robots.txt is a doorway. robots.txt was among the most-requested URLs on both sites, and it's where scanners learn what's allowed. Nothing to panic about, but "allow all AI agents" and "allow anything that says it's an AI agent" turn out to be the same sentence in practice.
Neither of these ruins the 12% finding — credential probes are a rounding error against 425 live fetches, and stripping them changes nothing about the pattern. But it's exactly the kind of caveat that gets dropped from a client deck, and dropping it is how a plausible number becomes a wrong one. Same discipline that made me throw out a much better headline about llms.txt once I checked what my instrument could actually see.
What would I do with this on a specific site?
The sequence I'd run, in order:
- Log at the server, split by purpose. Crawl, training, search-index and live-answer fetches in separate buckets. A blended "AI traffic" number hides the only bucket that matters — on site A the biggest agent by volume was doing pure background crawling.
- List the URLs that got live fetches. That's your existing citable surface, usually a much shorter list than anyone expects. 207 out of 1,703 here.
- Look at what they are. If they're all data or cost pages, that's your answer about what to build next.
- Publish one data page from proprietary numbers, with the base stated for every figure, and link it down to the relevant money page.
- Watch that URL's live-fetch count. It moves in days. That's the shortest feedback loop available in this work, and it's the reason I'd rather own the log than rent a dashboard.
Running those five steps on one site, with the logger left behind, is the audit. The wider frame — which four numbers people mean by AI visibility and which instrument sees each — is in what AI visibility actually means. Once you know which pages get pulled, writing the next ones so a machine can lift a whole claim out of them is answer engine optimization.
I build the measurement layer and the citable surface: server-side logging that separates a crawl from a live answer fetch, data assets built from a client's own records, and the attribution that ties a citation to an invoice. Scope and prices are public on pricing; the citation work is described under AI citation visibility. If you want to know which of your pages AI is actually reaching for, let's talk.
Related: