Which Pages AI Pulls Into Answers: AI Crawled 1,703 URLs and Used 207, and the Gap Is the Whole Job
AI tools read far more of your website than they ever use. On one repair company's site, AI programs looked at 1,703 pages in eight days but used only 207 of them to answer real customers, and those were almost all pages built from the company's own numbers, such as repair costs and brand failure rates, not its service or city pages. This post shows what the used pages have in common and how a service business can build the kind of page that gets its name into AI answers.
Every AI visibility conversation starts in the wrong place: are the bots crawling me (reading my pages automatically)? They are. Crawling is cheap, and AI companies do it at industrial scale.
The question that decides whether any of it matters is narrower. Out of everything a crawler took, how much does an assistant actually reach for when a real person asks something?
I can answer that from server logs on two live sites, because the request that an assistant makes while composing an answer carries a different user agent (the name a program gives when it asks for a page) than the background crawler. Eight days on one site, five on the other:
| Site A (appliance repair) | Site B (home services) | |
|---|---|---|
| Distinct URLs touched by AI agents | 1,703 | 1,290 |
| Distinct URLs pulled live to answer someone | 207 | 53 |
| Share | 12.2% | 4.2% |
The underlying per-agent and per-day tables are published as an open dataset. Eighty-eight percent of what got crawled on site A was never consulted for an answer. On site B, ninety-six percent. That gap is the actual work, and no third-party tool can show it to you because it only exists in your own log.
What do the pages that get pulled into answers have in common?
Here are the most-fetched URLs from the live-answer traffic on both sites, crawlers excluded:
| Site A: live fetches | Count | Site B: live fetches | Count |
|---|---|---|---|
/robots.txt |
33 | A brand failure-rate data page | 20 |
| A "most reliable brands" data page | 33 | A "most reliable brands" data page | 15 |
| A statistics page from the client's job records | 31 | /robots.txt |
5 |
| Homepage | 28 | A dishwasher repair cost page | 5 |
| A brand-specific compressor failure article | 18 | A smart appliance failure statistics page | 2 |
| A city-level repair cost page | 14 | A city repair cost blog post | 2 |
The pattern replicates across two unrelated domains, and it is not subtle: the pages assistants pull live are pages made of numbers.
On site A the top two content URLs are both built from the client's own service data: brand reliability and a statistics page assembled from tens of thousands of real jobs. On site B the top two are both from a /resources/ section of data pages, ahead of every single service and location page on the domain. Cost pages with actual figures show up on both.
What's conspicuously missing from both lists: service pages, location pages, "about us", anything written to convert rather than to inform. Those made up the overwhelming majority of the 1,703 and 1,290 URLs crawled. They were fetched, indexed, and then never reached for.
Three properties the fetched pages share:
They answer a question that has a factual answer. "How long do refrigerators last", "which brands fail most", "what does this repair cost". An assistant composing an answer needs a source for a claim, and a page whose whole purpose is a claim with a number is the cheapest source to use.
The number is specific and attributed. Instead of "appliances often fail early", it gives a figure with a base attached: a percentage across a named number of service visits. That's a sentence an assistant can lift intact, which is also why those pages accumulate more referring domains (other websites linking to them) than anything else on the domain. The same property produces two payoffs.
They're self-contained. The claim doesn't depend on the paragraph above it or a chart three screens down.
The service pages fail all three. That doesn't mean they're bad pages; they exist to convert the visitor who already decided. It means expecting them to earn citations is a category error, asking a page to do a job it was never built for, and it explains why so many businesses conclude AI visibility doesn't work for them: they built entirely the wrong kind of page and then measured for citations.
Does this mean location and service pages are wasted?
No, and the log is not the right instrument for judging them.
Service and location pages get found through classic search by people ready to buy. Their job is a phone call, not a citation. What the data says is that they are not the assets that get you into AI answers, and if your entire site is service and location pages, your citable surface is close to zero regardless of how well those pages are written.
The practical shape that follows: a layer of data pages built from whatever proprietary information the business already has, each linking down into the money pages. The data page earns the citation and the link; the money page (the page that sells the job) takes the visitor. On the client site above, that statistics page is simultaneously the most-fetched content URL by assistants and the biggest single source of referring domains, so one asset serves two channels.
If you have no proprietary data, you have job records, invoice amounts, call logs, warranty claims, parts orders. That is data. Most service businesses are sitting on a dataset nobody in their industry has published, and are instead publishing the same "5 signs your dishwasher is broken" article as everyone else.
What was hiding in the live-fetch log on the second site?
This is the part I did not expect, and it changes how you should read anyone's AI traffic report.
Sorting site B's live-answer fetches, a few URLs near the bottom were not content at all:
/wp-config.php.old
/wp-config.php.bak
/wp/.env
/z9x8c7v6b5-debug-trigger-...
Those are requests for credential files, arriving with a user agent claiming to be an AI assistant fetching a page for a person. No assistant asks for wp-config.php.bak. Someone is forging the user-agent string: either a vulnerability scanner using an AI agent name because it's usually allowed through, or something deliberately probing under cover.
Two consequences worth carrying:
The user-agent string is a claim, not evidence. Anybody can send any user agent. If your AI-visibility number comes from user-agent matching alone (mine does, and I'm saying it about my own instrument), some fraction of it is not AI. Verification means checking the request's IP (its network address) against the ranges the AI companies publish, or a reverse DNS lookup that confirms who owns the address. My logger doesn't do that yet; until it does, the honest word for these numbers is "claimed".
A permissive robots.txt is a doorway. robots.txt, the file that tells bots what they may read, was among the most-requested URLs on both sites, and it's where scanners learn what's allowed. Nothing to panic about, but "allow all AI agents" and "allow anything that says it's an AI agent" turn out to be the same sentence in practice.
Neither of these ruins the 12% finding: credential probes are a rounding error against 425 live fetches, and stripping them changes nothing about the pattern. But it's exactly the kind of caveat that gets dropped from a client deck, and dropping it is how a plausible number becomes a wrong one. It is the same discipline that made me throw out a much better headline about llms.txt once I checked what my instrument could actually see.
What would I do with this on a specific site?
The sequence I'd run, in order:
- Log at the server, split by purpose. Crawl, training, search-index and live-answer fetches in separate buckets. A blended "AI traffic" number hides the only bucket that matters: on site A the biggest agent by volume was doing pure background crawling.
- List the URLs that got live fetches. That's your existing citable surface, usually a much shorter list than anyone expects. 207 out of 1,703 here.
- Look at what they are. If they're all data or cost pages, that's your answer about what to build next.
- Publish one data page from proprietary numbers, with the base stated for every figure, and link it down to the relevant money page.
- Watch that URL's live-fetch count. It moves within days. That's the shortest feedback loop available in this work, and it's the reason I'd rather own the log than rent a dashboard.
Running those five steps on one site, with the logger left behind, is the audit. The wider frame (which four numbers people mean by AI visibility and which instrument sees each) is in what AI visibility actually means. Once you know which pages get pulled, writing the next ones so a machine can lift a whole claim out of them is answer engine optimization.
What to do with this
Keep your service and city pages for people ready to call, and add a small set of data pages for AI answers. Pull a few honest figures from your own job records, such as typical repair costs, how long common appliances last in your jobs, or which brands you service most, state how many jobs each figure is based on, and link each data page to the service page it supports. Avoid publishing numbers about your own callbacks or repeat visits. If you want to see which of your pages AI already uses, see AI visibility audit, with prices on the pricing page.
I build the measurement layer and the citable surface: server-side logging that separates a crawl from a live answer fetch, data assets built from a client's own records, and the attribution that ties a citation to an invoice. Scope and prices are public on pricing; the citation work is described under AI citation visibility. If you want to know which of your pages AI is actually reaching for, let's talk.
Related: