Home / Blog / GEO/AEO
GEO/AEOAI crawlers took 22,614 pages in six days — and sent 1,174 visits in ten months
Meta's crawler alone pulled 16,547 pages in six days and referred nobody. Two reports from my own servers: who takes your pages, and who actually sends people.
TL;DR
Across 64 sites, AI crawlers made 22,614 requests in six days — three quarters of them from Meta, which sent zero visits. Across 43 analytics properties, assistants sent 1,174 visits in ten months, 85 percent from ChatGPT and Perplexity. That is 0.083 percent of search traffic on my largest site. Optimise for the shortlist, not for the session count.
What the logs actually show
I run SEO and AI visibility for the sites on my own infrastructure — 64 of them keep separate nginx logs. Last week I pulled two reports: which AI crawlers take pages, and which assistants send people. The two lists barely overlap, and that changes how this channel should be measured.
Six full days, 31 October to 5 November 2026, requests by user agent:
- meta-externalagent — 16,547 requests across 7 sites
- ClaudeBot — 1,401 across 12 sites
- PerplexityBot — 1,078 across 7 sites
- Applebot — 984 across 15 sites
- OAI-SearchBot — 576 across 22 sites
- Amazonbot — 555 across 11 sites
- Bytespider — 453 across 17 sites
- Claude-SearchBot — 358 across 8 sites
- GPTBot — 244 across 13 sites
- Claude-User — 136 across 8 sites
- ChatGPT-User — 131 across 7 sites
- DuckAssistBot — 84 across 6 sites
- CCBot — 67 across 10 sites
22,614 requests in under a week. Meta made nearly three quarters of them. Six of those hosts are my own sites; drop them and 19,887 requests remain, with the same shape.
Three traps in counting this
Duplicate logs. My hosting panel writes every request twice — once to site-ssl.access.log, once to site-backend.access.log. Grep both and every agent doubles. My first pass reported 32,950 Meta requests, exactly twice the truth. Count one log family and reconcile the total against the line count for the day.
Google-Extended is not a crawler. I searched for it and found zero requests, which is the only possible result: Google-Extended is a robots.txt token, not a user agent. No request carries it. If it appears in your traffic report as a source, that report was not built from logs. MistralAI-User and YandexAdditional were genuinely absent too.
Bytespider hides. It is widely assumed to be gone; it made 453 requests across 17 of my sites. It does not announce itself as a bot — the user agent reads like an ordinary iPhone or Android browser, with the name buried at the end. A filter that looks for bot in the user agent string will never see it.
What analytics shows
Same projects, different report. Analytics referrals by source host, 43 properties, 1 January to 6 November 2026:
- chatgpt.com — 518 visits
- perplexity.ai — 483
- gemini.google.com — 114
- claude.ai — 55
- copilot.microsoft.com — 4
1,174 visits in ten months, spread across 10 properties out of 43. ChatGPT and Perplexity account for 85 percent of them. DeepSeek sent none. Meta sent none.
The two tables side by side
The windows differ, so read this as orders of magnitude, not as a ratio. Crawlers: six days. Referrals: ten months.
OpenAI's three agents made 951 requests that week and sent 518 visits over ten months. Anthropic's three made 1,895 requests and sent 55 visits. Perplexity made 1,078 and sent 483. Meta made 16,547 and sent nobody.
Reading a site and recommending a site are different activities. Most of the crawler traffic you pay hosting for comes from products that do not link out at all.
The number nobody puts on a slide
On my largest property — 816,970 visits in the period — assistants delivered 507 visits against 608,831 from search. That is 0.083 percent.
On the second largest: 640 assistant visits against 256,458 from search. That is 0.250 percent — three times better, and still a quarter of one percent.
I spent this year building AI visibility for these sites, and I am the one telling you the channel delivers a tenth of a percent of sessions. Both things are true. You are better off hearing the second one from your vendor now than finding it in your own analytics in March.
So why work on it at all
Because sessions are the wrong unit for this channel.
When a buyer asks an assistant to name vendors in your category, the answer is not a click. The answer is a shortlist. If you are on it, the visit arrives later — as a branded search or a direct hit — and gets attributed to neither you nor the assistant. I cannot measure that handoff cleanly, and I will not pretend assistant referrals capture the whole effect.
What I can measure is the two halves of the mechanism, and they break independently.
Readability. Your robots.txt either allows GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot or it does not. Your pages either ship text in the HTML or hide it behind JavaScript these agents do not execute. Your sitemap either reflects today's site or lies. On most sites I audit, exactly one of those three is broken, and the logs show the result of the fix the next day.
Citability. An assistant lifts a paragraph, not a page. That paragraph has to carry the whole answer: the number, the condition under which it holds, and the date. A price inside an image is invisible. A comparison table assembled in JavaScript is invisible. A 2,000-word page that circles the question for six paragraphs loses to someone else's 200-word block that simply answers it.
A site can pass the first test for months and fail the second. Logs full of GPTBot next to an empty referral report is exactly that picture — and that is where nearly all of the real work sits.
Five checks you can run this week
- Grep your access logs for GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot. Nothing there means robots.txt or your firewall, not your copy. Count one log family so you do not double.
- Build a referral report broken down by source host and look for chatgpt.com and perplexity.ai. Most analytics tools file them under link traffic, not search.
- Open your three highest-value commercial pages with JavaScript disabled. Anything that disappears, the assistant never saw.
- Ask ChatGPT and Perplexity what your buyer would ask. Write down who gets named. Do it from a logged-out session.
- Find the page where the price or the key number lives inside an image, and move it into text.
Want this run on your site?
Get a free audit →FAQ
Does blocking AI crawlers protect my content?
It stops the well-behaved ones. In my logs the agents that identify themselves — GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot — respect robots.txt. Bytespider does not announce itself as a bot at all, so a robots.txt rule never reaches it. Blocking also removes you from the answers those assistants give, which is the opposite of what most companies want.
Is 1,174 visits in ten months worth optimising for?
Not if you count sessions. The case for the work is the shortlist: when a buyer asks an assistant who to hire, you are either named or you are not, and the visit that follows arrives as a branded search. I measure what I can — readability in the logs, citability in the answers — and I do not claim the referral number captures the effect.
Why does Meta crawl so much and refer nothing?
Its crawler feeds model training and in-product features that do not link out. That is true of several agents here: Amazonbot and CCBot sent no referrals either. Crawl volume tells you who is reading you, not who is recommending you.
How do I know whether my pages are even readable to these agents?
Two checks settle it. Fetch a key page with curl and look for your main copy in the raw HTML; then load it in a browser with JavaScript disabled. Whatever is missing in either case is missing for the assistant. After that, the logs tell you whether the agents come back.
The Russian original of this study, with the same two tables: ИИ-краулеры и трафик из нейросетей.