AI Crawler Ledger: October 2026

What the requests were for

Cloudflare files each crawler under a category. Its own definitions are summarised under each bar. Bars are drawn to scale (linear), as a share of all 715,970 requests.

Crawled vs referred, by operator

Each operator's successful crawler requests set against the visits its AI assistant referred back. Crawlers do different jobs: training crawlers are not built to send readers, search crawlers index pages an assistant may cite with a link, and user-triggered fetchers read a page a person asked about.

Requests are every response; successful are 2xx responses. Ratios divide successful requests by the Cloudflare estimate and are rounded to three significant figures. Where no visit was recorded there is nothing to divide by, so no ratio is given.

Every crawler

All 20 crawlers Cloudflare counted, sorted by requests. Names, operators and categories are Cloudflare's.

The crawlers above add up to 712,720 requests; Cloudflare's total is 715,970, and it does not say which crawlers made the other 3,250. The success rate is over the 712,720 requests attributed to a listed crawler (the base of Cloudflare's status classes). The status classes cover only those requests, so the all-requests row has no success rate. Figures of 1,000 or more are accurate to about 5 requests.

Referrals from AI assistants

Visits that arrived from an AI assistant's site, by referrer. Cloudflare's figure is its estimate from a 10% sample of requests; GA4 counts sessions in the days it covers.

For comparison: search referrals

Search engines also sent only a few visits, a reminder that the site's human audience is small. Cloudflare estimates, same sample:

• www.bing.com120
• cn.bing.com50
• search.yahoo.com10
• au.search.yahoo.com10
• www.google.com10
• m.baidu.com10
• duckduckgo.com10

GA4 session sources in the same range: (direct) 242, (not set) 18, bing 2, duckduckgo 1 (246 sessions, 6 engaged).

How the site answered

Of the 712,720 requests attributed to a listed crawler (the base of Cloudflare's status classes), 39,370 (5.5%) did not get a successful response. By status code, over all 715,970 requests, 18,190 were 404 Not Found and 15,490 were 522, Cloudflare's code for a connection to the site's server that timed out. The ratios above use successful requests only, so these are left out of them.

Status classes above: over the 712,720 attributed requests. By code, over all requests: 200: 675,280; 204: 74; 301: 1,750; 308: 176; 403: 102; 404: 18,190; 499: 4,610; 502: 17; 522: 15,490; 523: 270. The four status classes add up to 712,720, the requests attributed to listed crawlers. The per-code counts add up to 715,959 before the small unlisted codes (520, 524, 500), so they appear to cover all 715,970 requests, including the 3,250 not attributed to a listed crawler. That would explain why status 200 (675,280) is higher than the 2xx class (673,350).

Every Block Crawler setting in AI Crawl Control was off (33 crawlers listed, 20 active in the period). There were 102 HTTP 403 responses in all. robots.txt was fetched successfully 1,830 times, and Cloudflare reported no robots.txt violations.

Most-crawled pages

Successful crawler requests by path, out of 4,751 paths:

1. /22,446
2. /about/editorial-process20,150
3. /news20,146
4. /topics19,142
5. /search19,084
6. /agency18,868
7. /ai18,867
8. /topics/business17,604
9. /topics/artificial-intelligence17,055
10. /about15,722

Methodology

Caveats

• Interestana is a small, new site, so every ratio here is one small site's illustration, not a measurement of the web.
• Cloudflare's 30-day window is rolling, and the figures kept moving while they were read: one path's count rose from 20,146 to 20,294 within minutes. Figures on the Overview page, read a few minutes apart, differ slightly from the Metrics page (OAI-SearchBot 246,990 there, 246,650 here); this ledger uses the Metrics page.
• The per-crawler requests add up to 712,720 but the total is 715,970. About 3,250 requests are not attributed to any listed crawler, and the dashboard does not explain the gap.
• Crawler identity is Cloudflare's classification. On the Free plan, which interestana.com uses, Cloudflare documents that AI Crawl Control identifies AI crawlers by user-agent string, and this view shows no per-request identity test, so a request that falsely claimed a crawler's name would be counted under that crawler.
• Categories are Cloudflare's (AI Search, AI Crawler, Search Engine Crawler, AI Assistant). Only category totals are shown, so each crawler was assigned to a category by reconciling the sums. They match within the dashboard's rounding, except for Claude-User, whose category is uncertain. Rounding also leaves room for two small crawlers to trade places: CCBot (222 requests) and Claude-SearchBot (213) could swap categories and both totals would still fit, so the sums alone do not settle those two. They are listed as their operators describe them.
• Successful means an HTTP 2xx response. Cloudflare labels these "Allowed", which does not mean the rest were blocked: nothing was blocked by AI Crawl Control. About 39,000 of the requests attributed to a listed crawler were unsuccessful. Over all requests, the largest unsuccessful codes were 404 Not Found (18,190) and 522, Cloudflare's code for a timed-out connection to the site's server (15,490). The 522 responses cluster in the last few days of the period.
• Fetches that people trigger through an assistant had mixed results: ChatGPT-User 3,510 requests and 2,830 successful, DuckAssistBot 306 and 72, Claude-User 424 and 20, Perplexity-User 343 and 6, MistralAI-User 265 and about 0; together 2,928 of 4,848 succeeded. Claude-SearchBot, a search crawler, made 213 requests and 1 succeeded. Counts of all requests overstate how often assistants read a page.
• Cloudflare Web Analytics is sampled: visits come from a 1% sample of page loads and referrers from a 10% sample. Every referrer count is a multiple of 10, so chatgpt.com's 10 most likely stands for one sampled visit. A 95% interval for one sampled event runs from about 0 to 56 actual visits, so the chatgpt.com ratio could be anywhere from about 4,400 upward. A referrer missing from the list could still have sent a few visits.
• Most Web Analytics visits are probably automated even with bots excluded: 38,800 of 40,400 came from Singapore, 40,820 had no referrer and 39,800 were Chrome. The number of human visits, and so any share of referrals, rests on a much smaller and unknown base. The one sampled chatgpt.com visit may itself have been automated.
• GA4 covers only about the last 3 of the 30 days. Google's tag runs only once Google's consent tool reports consent granted or not required, and Interestana does not load it for visitors who declined consent in the EEA, the UK or Switzerland, for browsers sending Global Privacy Control, for the site owner's own browsers, for pages marked noindex or on other hosts. Ad blockers stop it too. GA4's channel rows add up to 263, more than its 246 sessions, as GA4 displays it.
• Referral counts are a lower bound: a visit that arrives with no referrer (from some apps, for example) is counted as direct.
• The three sources measure different things and must not be added together: AI Crawl Control counts requests by crawlers at Cloudflare, Web Analytics counts page loads reported by a JavaScript beacon, and GA4 counts consented sessions.
• The AI Crawl Control Metrics page mentions referral trends, but this plan shows no AI-referral section. Referral figures come only from Web Analytics and GA4.

How to cite

The data and charts are licensed CC BY 4.0: reuse them freely, with credit to Interestana and a link to this page. Please quote the small-site caveat with any ratio.

Data and editions

Each edition's JSON holds everything on this page: per-crawler and per-operator counts, categories, status codes, referrals, the method and the caveats. The CSV holds the crawler table. Both are open to any site or script (CORS). Source: Cloudflare dashboard for interestana.com, read on 2026-10-13.

• #1 · October 2026Latest, shown above

  13 September to 13 October 2026 · published 13 October 2026

  JSONCSV

A new edition is added each month, covering the 30 days before it is read. More research