{
  "dataset": "Interestana AI Crawler Ledger",
  "edition": "2026-10",
  "editionNumber": 1,
  "available": true,
  "title": "AI Crawler Ledger #1: 13 September to 13 October 2026",
  "site": "interestana.com",
  "published": "2026-10-13",
  "url": "https://interestana.com/research/ai-crawler-ledger",
  "creator": "Interestana",
  "license": {
    "name": "CC BY 4.0",
    "url": "https://creativecommons.org/licenses/by/4.0/"
  },
  "source": "Cloudflare dashboard for interestana.com, read on 2026-10-13",
  "period": {
    "start": "2026-09-13T05:41:00Z",
    "end": "2026-10-13T05:41:00Z",
    "days": 30,
    "note": "Cloudflare's rolling \"Last 30 days\" window (AI Crawl Control and Web Analytics): 13 Sep 2026 08:41 to 13 Oct 2026 08:41 Arabian Standard Time (UTC+3), read from the dashboard date picker between about 08:41 and 08:55 UTC+3 on 13 Oct 2026. GA4: the calendar dates 13 Sep to 13 Oct 2026, with 13 Oct partial; its time zone was not noted. GA4 holds data only from about 11 Oct 2026."
  },
  "method": [
    "All figures were read from the Cloudflare and Google Analytics dashboard screens for interestana.com and transcribed. Nothing was exported, and nothing was changed beyond view controls (date ranges, tabs and list length).",
    "Cloudflare AI Crawl Control (Free plan) gives requests per crawler, per operator, per category and per HTTP status for the 30-day window. \"Requests\" here is every status; \"successful\" is a 2xx response.",
    "Cloudflare documents that on the Free plan AI Crawl Control identifies AI crawlers by their user-agent strings. A crawler's operator and category are Cloudflare's. The dashboard shows only category totals, so each crawler's category was assigned by reconciling the per-crawler figures with those totals.",
    "Referral visits come from Cloudflare Web Analytics (a JavaScript beacon on page loads, bots excluded, referrers from a 10% sample) and from GA4 session sources. A referral is a visit whose referring page, as the browser reports it, or whose GA4 session source names an AI assistant.",
    "Values of 1,000 or more are shown by the dashboard in thousands with two decimals (for example 246.65k) and are recorded here as whole numbers (246650). They are accurate to about 5 requests.",
    "No IP address, user identifier or other personal data was collected or is published."
  ],
  "totals": {
    "requests": 715970,
    "successful2xx": 673350,
    "attributedToListedCrawlers": 712720,
    "notAttributed": 3250,
    "byCategory": [
      {
        "category": "AI Search",
        "requests": 329320
      },
      {
        "category": "AI Crawler",
        "requests": 324170
      },
      {
        "category": "Search Engine Crawler",
        "requests": 54810
      },
      {
        "category": "AI Assistant",
        "requests": 4420
      }
    ],
    "byOperator": [
      {
        "operator": "OpenAI",
        "requests": 252110
      },
      {
        "operator": "Meta",
        "requests": 149460
      },
      {
        "operator": "Amazon",
        "requests": 90840
      },
      {
        "operator": "Anthropic",
        "requests": 52960
      },
      {
        "operator": "Apple",
        "requests": 51510
      },
      {
        "operator": "Microsoft",
        "requests": 50840
      },
      {
        "operator": "Perplexity",
        "requests": 31290
      },
      {
        "operator": "Huawei",
        "requests": 28450
      },
      {
        "operator": "Baidu",
        "requests": 2580
      },
      {
        "operator": "Google",
        "requests": 1420
      },
      {
        "operator": "ByteDance",
        "requests": 474
      },
      {
        "operator": "DuckDuckGo",
        "requests": 306
      },
      {
        "operator": "Mistral",
        "requests": 265
      },
      {
        "operator": "Common Crawl",
        "requests": 222
      }
    ],
    "byStatusClass": [
      {
        "statusClass": "2xx",
        "requests": 673350
      },
      {
        "statusClass": "3xx",
        "requests": 1910
      },
      {
        "statusClass": "4xx",
        "requests": 21730
      },
      {
        "statusClass": "5xx",
        "requests": 15730
      }
    ],
    "byStatus": [
      {
        "status": 200,
        "requests": 675280
      },
      {
        "status": 204,
        "requests": 74
      },
      {
        "status": 301,
        "requests": 1750
      },
      {
        "status": 308,
        "requests": 176
      },
      {
        "status": 403,
        "requests": 102
      },
      {
        "status": 404,
        "requests": 18190
      },
      {
        "status": 499,
        "requests": 4610
      },
      {
        "status": 502,
        "requests": 17
      },
      {
        "status": 522,
        "requests": 15490
      },
      {
        "status": 523,
        "requests": 270
      }
    ],
    "statusNote": "The four status classes add up to 712,720, the requests attributed to listed crawlers. The per-code counts add up to 715,959 before the small unlisted codes (520, 524, 500), so they appear to cover all 715,970 requests, including the 3,250 not attributed to a listed crawler. That would explain why status 200 (675,280) is higher than the 2xx class (673,350).",
    "blockedByAiCrawlControl": 0,
    "blockingNote": "Every Block Crawler setting in AI Crawl Control was off (33 crawlers listed, 20 active in the period). There were 102 HTTP 403 responses in all.",
    "robotsTxt": {
      "successfulFetches": 1830,
      "unsuccessfulFetches": 155,
      "contentSignalsSet": false,
      "violations": 0,
      "note": "interestana.com/robots.txt, per the AI Crawl Control Signals page, which reported \"No violations found\"."
    },
    "dataTransfer": [
      {
        "crawler": "OAI-SearchBot",
        "value": 17.64,
        "unit": "GB"
      },
      {
        "crawler": "Meta-ExternalAgent",
        "value": 6.07,
        "unit": "GB"
      },
      {
        "crawler": "Amazonbot",
        "value": 869.12,
        "unit": "MB"
      },
      {
        "crawler": "ClaudeBot",
        "value": 669.67,
        "unit": "MB"
      },
      {
        "crawler": "BingBot",
        "value": 653.99,
        "unit": "MB"
      },
      {
        "crawler": "the other 14 crawlers (as the dashboard words it)",
        "value": 2.36,
        "unit": "GB"
      }
    ],
    "dataTransferNote": "As displayed by AI Crawl Control. Its Overview page also reported that OpenAI crawlers accessed 18.27 GB of HTML.",
    "distinctSuccessfulPaths": 4751,
    "topSuccessfulPaths": [
      {
        "path": "/",
        "successful2xx": 22446
      },
      {
        "path": "/about/editorial-process",
        "successful2xx": 20150
      },
      {
        "path": "/news",
        "successful2xx": 20146
      },
      {
        "path": "/topics",
        "successful2xx": 19142
      },
      {
        "path": "/search",
        "successful2xx": 19084
      },
      {
        "path": "/agency",
        "successful2xx": 18868
      },
      {
        "path": "/ai",
        "successful2xx": 18867
      },
      {
        "path": "/topics/business",
        "successful2xx": 17604
      },
      {
        "path": "/topics/artificial-intelligence",
        "successful2xx": 17055
      },
      {
        "path": "/about",
        "successful2xx": 15722
      }
    ]
  },
  "crawlers": [
    {
      "bot": "OAI-SearchBot",
      "operator": "OpenAI",
      "category": "AI Search",
      "categoryAssignment": "reconciled",
      "requests": 246650,
      "successful2xx": 242910,
      "note": "Search indexing for ChatGPT search. Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "Meta-ExternalAgent",
      "operator": "Meta",
      "category": "AI Crawler",
      "categoryAssignment": "reconciled",
      "requests": 149460,
      "successful2xx": 144400,
      "note": "Collects pages for AI model training. Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "Amazonbot",
      "operator": "Amazon",
      "category": "AI Crawler",
      "categoryAssignment": "reconciled",
      "requests": 90840,
      "successful2xx": 73760,
      "note": "Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "ClaudeBot",
      "operator": "Anthropic",
      "category": "AI Crawler",
      "categoryAssignment": "reconciled",
      "requests": 52320,
      "successful2xx": 51660,
      "note": "Collects pages for AI model training. Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "Applebot",
      "operator": "Apple",
      "category": "AI Search",
      "categoryAssignment": "reconciled",
      "requests": 51510,
      "successful2xx": 50720,
      "note": "Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "BingBot",
      "operator": "Microsoft",
      "category": "Search Engine Crawler",
      "categoryAssignment": "reconciled",
      "requests": 50840,
      "successful2xx": 45110,
      "note": "Search engine crawler; Bing results also feed Microsoft Copilot. Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "PerplexityBot",
      "operator": "Perplexity",
      "category": "AI Search",
      "categoryAssignment": "reconciled",
      "requests": 30940,
      "successful2xx": 30460,
      "note": "Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "PetalBot",
      "operator": "Huawei",
      "category": "AI Crawler",
      "categoryAssignment": "reconciled",
      "requests": 28450,
      "successful2xx": 26760,
      "note": "Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "ChatGPT-User",
      "operator": "OpenAI",
      "category": "AI Assistant",
      "categoryAssignment": "reconciled",
      "requests": 3510,
      "successful2xx": 2830,
      "note": "Fetches a page in response to a ChatGPT user's request. Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "Baidu",
      "operator": "Baidu",
      "category": "Search Engine Crawler",
      "categoryAssignment": "reconciled",
      "requests": 2580,
      "successful2xx": 1800,
      "note": "Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "GPTBot",
      "operator": "OpenAI",
      "category": "AI Crawler",
      "categoryAssignment": "reconciled",
      "requests": 1950,
      "successful2xx": 1610,
      "note": "Collects pages for AI model training. Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "Googlebot",
      "operator": "Google",
      "category": "Search Engine Crawler",
      "categoryAssignment": "reconciled",
      "requests": 1390,
      "successful2xx": 975,
      "note": "Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "Bytespider",
      "operator": "ByteDance",
      "category": "AI Crawler",
      "categoryAssignment": "reconciled",
      "requests": 474,
      "successful2xx": 239,
      "note": "Collects pages for AI model training. Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "Claude-User",
      "operator": "Anthropic",
      "category": "AI Crawler",
      "categoryAssignment": "reconciled-uncertain",
      "requests": 424,
      "successful2xx": 20,
      "note": "Its operator documents it as a fetcher triggered by a Claude user's request. The category totals add up only if Cloudflare counts it as an AI Crawler, so its category is uncertain. Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "Perplexity-User",
      "operator": "Perplexity",
      "category": "AI Assistant",
      "categoryAssignment": "reconciled",
      "requests": 343,
      "successful2xx": 6,
      "note": "Fetches a page in response to a Perplexity user's request. Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "DuckAssistBot",
      "operator": "DuckDuckGo",
      "category": "AI Assistant",
      "categoryAssignment": "reconciled",
      "requests": 306,
      "successful2xx": 72,
      "note": "Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "MistralAI-User",
      "operator": "Mistral",
      "category": "AI Assistant",
      "categoryAssignment": "reconciled",
      "requests": 265,
      "successful2xx": 0,
      "successfulApproximate": true,
      "note": "Fetches a page in response to a Le Chat user's request. Absent from the 2xx view, so about 0 successful. Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "CCBot",
      "operator": "Common Crawl",
      "category": "AI Crawler",
      "categoryAssignment": "reconciled",
      "requests": 222,
      "successful2xx": 14,
      "note": "Builds an open web dataset that is used for AI training. Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "Claude-SearchBot",
      "operator": "Anthropic",
      "category": "AI Search",
      "categoryAssignment": "reconciled",
      "requests": 213,
      "successful2xx": 1,
      "note": "Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    },
    {
      "bot": "Google-CloudVertexBot",
      "operator": "Google",
      "category": "AI Crawler",
      "categoryAssignment": "reconciled",
      "requests": 23,
      "successful2xx": 6,
      "note": "Not blocked.",
      "source": "AI Crawl Control > Metrics, Requests over time, grouped by crawler, Last 30 days"
    }
  ],
  "referrals": [
    {
      "referrer": "chatgpt.com",
      "assistant": "ChatGPT",
      "operator": "OpenAI",
      "cloudflareEstimate": 10,
      "cloudflareSampledVisits": 1,
      "ga4Sessions": 0,
      "note": "Cloudflare's estimate from a 10% sample: most likely one sampled visit. GA4 recorded no session from chatgpt.com."
    },
    {
      "referrer": "perplexity.ai",
      "assistant": "Perplexity",
      "operator": "Perplexity",
      "cloudflareEstimate": 0,
      "ga4Sessions": 0,
      "note": "Not among the 12 referrers Cloudflare listed (its complete list); not a GA4 session source."
    },
    {
      "referrer": "claude.ai",
      "assistant": "Claude",
      "operator": "Anthropic",
      "cloudflareEstimate": 0,
      "ga4Sessions": 0,
      "note": "Not listed by Cloudflare or GA4."
    },
    {
      "referrer": "gemini.google.com",
      "assistant": "Gemini",
      "operator": "Google",
      "cloudflareEstimate": 0,
      "ga4Sessions": 0,
      "note": "Not listed by Cloudflare or GA4."
    },
    {
      "referrer": "copilot.microsoft.com",
      "assistant": "Microsoft Copilot",
      "operator": "Microsoft",
      "cloudflareEstimate": 0,
      "ga4Sessions": 0,
      "note": "Not listed by Cloudflare or GA4. Visits from Bing search are counted separately, under search referrals."
    },
    {
      "referrer": "meta.ai",
      "assistant": "Meta AI",
      "operator": "Meta",
      "cloudflareEstimate": 0,
      "ga4Sessions": 0,
      "note": "Not listed by Cloudflare or GA4."
    },
    {
      "referrer": "chat.mistral.ai",
      "assistant": "Le Chat",
      "operator": "Mistral",
      "cloudflareEstimate": 0,
      "ga4Sessions": 0,
      "note": "Not listed by Cloudflare or GA4."
    },
    {
      "referrer": "duckduckgo.com/aichat",
      "assistant": "Duck.ai",
      "operator": "DuckDuckGo",
      "cloudflareEstimate": 0,
      "ga4Sessions": 0,
      "note": "Not listed as such. duckduckgo.com appears in both Cloudflare and GA4 and is counted here as search; a referrer host alone cannot tell DuckDuckGo search from its AI chat."
    },
    {
      "referrer": "you.com",
      "assistant": "You.com",
      "operator": "You.com",
      "cloudflareEstimate": 0,
      "ga4Sessions": 0,
      "note": "Not listed by Cloudflare or GA4."
    },
    {
      "referrer": "deepseek",
      "assistant": "DeepSeek",
      "operator": "DeepSeek",
      "cloudflareEstimate": 0,
      "ga4Sessions": 0,
      "note": "No DeepSeek domain was listed by Cloudflare or GA4."
    }
  ],
  "searchReferrals": [
    {
      "referrer": "www.bing.com",
      "operator": "Microsoft",
      "cloudflareEstimate": 120
    },
    {
      "referrer": "cn.bing.com",
      "operator": "Microsoft",
      "cloudflareEstimate": 50
    },
    {
      "referrer": "search.yahoo.com",
      "operator": "Yahoo",
      "cloudflareEstimate": 10
    },
    {
      "referrer": "au.search.yahoo.com",
      "operator": "Yahoo",
      "cloudflareEstimate": 10
    },
    {
      "referrer": "www.google.com",
      "operator": "Google",
      "cloudflareEstimate": 10
    },
    {
      "referrer": "m.baidu.com",
      "operator": "Baidu",
      "cloudflareEstimate": 10
    },
    {
      "referrer": "duckduckgo.com",
      "operator": "DuckDuckGo",
      "cloudflareEstimate": 10
    }
  ],
  "webAnalytics": {
    "visits": 40400,
    "visitsSampling": "1% sample of page loads",
    "referrerSampling": "10% sample of requests",
    "referrers": [
      {
        "referrer": "(none, direct)",
        "estimate": 40820
      },
      {
        "referrer": "www.bing.com",
        "estimate": 120
      },
      {
        "referrer": "cn.bing.com",
        "estimate": 50
      },
      {
        "referrer": "www.facebook.com",
        "estimate": 20
      },
      {
        "referrer": "chatgpt.com",
        "estimate": 10
      },
      {
        "referrer": "forum.miata.net",
        "estimate": 10
      },
      {
        "referrer": "search.yahoo.com",
        "estimate": 10
      },
      {
        "referrer": "www.google.com",
        "estimate": 10
      },
      {
        "referrer": "m.baidu.com",
        "estimate": 10
      },
      {
        "referrer": "teams.public.onecdn.s… (name cut off in the dashboard)",
        "estimate": 10
      },
      {
        "referrer": "duckduckgo.com",
        "estimate": 10
      },
      {
        "referrer": "au.search.yahoo.com",
        "estimate": 10
      }
    ],
    "topCountries": [
      {
        "country": "Singapore",
        "visits": 38800
      },
      {
        "country": "China",
        "visits": 800
      },
      {
        "country": "United States",
        "visits": 500
      }
    ],
    "topBrowsers": [
      {
        "browser": "Chrome",
        "visits": 39800
      },
      {
        "browser": "Unknown",
        "visits": 1020
      },
      {
        "browser": "Edge",
        "visits": 180
      }
    ],
    "note": "Bots excluded by Cloudflare. Most of these visits still look automated (see caveats).",
    "source": "Cloudflare Web Analytics > Visits by source > Referers, bots excluded, Last 30 days (10% sample of requests; all 12 referrers listed)"
  },
  "ga4": {
    "sessions": 246,
    "engagedSessions": 6,
    "events": 859,
    "sessionSources": [
      {
        "source": "(direct)",
        "sessions": 242
      },
      {
        "source": "(not set)",
        "sessions": 18
      },
      {
        "source": "bing",
        "sessions": 2
      },
      {
        "source": "duckduckgo",
        "sessions": 1
      }
    ],
    "channels": [
      {
        "channel": "Direct",
        "sessions": 242
      },
      {
        "channel": "Unassigned",
        "sessions": 18
      },
      {
        "channel": "Organic Search",
        "sessions": 3
      }
    ],
    "aiAssistantSessions": 0,
    "coverageNote": "GA4 reported using 100% of available data, but shows no users before 10 Oct 2026 and about 150 a day from 11 Oct, so it covers about 3 of the 30 days.",
    "source": "GA4 > Traffic acquisition, Session source, 2026-09-13 to 2026-10-13 (data from about 2026-10-11 only)"
  },
  "caveats": [
    "Interestana is a small, new site, so every ratio here is one small site's illustration, not a measurement of the web.",
    "Cloudflare's 30-day window is rolling, and the figures kept moving while they were read: one path's count rose from 20,146 to 20,294 within minutes. Figures on the Overview page, read a few minutes apart, differ slightly from the Metrics page (OAI-SearchBot 246,990 there, 246,650 here); this ledger uses the Metrics page.",
    "The per-crawler requests add up to 712,720 but the total is 715,970. About 3,250 requests are not attributed to any listed crawler, and the dashboard does not explain the gap.",
    "Crawler identity is Cloudflare's classification. On the Free plan, which interestana.com uses, Cloudflare documents that AI Crawl Control identifies AI crawlers by user-agent string, and this view shows no per-request identity test, so a request that falsely claimed a crawler's name would be counted under that crawler.",
    "Categories are Cloudflare's (AI Search, AI Crawler, Search Engine Crawler, AI Assistant). Only category totals are shown, so each crawler was assigned to a category by reconciling the sums. They match within the dashboard's rounding, except for Claude-User, whose category is uncertain. Rounding also leaves room for two small crawlers to trade places: CCBot (222 requests) and Claude-SearchBot (213) could swap categories and both totals would still fit, so the sums alone do not settle those two. They are listed as their operators describe them.",
    "Successful means an HTTP 2xx response. Cloudflare labels these \"Allowed\", which does not mean the rest were blocked: nothing was blocked by AI Crawl Control. About 39,000 of the requests attributed to a listed crawler were unsuccessful. Over all requests, the largest unsuccessful codes were 404 Not Found (18,190) and 522, Cloudflare's code for a timed-out connection to the site's server (15,490). The 522 responses cluster in the last few days of the period.",
    "Fetches that people trigger through an assistant had mixed results: ChatGPT-User 3,510 requests and 2,830 successful, DuckAssistBot 306 and 72, Claude-User 424 and 20, Perplexity-User 343 and 6, MistralAI-User 265 and about 0; together 2,928 of 4,848 succeeded. Claude-SearchBot, a search crawler, made 213 requests and 1 succeeded. Counts of all requests overstate how often assistants read a page.",
    "Cloudflare Web Analytics is sampled: visits come from a 1% sample of page loads and referrers from a 10% sample. Every referrer count is a multiple of 10, so chatgpt.com's 10 most likely stands for one sampled visit. A 95% interval for one sampled event runs from about 0 to 56 actual visits, so the chatgpt.com ratio could be anywhere from about 4,400 upward. A referrer missing from the list could still have sent a few visits.",
    "Most Web Analytics visits are probably automated even with bots excluded: 38,800 of 40,400 came from Singapore, 40,820 had no referrer and 39,800 were Chrome. The number of human visits, and so any share of referrals, rests on a much smaller and unknown base. The one sampled chatgpt.com visit may itself have been automated.",
    "GA4 covers only about the last 3 of the 30 days. Google's tag runs only once Google's consent tool reports consent granted or not required, and Interestana does not load it for visitors who declined consent in the EEA, the UK or Switzerland, for browsers sending Global Privacy Control, for the site owner's own browsers, for pages marked noindex or on other hosts. Ad blockers stop it too. GA4's channel rows add up to 263, more than its 246 sessions, as GA4 displays it.",
    "Referral counts are a lower bound: a visit that arrives with no referrer (from some apps, for example) is counted as direct.",
    "The three sources measure different things and must not be added together: AI Crawl Control counts requests by crawlers at Cloudflare, Web Analytics counts page loads reported by a JavaScript beacon, and GA4 counts consented sessions.",
    "The AI Crawl Control Metrics page mentions referral trends, but this plan shows no AI-referral section. Referral figures come only from Web Analytics and GA4."
  ]
}