JavaScript Links Invisible to Most AI Search Crawlers

A controlled experiment conducted over 41 days has revealed that most AI search crawlers, including prominent ones like GPTBot and ClaudeBot, are unable to discover and index web pages that rely on JavaScript to render internal links. The experiment, detailed in a blog post, involved setting up a Brazilian business classification directory with approximately 2,400 pages. This directory's hierarchical structure, comprising 21 top-level sections branching into divisions, groups, and classes (totaling 1,062 pages within the hierarchy), was used to test link discoverability. The 1,062 hierarchical pages were divided into two groups: an HTML group with 11 sections where child links were hard-coded directly into the raw HTML, and a JavaScript (JS) group with 10 sections where all child links were injected by JavaScript after the page had loaded. In the JS group, the raw HTML contained only an empty div, meaning bots that do not execute JavaScript would perceive no links whatsoever. To ensure the test's integrity, all alternative discovery paths, such as sitemaps, were disabled. The results indicated that Google's crawler stack was the only one observed to execute JavaScript and follow the injected links. However, the majority of this JavaScript execution and link following was not performed by Googlebot, the crawler responsible for building the main search index. Googlebot itself only managed to reach 2% of the JavaScript-linked pages. In stark contrast, GPTBot, ClaudeBot, Bingbot, PerplexityBot, Meta's crawlers, Amazon's crawlers, and all other tracked bots found zero JavaScript-injected links. This suggests a significant portion of the web, particularly content relying on client-side rendering for navigation, may be effectively invisible to these AI-driven search engines. The experiment's methodology focused on isolating the impact of JavaScript link injection by removing other indexing mechanisms. The test site, a project owned and operated by the experimenter, provided a real-world structure for evaluating bot behavior. The findings underscore a critical gap in how AI assistants and emerging search technologies interact with web content compared to traditional web crawlers, potentially impacting content discoverability and SEO strategies for websites employing JavaScript-heavy architectures. The experiment's duration of 41 days provided a substantial period for bots to interact with the test site, lending statistical weight to the observed outcomes. The clear dichotomy between bots that executed JavaScript and those that did not highlights a fundamental difference in their crawling and indexing capabilities.