97% Of LLMS.txt Files Got No Requests, Ahrefs Data Shows via @sejournal, @MattGSouthern

Ahrefs analyzed 137,000 domains and found that 97% of llms.txt files received zero requests from web crawlers. This data indicates a significant underutilization or lack of engagement with these files, which are intended to guide AI bots on how to crawl websites. The study also revealed that AI retrieval bots constituted only 1% of the total requests made to these llms.txt files. This suggests that while AI is a growing presence online, its direct interaction with website-specific AI crawling instructions remains minimal. The findings were published by Search Engine Journal, highlighting the current landscape of AI bot behavior concerning website indexing protocols. The analysis underscores a gap between the existence of llms.txt files and their actual use by the majority of AI crawlers.