Common Crawl Finds llms.txt Files Misused

Common Crawl, a non-profit organization dedicated to web crawling and data archiving, has identified significant misuse of the `llms.txt` file, a convention intended to guide Large Language Models (LLMs) on how to interact with websites. In an analysis of 584,107 `llms.txt` files, Common Crawl researchers discovered that the vast majority were generated from templates, indicating a lack of genuine customization or intent by website owners. This widespread template usage means that many of these files do not reflect specific instructions for LLMs but rather generic placeholders.

Further investigation revealed that a substantial portion of these `llms.txt` files contained no actual links or directives. This absence of content renders the files ineffective for their intended purpose of providing LLMs with specific guidance. More critically, some of the analyzed `llms.txt` files included crawler rules that the `llms.txt` format is fundamentally incapable of enforcing. This suggests a misunderstanding or misapplication of the file's purpose, with website administrators attempting to use it for purposes beyond its technical capabilities, akin to how `robots.txt` is used for web crawlers.

The `llms.txt` file is a relatively new convention, emerging as LLMs became more prevalent and began to interact with web content. Unlike `robots.txt`, which is a well-established protocol for web crawlers to respect or ignore directives about which pages they can access, `llms.txt` is not a standardized protocol with universal enforcement mechanisms. Its effectiveness relies on the LLM's programming and the website owner's intent to provide specific instructions. The findings from Common Crawl highlight a potential gap in understanding and implementation of this emerging convention, leading to files that are either ineffective or misleading.

This analysis, reported by Search Engine Journal, underscores the challenges in establishing and adhering to best practices as AI technologies evolve. The misuse of `llms.txt` files could lead to inefficient or unintended interactions between LLMs and websites, potentially impacting data collection and the overall web ecosystem. The findings suggest a need for clearer guidelines and educational resources for website owners regarding the proper use and limitations of `llms.txt` files to ensure they serve their intended purpose effectively.