Microsoft Clarity Now Flags Bots That Ignore Robots.txt via @sejournal, @MattGSouthern

Microsoft Clarity introduced a new feature on March 18, 2024, that flags bots violating robots.txt directives. The Bot Analytics dashboard now identifies when bots attempt to access URLs that are explicitly disallowed by a website's robots.txt file. This update provides users with trends and filtering options, allowing them to analyze bot behavior based on specific bot types and operators. The addition aims to enhance website owners' control over bot traffic and improve data accuracy by highlighting non-compliant bots. This functionality is crucial for maintaining website integrity and ensuring that search engine crawlers and other automated tools adhere to specified crawling rules. The development was first reported by Search Engine Journal.