AI Observatory Launched to Track Real User Conversations

A new research initiative, the AI Observatory, has been launched to address a critical gap in understanding how individuals genuinely interact with artificial intelligence models. AI companies such as Anthropic and OpenAI frequently release reports detailing user engagement with their products, including Claude and ChatGPT. However, AI researchers contend that these reports selectively present data, omitting information that might not align with the companies' desired narratives. Anka Reuel, a Computer Science PhD candidate at the Stanford Trustworthy AI Research (STAIR) Lab and co-lead of the AI Observatory project, highlighted this issue, stating, “There is no independent source to corroborate it.” The AI Observatory aims to rectify this by establishing a public platform that aggregates and analyzes real AI conversations. These conversations are collected with explicit user consent through seven existing datasets, ensuring privacy while enabling comprehensive analysis. The platform's primary objective is to furnish independent data sources for researchers and policymakers, enabling them to accurately assess the diverse ways generative AI is being utilized. Reuel emphasized that significant decisions regarding AI's benefits and risks are currently being made based on insufficient and potentially biased data. Initial findings from the AI Observatory indicate that AI usage patterns vary considerably among different models and have evolved over time. The research reveals a broader spectrum of sensitive user behaviors than is typically captured in reports from major AI developers, which tend to emphasize work-related applications over personal use. The Anthropic Economic Index, a well-regarded and frequently cited source for AI usage data, has been identified as having significant blind spots. As its name implies, this index primarily focuses on work and productivity-related uses of Claude AI, actively filtering out conversations deemed unrelated to these professional contexts. When the AI Observatory team applied Anthropic's filtering methodology to their own aggregated dataset, they discovered that approximately 48% of the conversations would have been excluded. These filtered, non-work-related conversations were found to be more likely to encompass topics related to health and relationships (44.2% compared to 31.2% in Anthropic's analysis), adult or illicit subjects (7.9% versus 2.1%), instances of harassment and hate speech (27.5% versus 5.66%), and sexual content (16.7% versus 2.4%). This contrasts sharply with the limited reporting on such topics in company-generated analyses. For instance, OpenAI's 2025 report on ChatGPT usage indicated that only 30% of consumer interactions were work-related, suggesting a substantial portion of user engagement falls outside the scope of productivity-focused reporting.