How to make prompt tracking much more accurate

Prompt tracking for AI mentions and citations can be made significantly more accurate by implementing repeated runs, fixed sampling rules, and confidence intervals, according to a memo discussing AI SEO (AEO). Critics of prompt tracking point to the inherent variability of Large Language Models (LLMs), noting that identical prompts can yield different answers, with within-LLM variance from sampling alone ranging from 10% to 34%. Analysis of 815,000 prompt-page pairs, conducted in collaboration with AirOps, revealed that after running the same prompt three times in ChatGPT, only 2.2% of citations remained consistent. The memo argues that dismissing prompt tracking as unmeasurable due to its probabilistic nature is an oversimplification, drawing parallels to how probabilistic systems like weather and credit scores are still forecast and tracked. The average prompt is also considerably longer than traditional search keywords, making the probability of two individuals using the exact same prompt extremely low. While direct user prompt data is largely unavailable, search engines like Bing and Google currently offer some visibility into AI-generated content.