80% of ChatGPT product recommendations change when search is enabled: Study

ChatGPT's product recommendations shifted by 80.2% when its search functionality was enabled, according to a study of 20,000 responses conducted by Jeff Oxford, founder and CEO of Visibility Labs. Oxford tested 1,000 product-recommendation prompts, running each 10 times with search enabled and 10 times with search disabled. Only 19.8% of products recommended without search also appeared in recommendations generated with search enabled. The study found that even products ChatGPT consistently recommended without search rarely carried over when search was activated. Among products that appeared in 100% of search-disabled responses, just 15.8% remained in recommendations when search was enabled, a finding Oxford noted was contrary to his expectation that consistently recommended products would persist. The analysis also explored the correlation between products mentioned in ChatGPT's cited sources and their recommendation frequency, reporting a 0.4 Pearson correlation. Products mentioned more often in cited sources tended to have higher "Visibility Scores," a metric defined as the percentage of runs in which a product appeared for a given prompt. However, the study did not establish a causal link between cited-source mentions and recommendation frequency. When search was enabled, ChatGPT responses averaged 5.2 products, a decrease from the 6.2 products recommended when search was disabled. Across 10 runs per prompt, ChatGPT returned an average of 19 unique products with search enabled, compared to 21.8 unique products with search disabled. These findings suggest that products appearing in cited sources may gain increased visibility when ChatGPT's search is active, although the study does not quantify the impact of this increased visibility.