ChatGPT Citations Shift With Hidden Search Pipelines

Two independent analyses have revealed that ChatGPT's cited sources can change dynamically based on the specific, hidden retrieval pipelines it utilizes for generating answers. This finding introduces a new layer of complexity to tracking AI visibility, as the final output presented to users does not necessarily reflect the underlying source selection process. Researchers Chris Green and Suganthan Mohanadasan identified internal source-selection labels such as Labrador, Bright, Oxylabs, and SERP, which operate behind the scenes and are not visible in the citation cards presented to users.

Chris Green conducted a series of tests involving 1,000 prompts, each run up to 10 times, resulting in 9,946 completed search runs. His data indicated that Labrador served as the primary search source for 88.1% of prompts, followed by Bright (9.9%), Oxylabs (1.7%), and SERP (0.3%). However, Green observed that 11.6% of prompts experienced a shift in their primary search source across repeated executions. This change in source led to a significant drop in URL overlap, from 0.273 to 0.149, and domain overlap, from 0.265 to 0.155. These reductions represent approximately a 45% decrease in URL overlap and a 42% decrease in domain overlap when the search source was altered.

Suganthan Mohanadasan analyzed approximately 1,240 source records from two days of raw ChatGPT network traffic generated by a logged-in Pro account. He identified a 'result_source' field associated with web results, noting four distinct values: SERP, Labrador, Bright, and Oxylabs. Mohanadasan described Labrador as a source incorporating established publishers and reference sites. Bright was linked to Bright Data, Oxylabs to Oxylabs, and SERP was characterized as an open-web baseline, predominantly appearing in news-related results. While Green's repeated-prompt tests showed Labrador as the dominant source, Mohanadasan's sample indicated a more prominent role for Bright, particularly in queries related to commercial activities, shopping, finance, weather, and local information.

Furthermore, Mohanadasan's investigation uncovered that ChatGPT sometimes classifies certain queries before initiating a web search, utilizing a 'turn_use_case' field. In some instances, prompts were categorized as 'text' and consequently bypassed web search entirely, even when the query might have benefited from external information. This suggests that ChatGPT employs internal logic to determine the necessity of a web search, potentially relying on its internal knowledge base for certain types of queries.