Keenable AI Open-Sources Live Search Benchmark NEEDLE

Keenable AI has open-sourced NEEDLE, a novel benchmark designed to address the challenges of evaluating web search APIs, particularly when dealing with AI agents that can potentially access or already possess the answers. Traditional benchmarks often use static datasets, which can be exploited by agents that learn to bypass the retrieval process by accessing the "answer key" directly or by having the information encoded in their parametric memory. NEEDLE tackles this issue by dynamically rebuilding its query set from fresh public sources on an hourly and daily basis, ensuring that evaluations remain relevant and robust.

The benchmark's name, NEEDLE, is an acronym representing its five core verticals: News, Everyday, Expert, Deep-tail, and Legal Evaluation. Each vertical is crafted to model different agent intents and data sources. The News vertical generates queries hourly by scraping RSS feeds and analyzing Google Trends. The Finance vertical extracts registry facts from Wikidata and GLEIF, alongside single-quarter financial figures from SEC XBRL filings. The Scholar vertical transforms academic papers into four distinct query styles, including degraded titles, full-text-only details, natural-language clues, and hedged "tip-of-the-tongue" descriptions. The Deep-tail vertical focuses on rare-word queries sampled from public agent trajectory releases such as DeepResearchGym, OpenResearcher, and LRAT. Finally, the Legal vertical sources recent information from CourtListener, a public repository of legal documents.

NEEDLE operates by running fifteen different search APIs against the same query text under a unified protocol. The performance of these APIs is then assessed by "ultimate," a pooled oracle engine that aggregates and marks the findings from all tested engines. This approach provides a comprehensive and comparative evaluation. The benchmark is designed to be reproducible, functioning as an open-source evaluation harness rather than a commercial product. It is implemented as a Python command-line interface (CLI) that can be installed using `uv sync` and is driven by two subcommands per benchmark: `generate` and `run`. To utilize NEEDLE, users require an OpenRouter key for judging purposes and an individual API key for each search engine being tested. The system is capable of running on a standard laptop or within a continuous integration (CI) environment. Furthermore, NEEDLE allows for the recreation of all query streams used in the evaluation, in addition to providing quality judgments for the rankings.

This dynamic approach to benchmarking is crucial for accurately assessing the capabilities of modern AI search agents. By continuously updating the query set, NEEDLE mitigates the risk of agents "memorizing" answers or exploiting static datasets. The benchmark's modular design, with distinct verticals catering to different information needs, allows for a granular understanding of an API's performance across various domains. The open-source nature of NEEDLE also fosters transparency and collaboration within the AI research community, enabling developers to test and improve their search agents against a challenging and evolving standard. The benchmark's ability to run on common hardware and within CI pipelines makes it accessible for widespread adoption and continuous evaluation.