Needle in the Web: A Benchmark for Retrieving Targeted Web Pages in the Wild

AI summary

Overview

• Research area: Evaluation benchmarks for LLM-based web search agents, specifically exploratory/ambiguous information retrieval.
• Technical level: Intermediate — assumes familiarity with LLM agents, retrieval-augmented generation, and benchmark design, but the core idea is explained in accessible terms.
• Scope: Introduces and evaluates Needle in the Web, a 663-query benchmark that asks agents to find a single real webpage satisfying several vague, entity-masked criteria.

What This Paper Is About

Existing search-agent benchmarks such as BrowseComp and xBench-DeepSearch test Complex Reasoning Search: explicit, multi-constrained queries with one verifiable factual answer. The authors argue this misses Fuzzy Exploratory Search, where real users pose vague, multi-faceted requests and want the most relevant webpage rather than a factoid answer. Needle in the Web is built to measure that under-tested setting: the agent must locate exactly one webpage on the open web that mentions all of several deliberately vague criteria.

Key Contributions

1. A new task formulation. The paper formalizes Fuzzy Exploratory Search as a multi-constraint retrieval problem where every vague impression is an implicit semantic constraint, and the intersection of all constraints is expected to be a single webpage.
2. Needle in the Web, a 663-query benchmark. Queries span seven source websites (ArXiv, Open Library of Humanities, Wikipedia, CNN, Lonelyplanet, Pitchfork, Petapixel) and three difficulty levels — 222 easy, 229 medium, 212 hard.
3. A controllable, automated generation and validation pipeline. Factual claims are extracted from scraped articles, ranked by semantic similarity to the article using OpenAI's text-embedding-3-large, turned into masked criteria by replacing entity-specific content with generic placeholders (e.g., "someone", "somewhere", "a certain species"), and then filtered by an LLM validator.
4. An LLM-as-a-judge evaluation protocol. It defines semantic mention and query satisfaction, retrieving the webpage named in an agent's answer and labeling it a ground-truth match or a criteria match, both counted correct.

Main Findings

• Most systems fail the benchmark. GPT-4o scored 32.88 overall, Perplexity Sonar 33.18, DeepResearcher 32.88, Search-R1 30.77, Gemini 2.5-flash 30.17, and CognitiveKernel-Pro 12.37. A large portion of agents land below 35% accuracy and none is consistently strong across domains or difficulty levels.
• Accuracy collapses with difficulty. For GPT-4o, accuracy was 58.56 on Easy, 27.07 on Medium, and 12.26 on Hard; DeepResearcher went 57.66 / 27.51 / 12.74; Perplexity Sonar 53.60 / 31.44 / 13.68. This supports the claim that success in Complex Reasoning Search does not transfer to fuzzy retrieval.
• Answers are close to unique. Across 6 models attempting 663 queries, only 7 attempts found a valid answer distinct from the expected correct one.
• Domain matters more than model. Agents did better on structurally regular academic sources (ArXiv, Open Library of Humanities, Wikipedia) than on day-to-day sites (CNN, Lonelyplanet). For example, CNN Hard accuracy ranged from 3.45 to 13.79, while Wikipedia Easy ranged from 27.59 to 86.21.
• Closed-source models are more query-efficient. Accuracy was comparable between open and closed systems, but closed models typically needed only one or two search calls per query while open-source agents often needed more than five.
• Open-source agents struggle to fetch pages at all. Search-R1 (aiohttp, BeautifulSoup), CognitiveKernel-Pro (Playwright), and DeepResearcher (requests, markdownify) frequently returned empty responses or errors, even on heavily visited sites like CNN and ArXiv.
• Chunking hurts. Search-R1's chunk-level retrieval sometimes showed the correct page in the search results without returning it as the answer; similar cases appear in Sonar's intermediate steps.
• Repeated tool-use misunderstandings. Agents confused global search with domain-restricted search, issued searches for the entire query text, and — in CognitiveKernel-Pro's case — relied on string matching, which always fails because the queries never quote the ground-truth page's original wording.
• Human reference point. On 84 sampled queries with a 15-minute limit each, experts completed 85.7% of Easy queries (average 5 min 27 s), 78.6% of Medium (5 min 29 s), and 78.6% of Hard (6 min 41 s).

Methodology in Plain English

The authors chose seven websites with well-structured, moderate-length articles. They scraped article text with FireCrawl and wrote per-site scripts to strip out ads and other noise. For each article, an LLM extracted short factual claims, which were embedded and ranked by similarity to the whole article. Difficulty was then defined by which claims were used: the top three most central claims gave Easy queries, three mid-ranked claims gave Medium, and the three least central claims gave Hard. Each selected claim was then masked — names, places, and species replaced with generic expressions — producing the vague criteria the agent sees. A validation step asked an LLM whether the source article really supports every criterion; queries failing that check were discarded, which yielded the final 663 (222 / 229 / 212). Evaluation uses an LLM judge: it fetches the page the agent names, checks whether all criteria are mentioned, and if so whether the ground-truth claims are mentioned too. For the human baseline, 84 queries balanced across websites and difficulty levels were given to domain experts under a 15-minute cap.

Why This Matters

The paper argues that current search benchmarks systematically reward multi-hop factoid reasoning while ignoring the ambiguous, under-specified queries that dominate real usage, so reported progress may not reflect practical retrieval ability. Its benchmark, plus the finding that no system exceeds roughly 35% overall, gives the field a diagnostic target that separates retrieval quality from reasoning quality.

Real-world applications:

• Enterprise or internal search over documentation, where users often cannot name the exact document they want.
• News, research, and fact-checking workflows that need to surface a specific article from a fuzzy recollection of its content.
• Consumer assistants and shopping or travel agents that must map vague preferences onto specific pages.
• Academic literature discovery, where a researcher remembers a paper's finding but not its title, authors, or venue.

Industry relevance: the results directly bear on search-tool integration and agent scaffolding. The gap between closed and open systems appeared as a tooling and query-efficiency gap rather than a pure reasoning gap, and open-source agents broke on basic HTML fetching — both are engineering problems that vendors and framework maintainers can address.

Future Directions

• Improve fetching and tool interaction so agents can reliably obtain full pages and treat search APIs correctly, addressing the failures identified in the behavioral analysis.
• Build agents that recognize when the retrieved context is insufficient and keep exploring, rather than committing to fragmented snippets.
• Rework chunking and result presentation so a correct page visible in the search list is not lost before the final answer.
• Extend the benchmark beyond its limitations: the authors note it covers a fixed corpus at one point in time, and that interactive sites, social media, and non-English resources are underrepresented; they also suggest expansion to new domains, languages, and modalities. Whether performance gaps stem from reasoning or from toolchain artifacts remains an open question the paper explicitly flags as complicating fair comparison.

Target Audience

Researchers and engineers building LLM-based search agents, retrieval-augmented systems, and agent benchmarks will get the most from this paper, along with evaluation teams who need a benchmark that rewards retrieval under ambiguity rather than multi-hop factoid reasoning. Practitioners integrating web search into products will also find the agent behavioral analysis useful, since it isolates concrete tooling failures.

Authors’ abstract

Large Language Models (LLMs) have evolved from simple chatbots into sophisticated agents capable of automating complex real-world tasks, where browsing and reasoning over live web content is key to assessing retrieval and cognitive skills. Existing benchmarks like BrowseComp and xBench-DeepSearch emphasize complex reasoning searches requiring multi-hop synthesis but neglect Fuzzy Exploratory Search, namely queries that are vague and multifaceted, where users seek the most relevant webpage rather than a single factual answer. To address this gap, we introduce Needle in the Web, a novel benchmark specifically designed to evaluate modern search agents and LLM-based systems on their ability to retrieve and reason over real-world web content in response to ambiguous, exploratory queries under varying levels of difficulty. Needle in the Web comprises 663 questions spanning seven distinct domains. To ensure high query quality and answer uniqueness, we employ a flexible methodology that reliably generates queries of controllable difficulty based on factual claims of web contents. We benchmark three leading LLMs and three agent-based search systems on Needle in the Web, finding that most models struggle: many achieve below 35% accuracy, and none consistently excel across domains or difficulty levels. These findings reveal that Needle in the Web presents a significant challenge for current search systems and highlights the open problem of effective fuzzy retrieval under semantic ambiguity.