InteractComp: Evaluating Search Agents With Ambiguous Queries

AI summary

Overview

• Research area: Natural Language Processing — specifically LLM-based search agents, interactive clarification, and benchmark design for agent evaluation.
• Technical level: Intermediate. Readers should be familiar with LLM agents, tool-use/ReAct-style loops, and standard benchmark metrics, but the paper's core argument is conceptual and accessible without deep technical background.
• Scope: The paper introduces InteractComp, a 210-question bilingual benchmark that measures whether search agents can recognize ambiguous queries and actively ask clarifying questions to resolve them, and it uses that benchmark to diagnose a systematic failure across 17 models.

What This Paper Is About

Existing search-agent benchmarks like GAIA and BrowseComp assume the user's query is complete and unambiguous, so agents can succeed purely through retrieval and reasoning. In reality, users often start with underspecified requests whose intended target cannot be identified without follow-up questions. The paper builds a benchmark where ambiguity is deliberately engineered so that no amount of searching alone can resolve it, then measures whether agents recognize this and interact to disambiguate before answering.

Key Contributions

1. A new benchmark (InteractComp) of 210 expert-curated, bilingual (139 English, 71 Chinese) questions across 9 topic domains, purpose-built so that answers are short and verifiable once the right context is obtained, but genuinely underdetermined without interaction.
2. A target-distractor construction methodology that pairs a lesser-known target entity with a popular look-alike sharing the same attributes, so that questions alone admit multiple plausible answers while distinctive attributes are hidden and recoverable only through clarification.
3. A diagnostic evaluation of 17 models spanning open-weight and proprietary families, isolating the incremental value of knowledge recall, search, and interaction through four controlled agent configurations.
4. Evidence that interaction failure is strategic, not capability-based, supported by forced-interaction experiments, natural-language interaction variants, error attribution, and a 15-month longitudinal comparison against BrowseComp.

Main Findings

• A five-fold gap reveals the core problem. With complete disambiguating context, the best model reaches 71.50% accuracy (OpenAI o3); in the full search-plus-interaction setting, the best model (GPT-5) reaches only 13.73%. The gap indicates models can solve the task once informed but fail to obtain the information themselves.

• Search alone barely helps. In ablations, answer-only accuracy ranges from 1.43% to 7.62%, and adding web search raises it only to roughly 6.74%–9.52%. Retrieval cannot resolve the ambiguity because the distinguishing attributes are absent from the initial query by design.

• Asking beats searching. The ask-only configuration (6.67%–25.24%) outperforms search-only for every tested model, because clarification directly yields the missing discriminative attributes. This does not mean search is unnecessary, but that search cannot succeed before disambiguation occurs.

• Models underuse interaction even when it is available. Interaction rates vary wildly, from 0.25% of rounds for GLM-4.5 to 73.95% for GPT-4o-mini, and high interaction rates do not automatically translate into accuracy. GPT-4o-mini asks constantly yet scores only 7.14%, suggesting unfocused questioning is as ineffective as silence.

• More opportunities do not fix the behavior. Quadrupling the round budget from 5 to 20 moves GPT-5's average interactions only from 1.14 to 1.90 and accuracy from 14% to 20%. Claude-Sonnet-4 barely interacts at all across budgets.

• Forcing interaction roughly doubles performance. Requiring GPT-5 to ask at least 8 clarifying questions before answering raises accuracy from 20% to 40%, confirming the capability is latent rather than absent. Not all models benefit equally — Claude-Sonnet-4 shows only modest gains — indicating information acquisition is a distinct capability that differs by architecture.

• Richer responses do not close the gap. Allowing free-form natural-language replies instead of yes/no/unknown raises GPT-5 to 25.71% and DeepSeek-V3.1 to 17.14%, but both remain far below their with-context ceilings of roughly 60–72%. Bandwidth is not the binding constraint; deciding what to ask is.

• Two distinct failure modes emerge from error attribution. GPT-5 fails mainly on strategic errors (70.5% overconfidence — answering with minimal interaction), while Claude-Sonnet-4 fails mainly on tactical errors (87.8% identification errors — asking and searching but locking onto the wrong candidate). Gemini-2.5-Pro falls between them.

• Interaction capability has stagnated while search has surged. Over 15 months, BrowseComp performance improved roughly seven-fold (about 10% to 70%), while InteractComp performance stayed flat in the 6–14% band across GPT-5, DeepSeek-R1, GPT-4.1, and others.

• Interaction correlates with better calibration. Models that ask more questions tend to produce more realistic confidence estimates — GPT-4o-mini's calibration error is 37.44 versus 84.35 for low-interaction Doubao-1.6 — even when their raw accuracy does not improve.

• Open-weight models are markedly more conservative. All open-weight models interact in under 45% of rounds, most under 32%, while proprietary models such as GPT-4.1 and GPT-5 ask in roughly 31–34% of rounds. This pattern suggests open-weight training may discourage seeking clarification.

Methodology in Plain English

The researchers start from a simple observation: ambiguity is worst when two entities look alike but one is far more famous. So human annotators (master's and PhD students) were asked to find a pair of similar entities differing in popularity — for example, a well-known sport and an obscure one sharing surface features. The question is then written using only the attributes both entities share, making it genuinely ambiguous. The attributes unique to the lesser-known target are set aside as hidden "context" that a simulated user will reveal if asked.

To keep interaction meaningful, the protocol between agent and simulated user is restricted to binary answers: yes, no, or "I don't know." This is deliberate. If the user could dump all context at once, the task would collapse into ordinary retrieval. Binary answers force the agent to commit to specific hypotheses and tests whether it can generate the right discriminating questions rather than merely asking many questions.

Verification runs in two stages. First, independent annotators confirm the target satisfies every stated attribute, that question-plus-context admits exactly one valid answer, and that paraphrasing the question does not change the candidate set. Second, interaction necessity is validated by hand-checking the first five pages of Google results and by running GPT-5, GPT-5-mini, and Claude-Sonnet-4 for five rounds without interaction — any question those models answer too easily gets rewritten to be more ambiguous.

Evaluation uses a ReAct-style agent with four configurations (answer-only, answer-plus-search, answer-plus-interact, and the full stack), run for up to 10 rounds per instance, with GPT-4o simulating the user. Metrics cover accuracy, calibration error across five confidence bins, interaction rate, average rounds, and dollar cost. The team also ran scaling tests at 5, 10, and 20 rounds, a natural-language interaction variant, forced-interaction protocols requiring 2 to 10 questions before answering, manual error attribution on failed trajectories, and a longitudinal comparison against BrowseComp over 15 months.

Why This Matters

Impact on research. The paper reframes a capability gap as a measurement gap. Search benchmarks have driven genuine progress because they are easy to grade, but they systematically exclude the ambiguity that real users bring. InteractComp shows that this omission has real consequences: a capability that models clearly possess is essentially untrained and unevaluated. Because final answers are short and binary-correct, the benchmark produces clean reward signals suitable for reinforcement learning from verifiable rewards, giving the community a training substrate rather than just a leaderboard.

Real-world applications:

• Conversational assistants and chatbots that currently guess at underspecified requests instead of asking a targeted follow-up, especially in customer support and technical troubleshooting where a single disambiguating question saves a long, wrong trajectory.
• Enterprise and legal search, where a query like "the 2023 compliance filing" may match dozens of documents and where asking "do you mean the EU or US version?" is vastly cheaper than returning the wrong one.
• Medical and clinical information lookup, where ambiguous drug names, procedure names, or patient descriptors carry real risk if the system confidently commits to the wrong interpretation.
• Agentic research and data-gathering pipelines, where wasted search calls and tokens are a direct cost, and where premature confident answers propagate errors downstream into reports or decisions.

Industry relevance. The cost data in the paper make the business case concrete: some models burn $60–$115 per evaluation run while achieving single-digit accuracy, largely because they search and answer without ever asking. Interaction is not just an accuracy feature — it is a resource-allocation policy. The finding that models trained to be decisive may be suppressing clarification behavior is directly actionable for anyone fine-tuning or deploying agents, particularly vendors building search products and agent frameworks.

Future Directions

• Training interaction policies with reinforcement learning. The paper explicitly positions InteractComp as an RLVR environment. An open question is whether reward shaping on verifiable outcomes can teach models when to ask, not just how to ask, and whether that transfers to domains outside the benchmark.
• Separating strategic from tactical interaction skill. Error attribution shows different models fail at different layers — some never ask, others ask badly. Whether these two abilities can be trained independently, and which is the harder bottleneck, remains unresolved.
• Reducing user burden while preserving information gain. Binary yes/no interaction is a deliberate constraint, but it imposes many turns. Designing protocols that extract maximum discriminative information per question — or that let users volunteer partial information naturally — is an open design problem with direct implications for interaction fatigue.
• Combining interaction training with reasoning-efficiency methods. The paper gestures at pairing better questioning with techniques that suppress redundant deliberation and long traces, so agents ask sharper questions without inflating computational cost.
• Testing generalization beyond English and Chinese. The benchmark is bilingual across 9 domains, but whether interaction competence learned here transfers to other languages, multimodal queries, or multi-user settings is untested.

Target Audience

Researchers and engineers working on LLM agents, tool-use, and information retrieval will find the benchmark directly usable, as will teams building conversational search products or evaluating agent reliability. Benchmark designers and evaluation researchers will benefit from the target-distractor construction methodology and the two-stage verification protocol as a template for building ambiguity-aware datasets. Reinforcement learning practitioners interested in verifiable-reward environments and multi-turn credit assignment are a secondary audience, since the paper argues InteractComp provides clean binary signals for training. Finally, product managers and technical decision-makers evaluating agent deployments will find the calibration and cost data useful for understanding why confident answers are not the same as correct ones.

Authors’ abstract

Language agents have demonstrated remarkable potential in web search and information retrieval. However, many search-agent benchmarks assume that user queries are complete and unambiguous. This assumption leaves under-tested a practical failure mode: agents may face ambiguous requests where the intended target cannot be identified without clarification. Yet most agents lack interactive mechanisms during the search process, and existing benchmarks cannot assess this capability. To address this gap, we introduce InteractComp, a benchmark designed to evaluate whether search agents can recognize query ambiguity and actively interact to resolve it during search. Following the principle of easy to verify, interact to disambiguate, we construct 210 expert-curated questions across 9 domains through a target-distractor methodology that creates controlled ambiguity resolvable only through interaction. Evaluation of 17 models reveals striking failure: the best model achieves only 13.73% accuracy despite 71.50% with complete context, exposing systematic overconfidence rather than reasoning deficits. Forced interaction produces dramatic gains, demonstrating latent capability current strategies fail to engage. Longitudinal analysis shows interaction capabilities stagnated over 15 months while search performance improved seven-fold, revealing a critical blind spot. This stagnation, coupled with the immediate feedback inherent to search tasks, makes InteractComp a valuable resource for both evaluating and training interaction capabilities in search agents. The code is available at https://github.com/FoundationAgents/InteractComp.