GISA: A Benchmark for General Information-Seeking Assistant

AI summary

Overview

• Research area: Natural Language Processing — specifically evaluation benchmarks for LLM-driven search agents that autonomously gather information through multi-turn web interaction.
• Technical level: Intermediate. The paper introduces no new model architecture; its technical content is benchmark construction, evaluation protocol design, and agent system configuration, with substantial experimental detail.
• Scope: The paper presents GISA, a 373-query human-constructed benchmark for evaluating general information-seeking assistants across item, set, list, and table answer formats, spanning both deep and wide search, with a periodically updated live subset and complete human search trajectories for every query.

What This Paper Is About

Existing benchmarks for web search agents are built in ways that make them poor proxies for real user needs: many are reverse-engineered by starting from a chosen answer and writing a query backward from it, most evaluate either deep navigation to a single fact or broad aggregation across sources but not both, and most rely on static answer sets that LLMs may have memorized during pre-training. GISA was built to address all three problems at once, by having humans design queries from genuine browsing experiences, requiring both deep reasoning and broad collection within the same task, and maintaining a live subset whose answers are periodically updated. The result is a benchmark whose answers are structured (item, set, list, table) so that grading can be deterministic rather than dependent on an LLM judge.

Key Contributions

1. A human-constructed benchmark of 373 queries with structured, deterministically gradable answers. Answers are cast into four formats — item, set, list, and table — allowing strict matching metrics instead of subjective LLM-based judgment.
2. Unified evaluation of deep and wide search. Queries are designed so that an agent must both aggregate information broadly (for example, gathering basic facts about all presidents of a country) and reason deeply (for example, verifying each one's historical record), rather than doing only one or the other.
3. A dynamic, anti-contamination design. Queries are split into a stable subset (answers immutable for at least three years) and a live subset subject to change, with the live subset's ground truth maintained on a monthly schedule. A memorization check using DeepSeek-V3.2 with reasoning and web search disabled removes queries answerable from parametric knowledge alone.
4. Complete human search trajectories for every query. Interaction logs — Google search queries issued, search results page content, click-through behavior, and precise timestamps — are captured with a custom browser extension and serve as gold-standard references for process-level supervision, process reward modeling, and imitation learning, while also confirming each task is solvable by realistic search behavior.

Main Findings

• Best performing model reaches only 19.30% overall exact match. Claude 4.5 Sonnet (thinking) is the top LLM-based ReAct agent, with 63.64% EM on item questions but only 13.04% EM on table questions. The paper concludes that current search agents are far from reliably solving complex information-seeking tasks.
• Performance degrades as the breadth of required information grows. Agents do reasonably well on item-type questions but struggle on table-type tasks that require collecting and organizing information across multiple dimensions. The paper notes that in GISA the amount of information to gather does not necessarily correlate with question difficulty, and that the drop on lists and tables reflects difficulty in answer organization and formatting, not only collection.
• Excessive tool use does not help. Claude 4.5 Sonnet (thinking) uses moderate numbers of search (7.57) and browse (4.63) calls, while DeepSeek-V3.2 and GLM-4.7 invoke substantially more tools but score lower; the paper attributes this to noise from irrelevant retrieved content and longer context harming reasoning quality.
• Thinking mode gives consistent but costly gains. Claude 4.5 Sonnet improves from 16.36% to 19.30% overall EM when thinking is enabled, and DeepSeek-V3.2 from 11.53% to 14.47%, at the cost of significantly higher token consumption.
• Commercial search and deep research systems underperform LLM-based ReAct agents. GPT-4o Search Preview (5.63% overall EM), OpenAI o4 Mini Deep Research (7.78%), Perplexity Sonar Pro Search (7.51%), and Google Search AI Mode (9.38%) all trail the best ReAct agent. The deep research systems are described as suffering from poor instruction-following and consequent formatting errors; Google Search AI Mode is noted as relatively better among commercial systems and faster in response speed.
• Models and humans search differently. Humans issue fewer queries (3.53 on average) but browse substantially more pages (19.03), whereas models search more frequently (7.57 queries) but browse far fewer pages (4.63). Humans also show higher adjacent query overlap (0.31 vs. 0.22), indicating more targeted query refinement.
• Behavioral similarity to humans correlates with performance. High-similarity cases achieve an average F1 of 0.76 versus 0.56 for low-similarity cases, and successfully solved cases show higher URL overlap (0.31 vs. 0.15).
• The live subset exposes likely memorization. Kimi K2.5, the most recently released model evaluated (Jan. 28, 2026), scores 11.33% overall EM on the live subset versus 18.39% on the stable subset, while Claude 4.5 Sonnet shows no significant difference between subsets.
• Inference-time scaling helps but answer selection lags. On 40 randomly sampled queries with Qwen3-Max, Best@k rises from 8.90% to 22.22% at k=16, a 2.5× improvement, while Majority@k improves but consistently lags behind Best@k, indicating that choosing the correct answer among multiple candidates remains difficult.
• Most errors occur at the search stage and the output stage. In a manual analysis of 50 error cases from Claude 4.5 Sonnet with thinking: query misunderstanding accounts for only 3.2%; the search level accounts for 49.2% (inability to formulate effective queries 14.3%, failure to exploit in-page hyperlinks 17.5%, failure to issue verification queries when sources conflict 17.5%); and the output level includes information extraction failures at 15.9% and instruction-following errors at 31.7%, the largest single category. A single sample may exhibit multiple error types.

Methodology in Plain English

GISA was built entirely around human annotators rather than automated generation, in four stages:

1. Brainstorming. Using the ten-domain taxonomy from BrowseComp (TV Shows & Movies, Science & Technology, Art, History, Sports, Music, Video Games, Geography, Politics, and Other), annotators browsed domain-specific websites — news aggregators for Politics, encyclopedic archives for History — and wrote down questions that arose naturally from what they read.
2. Query refinement. Raw questions were converted into formal queries with a chosen answer format (item, set, list, or table). Each seed question produced exactly one query. Annotators ran a preliminary feasibility check to confirm the answer was not already available pre-aggregated on an existing page, discarding trivial questions. List queries required an explicit sorting criterion in the prompt; table queries required fully specified column names, a designated primary sort column, and secondary and tertiary keys as tie-breakers so row order is unique. Queries were labeled stable or live.
3. Human annotation. Fifteen information-retrieval graduate students used a custom browser extension that logged Google queries, search results page content, click-throughs, and timestamps. They were restricted to Google Search and forbidden from using other search engines or LLMs, and instructed to ignore Google's built-in AI-generated summaries. Answers were saved as CSV, trajectories exported as JSON logs, and a post-processing script standardized the trajectories. A five-question pilot phase (excluded from the final data) trained annotators on

Authors’ abstract

The advancement of large language models (LLMs) has significantly accelerated the development of search agents capable of autonomously gathering information through multi-turn web interactions. Various benchmarks have been proposed to evaluate such agents. However, existing benchmarks often construct queries backward from answers, producing unnatural tasks misaligned with real-world needs. Moreover, these benchmarks tend to focus on either locating specific information or aggregating information from multiple sources, while relying on static answer sets prone to data contamination. To bridge these gaps, we introduce GISA, a benchmark for General Information-Seeking Assistants comprising 373 human-crafted queries that reflect authentic information-seeking scenarios. GISA features four structured answer formats (item, set, list, and table), enabling deterministic evaluation. It integrates both deep reasoning and broad information aggregation within unified tasks, and includes a live subset with periodically updated answers to resist memorization. Notably, GISA provides complete human search trajectories for every query, offering gold-standard references for process-level supervision and imitation learning. Experiments on mainstream LLMs and commercial search products reveal that even the best-performing model achieves only 19.30\% exact match score, with performance notably degrading on tasks requiring complex planning and comprehensive information gathering. These findings highlight substantial room for future improvement.