LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services

AI summary

Overview

Research area: Large reasoning models (LRMs), agentic search systems, retrieval-augmented generation, and benchmark design for vertical/domain-specific AI applications.

Technical level: Intermediate. Understanding the paper's core claims requires some familiarity with LLM agents, tool calling, multi-hop reasoning, and retrieval evaluation metrics (NDCG, MRR), though the problem framing is accessible to non-specialists.

Scope: This paper introduces LocalSearchBench, the first comprehensive benchmark for evaluating agentic search systems in local life services, comprising a 1.35M-entry merchant database, a retrieval tool (LocalRAG), and 900 curated multi-hop question-answer tasks, along with an evaluation environment (LocalPlayground) used to test 16 leading reasoning models.

What This Paper Is About

Agentic search systems use large reasoning models to break a query into sub-questions, call tools like web search or databases, and synthesize an answer across multiple steps. Most existing benchmarks test these systems on general information-seeking tasks and ignore vertical domains such as local life services, where queries are ambiguous, involve geographic and temporal constraints, and require reasoning across multiple merchants and service categories. This paper builds a benchmark and evaluation environment specifically for that domain, then measures how far current models fall short.

Key Contributions

1. The first comprehensive offline benchmark for agentic search in local life services. LocalSearchBench combines a 1,354,185-entry anonymized merchant database spanning 6 service categories (dining, lifestyle, shopping, accommodation, healthcare, tourism) across 9 major Chinese cities with 900 expert-validated multi-hop QA tasks derived from real user queries.

2. A systematic evaluation framework, LocalPlayground, plus a domain-specific retrieval tool (LocalRAG). LocalPlayground provides a ReAct-style search agent with merchant retrieval and web search tools, and a separate LLM-as-judge validation agent scoring both answer quality (7 metrics) and reasoning-trajectory quality (4 dimensions).

3. A reusable data construction pipeline. The authors designed multi-agent procedures for merchant seed collection, data augmentation (12 fields expanded to 29), privacy anonymization, question instantiation across intelligence levels L1–L5, and two-stage QA validation, all with reported inter-annotator agreement statistics.

4. Extensive empirical evaluation and failure analysis. Sixteen state-of-the-art models were benchmarked, with ablation and sensitivity studies on tool integration and retrieval hyperparameters, plus expert annotation of 580 failure trajectories to categorize why models fail.

Main Findings

• Current models perform poorly on this domain. The best model, DeepSeek-V3.2 (with thinking), achieved only 35.60% correctness, 77.56% completeness, and 39.78% faithfulness. Most models showed a large gap between fluent output and factually grounded output (average faithfulness 30.72%).

• Thinking models consistently outperform non-thinking variants. Across the panel, thinking-mode models reached 30.49% average correctness versus 21.30% for non-thinking modes, and 64.59% versus 49.10% completeness. Trajectory-quality advantages were smaller.

• Expert-level (L4) tasks are harder than intelligent-level (L3) tasks. Correctness dropped from 30.57% on L3 to 25.10% on L4, confirming that personalized planning and hard-to-access external dependencies pose extra difficulty.

• Web search cuts both ways. Adding web search to the LocalRAG toolchain on average raised correctness by 8.92 percentage points and completeness by 7.41 points, but reduced faithfulness by 2.83 points, indicating that real-time retrieval introduces noise into multi-hop chains.

• Five interaction rounds is the sweet spot. Varying the maximum round budget from 2 to 8 showed N=5 maximizes correctness (35.60%); both fewer rounds (insufficient evidence gathering) and more rounds (context noise) degraded performance.

• Retrieval hyperparameters matter with diminishing returns. The best LocalRAG configuration was top-N=100 dense candidates and top-K=20 after reranking, achieving NDCG@10 of 0.84 and MRR@10 of 0.82. Scaling to (150, 30) hurt correctness by 3.7% and search efficiency by 5.4%, an information-overload effect.

• Four dominant failure modes. Expert annotation of 580 failure trajectories found: unstable tool-call strategy (38.6%), missing explicit multi-hop planning (30.9%), long-context noise (18.8%), and query generation that omits critical constraints (11.7%).

• LLM-as-judge is reasonably reliable. Human evaluation showed substantial agreement with the automated judge (Cohen's kappa = 0.79 on answer scoring; 0.74 on data quality across 20 fields), with 88.34% raw agreement on the data-quality dimensions.

Methodology in Plain English

The authors started with raw merchant data from a leading Chinese local life service platform — over 1.6 million records sampled across nine cities in 2025. Because the original records had only 12 sparse fields, they used a data-augmentation agent to expand each merchant into 29 fields covering basic, locational, service, and product information, then a privacy-rewriting agent to anonymize seven sensitive fields (identity, exact location, contact details) while keeping the business signal intact. A validation agent and human spot-checks scored the resulting data on completeness, consistency, geographic accuracy, and anonymization degree.

For the question set, the team collected 1,200 real single-hop questions, classified them into five intelligence levels, and focused on L3 and L4 — levels that require genuine multi-step reasoning but can be simulated offline. Human annotators then grounded roughly 100 questions per city, expanding each into a multi-hop query with 2–4 reasoning segments such as cross-merchant comparison, schedule planning, and constraint resolution. Reference answers were produced by running each hop through LocalRAG and, when real-time signals were needed, web search, then having Claude Opus 4.5 synthesize a final answer under strict grounding rules. Every QA pair passed hop-by-hop validation and independent review by three domain experts, yielding 900 final tasks.

To evaluate models, the authors built LocalPlayground: a search agent that iteratively calls LocalRAG or web search over up to N rounds, recording the full reasoning trajectory, and a validation agent that scores both the final answer and the trajectory. Sixteen models — including GPT-4.1, o3, Gemini-2.5 Pro/Flash, DeepSeek-V3.2, GLM-4.6, Qwen3 variants, and the LongCat family — were run under identical settings.

Why This Matters

Research impact. The paper exposes a blind spot in agentic-search evaluation: general-domain benchmarks vastly overstate how well today's models handle geographically and temporally constrained, multi-service queries. It also supplies open-source infrastructure (benchmark, retrieval tool, evaluation environment, and construction pipeline) that other researchers can extend to new cities, languages, or verticals.

Real-world applications:

• Travel and itinerary planning: Building assistants that must coordinate restaurants, hotels, transport, and attractions under time and distance constraints.
• Local commerce and recommendation: Multi-constraint merchant recommendation that respects budget, dietary, location, and timing requirements simultaneously.
• Event and group coordination: Bundling multiple services (booking, catering, entertainment) into a single coherent plan for a group of people.
• Customer-service agents for platforms: Diagnosing which step a multi-hop user request failed at, using the trajectory-quality metrics to target retraining or tool improvements.

Industry relevance. Platforms that operate local life services — food delivery, on-demand services, travel booking, and location-based marketplaces — have direct stakes in this benchmark. The failure analysis tells engineering teams where to invest: deterministic tool-call policies instead of free-form agent decisions, explicit hop planners, constraint-preserving query rewriting, and retrieval pipelines that filter context rather than naive top-K expansion.

Future Directions

• Domain-specific agent training. The paper shows off-the-shelf LRMs plateau at 35.60% correctness; reinforcement learning or fine-tuning on local life service trajectories is the natural next step, and the released framework supplies the training signal.
• Explicit multi-hop planner architectures. Since missing hop planning accounts for 30.9% of failures, structured planners that decompose queries into validated sub-goals before any retrieval may outperform ReAct-style free-form loop decisions.
• Better tool-routing and tool-use discipline. Unstable tool-call strategy was the single largest failure category (38.6%); learned routing policies that decide when to use LocalRAG versus web search, and how often, are an open problem.
• Scaling to L5 and dynamic environments. The current benchmark deliberately excludes L5 (AGI-level) cross-platform coordination because it is hard to simulate offline. Extending to live, stateful platforms with real-time inventory, pricing, and availability would test the ceiling of these systems.

Target Audience

This paper is most valuable for researchers and engineers working on LLM agents, tool-augmented reasoning, and retrieval-augmented generation — especially those building systems for location-based or transactional domains where constraint satisfaction matters more than open-ended text generation. It is also directly relevant to product teams at local life service platforms evaluating whether current models are ready for user-facing deployment, and to benchmark designers looking for a template for constructing vertical-domain evaluation suites with rigorous human validation.

Authors’ abstract

Recent advances in large reasoning models LRMs have enabled agentic search systems to perform complex multi-step reasoning across multiple sources. However, most studies focus on general information retrieval and rarely explores vertical domains with unique challenges. In this work, we focus on local life services and introduce LocalSearchBench, which encompass diverse and complex business scenarios. Real-world queries in this domain are often ambiguous and require multi-hop reasoning across merchants and products, remaining challenging and not fully addressed. As the first comprehensive benchmark for agentic search in local life services, LocalSearchBench comprises a database of over 1.3M merchant entries across 6 service categories and 9 major cities, and 900 multi-hop QA tasks from real user queries that require multi-step reasoning. We also developed LocalPlayground, a unified environment integrating multiple tools for LRMs interaction. Experiments show that even state-of-the-art LRMs struggle on LocalSearchBench: the best model (DeepSeek-V3.2) achieves only 35.60% correctness, and most models have issues with completeness (average 60.32%) and faithfulness (average 30.72%). This highlights the need for specialized benchmarks and domain-specific agent training in local life services. Code, Benchmark, and Leaderboard are available at https://localsearchbench.github.io/.