HALO: Human-Grounded Granular Evaluation of Proactive Agents

AI summary

Overview

• Research area: Natural Language Processing, specifically evaluation of LLM-based proactive agents that act without explicit user requests.
• Technical level: Advanced. The paper assumes familiarity with LLM agents, tool use, retrieval, and benchmark design, although its main conclusions are stated in accessible terms.
• Scope: The paper introduces HALO, an executable benchmark (HALO-Arena) and a human-annotated response dataset (HALO-8K), to diagnose whether, when, and how far proactive agents should intervene.

What This Paper Is About

Proactive agents must decide whether to help, when to help, and how much to do without being asked. Existing evaluations often reduce this to automated scores or binary accept/reject verdicts, which do not explain whether an agent used evidence correctly or why a particular intervention failed. HALO addresses this by pairing simulated personal environments and evidence-retrieval tasks with fine-grained human judgments about action level, timing, intensity, and other failure modes.

Key Contributions

1. HALO-Arena: An executable benchmark covering 4 proactive need types (Dependency, Blockage, Risk, and Opportunity), 10 everyday domains, 50 personal environments, and 400 scenarios. Agents interact with email, calendar, chat, contacts, and wallet apps, plus SQLite databases and text files, using 29 interaction commands and 19 action commands.
2. HALO-8K: A dataset of 8,000 responses from 10 models across 400 scenarios and 2,160 checkpoints, annotated by 20 experts. Annotations include action level, acceptability verdicts, and diagnoses of rejected responses covering timing, intensity, and other issues.
3. A 10-model evaluation: The paper compares 5 proprietary and 5 open-source models on acceptability, action choices, timing and intensity errors, reasoning tokens, and retrieval rounds.
4. A central empirical finding: Over-proactivity, defined as Premature timing or Excessive intensity, accounts for 61.20% of rejected responses, making it the dominant observed failure mode.

Main Findings

• Acceptability varies widely across models. Scores range from 30.38% for GPT-OSS-20B to 72.88% for Gemini3.8-Flash, a 42.50-point gap. DeepSeekV4-Pro leads open-source models at 56.63%. Even the strongest model leaves over 25% of responses unacceptable.
• Proprietary models average higher but the groups overlap. Proprietary models average 63.08% acceptability versus 47.55% for open-source models, a 15.53-point gap. DeepSeekV4-Pro exceeds both Gemini3.1-Pro and GPT5.6-Luna, so group membership is an incomplete account of quality.
• Over-proactivity dominates rejections. Premature timing or Excessive intensity occurs in 2,188 of 3,575 rejected responses (61.20%). Model-wise shares range from 44.23% to 79.66%, exceeding 50% of rejections in 6 of 10 models. Premature timing is almost 3 times as common as Belated timing, and Excessive intensity is about 2 times as common as Insufficient.
• Attention is the most common action level. L1 Attention accounts for 69.20% of all responses, L2 Preparation for 4.18%, and L3 Execution for 16.09%. L1 exceeds 50% of responses in 9 models and ranges from 48.00% for Claude-Opus5.5 to 87.00% for GPT5.6-Luna.
• Execution frequency alone does not explain the acceptability gap. L3 shares are almost identical across the proprietary and open-source groups at about 16.1%, despite the 15.53-point acceptability gap. Proprietary models choose L0 Observation about 2 times as often, while open-source models choose L1 more often.
• Restraint can matter as much as action. In paired comparisons at shared checkpoints, Qwen3.5-9B raises intervention frequency by 27.36 points and acceptability by 14.43 points, while Gemini3.8-Flash raises intervention frequency by 28.89 points but acceptability falls by 14.77 points. The Gemini loss combines a 5.94-point gain from accepted interventions with a 20.71-point decline in accepted L0. The paper notes these comparisons do not isolate a causal effect.
• Need types elicit different restraint. Opportunity and Dependency have similar acceptability (55.85% vs. 55.05%), but L0 accounts for 13.60% of Opportunity responses versus 7.20% of Dependency responses. All 10 models choose L0 more often for Opportunity than Dependency.
• Evidence acquisition is associated with better help. Across 10 models, mean retrieval rounds correlate with acceptability (r = 0.885, ρ = 0.830). Among 5,536 L1 responses, content access is associated with 59.0% acceptability versus 40.8% without it, and Insufficient rates of 5.2% versus 15.5%. A fixed-effects model using 5,232 L1 responses at 1,730 checkpoints estimates +9.25 points in acceptability and −

Authors’ abstract

Proactive agents must decide whether, when, and how far to assist without an explicit request. Existing evaluations often rely on automated scores or binary verdicts, offering limited human diagnosis of evidence use and intervention quality. To address these gaps, we introduce HALO, Human-Grounded GrAnular EvaLuation of PrOactive Agents. HALO-Arena covers 4 proactive need types, Dependency, Blockage, Risk, and Opportunity, across 10 everyday domains and 50 personal environments. Scenarios require agents to retrieve evidence from structured databases or unstructured documents before judging whether assistance is warranted and choosing its scope. On HALO-Arena, we construct HALO-8K, a dataset of 8,000 responses from 10 models across 400 scenarios, annotated by 20 experts for action level, acceptability verdicts, and human diagnosis. For rejected responses, human diagnosis covers timing, intensity, and other issues. Recorded reasoning tokens and retrieval rounds measure the effort behind these responses. The best proprietary and open-source models achieve 72.88% and 56.63% acceptability, respectively. Our central finding is that over-proactivity is the dominant observed failure mode, with Premature timing or Excessive intensity accounting for 61.20% of rejected responses. Further analyses yield complementary findings on the balance between intervention and restraint, evidence acquisition, and the calibration of timing and intensity.