HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference

AI summary

Overview

Research area: AI evaluation methodology — specifically sample-efficient benchmarking of software-engineering agents, combining item-response theory, Bayesian hierarchical modeling, and active (adaptive) data acquisition.

Technical level: Advanced. The paper assumes familiarity with item-response/latent-trait models, Beta–Binomial conjugacy, importance sampling, SVD-based factor models, and variance-based acquisition criteria.

Scope: The paper proposes HSPR, a hierarchical posterior-inference procedure that estimates a software agent's full-benchmark score from roughly 10% of task executions, and evaluates it on four task banks from the SWE-bench family.

What This Paper Is About

Running a software agent on an entire benchmark is expensive, since each task can require repository exploration, code editing, tool use, and execution. Evaluating only a small subset of tasks is cheaper and can preserve the ranking of agents, but the resulting scores are often badly off — the paper notes that a Mid-Range subset on SWE-bench Pro preserves ordering well (Kendall's τ_b = .967) while its raw subset mean has a mean absolute error of 22.798 percentage points. HSPR addresses this gap: given a budget of only a few observed task outcomes, it recovers an accurate numerical estimate of the agent's full-benchmark score by exploiting historical responses from other agents and by choosing which tasks to run next.

Key Contributions

1. A hierarchical full-score model (HSPR) that combines source-fitted response factors (an intercept, an ability slope, and residual loading directions per task) with uncertain target-specific success rates inside groups of tasks that share an identical historical response signature.

2. A posterior-guided acquisition rule for score recovery, whose criterion (Equation 10) is proven to equal the expected one-step reduction in the posterior variance of the full-benchmark score; a latent-state alternative is also defined, and source-only validation selects between them before any target evaluation.

3. A predictive decision procedure: observed successes plus posterior-predictive counts for unqueried tasks define a distribution over the full score, whose median (with optional source-selected calibration) is the reported estimate.

4. A controlled empirical study across four SWE-bench task banks, including end-to-end hierarchy controls, estimation and acquisition ablations, paired statistical analysis, mechanism diagnostics, and 5–20% budget sweeps.

Main Findings

• Lowest error on three of four banks: At a matched 10% task budget, HSPR attains the lowest MAE on Lite (1.927 pp), Full (0.717 pp), and Pro (2.098 pp). On Verified its MAE is 1.915 pp, within 0.055 pp of ATLAS-port (1.861 pp).
• Lowest worst-case error everywhere: HSPR has the lowest seed-averaged MaxErr on all four banks — 5.683 (Lite), 6.000 (Verified), 2.638 (Full), and 4.601 (Pro).
• Strong aggregate standing: Equal-bank macro MAE is 1.664 pp, versus 2.478 pp for PTA-IRT, described as the strongest baseline on that aggregate; HSPR ranks in the top two across all 12 benchmark–metric combinations and leads eight.
• Ranking is not score accuracy: On Pro, raw Mid-Range selection has τ_b = .967 yet MAE of 22.798 pp, compared with 2.098 pp for HSPR; applying a source-fitted Ridge estimator to the same Mid-Range outcomes lowers MAE to 3.434 pp.
• Group uncertainty helps unevenly: Removing group uncertainty raises macro MAE from 1.664 to 1.808 pp; the gain is largest on Full (MAE 1.331 → 0.717 pp). The paired bootstrap interval for the 0.143 pp macro difference is [−0.012, 0.302], i.e. it includes zero.
• Adaptive querying is the biggest lever: Freezing the HSPR reconstructor but replacing adaptive queries with Uniform or deterministic Mid-Range queries raises macro MAE to 3.085 and 2.934 pp respectively.
• The median decision is marginally better: Using the posterior mean instead of the median on identical query paths gives macro MAE 1.674 pp versus 1.664 pp.
• Group structure explains the pattern: SWE-bench Full's 2,294 tasks form about 276 source-signature groups on average (N/Ḡ = 8.32), while Verified retains 426 groups among 500 tasks (1.17), leaving less within-group sharing to exploit.
• Identical historical signatures, different target outcomes: On Full and Pro, 41.3% and 36.7% of tasks have no successes among source agents, yet target success rates in those sets are 1.84% and 2.07%.
• Hierarchy changes where evidence is gathered: In the controlled Full comparison, source-zero tasks receive 13.2% of queries with hierarchy versus 0.19% without it.
• Budget sweep: Equal-bank macro MAE falls from 2.562 pp at a 5% budget to 1.262 pp at 20%, with 10% settings held fixed across checkpoints.

Methodology in Plain English

The method works in three stages.

Stage one — learn from history. The researchers take an archived matrix of binary pass/fail outcomes for many earlier agents across all tasks in a benchmark. They fit a simple logistic "item model" in which each task has a baseline difficulty and each agent has an ability level, then take a singular value decomposition of what that model fails to explain, yielding a few residual directions per task. Tasks with exactly the same pattern of historical outcomes are pooled into "source-signature groups," and each group's task parameters are averaged. All of this is fixed before the new agent is evaluated.

Stage two — evaluate the new agent adaptively. The new agent's behaviour is described by a small latent state: one ability number plus a few residual coordinates. Each group of tasks has its own success rate, drawn from a Beta distribution centered on what the latent state predicts, with a tunable concentration controlling how much rates may drift from that prediction. Every observed outcome updates both the global latent state and that group's rate, so one result informs other tasks in the same group and, through the global state, tasks in other groups. To pick the next task, HSPR computes for each candidate group the squared covariance between the full score and a random outcome from that group, divided by that outcome's variance — a quantity shown to equal the expected reduction in score variance. It picks the best group and samples a task uniformly inside it, repeating after each observation. Source-only leave-one-agent-out validation decides the settings, including whether to use the score-variance rule or a latent-state rule (13 of 16 deployed folds chose Score, three chose Latent).

Stage three — report a number. The observed successes are combined with simulated counts for all unqueried tasks to build a distribution over the complete benchmark score. The median of that distribution is reported, optionally shifted by a small correction term scaled by λ ∈ {0, .5} and by the unobserved fraction (1 − B/N), then clipped to [0, 1]. Computation uses 8,192 importance-sampling nodes drawn from a scrambled Sobol sequence and 131,072 complete score draws, with the simulation seed 20261005.

Why This Matters

Impact on research. The paper makes a sharp distinction between ranking fidelity and score fidelity, and shows that a cheap subset can look excellent on the former while being off by more than 20 percentage points on the latter. It also offers a principled alternative to raw subset means: a model-conditional posterior over the full score, with an acquisition rule derived from an exact variance identity rather than heuristic informativeness. The authors explicitly recommend that recovered scores be reported alongside the evaluation budget and clearly labelled as recovered rather than measured.

Real-world applications:

• Cheap leaderboard maintenance: benchmark maintainers could refresh rankings of many agent configurations at a fraction of the task-execution cost, while still reporting calibrated scores.
• Faster iteration during agent development: teams could compare scaffold or prompt variants without paying for a full SWE-bench-scale run at every checkpoint.
• Regression testing of agent releases: a fixed small, adaptively chosen task set per agent could flag score drops after a change.
• Budget-aware procurement or auditing: third parties assessing an agent under a fixed execution budget could obtain an interval-aware score estimate instead of an unqualified subset mean.

Industry relevance. The economics of agent evaluation are dominated by per-task execution cost (tokens, sandboxes, wall-clock time). The paper measures budget purely in distinct target-task executions and reports evaluator inference overhead separately; it explicitly cautions that token, wall-clock, and monetary savings "cannot be inferred directly from task-count reduction," and that historical data collection, source fitting, and configuration selection add cost. The relevance is therefore strongest for organizations that already hold large historical response matrices and want to amortize them.

Future Directions

• Generalization beyond SWE-bench: the authors state that behaviour under substantial shifts in model families or agent scaffolds "remains to be evaluated"; testing on non-software agent benchmarks is the obvious next step.
• Reducing dependence on source coverage: response factors and signature groups come from the historical bank, so sparse or unrepresentative history is an open failure mode worth quantifying.
• Relaxing within-group exchangeability: posterior uncertainty is conditional on the assumed model, including the assumption that tasks sharing a signature are exchangeable draws from a shared rate — an assumption the source-zero analysis shows is only approximately true.
• Accounting for full evaluation cost: converting task-count budgets into token, wall-clock, and monetary budgets, and charging configuration-selection cost explicitly, would make the efficiency claims directly actionable.

Target Audience

Researchers and practitioners in LLM/agent evaluation who need accurate benchmark scores under tight execution budgets; methodologists interested in psychometric and Bayesian approaches to benchmark compression (item-response models, active testing, surrogate estimators); and benchmark maintainers or engineering teams who already possess large historical response matrices and want to reuse them. Readers without a background in latent-variable modeling will find the mathematical sections demanding, though the empirical comparison in Table 1 and the ablation structure in Table 2 are accessible on their own.

Authors’ abstract

Evaluating software agents on a full benchmark is expensive. A small task subset can preserve agent rankings while yielding inaccurate scores. Historical responses provide useful structure, but tasks with identical source responses can have different outcomes for a new target agent. We introduce Hierarchical Score Posterior Recovery (HSPR), which combines source-fitted response factors with uncertain group success rates. Observed target outcomes update the posteriors over the target latent state and group rates; posterior-guided acquisition selects subsequent tasks. Its score-based criterion equals the expected one-step reduction in full-score posterior variance. Observed successes and posterior-predictive counts for unqueried tasks yield a full-score posterior. The final estimate uses its median with source-selected calibration. At a 10% task budget on four task banks from the SWE-bench family, HSPR achieves the lowest mean absolute error (MAE) on three banks and the lowest maximum absolute error (MaxErr) on all four among the methods compared. Removing group uncertainty or replacing adaptive queries increases macro MAE in the respective controls. Macro MAE decreases across 5-20% budgets.