AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling

AI summary

Overview

Research area: Evaluation methodology for large language models — specifically LLM-as-a-judge reliability in agentic tool-calling settings.

Technical level: Advanced. The paper assumes familiarity with LLM benchmarking, function/tool-calling, directed acyclic graphs, Cohen's kappa, bootstrap confidence intervals, and chain-of-thought prompting.

One-sentence scope: AgentJudgeBench is a 3,808-record benchmark that measures how reliably six LLM judges score agentic tool-calling outputs across six DAG topologies, three difficulty tiers, five generators, and paired with- versus without-ground-truth conditions.

What This Paper Is About

Practitioners increasingly use LLMs as automatic judges to score agentic systems that call tools and orchestrate multi-step workflows, but nobody has checked how reliable those judges actually are in this structured setting. Unlike open-ended text evaluation, tool-calling correctness requires the right tools, well-formed arguments, correct ordering, and full query coverage — four aspects that can fail independently. This paper builds a controlled benchmark to measure judge alignment with a deterministic programmatic reference, and to test whether chain-of-thought, judge temperature, ground-truth exposure, or prompt structure can improve it.

Key Contributions

1. A dataset of 3,808 records spanning six DAG topologies (linear, fan-out, fan-in, diamond, optional enrichment, loop-like) and three difficulty tiers (easy, medium, hard), each with a programmatically verified ground-truth trace, across 15 enterprise domains with 8–19 tools per record.

2. A four-metric evaluation framework — tool selection, parameter structure, sequence accuracy, query coverage — with a paired with-ground-truth/without-ground-truth protocol and bootstrap confidence intervals.

3. A six-research-question empirical study producing actionable guidance: with ground truth, QwQ-32B best matches the programmatic reference while GPT-OSS-120B is the most human-aligned judge in a 120-record validation study; without ground truth, frontier judges lead only marginally and a convergence ceiling limits the practical difference.

4. Public release of the code (built on the SyGra graph-oriented synthetic-data-generation framework) and dataset via GitHub and Hugging Face, including all seven judges' with-GT/without-GT verdicts.

Main Findings

• Monotone difficulty degradation. All 30 (generator, judge) pairs show strictly monotone alignment degradation from easy to hard under both conditions. The degradation is roughly 1.5× larger without ground truth than with it. On hard without-GT records, all six judges converge to a narrow 77–82% band across four of five generators, including frontier GPT-5.4.

• The 77–82% ceiling is task-level, not judge-level. The C2 recalibrated-prompt ablation (a 0.5-default instead of the standard 1.0-default without-GT prompt) moves the three strongest generators by ≤ +1.0 pp (Llama-3.3-70B +0.4, Qwen3-32B +1.0, GPT-5.4 +0.5), confirming structural task difficulty as the primary ceiling driver. For weaker generators the prompt matters much more: Llama-3.1-8B +4.1 pp and SmolLM3-3B +5.6 pp.

• Ground-truth exposure is not uniformly beneficial. GT lift is positive for QwQ-32B and GPT-OSS-120B but negative for GPT-5.4 (−1.5 pp) and Gemini-2.5-Pro (−3.9 pp), with non-overlapping bootstrap confidence intervals. The effect concentrates in sequence accuracy: frontier judges anchor to the GT trace's ordering and penalise functionally equivalent but structurally deviant sequences, consistent with over-anchoring rather than independent judgement.

• A corrupted-GT control confirms pure anchoring. When the reference is replaced with a wrong GT from a different record, Gemini-2.5-Pro aligns identically under standard and corrupted GT, while QwQ-32B tracks its without-GT alignment within 0.2 pp.

• Chain-of-thought reasoning in the judge is negligible. Across 24 paired cells (QwQ-32B with thinking on vs. off, four open-weight generators, three difficulties), the mean GT gap is +0.11 pp and no cell exceeds 0.3 pp on either condition. QwQ-32B's advantage therefore reflects training distribution rather than inference-time compute.

• Judge temperature has negligible effect. Qwen3-32B alignment is insensitive across T ∈ {0.3, 0.7, 1.0}, with a maximum spread of 0.6 pp across all (difficulty, condition) cells. A second pairing (GPT-OSS-120B on Llama-3.1-8B-Instruct) shows a maximum spread of 0.25 pp across all three difficulty tiers.

• Prompt format is the largest configuration lever tested, but not universal. The structured per-metric JSON prompt beats a free-form variant by +4.8 to +6.5 pp with GT on the Qwen3-32B / Llama-3.3-70B pairing. A second pairing (QwQ-32B on SmolLM3-3B) partially replicates this (+3.9 pp easy, +2.4 pp medium) but reverses on hard (−0.8 pp), so the authors do not treat prompt format as a uniformly dominant, difficulty-independent lever.

• Inter-judge agreement is moderate. Mean agreement is 79.1% (κ = 0.419) with GT and 92.6% (κ = 0.559) without GT; the higher without-GT value reflects prompt-driven verdict compression rather than genuine consensus. Highest agreement is QwQ-32B × GPT-OSS-120B (89.4%, κ = 0.606); lowest is Claude Sonnet 4.5 × GPT-OSS-20B (70.4%, κ = 0.225). Without GT, pairwise κ decreases monotonically with judge tier separation (ρ = −0.825, p < 0.01); this relationship vanishes under GT (ρ = −0.171), where GT acts as a shared anchor. Disagreement concentrates at the 0.5 partial-credit boundary (94–97% of off-diagonal entries).

• Ensembling does not help. A soft jury of all six judges matches but does not exceed the top individual judge (82.5% with GT hard); the six-judge ensemble achieves 79.5% on hard without GT, within 0.4 pp of the best individual judge (GPT-5.4, 79.8%), confirming correlated structural failure.

• Best judge depends on configuration. QwQ-32B leads GT alignment in 10 of 15 (generator, difficulty) cells but is never the top without-GT judge. Gemini-2.5-Pro and GPT-5.4 lead without-GT on stronger and weaker generators, respectively.

• Metric and topology difficulty orderings. With GT, sequence accuracy is the weakest dimension with substantial inter-judge spread; without GT, all judges compress to a narrow band. DAG topology imposes a judge-independent difficulty ordering: fan-out easiest, loop-like and fan-in hardest.

• The judge-specialised baseline behaves differently. Prometheus-2 reaches only 42–52% exact agreement with κ = 0.01–0.07 against every judge, well below the 0.225–0.606 range of the six main judges, indicating it forms its own bloc rather than being a lower-alignment member of the general-purpose cluster.

• Scorer validation. A 120-record human annotation study on hard-difficulty records, stratified across all six topologies, validates the programmatic scorer at 92.7% metric-level agreement with independent human judgement.

• Difficulty-rewrite validation is uneven. The medium-to-hard step is unanimously validated on 93.9% of records, but the easy-to-medium step on only 58.1%; in roughly 41.9% of cases medium is better characterised as a paraphrase than a strict difficulty increase, so the primary evidence comes from the medium-to-hard step.

• Metric weighting is robust. Spearman ρ ≥ 0.83 between the equal-weight judge ranking and five alternative weighting schemes (seq ×2, param ×2, cov ×0.5, seq ×2 + param ×2).

Methodology in Plain English

The authors built their data synthetically rather than mining real enterprise traces, because a controlled reliability study needs a certifiably correct reference trace for every record plus systematic coverage of topologies and difficulty levels — properties that real logs rarely provide. Starting from an enterprise domain label, the pipeline generates use-case scenarios, synthesises a typed tool inventory, produces executable pseudocode linking tool dependencies, pairs a natural-language user utterance with input/output specifications, and captures an ordered execution trace of sequential and parallel calls.

Each record is then expanded into easy, medium, and hard variants by increasing query ambiguity while holding task structure and the ground-truth trace fixed. Ground-truth traces pass a two-level quality gate — structural JSON-schema validation, argument type checking, and trace consistency first, then per-pair checks for argument sufficiency, grounding alignment, and query naturalness — with no LLM involved in either gate, ruling out self-reinforcing bias. Failing records are regenerated.

For scoring, a deterministic programmatic judge emits four independent per-record scores in [0,1]: tool-selection accuracy penalises set-level mismatch; sequence accuracy measures position-by-position identity normalised by expected length; parameter-structure accuracy gives each predicted call 1.0 (exact key set), 0.5 (correct keys plus extras), or 0.0 (missing required keys or unknown tool), averaged over calls; query coverage is the fraction of expected tool identifiers covered, ignoring extras. Where a scalar is needed, the four are averaged with equal weights.

Every LLM judge then receives the original user query, the full set of available tool schemas, and the generator's predicted tool-call sequence, and returns a single JSON object scoring each of the four metrics on a {0, 0.5, 1} scale plus a one-sentence justification per metric. Two prompt variants exist: the with-GT prompt adds the ground-truth tool-call sequence as a reference block, the without-GT prompt omits it entirely, forcing the judge to infer correctness from the query and schemas alone. Judge decoding is held constant across all (generator, judge, difficulty, condition) configurations.

Alignment is computed as 1 − |programmatic value − judge verdict| per metric, averaged across the four metrics, then averaged across records and scaled to a percentage. Perfect reproduction of the programmatic vector yields 100%; an independent judge attains 50% in expectation. The design is a fully-crossed factorial over generator (five models, 3B to frontier), judge (six LLMs), and difficulty, observed under both GT conditions, yielding 90 factorial cells and 321,648 valid tuples.

Why This Matters

Impact on research. This is the first benchmark to systematically study LLM-as-a-judge reliability for agentic tool-calling over workflow DAGs, as distinct from the broader LLM-as-judge literature on open-ended text or preference evaluation. Prior tool-calling benchmarks reported judge reliability only as a single aggregate figure on a small sample, varying neither task complexity, difficulty, nor ground-truth availability. The paper shows that aggregate pass-rate agreement hides a structural ceiling that model capacity cannot overcome, and that a judge reliable at one failure dimension can be blind to another.

Real-world applications:

• Selecting and configuring a judge for an internal agentic evaluation pipeline, using the deployment decision table the authors provide.
• Auditing existing tool-calling benchmarks whose scores were produced by LLM judges, to understand how much of a reported pass rate reflects real capability versus judge leniency.
• Designing prompt templates for automated QA of enterprise workflow agents, where the structured per-metric rubric outperformed free-form prompts by 4.8–6.5 pp on one pairing.
• Diagnosing whether a deployed agent's failure is in tool selection, argument structure, ordering, or query coverage, using the per-metric decomposition.

Industry relevance. The work is authored by ServiceNow AI (Hyderabad, India) and targeted at enterprise agentic workflows across 15 domains including IT service management, contract lifecycle management, and energy grid operations. The finding that weak generators are substantially over-credited by the standard without-GT prompt (up to +5.6 pp when recalibrated) is directly actionable for teams evaluating low-quality or early-stage agent checkpoints. The finding that ground-truth exposure reduces alignment for GPT-5.4 and Gemini-2.5-Pro warns against the common assumption that showing the judge the answer always helps.

Future Directions

• Replace the non-reproducible frontier generator. Four of five generators are open-weight; GPT-5.4 is a non-reproducible Azure snapshot that also serves as judge and rewrite meta-judge. The paper reports a post-hoc correlational self-bias signal on sequence accuracy (+0.172 versus ≤ +0.008) that it cannot rule out or fully investigate.
• Address the domain-drift and interactive multi-turn gaps created by synthetic data. The authors explicitly note that generating records synthetically trades off domain drift and interactive multi-turn execution, and point to the Limitations section for the validation evidence for that choice.
• Determine whether the 77–82% hard without-GT ceiling can be broken at all. The C2 ablation shows the ceiling height is partly prompt-dependent for weaker generators, leaving open whether any prompt design or judge architecture could raise it for capable generators.
• Reconcile the divergence between programmatic and human notions of correctness. QwQ-32B best matches the programmatic reference with GT, while GPT-OSS-120B is the most human-aligned judge in the 120-record validation study — a disagreement between the two reference notions that the paper does not resolve.
• Calibrate the medium difficulty tier. Because the easy-to-medium rewrite is unanimously validated on only 58.1% of records, medium data functions as a robustness check rather than a calibrated mid-point, and the authors flag this as a limitation to be corrected in future work.

Target Audience

Researchers and engineers building or auditing agentic tool-calling systems, benchmark designers who need to choose and configure LLM judges, and evaluation teams at enterprise software vendors deploying multi-step workflow agents. It is also relevant to the broader LLM-as-judge community interested in how judge reliability transfers from text-centric preference tasks to structured correctness tasks. The paper is written at an advanced technical level and assumes comfort with evaluation statistics and function-calling schemas.

Authors’ abstract

LLM judges are widely used to evaluate agentic tool-calling systems, yet their reliability on structured, dependency-driven workflows remains largely unexamined. We present AgentJudgeBench, the first benchmark to systematically study LLM-as-a-judge reliability for agentic tool-calling over workflow DAGs, as distinct from the broader LLM-as-a-judge task of open-ended text or preference evaluation. The benchmark comprises 3,808 instances spanning six DAG topologies and three difficulty tiers, evaluated with five generators (3B-70B open-weight models and GPT-5.4) and six judges (20B to frontier scale) under paired with- and without-ground-truth conditions. Judge alignment degrades monotonically with task difficulty, 1.5x faster without ground truth, and on hard queries without ground truth all six judges converge to a narrow 77-82% band regardless of scale, revealing a structural ceiling driven primarily by task difficulty, though its height is partly prompt-dependent for weaker generators, that model capacity alone cannot overcome. Ground-truth exposure is not uniformly beneficial: it reduces alignment for GPT-5.4 (1.5 pp) and Gemini-2.5-Pro (3.9 pp), consistent with over-anchoring. Among mitigation strategies, chain-of-thought reasoning and judge temperature both have negligible effect, while structured evaluation rubrics improve alignment by up to 6.5 pp but do not generalize uniformly across judge-generator pairs. With ground truth, QwQ-32B best matches the programmatic reference, while a human validation study identifies GPT-OSS-120B as the most human-aligned judge; without it, frontier judges lead only marginally within the shared ceiling. These results expose fundamental limitations of current LLM judges and yield practical guidelines for reliable evaluation in agentic systems.