JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks

AI summary

Overview

Research area: Evaluation methodology for agentic AI systems working on open-ended professional tasks; LLM-as-a-judge; benchmark design for deep-research agents.

Technical level: Intermediate. The framework is described with formal notation and scoring equations, but the core idea is conceptual and readable without deep mathematical background.

Scope: The paper proposes JADE, a two-layer evaluation framework that combines stable expert-authored evaluation skills with response-specific, claim-level verification, and it introduces BizBench (150 strategic sourcing queries) to validate the approach, with transfer tests on HealthBench and DR.BENCH.

What This Paper Is About

Evaluating AI agents on open-ended professional work is caught between two bad options: static rubrics are reproducible but cannot accommodate the many valid ways a report might be written, while LLM-as-a-judge is flexible but unstable and biased, and neither is grounded in expert principles or explicit evidence checking. The authors call this the stability–adaptivity dilemma. JADE's goal is to resolve it by separating reusable expert evaluation principles (Layer 1) from case-specific, claim-level assessment of each individual report (Layer 2), with gating that zeroes out conclusions built on refuted factual claims.

Key Contributions

1. JADE, a two-layer evaluation framework. Layer 1 encodes expert knowledge as reusable evaluation skills that are activated deterministically, producing a query-specific checklist. Layer 2 generates a report-specific checklist of typed items and performs claim-level verification, with evidence-dependency gating.

2. BizBench. A benchmark of 150 labeled strategic sourcing queries drawn from naturally occurring B2B workflows, annotated with a three-level hierarchical taxonomy, for evaluating temporally dynamic professional tasks.

3. Discovery of systematic agent failure modes. Structured evaluation on BizBench empirically surfaces a large evidence–reasoning gap and pervasive citation laundering, both of which holistic LLM evaluators miss.

4. Cross-domain and cross-backbone validation. The framework is shown to transfer to DR.BENCH (10 professional domains) and HealthBench (medical), and to remain effective with different judge backbones.

Main Findings

• Average BizBench score is low. Across the leaderboard, the average final score is 42.6%. The top systems, Gemini Deep Research (57.1%) and Shopping Research (56.2%), still fall short of the 80%+ threshold the authors describe as expected for professional deliverables.

• Agentic deep research systems outperform API-based agents. Gemini Deep Research leads at 57.1%. Open-source models match proprietary alternatives (DeepSeek V3.2 reaches 55.8% with tool use and 46.1% without).

• Evidence–reasoning gap. Across all 18 configurations in the leaderboard, the average evidence score is 84.0% while reasoning averages only 50.5% — a gap of 33.5 percentage points. In a concrete case, some systems retrieved highly accurate data (evidence: 100%) but provided limited synthesis (reasoning: 5.3%).

• Source credibility deficiency and citation laundering. Evidence scores average 84.0% but source credibility is 48.2%, a 35.8% gap. Many agents rely on low-credibility intermediaries instead of original documents, weakening evidential grounding despite factual correctness.

• Strong expert alignment. Full JADE achieves Pearson r = 0.858 with human experts in a study on 180 reports (30 tasks × 6 models) annotated by 5 domain experts, a 28.6% improvement over vanilla LLM-as-a-judge (r = 0.667).

• Backbone robustness. JADE reaches r = 0.827 with a GPT-4.1 backbone and r = 0.841 with Gemini-3-flash, both beating the vanilla baseline.

• Ablation results. Adding a flat generated checklist raises r from 0.667 to 0.769. Skill activation (Layer 1) alone raises it to 0.844 (+9.8%); report-specific checklists (Layer 2) alone raise it to 0.786 (+2.2%); combining both layers adds a further +1.4% synergy.

• Reproducibility. Variance across runs is 1.12% for full JADE, versus 1.45% for vanilla judging.

• Score calibration. Vanilla LLM-as-judge tends to score substantially higher than human experts on average, while full JADE produces stricter scores closer to professional quality levels.

• DR.BENCH transfer. Over 214 expert-curated tasks across 10 domains, JADE reaches Pearson r = 0.736 and Spearman ρ = 0.789, with statistically significant correlations in 9 of 10 domains. Sports & Competitions is the only non-significant domain (p = 0.297). Environment & Sustainability is strongest (n = 12, r = 0.882, ρ = 0.941); Business & Finance (n = 35) shows r = 0.733, ρ = 0.848; Health & Medicine (n = 19) r = 0.758, ρ = 0.609.

• HealthBench transfer. On a 474-instance subset with checklists of at least five items, adding theme-summarized skills improves Spearman ρ from 0.154 to 0.228 (+48%). After removing 10% outlier items with highly specific clinical details, ρ rises to 0.379 (+358%); with 20% trimming, ρ reaches 0.530 (+183%).

• Operational cost. JADE costs about $0.126 per query–response pair across 19.6 LLM calls on average, versus $0.004–$0.013 for a vanilla pointwise judge. A DR.BENCH protocol baseline costs $0.108 across roughly 76 calls.

• Expert effort is amortized. For BizBench, JADE required less than 15 person-days for a 17-skill taxonomy reused across all 150 queries, compared with HealthBench's 48,562 expert criteria from 262 physicians over 11 months. On average 5.0 labels are activated per query out of 17 total skills.

• Evidence gating in action. In one case, an agent report claimed a price range of $999–$1,199 CAD corresponded to roughly $730–$875 USD; JADE verified the premise with real-time exchange rates, detected a material deviation, and gated the dependent pricing justification to zero credit.

Methodology in Plain English

The authors start from how human experts actually evaluate reports: they do not compare against a single ideal answer, they apply domain principles and then check specific claims as they read. JADE copies that structure.

Layer 1 handles stability. Each of the 17 evaluation skills encodes a professional principle and comes with an expert-written rubric template. When a query arrives, a deterministic, taxonomy-based mapping activates the relevant skills, and their templates are composed into a query-level rubric. Because activation depends only on the query, the same query always produces the same rubric — no randomness from the LLM.

Layer 2 handles adaptivity. Given the query, the response, and the composed rubric, a model generates a query-specific checklist of atomic Yes/No items with weights (positive weights for quality requirements, negative weights for critical flaws). A second checklist is generated from the actual report and splits into evidence items (verifiable factual claims such as entity existence, quantitative attributes, source attributions) and reasoning items (whether conclusions are supported, assumptions stated, logic valid).

Scoring runs on two tracks. A verification agent performs real-time web search and content analysis on each evidence item and returns a verification score and confidence. An LLM judge scores query-level and reasoning items on a three-point scale (1 = Yes, 0.5 = Partial, 0 = No). Dependency gating then connects the tracks: if any evidence item a reasoning item depends on has verification confidence below a threshold, that reasoning item's score is forced to zero. The final score multiplies a clipped reasoning score by the mean evidence reliability, so strong reasoning without reliable evidence — or reliable evidence with poor reasoning — is penalized either way.

BizBench was built by starting from roughly 10,000 naturally occurring B2B queries, reducing them to about 3,200 through automated deduplication, de-identification, and noise filtering, screening down to about 350 candidates on analytical depth and multi-step reasoning by domain experts, and finally validating 150 high-quality queries. Each query carries a three-level multi-label taxonomy: primary intent (4 categories), information need (7 categories), and operational constraint (6 categories). Average query length is 113.2 tokens (min 16, max 934), with 5.0 labels per query and 5 total languages.

Why This Matters

For research, JADE reframes agent evaluation as compositional rather than monolithic: stable expert principles and adaptive case-specific verification are handled by separate layers and linked by explicit fact-to-reasoning dependencies. Its structured scores expose diagnostic failure modes — unsupported reasoning, hallucinated citations, weak source use, and citation laundering — that a single holistic score hides. The public release of BizBench, rubrics, and evaluation code is intended to enable reproducible comparison on temporally dynamic, reference-free tasks.

Real-world applications:

• Strategic sourcing and procurement, where agents research suppliers, certifications, prices, and shipping constraints against live-changing conditions, and where vendor rankings may be influenced by evaluation outcomes.
• Market and competitor analysis, where a report must synthesize retrieved data rather than merely collecting it — the exact gap JADE measures.
• Medical information evaluation, via the HealthBench transfer, where expert-authored clinical checklists can be turned into reusable theme-level evaluation skills.
• Multi-domain professional research, as demonstrated on DR.BENCH's 10 domains including business and finance, environment and sustainability, health and medicine, and law.

For industry, the framework offers a way to audit agent outputs before they influence consequential decisions, and the cost figures (about $0.126 per evaluation) give practitioners a concrete sense of what structured judging costs relative to a single-call judge. The paper explicitly cautions that JADE scores are not quality certifications and should not replace domain experts for legal, medical, or safety-critical decisions.

Future Directions

• Reducing domain adaptation cost. The authors state that deploying JADE in a new domain requires experts to first define a compact reusable skill taxonomy, a nontrivial cost, and note that effectiveness may decrease in knowledge-intensive fields such as medicine and education. The weak HealthBench correlations on the untrimmed set point to open work on capturing highly specific professional details.

• Handling closed-form factual recall. JADE is designed for open-ended judgment and proves weaker where exhaustive answer keys are needed, as reflected by the sports domain result on DR.BENCH (the only non-significant domain). Extending or hybridizing the framework for such tasks is left open.

• Mitigating inherited judge biases. The paper acknowledges that JADE, as an LLM-based judge, inherits known length and preference effects, and reports density normalization, deterministic skill activation, and criterion-level scoring as mitigations. Further reduction of these biases remains a question.

• Addressing multi-turn and style sensitivity. BizBench's primarily single-turn and underspecified queries make evaluation sensitive to response style, since some agents favor detailed one-shot reports while others rely on multi-turn interaction. Extending the benchmark and evaluation to multi-turn workflows is a logical next step.

• Complexity and cost reduction. The multi-stage pipeline increases system complexity and per-query cost relative to a single-call judge, leaving room for more efficient decomposition.

Target Audience

Researchers and engineers building or evaluating autonomous research and agentic systems, particularly those working on LLM-as-a-judge methodology, benchmark construction, or deep-research agents. Also relevant to evaluation and procurement practitioners in professional domains — sourcing, market analysis, consulting, and healthcare — who need auditable, claim-level assessments of AI-generated reports. Readers with a background in machine learning evaluation or NLP benchmarking will get the most out of the methodological sections.

Authors’ abstract

Evaluating agentic AI on open-ended professional tasks faces a fundamental dilemma between rigor and flexibility. Static rubrics provide rigorous, reproducible assessment but fail to accommodate diverse valid response strategies, while LLM-as-a-judge approaches adapt to individual responses yet suffer from instability and bias. Human experts address this dilemma by combining domain-grounded principles with dynamic, claim-level assessment. Inspired by this process, we propose JADE, a two-layer evaluation framework. Layer 1 encodes expert knowledge as a predefined set of evaluation skills, providing stable evaluation criteria. Layer 2 performs report-specific, claim-level evaluation to flexibly assess diverse reasoning strategies, with evidence-dependency gating to invalidate conclusions built on refuted claims. Experiments on BizBench show that JADE improves evaluation stability and reveals critical agent failure modes missed by holistic LLM-based evaluators. We further demonstrate strong alignment with expert-authored rubrics and effective transfer to HealthBench and DR.BENCH, covering medical and 10-domain professional evaluation settings. Code and data are available at https://github.com/smiling-world/JADE.