AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems

AI summary

Overview

Research area: Evaluation infrastructure for natural language processing — specifically benchmarking, LLM-as-judge scoring, agent/tool-use assessment, and retrieval-augmented generation (RAG) evaluation.

Technical level: Intermediate. The core ideas are accessible to anyone who runs model evaluations; the agent-episode layer and matching algorithms assume some familiarity with tool-calling traces and retrieval metrics.

Scope (one sentence): The paper presents AuditKIT, a source-available evaluation library that records benchmark accuracy, judge scores, runtime performance, tool-call behavior, and retrieval quality in a single result object, with explicit metric direction, configuration fingerprints, and "unscored" handling for missing evidence.

What This Paper Is About

Evaluating a language model or agent requires many different measurements — benchmark accuracy, judge-based quality, latency, tool-call correctness, retrieval quality — and today these live in separate tools with different interfaces, metric definitions, and result formats. That fragmentation makes results hard to compare and can hide real failures, such as an agent that writes a plausible answer while the underlying task fails, or a trace field that was never recorded being silently read as a zero. AuditKIT's goal is to put all of these measurements behind one evaluation contract and one result representation, so that comparisons are direction-correct, reproducible at the configuration level, and honest about what could not be measured.

Key Contributions

1. A single evaluation interface. A call to evaluate() takes a dataset, adapter, model, optional annotator, and metrics, and returns one RunResult; compare_models() consumes those results for pairwise and N-model comparison, selecting winners per metric.

2. Direction-aware comparison plus provenance. Every metric must declare MAXIMIZE or MINIMIZE, enforced at class definition (omitting it raises TypeError), and that declaration is used by every grading, diffing, and leaderboard operation. A SHA-256 fingerprint over model, data, adapter, metrics, and settings serves as both cache key and comparison key.

3. Agent and RAG evaluation beyond text matching. Turn-aware tool-call metrics using maximum bipartite matching, separate scoring for parallel versus sequential calls, multi-route references, an episode layer with verified outcome oracles and trial-reliability reporting, ranked retrieval metrics, judged grounding, and seven model-free RAG evidence checks.

4. Interoperability with zero required dependencies. Read-only importers for AgentTune, AlignTune, SafeTune, CuratorKIT, and CircuitKIT artifacts; the core declares an empty dependency list, and the agent layer, tool-call parser, retrieval metrics, and RAG checks are stdlib-only.

Main Findings

• No reviewed tool ties metric direction to winner selection. Surveying ten tools (lm-evaluation-harness v0.4.12, HELM v0.5.16, LangSmith, DeepEval v4.2.0, Braintrust, OpenAI Evals, Inspect AI v0.3.260, Arize Phoenix v17.2.0, Promptfoo v0.122.0, PyRIT v1.1.0), the authors found several that store direction for some metrics, but none that declare direction in the metric definition and apply it in every model comparison. HELM's Mean Win Rate is direction-correct but a fixed aggregate; LangSmith lets users mark a feedback key higher-is-better outside a metric-definition contract.

• Existing agent scoring flattens or greedily pairs calls. In the scoring code of RAGAS, DeepEval, TruLens, Phoenix, BFCL, agentevals, Inspect, and tau-bench, calls are paired greedily or compared as sorted sets, and calls are flattened across turns, with none separating serialized from batched calls under the same call-level score. AuditKIT uses maximum bipartite matching, so a greedy first fit never rejects a correct run.

• Three runs with identical call-level F1 can differ structurally. In the paper's worked example, a correct run, a run that serializes independent calls, and a run that batches a dependent call early all score tool_call_f1 = 1.0, but are separated by parallel recall and precision (serialized: recall 0.0, no precision value; batched: recall 1.0, precision 0.0).

• RAG average precision in common practice hides misses. The paper notes that RAG average precision often divides by the relevant items actually retrieved; AuditKIT divides by all relevant items so an unretrieved gold chunk lowers the score.

• 68 registered metrics across 15 families. These include 9 agent/tool-use metrics, 8 retrieval and judged-RAG metrics, 7 deterministic RAG checks, 9 code/deterministic metrics, 6 generation-quality metrics, and 6 LLM-as-judge metrics. The authors describe this as registry size, not a count of independently validated metrics.

• Eight built-in adapters cover generation, multiple choice, chat, few-shot, instruction, RAG, custom templates, and tool calling.

• Missing evidence is excluded, not zeroed. Metrics with nothing to measure return no score; unparseable judge verdicts are errors; unknown and not_tested results are excluded from means and counted separately; an oracle with no state to check returns unknown, not failure.

• Reliability reporting goes beyond a single pass rate. Episode evaluation reports verified success over all cases and over decided cases, plus pass@k (following Chen et al.) and all-k (the pass^k measure of tau-bench), with all-k flagged unless the caller confirms a reset between trials.

• The fingerprint is a configuration identifier, not a reproducibility certificate. Identical fingerprints are described as necessary but not sufficient for identical results, and trading off heterogeneous metrics (say, accuracy against latency) remains a separate unsolved problem.

• Benchmark scores are not rankings. The seven case studies are explicitly framed as illustrations of what the library distinguishes, not model rankings; sample sizes range from 6 to 100. Studies 1 to 3 ran on the earlier auditkit 0.3.0 line (44 metrics, 7 adapters, no agent layer, not published on PyPI); Studies 4 to 7 used release 1.1.2. Study 1 examines three community-pruned MLP-neuron variants of Llama-3.2-3B; Study 4 serves one model family through four runtimes (via the vLLM and SGLang integration code); Studies 5 to 7 cover a multi-hop question-answering assistant, Lexsi Labs stack artifacts, and tool-calling model choice. The measured results of these studies are not included in the text available for this summary.

Methodology in Plain English

The authors built a library around four small interfaces — adapter, model, annotator, and metric — and made custom components use the same registration path as built-ins (a subclass plus one decorator, or a decorated plain function). Every dataset is converted into Sample objects, adapters turn samples into model-ready requests, the runner calls the model and copies its trace into a scoring context, metrics emit Score objects stamped with their direction, and everything rolls up into a RunResult holding predictions, fingerprint, configuration, errors, unscored counts, and measured latency, tokens, size, and cost.

For agents, the paper models a reference as a list of turns, where each turn is the set of calls made in one response. Calls are matched to references with maximum bipartite matching under a configurable argument predicate (exact, subset, name-only, or custom), and parallel behavior is scored with three separate quantities: parallel recall (were independent calls actually batched), parallel precision (were batched calls actually independent), and a detection flag. A second, episode-level layer separates verified task outcomes — from final-state assertions, artifact assertions, answer assertions, custom predicates, or judge-based assertion oracles — from trajectory diagnostics, and records per-field coverage (observed, inferred, or unavailable) so a scorer only runs on fields that exist.

To evaluate how the library compares, the authors read the public documentation and scoring code of ten existing tools at pinned versions, built a capability table, and then ran seven small or controlled case studies to show where answer extraction, serving runtime, retrieval metric choice, and trace structure change the interpretation of a result.

Why This Matters

Impact on research. Comparisons across papers and leaderboards often mix metrics with opposite directions and loosely specified computation, and agent results are especially vulnerable because a final answer can look right while the trajectory fails. Making direction part of the metric definition, marking unmeasurable evidence as unscored, and separating verified outcomes from structural diagnostics gives reviewers and replication efforts something firmer to check.

Real-world applications.

• Model and vendor selection: comparing hosted and local models on accuracy, latency, error rate, and cost in one direction-correct leaderboard rather than stitching together spreadsheets from several tools.
• Agent deployment monitoring: checking whether a deployed agent's tool calls are schema-valid, redundant, looping, or violating a least-privilege policy, alongside a state-based check that the task actually completed.
• Regulated RAG systems (banking, insurance): verifying that retrieved evidence is complete, current as of the decision date, within the caller's access rights, correctly cited, and that the system abstains when a question is unanswerable.
• Serving-runtime evaluation: comparing the same model across inference stacks (for example vLLM and SGLang) under a shared measurement path.

Industry relevance. The zero-dependency core, lazy optional extras, read-only importers for an existing internal tool stack, and sidecar output that never writes into source data are all choices aimed at teams that need to adopt an evaluation layer without displacing what they already run. The paper positions AuditKIT as narrower than red-teaming tools: its probes are basic, and it targets measurement and comparison rather than attack generation.

Future Directions

• Metric trade-offs across heterogeneous objectives. The paper states that ranking several metrics correctly per metric does not solve trading off accuracy against latency against cost, and leaves that as an open problem.

• Stronger reproducibility guarantees. The authors are explicit that a matching fingerprint is necessary but not sufficient for identical results; deciding what else is required (and how to certify it) is unresolved.

• Closing the AgentTune metric gaps. Self-correction, tool efficacy, plan adherence, and path edit distance have no AuditKIT counterpart yet, and AgentTune healing-audit records are refused as too lossy to import.

• Extending beyond basic red teaming. Red-teaming coverage is marked partial in the capability table, and the authors note AuditKIT measures rather than generates adversarial probes, leaving that direction open.

Target Audience

Engineers and researchers who build or run evaluation pipelines for language models and agents — particularly teams comparing multiple models or serving stacks, teams evaluating multi-turn tool-calling agents where trajectory structure matters, and teams deploying RAG in regulated settings where evidence completeness, freshness, and access control must be checked. It is also useful for reviewers and benchmark maintainers who need to reason about why two evaluation numbers are or are not comparable.

Authors’ abstract

Evaluating language models and agents requires combining benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality. Differences in interfaces, metric semantics, and trace formats complicate these comparisons and can hide failures in agentic workflows. We present AuditKIT, a source-available evaluation library that records these measurements in one result representation, with direction-aware metric contracts, configuration fingerprints, and extensible metrics and backends. For agents, it provides turn-aware tool-call matching, separates parallel from sequential execution, and distinguishes verified task outcomes from trajectory-level diagnostics. Missing or unobservable evidence is marked unscored rather than counted as failure. For retrieval-augmented generation, it combines ranked retrieval metrics, judged grounding, and deterministic evidence checks. Seven case studies spanning model comparison, serving runtimes, retrieval, tool calling, and interoperability show how answer extraction, runtime behavior, and metric selection change evaluation outcomes.