DICE: Discrete Interpretable Comparative Evaluation with Probabilistic Scoring for Retrieval-Augmented Generation

AI summary

Overview

Research area: Evaluation methodology for Retrieval-Augmented Generation (RAG) systems, positioned at the intersection of automated text-generation evaluation, LLM-as-a-judge methods, and responsible/trustworthy AI.

Technical level: Advanced. The paper assumes familiarity with RAG architectures, embedding and generation models, Elo rating systems, pairwise preference evaluation, and statistical agreement measures such as Cohen's kappa.

Scope: The paper introduces DICE, a two-stage evaluation framework that replaces scalar RAG scoring with evidence-grounded pairwise comparisons expressed over the decision vocabulary {A, B, Tie}, coupled with probabilistic confidence scoring and a Swiss-system tournament for scalable multi-system ranking.

Venue note: The paper content carries the workshop title "ResponsibleFM," is distributed under a CC BY-NC-ND 4.0 license, and is listed as arXiv:2512.22629v1 [cs.AI], dated 27 Dec 2025. Code is stated to be available at https://github.com/shiyan-liu/DICE. Authors are Shiyan Liu, Jian Ma, and Rui Qu of the School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.

What This Paper Is About

RAG systems are being deployed in high-stakes domains such as finance, law, healthcare, and scientific research, but the metrics used to judge them — surface-overlap scores, embedding metrics, and scalar LLM-based scores like RAGAS — give a single opaque number that hides why one system beat another and how certain the judge actually was. DICE addresses this by reframing RAG evaluation as a comparative task: a judge analyzes two system outputs against retrieved evidence, issues a discrete judgment from {A, B, Tie}, and reports a calibrated probability distribution over those three outcomes. The framework also tackles the cost of comparing many systems at once, replacing exhaustive round-robin tournaments with a Swiss-system schedule.

Key Contributions

1. A two-stage, evidence-coupled evaluation framework. DICE separates deep analytical reasoning (Stage I) from probabilistic, confidence-aware scoring (Stage II), producing transparent reasoning traces and probability-grounded {A, B, Tie} judgments that support systematic error diagnosis and accountable system improvement.

2. A standardized, reproducible evaluation protocol. The authors specify judge prompts, the decision vocabulary {A, B, Tie}, hard and soft scoring rules, fixed judge model versions and decoding parameters, controlled random seeds, and pairing/tournament procedures so that comparisons can be reproduced across systems and studies.

3. A curated Chinese financial QA benchmark with eight RAG systems. The dataset consists of 70 QA pairs derived from real-world Chinese financial news articles and commentaries, benchmarked against a 2×2×2 factorial design of eight systems. DICE reaches 85.7% accuracy and a Cohen's kappa of 0.742, outperforming the RAGAS baseline (45.7% accuracy, kappa 0.096).

4. A Swiss-system tournament for scalable ranking. The tournament reduces computational complexity from O(N²) to O(N log N), achieving a 42.9% reduction in the eight-system evaluation (16 comparisons versus 28 in exhaustive round-robin) while preserving identical rankings against the exhaustive baseline.

Main Findings

• Strong human agreement on the Chinese financial QA set: Across 70 QA pairs and three human experts, DICE achieved 85.7% accuracy with a Cohen's kappa of 0.742, versus RAGAS at 45.7% accuracy and kappa 0.096. The kappa value of 0.742 is described as substantial agreement beyond chance.

• Disagreements are conservative, borderline cases: Analysis of the confusion matrix showed DICE's remaining disagreements with experts were concentrated in borderline cases where DICE applied more stringent criteria — it appeared more conservative in tie classifications than human evaluators.

• Generation model capacity dominates under DICE: In the eight-system tournament, systems using Qwen2.5-7B (S1–S4) consistently outperformed those using Qwen2.5-0.5B (S5–S8). Within each generation tier, embedding model and chunk size gave incremental variation, with S1 (bge-small-zh, 256-token chunks, Qwen2.5-7B) ranking highest.

• DICE and RAGAS produce different rankings: RAGAS showed different sensitivities to embedding choices within the same generation tier, placing emphasis on specific configuration combinations rather than the consistent architectural progression DICE revealed.

• Swiss-system ranking is faithful but more stable: Validation against the exhaustive 28-comparison baseline confirmed identical rankings across all systems, and the Swiss-system run showed minimal variance with shorter error bars than the exhaustive approach — described as superior precision and stability alongside a 42.9% reduction in computational cost.

• Confidence scoring behaves as designed in the worked example: In the appendix TACO-trade example, the judge assigned P(A)=0.83, P(B)=0.00, P(Tie)=0.16, a high-confidence margin of 0.83 − 0.16 = 0.67, and hard scores A=1.0, B=0.0. All three human experts also chose A, a 3/3 consensus.

• The confidence threshold was tuned empirically: The 0.1 probability-margin threshold distinguishing high-confidence from low-confidence judgments was determined through grid search on a validation set to optimize agreement with human expert judgments.

Methodology in Plain English

Stage I — Analysis before verdict. Instead of asking a judge model for a score, DICE first has it examine each system's answer against the retrieved evidence, considering factual accuracy relative to ground truth, completeness in addressing the query, and how well the answer integrates the retrieved documents. This produces a written reasoning trace. The judge then picks a single token from {A, B, Tie}. Answers are implicitly sorted into a hierarchy: fully correct beats partially correct, which beats information insufficiency (an explicit "I don't know"), which beats completely incorrect.

Stage II — Turning a verdict into a probability. The model's logits for the tokens A, B, and Tie are passed through a softmax to get a probability distribution. The margin between the top and second-highest probabilities, ΔP = P_max − P_second, decides which scoring rule applies. If ΔP ≥ 0.1, the judgment is treated as high-confidence and scored decisively: (1, 0) for A, (0, 1) for B, or (0.5, 0.5) for Tie. If ΔP < 0.1, a soft scheme proportionally redistributes the tie probability between A and B (S_soft = (p_A + p_Tie · p_A/(p_A+p_B), p_B + p_Tie · p_B/(p_A+p_B))).

Ranking many systems cheaply. Rather than running every system against every other system, DICE runs a Swiss-system tournament. Systems start at an Elo rating of 1500 and are paired each round by current rating while avoiding repeat matches. Ratings update via R' = R + K · (S_total/N_questions − E), where E follows the standard Elo expectation 1/(1+10^((R_B−R_A)/400)). Weighted updates amplify the K-factor when a lower-rated system beats a higher-rated one, and cumulative soft win scores drive the updates.

Single-system mode. For evaluating one new system, DICE compares it against three representative baseline systems selected from earlier tournaments as High, Medium, and Low quality tiers, using the same pairwise protocol. This converts an absolute measurement into a relative position on a known quality spectrum.

Experimental setup. The judge model was DeepSeek-R1, chosen for reasoning ability and multilingual performance on Chinese financial content. Human validation used systems S1 and S8, judged independently by three financial professionals with advanced degrees and at least five years of experience; disagreements were resolved by majority vote and tie cases by discussion. The RAGAS baseline combined Faithfulness, Answer Relevancy, and Context Relevance as an unweighted arithmetic mean, with pairwise {A, B, Tie} labels derived using a threshold of Δ = 0.15.

Why This Matters

Impact on research. DICE argues that single scalar metrics obscure meaningful differences between RAG systems and offer little explanatory power for responsible deployment. By grounding judgments in retrieved evidence, quantifying uncertainty over discrete outcomes, and publishing the exact prompts, decision vocabulary, and scoring rules, the framework pushes RAG evaluation toward transparent, reproducible, and statistically principled comparison — addressing the reliability concerns that affect LLM-as-a-judge approaches such as position bias and prompt sensitivity.

Real-world applications:

• Financial question answering: evaluating and selecting RAG systems that must answer market analysis, corporate performance, and regulatory questions accurately, where factual errors carry practical consequences.
• High-stakes domain deployment: the paper names law, healthcare, and scientific research as domains where trustworthy, evidence-grounded evaluation is necessary before deployment.
• Model selection for engineering teams: DICE's finding that generation model capacity dominated over embedding and chunking choices gives practitioners a concrete prioritization signal for financial QA builds.
• Cost-constrained benchmarking: the Swiss-system tournament makes comparing large cohorts of RAG systems feasible within a fixed evaluation budget, cutting comparisons by 42.9% in the eight-system study.

Industry relevance. The framework is designed for practitioners who need actionable error diagnostics rather than opaque numbers. Its single-system baseline mode lets a team position a newly built RAG pipeline against established quality tiers, and its published protocol makes vendor-to-vendor or internal A/B comparisons reproducible. The acknowledgements note that the first author's prior internship at JD.com, working with real-world RAG systems, inspired aspects of this research direction, with the work itself conducted independently at Huazhong University of Science and Technology afterward.

Future Directions

• Broaden beyond a single domain. The current evaluation uses 70 QA pairs in Chinese finance only, which constrains out-of-distribution generalization; the authors plan to extend to broader domains and larger system cohorts.

• Calibrate thresholds per domain. The 0.1 confidence margin and related scoring choices may need domain-specific calibration — an open question is how much calibration effort transfers across languages and domains.

• Harden the judge model. The paper notes the judge model can introduce bias or be vulnerable to adversarial inputs, and proposes bias audits, privacy-preserving retrieval, and red-teaming as next steps.

• Improve reasoning traces on edge cases. Reasoning traces may be incomplete on borderline cases; the authors propose adaptive calibration and ensemble strategies to improve robustness while preserving explainability and efficiency.

Target Audience

This paper is most useful to RAG researchers and engineers who need to compare multiple retrieval-generation pipelines and want more than a single number; to evaluation and benchmark researchers working on LLM-as-a-judge reliability, uncertainty quantification, and reproducible protocols; to applied teams in finance and other high-stakes domains choosing among RAG configurations; and to responsible-AI practitioners interested in explainability, confidence calibration, and ethically documented benchmarks. Readers should be comfortable with Elo rating mechanics, softmax probability outputs, and agreement statistics like Cohen's kappa.

Authors’ abstract

As Retrieval-Augmented Generation (RAG) systems evolve toward more sophisticated architectures, ensuring their trustworthiness through explainable and robust evaluation becomes critical. Existing scalar metrics suffer from limited interpretability, inadequate uncertainty quantification, and computational inefficiency in multi-system comparisons, hindering responsible deployment of RAG technologies. We introduce DICE (Discrete Interpretable Comparative Evaluation), a two-stage, evidence-coupled framework that advances explainability and robustness in RAG evaluation. DICE combines deep analytical reasoning with probabilistic $\{A, B, Tie\}$ scoring to produce transparent, confidence-aware judgments that support accountable system improvement through interpretable reasoning traces, enabling systematic error diagnosis and actionable insights. To address efficiency challenges at scale, DICE employs a Swiss-system tournament that reduces computational complexity from $O(N^2)$ to $O(N \log N)$, achieving a 42.9% reduction in our eight-system evaluation while preserving ranking fidelity. Validation on a curated Chinese financial QA dataset demonstrates that DICE achieves 85.7% agreement with human experts, substantially outperforming existing LLM-based metrics such as RAGAS. Our results establish DICE as a responsible, explainable, and efficient paradigm for trustworthy RAG system assessment.