WORTH Benchmark: Evidence Discovery under Not-Fully-Cooperative Information -- A Two-Episode Pilot

AI summary

Overview

• Research area: Large language model evaluation / reasoning benchmarks, specifically evidence discovery and pragmatics-informed evaluation design in NLP.
• Technical level: Intermediate — readers should be comfortable with benchmark design, scoring schemes, and LLM reasoning evaluation terminology, but no mathematics or model internals are required.
• Scope: A two-episode pilot benchmark (WORTH) in which models read unannotated, professionally authored Chinese mystery narratives, select a limited quota of key clue sentences, and identify case participants, scored separately on each stage.

What This Paper Is About

Most reasoning benchmarks hand the model the information it needs, already organized as premises, and ask only whether it can reason from there. This paper asks an earlier question: when nobody points out the evidence, and the text itself is written to delay rather than reveal the truth, can a model find where its reasoning should start? The authors build a small benchmark out of original mystery stories whose clues are deliberately buried by narrative technique, and they score clue-finding and culprit-identification as two separate things.

Key Contributions

1. A conceptual reframing of reasoning evaluation. The paper introduces two concurrent conditions that existing benchmarks rarely combine: autonomous evidence discovery (the model must decide which sentences are worth treating as evidence) and not-fully-cooperative informational relationships (factually true text whose presentation — placement, salience, causal framing — impairs correct judgment). The authors connect this distinction to Grice's Cooperative Principle, epistemic vigilance theory, and the survey of Gricean maxims in NLP.

2. An executable evaluation framework built on original narratives. Rather than reusing public-domain novels, materials are written from scratch by a professional author, so every key clue sentence has a documented inferential function, position in the evidence chain, and deliberately reduced salience — designed before composition rather than annotated afterwards.

3. Itemized, decomposed scoring. Clue hits and participant attributions are scored separately, with a quota mechanism for clues (selecting non-key sentences costs no points but consumes quota) and explicit rewards and penalties for participant selection. This makes visible whether "what was found" and "what was concluded" agree.

4. A two-episode pilot with reported diagnostic phenomena. 70 formal runs across two episodes (67 content-scorable), documenting three classes of behavior that premise-provided evaluations do not surface.

Main Findings

• Clue hits and participant judgments are not synchronized. A model can identify more key clue sentences yet still hold an incorrect theory in participant judgment; conversely, a model can be conservative in participant selection, make fewer errors, and still miss key evidence.

• A shared error attractor across different models. Across the 37 content-scorable runs in Episode 02, 31 runs simultaneously selected the same set of incorrect participants — a combination that superficially enjoys ample evidentiary support but diverges from the ground truth.

• Salience asymmetry between suspicion and exoneration. Adjacent suspicion-oriented and exoneration-oriented sentences were selected 25 times and 0 times, respectively, indicating that models favor high-salience suspicion information and overlook low-salience counter-evidence.

• Scale of the pilot. Episode 01: 120 sentences, 12 key clue sentences, 5 runs each on 6 models. Episode 02: 134 sentences, 20 key clue sentences, 5 runs each on 8 models. Total: 70 formal runs, 67 content-scorable.

• Deliberate limits on claims. The authors explicitly state the study does not establish stable model rankings, does not infer general mechanisms, reports only the full-story condition, and does not report human-model comparisons because human player records have not been exported and cleaned.

Methodology in Plain English

Each episode is an original Chinese murder-mystery narrative presented as numbered sentences ([S001] to [Snnn]), with instructions stating that the numbers are for reference only and do not mark clues.

The model is asked to do three things: pick a limited number of sentences it believes are the key clues (at most 12 in Episode 01, at most 20 in Episode 02), name which of the candidate characters were involved in the crime, and write out its reasoning in natural language. The output must be a single structured object with seven mandatory fields — episode identifier, clue-sentence IDs, character IDs, sentence-by-sentence clue explanations, combined evidence analysis, final conclusion, and a confidence score — plus a self-check that IDs are non-duplicate and drawn from the right lists.

Only the first and third outputs are scored. Clue scoring is overlap with the author's designated key-sentence set; participant scoring rewards correct characters and penalizes incorrect ones. The middle stage (the written analysis) is deliberately left unscored, because reliably grading free-form reasoning would require expensive manual coding or a second model as judge, and because forcing a score onto the process could distort it into optimization for grading heuristics. The natural-language explanations are kept for diagnosis — for instance, to see whether a model understood what a clue sentence was for.

A key design choice is asymmetric information about the payoff matrix: models are told the total number of key clue sentences, the selection ceiling, and that wrong clues cost nothing while wrong characters cost points — but not the differentiated clue weights (Episode 02), not the differentiated penalty magnitudes for individual incorrect characters, and not the full payoff matrix. The authors argue that revealing it would turn narrative reasoning into risk management. The full matrix is disclosed in the paper for reproducibility.

Scoring parameters differ between episodes, and the authors state the two are not comparable: Episode 01 gives every clue 3 points, rewards the three correct characters 7/5/5, applies a flat −5 for incorrect characters, with a maximum of 53 and a theoretical minimum of −15. Episode 02 gives clues 1–4 points each (clue component totaling 41), rewards each correct character 2 points, and varies incorrect-character deductions from −2 to −8 (heaviest for the most plausible distractors), with a maximum of 47.

The story design itself is the experimental manipulation. In one Episode 02 example, a character is injured in the left leg but supports his crutch with his left hand — the fact is on the page, but the plausible "fight → injury → crutch" chain absorbs the reader's attention, so the anomaly must be discovered rather than pointed out.

Why This Matters

Impact on research. The paper argues that current benchmarks conflate difficulty with adversariality: adding length or distractors raises difficulty but keeps the text functionally cooperative. WORTH separates these, and its decomposition shows that "found the evidence" and "reached the right conclusion" can come apart — something a single accuracy number would hide. The finding that 31 of 37 scorable runs in one episode converged on the same wrong character set is a concrete argument for component-wise scoring.

Real-world applications:

• AI-assisted investigation and compliance review — systems that must sift factually accurate but selectively framed documents (depositions, incident reports, audit trails) where no one has flagged the relevant lines.
• Medical and legal document triage — where salient alarming details can crowd out low-salience exonerating information.
• Information literacy and misinformation tooling — distinguishing "false information" from "true information arranged to mislead," which is the paper's core definitional move.
• Narrative and game design analytics — the two episodes run as interactive mystery games on Bilibili's Toy platform, so clue-salience design is also a measurable property of the product.

Industry relevance. The paper's central empirical pattern — models over-selecting high-salience suspicion cues and never selecting adjacent exonerating evidence — is directly relevant to anyone deploying LLMs on decisions where false accusation is costly. The quota mechanism and the concealment of the payoff matrix are also practical templates for benchmark designers worried about strategic gaming.

Future Directions

• Implement the two unimplemented verification layers. The paper describes but does not run an evidence-cueing condition (keep the story, explicitly mark the author's key clue sentences, and compare participant attribution before and after prompting) and an evidence-modification condition (alter key evidence while preserving narrative coherence and changing the answer, to test responsiveness to textual change).
• Complete the human baseline. Both episodes have accumulated thousands of human player score records on Bilibili's Toy platform; the paper states these have not been exported or cleaned, so human-model comparison remains open.
• Formalize the narrative design methodology. The authors note that designing salience gradients and feeding model results back into evidence-structure calibration currently has no standardized workflow, and that a single small team carried the narrative, evaluation-methodology, and diagnostic-analytics roles simultaneously.
• Extend beyond two episodes and one condition. With only two episodes and the full-story condition, the authors explicitly decline to infer general mechanisms or produce stable rankings; whether the observed desynchronization and shared error attractors generalize is unresolved.

Target Audience

Benchmark designers and evaluation researchers in NLP who are dissatisfied with premise-provided reasoning tasks; researchers working on pragmatics-informed model evaluation, evidence attribution, or faithfulness of explanations; and practitioners deploying LLMs in investigative, legal, or compliance settings where selecting the wrong evidence — or missing the quiet exculpatory detail — carries real cost. Readers looking for leaderboard rankings or human-model comparisons will not find them in this version.

Authors’ abstract

Existing reasoning evaluations typically provide task-relevant information to models as available premises, rarely examining concurrently how models discover evidence autonomously and how information presentation within narratives guides or obscures judgment. This paper proposes the WORTH Benchmark: models read original mystery narratives without clue annotations, autonomously select key clue sentences, identify case participants, and explain their reasoning process. The narrative materials are designed by professional authors such that evidence facilitating reasoning coexists with narrative arrangements that delay the revelation of truth. We report a pilot experiment under full-story conditions across two episodes, conducting a total of 70 formal runs on models, of which 67 runs are content-scorable. Scoring separately records hits on author-designated key clue sentences and participant selections; natural language explanations are used for case diagnosis and are not scored independently. The results show that clue hits and participant judgments are not synchronized; across 37 content-scorable runs in Episode 02, 31 runs simultaneously selected the same set of incorrect participants, while adjacent suspicion-oriented and exoneration-oriented sentences were selected 25 times and 0 times, respectively. These phenomena illustrate the diagnostic value of component-wise analysis and suggest that not-fully-cooperative informational relationships may influence evidence selection and judgment. This study only implements full-story conditions for two episodes, and does not establish stable model rankings or infer general mechanisms on this basis; human player records have not yet been cleaned, and this version does not report human-model comparisons.