Jev-ToM: Evaluating Theory of Mind in Typed Decision Models

AI summary

Overview

Research area: Natural Language Processing, specifically evaluation of Theory of Mind (ToM) reasoning and "typed decision models" as an alternative to generative large language models for bounded selection tasks.

Technical level: Intermediate. Readers benefit from familiarity with LLM benchmarking, accuracy metrics, confidence calibration, and cost/latency comparisons, but no specialized mathematical background is required.

Scope (1 sentence): The paper introduces Jev-ToM, a framework that compares five typed decision models against seven generative baselines on 11,117 English questions drawn from nine existing Theory of Mind benchmarks organized into six capability groups, and then diagnoses reliability through confidence, reason, candidate-expansion, and resource experiments.

What This Paper Is About

Dialogue and collaborative assistants repeatedly need to judge what a partner knows, believes, or intends, and these judgments often have a finite set of possible answers even when the final response must be open-ended language. Generative LLMs currently handle this mental-state selection, but repeated calls and long prompts drive up waiting time and cost. The paper asks how competitive and reliable a typed decision model — TypeSafe Jev (hereafter Jev), which returns a selection plus candidate probabilities — is for this bounded Theory of Mind role.

Key Contributions

1. A cross-benchmark comparison. The authors compare Jev and the other typed models against generative LLMs across ToM tasks, measuring accuracy, reference cost, and observed response time on a harmonized collection of 11,117 questions from nine benchmarks.
2. A reliability diagnosis. They separate selection errors from output-validity failures using confidence-based answer selection and candidate interventions, rather than relying on aggregate accuracy alone.
3. A reason and mental-state feedback study. They investigate supplied reasons with different attachments (aligned, swapped, shared, crossed) and test whether related-question cues and predicted mental states each contribute to better action prediction.
4. A resource comparison. They price models against frozen OpenRouter reference tariffs and record observed response times to test whether the typed interface reduces cost and waiting for repeated ToM selection.

Main Findings

• Jev is competitive across all six capabilities. TypeSafe Jev reaches 73.2% source-balanced accuracy including native controls and ranks first or second in every capability group, leading beliefs, goals/desires/intentions, and emotions. Its largest lead over the strongest generative baseline is in emotions: 84.8% against GPT-3.5's 79.7%. Its lowest-scoring capabilities are beliefs and mental-state application.

• Typed models are not interchangeable. Open-Jev-2B leads the other typed models in four capabilities, Valen leads beliefs, and OpenJev-0.6B leads social pragmatics. Tiny-Jev exceeds OpenJev-0.6B on emotions (55.2% against 43.5%) but trails it on application. Mental-state application is the strongest capability for Open-Jev-2B, Valen, and OpenJev-0.6B, while Tiny-Jev favors emotions.

• Benchmark rankings reverse depending on aggregation. Removing 920 native controls drops Jev from 73.2% to 69.1% and GPT-3.5 from 71.6% to 70.9%, reversing their order; the paired difference is −1.79 points [−3.48, −0.07]. Within ToMi, Open-Jev-2B reaches 72.0% against Jev's 69.7% despite trailing substantially overall.

• Confidence is benchmark-dependent. At a descriptive selected-answer probability cutoff of 0.90, seven benchmarks fall below 10% retained error, but SimpleToM and OpenToM remain at 34.9% and 29.0%. ToMATO and SimpleToM both retain about 47% of answers, yet their retained errors are 1.8% and 34.9%. Hi-ToM's zero observed errors concern only 33 retained answers. Full-coverage error is similar for OpenToM (40.0%) and SimpleToM (40.9%), but retaining about half their answers yields 23.3% error at 48.8% coverage and 35.3% at 48.9% coverage respectively.

• Supplied reasons can repair or overturn selections. On 249 historically correct questions, aligned reasons raise fresh accuracy from 96.4% to 99.0%, while swapping attachments lowers it to 25.3%; neutral text preserves 96.8%. On historically incorrect questions, aligned and shared keyed reasons reach 71.0% and 69.4% against a fresh 10.0% baseline.

• Net effects hide opposing changes. In an eight-model study over 1,803 error-enriched "reused" materials and new reasons for 2,958 other questions, Jev's aligned gain falls from 51.4 points to 3.7 points on new materials, below Open-Jev-2B's 8.3-point gain. Crossed gains are lower for all eight models and negative for five. From aligned to crossed, repair stays near seven points while damage rises from 3.3 to 7.1, so 13.8% of weighted outcomes change strict correctness despite a net −0.4 points.

• Keyed reasons can accompany wrong answers. Across 4,761 crossed-pair materials, every model selects only the keyed reason more often than only the keyed answer; for Llama the shares are 23.8% and 13.0%. Jev and GPT-5.6-Luna achieve similar both-correct outcomes (78.2% and 77.6%) but differ sharply in invalid or failed shares (7.5% and less than 0.1%). Open-Jev-2B rarely fails validation yet has a 13.5% keyed-reason/wrong-answer share.

• Candidate expansion hurts selection and validity. On 108 questions (one per family), reattachment helps both tested models while expanding an eight-pair baseline to sixteen combinations reduces accuracy, including a 13.9-point GPT-3.5 loss despite valid outputs. Jev's strict accuracy falls from 81.0% to 76.4%, and the strict-to-saved gap widens from 3.2 to 6.0 points.

• Related-question cues drive most of the feedback gain. On 400 behavior-prediction questions (200 BigToM, 200 SimpleToM), question cues account for 10.75 of Jev's 14.5-point gain, with a further 3.75 points from adding the predicted answer. That extra gain concentrates on SimpleToM, where a fixed-No replay also reaches 88.5%; BigToM changes little. Predicted feedback reduces accuracy for Open-Jev-2B, Valen, and OpenJev-0.6B relative to their original controls.

• Cost and time favor Jev against nano and mini. At US$0.0295 per 1,000 questions on 11,117 questions, Jev costs 32.7% less than nano and 56.3% less than mini, with 11.7- and 6.7-point accuracy advantages. DeepSeek offers US$0.0138 at 65.4% accuracy, giving Jev 7.6 more points at 2.14 times the cost. Jev's timing run records 73.0% accuracy and a 282 ms mean response time, 83.9% below nano and 82.3% below mini; Open-Jev-2B follows at 464 ms with 56.6% accuracy.

Methodology in Plain English

The authors assembled a shared evaluation from nine existing English ToM benchmarks that have discrete candidate answers: BigToM, OpenToM, ToMBench, SimpleToM, FANToM, ToMATO, CogToM, Hi-ToM, and ToMi. Starting from 11,167 questions, they excluded 50 BigToM items whose stories omit a subsequent event required by the reference answer, leaving 11,117 questions across 2,268 question families with 2, 3, 4, or 15 candidate options depending on the source. Source-specific adapters convert each question into a common record containing the context, question, candidate texts, a reference answer, and identifiers that group related questions.

Questions are organized into six capability groups adapted from Abilities in Theory of Mind Space (ATOMS): knowledge access, beliefs, goals/desires/intentions, emotions, social pragmatics, and mental-state application. Coverage is unequal — beliefs draw on eight benchmarks while social pragmatics draws on two — and CognToM's native Percept items remain in benchmark totals but outside the capability groups.

Scoring is strict: an answer counts only if the output is valid and matches the reference. For typed models, probabilities must cover exactly the offered choices, lie in [0,1], sum to one within 10^-4, and assign the selected choice a probability within 10^-6 of the maximum. Accuracy is reported as exact match, averaged equally across the nine benchmarks (source-balanced overall EM) and across contributing benchmarks per capability, so larger datasets cannot dominate. Uncertainty is estimated by bootstrapping source-defined families — 5,000 draws for primary, material, and feedback comparisons, 2,000 for the six-condition study, and 10,000 for matched candidates — with 95% intervals from percentiles 2.5 and 97.5.

Beyond the primary comparison, the authors run targeted interventions: a six-condition study varying whether supplied reasons are keyed or foil, aligned or swapped, shared or neutral; an eight-model study over 1,803 error-enriched reused materials and new reasons for 2,958 other questions, with three coauthors assessing reason support on an initial 450-question sample; a 108-question candidate-expansion test; and a 400-question mental-state feedback test with five typed models.

Why This Matters

The paper's central claim is that high aggregate accuracy is not sufficient evidence for adopting a decision component, because confidence errors, sensitivity to supplied reasons, and output-validity failures vary by task. This has direct consequences for how ToM components are selected and integrated.

Impact on research: The work connects cross-benchmark ToM profiles with confidence-based selection, bidirectional changes under supplied reasons, and output validity, and shows that aggregate rankings can reverse under control removal or benchmark-specific comparisons. It also provides a harmonization recipe, family-based bootstrap estimates, and a planned versioned release of evaluation materials, candidate-order mappings, per-question predictions, de-identified annotation records, and scoring and resampling code.

Real-world applications:

• Dialogue assistants that track a partner's beliefs and outdated information to decide what to say next.
• Collaborative agents that update explicit belief states from observations for cooperative planning.
• Persuasion-style dialogue agents that maintain candidate interpretations of a partner's beliefs, desires, and intentions and condition responses on the highest-confidence candidate, such as donation-encouragement scenarios.
• Retrieval-style pipelines where a typed model performs record-level judgments while an LLM plans searches and composes responses.

Industry relevance: The resource comparison is framed around repeated mental-state selection, where the paper notes that repeated LLM calls and long prompts are sources of waiting time and cost. Jev's per-1,000-question reference price and 282 ms mean observed response time position the typed interface as a candidate for latency-sensitive or high-volume deployment, though the authors state that the overall system benefit depends on candidate generation and response composition, which the experiment does not measure.

Future Directions

1. Task-specific confidence calibration. The descriptive 0.90 cutoff is not a validated acceptance policy, and retained error varies widely across benchmarks at similar coverage.
2. Candidate design and output validation. Expansion to sixteen pairs impaired both selection and output validity even when the correct pair remained available, making both candidate construction and probability validation integration priorities.
3. Integration testing inside complete assistants. The evaluation treats Jev as a bounded decision component; candidate generation, partner-state updates across turns, and interactive task success remain untested, as do non-English, culturally different, and multimodal settings.
4. Building on related-question cues rather than predicted states. Question cues account for most of Jev's gain, and predicted feedback reduced accuracy for three of the typed models, so whether instance-specific mental-state predictions add value remains an open question.

Target Audience

Researchers and practitioners working on Theory of Mind evaluation, LLM benchmarking, and dialogue or collaborative agent systems; engineers choosing between typed decision models and generative LLMs for bounded selection tasks; and evaluators interested in confidence calibration, prompt-sensitivity testing, candidate design, and cost-latency tradeoffs. Readers primarily interested in open-ended generation or multimodal social reasoning will find the paper's English-only, text-based, supplied-candidate scope explicitly limiting.

Authors’ abstract

Dialogue and collaborative assistants repeatedly judge what a partner knows, believes, or intends. Typed decision models offer a potential alternative to generative large language models for bounded Theory of Mind (ToM) selection. We present Jev-ToM, comparing five typed decision models and seven generative baselines on 11,117 English text questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy including native controls and ranks first or second across all six capabilities, leading beliefs, goals/desires/intentions, and emotions. Other typed models show distinct strengths across tasks. Separate resource comparisons suggest that Jev could reduce the cost and waiting time of repeated ToM selection. Confidence-based selection remains benchmark-dependent, and supplied reasons can repair errors or overturn correct answers depending on their attachment. Candidate expansion reveals distinct selection and output-validity problems. Related-question cues account for most of Jev's downstream gains, with task-dependent effects from predicted states. The findings motivate testing Jev as a ToM selection component in assistants, with task-specific confidence calibration, candidate design, and output validation as integration priorities.