WOLF: Werewolf-based Observations for LLM Deception and Falsehoods

AI summary

Overview

Research area: Multi-agent systems, LLM safety and alignment, deception evaluation benchmarks. The paper sits at the intersection of social-deduction game simulation, agentic LLM evaluation, and trust/calibration measurement.

Technical level: Intermediate. The core concept (a Werewolf game with LLM players) is accessible, but the evaluation machinery — LangGraph state machines, exponential smoothing of suspicion, Brier scores, ROC AUC and AUPRC — assumes some familiarity with agent frameworks and classification metrics.

Scope: WOLF is a reproducible, statement-level benchmark that separately measures how often LLM agents produce deceptive statements and how well they detect deception in peers during repeated rounds of the social deduction game Werewolf.

What This Paper Is About

Most evaluations of LLM deception reduce it to a static classification task — given a text, is it a lie? That framing misses the interactive, adversarial and multi-turn nature of real deception, where agents must both conceal their own intentions and spot concealment in others. The authors build WOLF, a Werewolf-based multi-agent environment in which every public statement is annotated by the speaker (self-assessed honesty and deception type) and by every listening peer (deceptiveness plus a continuous suspicion score that evolves over rounds), so that deception production and deception detection can be measured separately and tracked over time.

Key Contributions

1. A LangGraph-based Werewolf implementation with role-conditioned agents (four Villagers, two Werewolves, one Seer, one Doctor in every game), strict night–day phase transitions, bid-based debate ordering, majority voting, and structured event logs for full reproducibility.

2. A deception measurement protocol at the statement level, pairing every public statement with a speaker self-assessment and concurrent peer analysis, and categorizing deception using a standardized taxonomy of omission, distortion, misdirection, and fabrication (alongside a "none" baseline).

3. A longitudinal suspicion model, in which peer suspicion scores are aggregated with exponential smoothing (α = 0.7) to produce trajectories of trust and doubt that carry across rounds rather than being recomputed from scratch.

4. A metrics suite covering deception production rates, detection accuracy against self-reports, calibration (Brier score, ROC AUC, AUPRC), a cross-perception matrix of observer-to-target suspicion, and threshold analyses.

Main Findings

• Werewolves win most games: Across 100 simulated games, Werewolves won 70% (70/100), Villagers won 10% (10/100), and 20% (20/100) ended without a declared winner.

• Game scale and structure: Games averaged 3.4 nights (± 1.3), 16.1 debate turns (± 6.1), 2.6 voting rounds (± 1.4), and 32.4 analyzed statements (± 12.3), reported as totaling 324 deception-analysis events per the results section; the abstract states the study covered 7,320 statements and 100 runs, and the conclusion says "over 7,200 statements." The per-game figure and the headline figure are not reconciled in the text.

• High self-reported deception base rate: Speakers self-labeled 69.4% of statements as strategically deceptive rather than strictly truthful. The abstract separately states that Werewolves produced deceptive statements in 31% of turns.

• Detection is moderate and asymmetric: Peer judgments reached 71–73% precision but only 48–61% recall, for 0.52 overall accuracy. In the role breakdown, precision ran 0.67 to 0.82 while recall ran 0.51 to 0.61, and overall recall was 0.48. Cautious but honest speech was over-flagged while subtle lies were missed.

• Deceptive roles are only partially distinguishable: Werewolves drew 53.3% average suspicion and were flagged by 60.1% of observers, versus Villagers at 47.9% suspicion and 49.8% flagged. But the Seer (55.1% suspicion, 61.8% flagged) and Doctor (60.8% suspicion, 60.5% flagged) were also heavily suspected — honest roles that withhold or hedge are penalized too.

• Withholding costs honest roles: Suspicion of the Seer was 57.2% before revealing information and 48.5% afterward. The Doctor was judged at 62.1% suspicious when protections failed versus 54.3% when they succeeded.

• Suspicion is diffuse across the group: In the final cross-perception matrix, all observer-to-target suspicion values clustered between roughly 46% and 59%. Werewolves rated each other at nearly 59%, and Villager-to-Werewolf suspicion (46.0%) was only slightly above Villager-to-Villager-style baseline suspicion, which the authors argue explains why Werewolves still win most games.

• Suspicion toward liars grows over time: Suspicion toward Werewolves rose from about 52% in early rounds to 62% by the end, while suspicion toward Villagers and the Doctor stayed flat around 44–46%. The Theil–Sen slope for Werewolf suspicion was +1.6 percentage points per round, near zero for Villagers and the Doctor. Villager and Werewolf confidence intervals overlapped in the first two rounds and diverged with non-overlap by round 3, with the gap widening to 16–18% by the final rounds. Final-round suspicion was 62.0% ± 1.2% (Werewolf) versus 45.0% ± 0.9% (Villager).

• Accusation willingness rises: The fraction of observers flagging deception increased from 35% to 47% across rounds.

• Suspicion is predictive but underconfident: Brier scores ranged from 0.26 to 0.29 across roles, ROC AUC from 0.55 to 0.58, and AUPRC from 0.75 to 0.77. The authors read this as weak discrimination but usable thresholding, with calibration adjustments as a possible fix that would not alter gameplay.

• Subtle deception survives longest: By speaker self-labeled type, omission (129 statements) drew 61.9% suspicion and 57.5% flagging, misdirection (51) drew 54.6% and 45.5%, fabrication (28) drew 51.7% and 40.1%, distortion (76) drew 50.3% and 35.5%, and non-deceptive statements (426) drew 57.1% suspicion and 51.4% flagging. Omissions and hedges resemble natural uncertainty, while fabrications conflict with accumulated context and are flagged more reliably.

• Compute footprint: Experiments ran on one NVIDIA A100 GPU (40GB), 64 CPU cores, and 256GB RAM. Each full game took roughly 12–13 minutes with LLM calls and under 30 seconds with the deterministic mock analyzer. One hundred logged games produced about 6GB of NDJSON data and required roughly 21 GPU-hours, with preliminary trials and ablations adding less than 20% additional compute cost.

Methodology in Plain English

The researchers implemented Werewolf as a programmable state graph. Every game has a fixed roster of eight agents — four Villagers, two Werewolves, one Seer, and one Doctor — so that conditions are identical across runs, unlike prior implementations that randomized roles. Night phases resolve simultaneously: Werewolves pick a victim, the Doctor protects someone, and the Seer inspects a role, with only survival or elimination made public. Day phases consist of debate followed by a majority vote to exile a suspect; Villagers win by eliminating all Werewolves, Werewolves win once they match or outnumber the Villagers.

Speaking order is decided by bidding, a mechanism borrowed from Werewolf Arena: each player bids an integer from 0 to 10 for urgency, higher bids tend to speak earlier, overbidding erodes later influence, and ties are broken with a slight bias toward players mentioned in the previous turn.

The measurement layer is the distinctive part. Each agent is bound to a role prompt (Werewolves conceal and coordinate, Villagers cooperate and flag, the Seer decides when to reveal findings, the Doctor protects) and keeps a private chain-of-thought scratchpad that is logged for researchers but never shared with other agents, so deception comes from role incentives rather than leaked privileged information. After each public statement, the speaker self-assesses whether the statement was deceptive, reports confidence, and picks a deception type if applicable; every other player independently judges the speaker's honesty and assigns a continuous suspicion score from 0 (full trust) to 1 (certain deception). Suspicion is then updated with exponential smoothing at α = 0.7, so new evidence carries substantial weight but past history prevents overreaction to a single remark.

To separate model behavior from framework behavior, the authors ran two modes: stochastic runs with real LLM agents, and subsystem ablations using a deterministic mock analyzer that applies fixed rules to assign labels and suspicion from text cues. All prompts, outputs, bids, votes, and state snapshots are archived as NDJSON streams so runs can be reconstructed or relabeled. Metrics include deception production rate, detection accuracy against self-reports, Brier score calibration, the cross-perception matrix of observer-to-target suspicion, and ROC/AUPRC threshold analysis.

Why This Matters

Impact on research. WOLF argues that deception evaluation should be dynamic and adversarial rather than static. By labeling which specific statements are deceptive — and by whom they were detected — it goes beyond the game-level outcome metrics (wins, eliminations) used by frameworks such as Werewolf Arena, AvalonBench, AmongAgents, and The Traitors. The reported asymmetry between strong deception production and weak detection supports the paper's framing that deception capabilities scale faster than detection capabilities, and the longitudinal suspicion traces offer a way to study trust formation across rounds rather than only at the end of a game.

Real-world applications:

• Content moderation and misinformation screening — the finding that omission and misdirection evade detection better than outright fabrication identifies which deceptive styles current detectors are most likely to miss.
• Fact-checking and verification pipelines — the calibration results (Brier 0.26–0.29, ROC AUC 0.55–0.58) indicate that raw suspicion signals need calibration before they can be used as decision thresholds.
• Negotiation and multi-agent coordination systems — the setup directly models agents with competing incentives who must decide how much to reveal about private information.
• Human–AI oversight and auditing — since honest roles like the Seer and Doctor are penalized for withholding, the paper warns that suspicion heuristics can punish legitimate discretion, a failure mode relevant to any system that scores trustworthiness.

Industry relevance. For teams deploying multiple LLM agents that interact with each other or with people — customer service escalation, autonomous trading or bidding, collaborative planning — the paper provides both a testbed and a caution. Detection precision around 72% with recall around 48% means roughly half of deceptive statements pass unflagged, and the diffusion of suspicion across roles means a raised alarm is not reliable evidence on its own.

Future Directions

• Larger and more adaptive games. The authors explicitly name larger, more adaptive games as a next step, addressing the limitation that fixing role distribution, player set, and debate length may under-sample longer-horizon or coalition-based strategies.

• Human or externally validated labels. Labels currently come from model self-assessments, so they reflect subjective judgments; the released logs are intended to let others re-label and externally check them.

• Calibration as a deployment step. Because AUPRC near 0.75 suggests suspicion can be thresholded but Brier and ROC AUC suggest underconfidence, researching calibration adjustments that improve detection without changing gameplay is a natural follow-up.

• Transfer to applied domains. The paper points toward moderation, fact-checking, and negotiation, raising the open question of whether the deception and detection dynamics observed in a stylized Werewolf setting generalize outside it.

Target Audience

Researchers working on multi-agent LLM systems, AI safety and alignment, and deception or theory-of-mind evaluation will get the most from this paper, particularly those who need a reproducible, logged benchmark rather than a one-off game simulation. Practitioners building agent-to-agent or agent-to-human systems with competing incentives — moderation, verification, negotiation, and oversight tooling — will find the detection accuracy and calibration numbers directly relevant to deployment decisions. Readers looking for a gentle starting point should note the paper assumes familiarity with agent frameworks and standard classification metrics; the game rules themselves are simple, but the measurement protocol and tables carry real technical density.

Authors’ abstract

Deception is a fundamental challenge for multi-agent reasoning: effective systems must strategically conceal information while detecting misleading behavior in others. Yet most evaluations reduce deception to static classification, ignoring the interactive, adversarial, and longitudinal nature of real deceptive dynamics. Large language models (LLMs) can deceive convincingly but remain weak at detecting deception in peers. We present WOLF, a multi-agent social deduction benchmark based on Werewolf that enables separable measurement of deception production and detection. WOLF embeds role-grounded agents (Villager, Werewolf, Seer, Doctor) in a programmable LangGraph state machine with strict night-day cycles, debate turns, and majority voting. Every statement is a distinct analysis unit, with self-assessed honesty from speakers and peer-rated deceptiveness from others. Deception is categorized via a standardized taxonomy (omission, distortion, fabrication, misdirection), while suspicion scores are longitudinally smoothed to capture both immediate judgments and evolving trust dynamics. Structured logs preserve prompts, outputs, and state transitions for full reproducibility. Across 7,320 statements and 100 runs, Werewolves produce deceptive statements in 31% of turns, while peer detection achieves 71-73% precision with ~52% overall accuracy. Precision is higher for identifying Werewolves, though false positives occur against Villagers. Suspicion toward Werewolves rises from ~52% to over 60% across rounds, while suspicion toward Villagers and the Doctor stabilizes near 44-46%. This divergence shows that extended interaction improves recall against liars without compounding errors against truthful roles. WOLF moves deception evaluation beyond static datasets, offering a dynamic, controlled testbed for measuring deceptive and detective capacity in adversarial multi-agent interaction.