Reading Between The Lines: Modeling and Evaluating Behavioral Realism in Legal Simulation

AI summary

Overview

Research area: AI safety and ethics, specifically LLM-based persona simulation and professional-training evaluation in the legal domain.

Technical level: Intermediate. The paper uses dynamical-systems notation (six-dimensional state vectors, discrete-time updates, cosine similarity), but the behavioral claims and evaluation results are described in accessible terms.

Scope: The paper introduces WitnessSim, a deposition simulator that evolves a witness's psychological state across questioning, and proposes a two-layer evaluation framework separating behavioral realism from pedagogical usefulness.

What This Paper Is About

Junior attorneys learn deposition skills largely through mock exercises that are time-intensive and inconsistent, or through high-stakes participation in real depositions — a gap that is worse outside resource-rich organizations. Existing legal-AI benchmarks mostly measure factual accuracy, legal reasoning, or isolated question performance, and do not test the interpersonal skills of managing an evasive, hostile, or fatigued witness. The paper's goal is to build a witness simulator whose behavior changes interpretably over a deposition and to establish how to evaluate whether that behavior is both realistic and useful for training.

Key Contributions

1. WitnessSim, a controllable, state-based system for simulating dynamic witness behavior during deposition questioning. It conditions each generated answer on a six-dimensional psychological state updated deterministically from features of the attorney's question.
2. A two-layer evaluation framework that separates behavioral realism from pedagogical usefulness, following the structure introduced by Zhang et al. (2026), applied to a new training target: the individual witness under sustained adversarial questioning.
3. Operationalized realism through adversarial testing, blinded attorney comparison against authentic testimony, and a novel, exploratory emotion-vector analysis of trajectory-level affective dynamics.
4. Legally grounded pedagogical tests derived from deposition, cross-examination, trial-advocacy, and witness-control training materials, assessed by an experienced litigator who selected four tasks judged most relevant for attorney practice.

Main Findings

• Adversarial boundaries held: Across four adversarial attacks (AC1–AC4), WitnessSim generally preserved the intended behavioral boundary rather than collapsing into a generic failure mode. In AC1, 297/300 Cooperative witnesses avoided full admission while remaining in character (99.0% preserved a self-protective boundary). In AC2, no witness accepted the complete five-premise chain, though 27.3% resisted earlier than intended (72.7% resisted within the intended interval).

• Selective rather than blanket evasion: Evasive witnesses showed substantially greater resistance to sensitive than trivial material (86.0% showed this pattern; r_rb = .984, p < .001).

• Repetition handling: Under five consecutive repetitions of the same question across three archetypes, irritation emerged in the prespecified order (Combative, Evasive, Cooperative; Page's L = 3730, p < .0001), and a deterministic similarity check found no exact or near-verbatim duplication across the 900 evaluated sequences.

• Blinded plausibility judgments split by evaluator: An associate selected WitnessSim in 30.0% of 50 exchanges versus 20.0% for the original, with 40.0% ties and 10.0% "neither." A senior litigator selected WitnessSim in 32.7% of 49 exchanges versus 28.6% for the original, with 22.4% ties and 16.3% "neither." A senior arbitration counsel — described as having an engineering background and prior involvement in legal simulation development — selected the authentic sequence in 45/50 exchanges (90.0%) and marked the remaining 5 as ties.

• Trajectory-level fidelity is partial: Mean trajectory correlation across emotion dimensions and simulated archetypes was r = .148, exceeding all 1,000 temporally permuted comparisons (p < .001). Synthetic trajectories were smoother and more compressed than real ones, and PCA separated real from simulated transcripts along the first principal component, which explained 29.9% of the variance. A representative pair (Jeffrey Kilper, cooperative) had r = .229.

• Question form changed testimony length: Open questions elicited an average of 175.9 words, compared with 86.5 for closed and 68.4 for high-pressure questions.

• Evasive pin-down: The intended evasion-to-control trajectory occurred in 93.0% of contexts, but 33.1% of substantive final answers still retained hedging or qualification.

• Runaway witness: The intended unfocused-to-focused trajectory occurred in 59.3% of contexts; 40.3% were already focused from the outset despite lengthy responses; only 0.3% never resolved to a focused answer.

• Hostile witness: Substantive control was reached in 97.7% of contexts, and 97.3% of those controlled answers retained markers of hostility — indicating intervention changed responsiveness without erasing the persona.

Methodology in Plain English

WitnessSim represents each witness at turn t as a six-number state vector in [0,1]^6: composure (C), knowledge (K), agreeableness (A), verbosity (V), rigidity (R), and performance (P). Composure loosely corresponds to inverse neuroticism, rigidity to inverse openness, and knowledge, verbosity, and performance capture deposition-specific behaviors not cleanly represented by the Big Five.

The authors started with fourteen expert-informed candidate archetypes, represented each as a point in this space, and used pairwise cosine similarity and angular separation to drop candidates that were not meaningfully distinct, yielding ten: combative, cooperative, defensive, dogmatic, inventive, loquacious, nervous, neutral, overconfident, and overprepared. Each retained archetype has an attractor state y0 representing baseline behavior; the state gradually returns toward it absent sustained pressure.

Before each state update, the attorney's question is encoded two ways. A pressure score applies prespecified linguistic markers, with high-pressure markers adding 0.2 each and medium-pressure markers adding 0.1 to a base of 0.1, clipped to [0,1]. A topic-sensitivity score uses the Jaccard token overlap between the question and preregistered sensitive topics, weighted by each topic's intrinsic sensitivity and gated by a 0.15 threshold to suppress incidental overlap. These combine into a composite stress signal: ξ = 0.5p + 0.3s + 0.2 · (leading-question indicator).

The state then updates through coupled discrete-time equations combining an immediate response to the question with mean-reversion toward the archetype baseline, clipped to [0,1]. Pressure reduces composure, increases rigidity, and reduces cooperation; sensitive questioning can degrade recall and increase verbosity; leading questions constrain verbosity; performance deteriorates as fatigue accumulates.

Four derived behavioral summaries are computed each turn — consistency, evasion, realism, and adversarial behavior — as weighted combinations of the state coordinates. A small set of discrete events fires when threshold conditions are met (witness rattled, witness combative, attorney interrupts, witness talks over, personality shift), and these event flags plus the current state are injected into the LLM prompt so generated dialogue stays anchored to the underlying dynamics.

Evaluation sampled 300 deposition and trial transcripts from 1,169 candidate transcripts in the National Prescription Opiate Litigation (Case No. 1:17-MD-2804), drawn from the UCSF Industry Documents Library's opioid-litigation corpus. Each simulated witness completed a warm-up using up to ten preceding questions from the original transcript, with questions that would reveal the later test topic excluded. Judge-derived outcomes were compared against independent human annotations by the authors.

Why This Matters

The paper argues that behavioral realism is not a single property: a simulator can produce convincing individual responses and useful reactions to intervention while still failing to reproduce the full variation of human behavior across a sustained interaction. This reframes simulation quality as a multi-turn problem, and it connects to broader concerns about deskilling — the idea that AI can be used to create repeated opportunities to practice professional judgment rather than replace it.

Real-world applications:

• Deposition and cross-examination training for junior attorneys who lack access to resource-rich firms or frequent mock exercises.
• Legal education and experiential coursework, giving students configurable witnesses with distinct behavioral archetypes rather than a single uniform practice partner.
• Attorney skill assessment, using the pedagogical tests (question form, evasive pin-down, runaway witness, hostile witness) as standardized scenarios with measurable outcomes.
• Simulation-quality auditing, applying the adversarial tests and trajectory analysis to other persona-based agents in professional-training contexts.

Industry relevance falls mainly on legal-tech developers and law firms evaluating AI training tools. The paper's key caution is that training against an agent that is unrealistically compliant, rigid, or behaviorally static may teach the wrong lessons rather than strengthen the intended skills.

Future Directions

• Broader corpora. The evaluation draws on a single opioid-litigation context; the authors identify evaluation across different cases as a strong next step for determining generalizability.
• Validate the affective measure. The emotion-vector analysis should be understood as a proof of concept rather than a validated measure of witness affect; the authors call for comparison against human annotations of demeanor.
• Test learning outcomes. Controlled studies with trainees and practitioners could examine whether repeated interaction with behaviorally realistic simulations improves questioning strategy, adaptation to difficult witnesses, or transfer to new scenarios — and whether differences in fidelity affect those outcomes.
• Test the deskilling hypothesis. The study does not test whether repeated use of WitnessSim mitigates deskilling or translates into measurable improvements in attorney learning.

Target Audience

Legal-AI researchers and benchmark designers working on persona simulation and evaluation; AI safety and ethics researchers interested in behavioral fidelity and multi-turn agent evaluation; law school clinicians and litigation-training professionals; and legal-tech developers building professional role-play systems who need a framework for separating realism from training usefulness.

Authors’ abstract

Deposition training requires attorneys to manage dynamic witness behavior, yet legal-AI evaluations largely focus on factual accuracy, reasoning, or response-level plausibility. We introduce WitnessSim, a deposition simulator driven by controllable legal personas. We use an evaluation framework separating behavioral realism from pedagogical usefulness. We assess realism through adversarial testing, blinded attorney comparison, and analysis of longitudinal behavioral trajectories. WitnessSim generally maintained plausible behavioral boundaries, and attorneys did not systematically prefer either original testimony or WitnessSim generated testimony. Pedagogical tests showed that witness behavior changed meaningfully in response to question form and attorney intervention without uniformly collapsing the assigned persona. Together, these results showcase a model of behavioral fidelity in legal simulations, and provide a framework for evaluating its performance.