PragWorld: A Benchmark Evaluating LLMs' Local World Model under Minimal Linguistic Alterations and Conversational Dynamics

AI summary

Overview

• Research area: Evaluation of large language models' implicit ("local world model") representations in multi-turn conversation, conversational pragmatics, robustness benchmarking, and mechanistic interpretability of transformer layers.
• Technical level: Intermediate. The problem framing is accessible, but parts of the analysis assume familiarity with transformer architecture, residual streams, MLP sublayers, causal patching, ablation, and fine-tuning losses.
• Scope (one sentence): The paper builds PragWorld, a 2,614-instance yes/no conversational QA benchmark created by applying seven minimal linguistic alterations to conversations from the GRICE and CICERO datasets, uses it to show that nine open and one closed source LMs are not robustly accurate, and then uses layer-level interpretability to identify "useful" and "harmful" layers and suppress the harmful ones.

What This Paper Is About

Conversations contain entity mentions, references, implicatures, and changing states of objects and people, and answering questions about them requires tracking a "local world model" that updates as the dialogue proceeds. The paper asks whether language models actually build and maintain such a model, and whether that model is malleable — that is, whether it updates correctly when a conversation is changed in small but semantically meaningful ways. To answer this, the authors construct a benchmark where each conversation is paired with minimally altered versions, so a robust model must answer both the original and the altered versions correctly.

Key Contributions

1. A new benchmark (PragWorld) built from seven minimal linguistic alterations. The authors apply negation, variable swap, quantity change, variable substitution, quantifier change, logical connective change, and injecting inconsistent data to seed conversations drawn from GRICE (Zheng et al. 2021) and CICERO (Ghosal et al. 2022), producing manual and synthetic splits totaling 2,614 distinct instances.

2. A robustness evaluation of ten LMs. Nine open-source and one closed-source model are evaluated under a "robust accuracy" criterion that requires correctness on both the original conversation and all of its altered variants, revealing systematic brittleness and strong Yes/No answer biases.

3. A dual-perspective interpretability framework. Direct Effect Patching and MLP zero-out Ablation are used to classify transformer layers as useful (removing them lowers accuracy) or harmful (removing them raises accuracy), and to identify which alteration categories are most affected by harmful layers.

4. Two layer-regularization fine-tuning strategies. Useful Layer Amplification (ULA) and Harmful Layer Suppression (HLS) are proposed to amplify useful layers or penalize harmful ones, with the goal of improving robustness to linguistic alterations.

Main Findings

• Models are not robustly accurate. On the manual split, robust accuracy was low across models: GPT-3.5-Turbo 42.86, DeepSeek-Inst 46.94, Phi-3-mini-4k-ins. 47.96, Phi-3.5-mini-ins. 48.98, Llama-3.2-1B-Ins. 14.29, Llama-3.2-3B-Ins. 20.41, Llama-3.1-8B-Ins. 48.98, Qwen2.5-0.5B-Ins. 19.39, Qwen2.5-1.5B-Ins. 22.45, and Qwen2.5-7B-Ins. 37.76. The paper states the Phi series performs best on robust accuracy, surpassing larger models such as GPT-3.5.

• Robust accuracy is higher on the synthetic split. Reported values range from 47.77 (Llama-3.2-1B-Ins.) to 67.21 (GPT-3.5-Turbo), with most models clustering between roughly 55 and 65.

• A pronounced Yes/No bias. Most models show a large gap between Yes Accuracy and No Accuracy. This is especially strong in smaller models: Llama-3.2-1B-Ins. reports 10.83 Yes accuracy versus 95.96 No accuracy on the manual split, and Llama-3.2-3B-Ins. reports 16.97 versus 98.65. GPT-3.5 and DeepSeek-Inst. more frequently answer "No".

• Fine-tuning on synthetic data improves robustness on the manual split. Every tuned model improved in robust accuracy, with the largest relative gains for the Llama-3 and Qwen-2.5 series: Llama-3.2-3B-Ins. +28.57, Qwen-2.5-1.5B-Ins. +25.51, Llama-3.2-1B-Ins. +18.36, Qwen-2.5-7B-Ins. +17.34, Llama-3.1-8B-Ins. +10.2, Phi-3.5-mini-ins. +3.06, Phi-3-mini-4k-ins. +2.04, Qwen-2.5-0.5B-Ins. +3.06. The Yes/No gap persists, so fine-tuning alone does not remove inherent bias.

• Entity tracking is present but imperfect, and improves with scale and fine-tuning. In an experiment on 364 instances built from 15 original conversations, fine-tuning consistently improved entity tracking across all models, especially smaller ones such as Llama-3.2-1B-Ins. and Qwen2.5-0.5B-Ins., while larger base models like Llama-3.1-8B-Ins. and Qwen2.5-7B-Ins. already performed well.

• Conversation length matters non-monotonically. Bucketing into Short (fewer than 11 utterances), Medium (11 to 15), and Long (more than 15), GPT-3.5 accuracy rises from Short to Medium and then falls for Long conversations, except for "No" accuracy in the synthetic split.

• Specific layers are harmful and specific alterations are hit hardest. For base Phi-3.5-mini-instruct, MLP zero-out at layers 2, 9, and 16 decreased accuracy (useful layers), while ablating layers 5, 6, 7, 11, 13, 17, and 31 improved accuracy (harmful layers). Logical Connective was the alteration most impacted by harmful layers, and Variable Swap the least.

• Fine-tuning helps suppress harmful layers. In the fine-tuned Phi model, Logical Connective is less affected in early layers compared to the base model, and Variable Swap also changes, indicating fine-tuning reduces harmful-layer effects.

• Direct Effect Patching shows sharp layer transitions. For a correctly answered example with Phi-3.5-mini-instruct, the fine-tuned model is robust to patching up to layer 16, with confidence changing dramatically between layers 16 and 17. For Phi and Qwen fine-tuned versions, confidence rises sharply between layers 14 and 15, while both Llama versions show the opposite trend — an early increase at layer 5, then a drastic decrease moving from layer 15 to 16.

• The regularization strategies target identified layers. HLS was applied to harmful layers 7, 11, 13, 17, and 31 with harmful weight 1.0e-3 on Phi-3.5-mini-instruct, and ULA to useful layers 0, 3, and 13 with useful weight 1.0e-3 on Qwen2.5-7B-Instruct.

Methodology in Plain English

The researchers started from two existing conversational QA datasets, GRICE and CICERO, and filtered them down to yes/no questions, because yes/no answers make it easy to score a model as right or wrong. They then hand-picked 44 seed conversations from GRICE and 33 from CICERO and rewrote them in seven small ways — for example, flipping a negation ("he didn't" to "he did"), swapping two objects between locations, changing a number, swapping a quantifier like "all" for "some", changing "and" to "or", or inserting a fact that conflicts with common sense. Authors annotated each altered item and a second author reviewed it, and only items both agreed on were kept, giving a 500-conversation manual split (300 from GRICE, 200 from CICERO).

For the larger synthetic split, they took 104 seed conversations (49 from GRICE, 55 from CICERO) and prompted GPT-4-turbo-2024-04-09 to generate new conversations in the same style, then applied the alterations with deterministic algorithms and generated questions automatically from templates covering three question types: quantity questions, universal quantifier questions, and existential quantifier questions. Human annotators then verified the answers, producing 2,114 synthetic conversations (1,074 "Yes" and 1,040 "No").

Evaluation used a stricter metric than plain accuracy: robust accuracy counts an example as correct only if the model gets both the original conversation and every altered version right. Alongside this they reported accuracy on alterations meant to flip the answer (Flip Accuracy), alterations meant to preserve it (Invariant Accuracy), original and altered accuracy, and per-label Yes and No accuracy.

To understand why models fail, the authors intervened inside the network. In Direct Effect Patching, they ran an altered conversation and a patched its internal activations into the original run layer by layer, measuring how the model's confidence in the correct altered answer changed. In MLP zero-out Ablation, they disabled the feed-forward submodule at a single layer and re-measured dataset-wide accuracy; a layer whose removal hurts accuracy is called useful, and one whose removal helps is called harmful. Guided by which layers looked harmful, they fine-tuned models with two added penalties: ULA attaches small two-layer classification heads to useful layers' MLP outputs and averages those losses with the next-token prediction loss, while HLS adds an L2 penalty on the MLP output of harmful layers.

Why This Matters

The paper reframes "does an LLM understand the conversation?" as a measurable robustness question: a model that answers the original conversation correctly but breaks under a minimal, meaning-preserving or meaning-changing edit does not have a stable internal model of what was said. This matters because it links benchmark scores to specific internal layers that can be regularized, offering a concrete route from diagnosis to mitigation rather than only reporting failure rates.

Real-world applications:

• Conversational assistants and customer-support agents, where a user's paraphrase, correction, or changed quantity must be tracked correctly rather than silently ignored.
• Dialogue-based information extraction, such as call-center or meeting transcripts where who moved what, where, and how many is operationally important.
• Safety and reliability auditing of deployed chat models, using minimal-alteration tests to detect answer biases (for example, an over-preference for "No") before deployment.
• Tutoring and educational dialogue systems, where a system must follow a student's evolving statements across turns.

Industry relevance centers on the fact that the benchmark is cheap to apply — minimal edits to existing conversations rather than entirely new datasets — and that the regularization methods are simple add-on loss terms to standard fine-tuning, making them practical for teams that already fine-tune open models. The released code is at https://github.com/SachinVashisth/PRAGWORLD.

Future Directions

• Broadening the alteration set and languages. The benchmark uses seven alterations on English dyadic conversations; whether the same fragility and the same harmful layers appear for other languages, other alteration types, or multi-party dialogue is not established here.
• Extending the interpretability findings beyond the models studied. The layer-level analysis is not reported for the closed-source GPT-3.5, and the paper does not report whether the useful/harmful layer patterns generalize across all ten evaluated models.
• Closing the Yes/No bias gap. Fine-tuning improved robust accuracy but did not remove the imbalance between Yes and No accuracy, which the paper flags as an unresolved issue.
• Scaling the regularization strategies. ULA and HLS were demonstrated with specific layer sets and weights on specific models; how they behave with other layer selections, weights, and larger models remains an open question. The paper also points to confounding effects of pragmatic phenomena investigated in Supplementary Section J.

Target Audience

This paper is most useful for researchers and engineers working on LLM evaluation, conversational NLP, and mechanistic interpretability, as well as practitioners who fine-tune open-source models and want diagnostic tools for robustness. Readers who need only a high-level view of where conversational models break will benefit from the benchmark construction and the robust-accuracy results, while those interested in internal representations will find the patching, ablation, and layer-regularization sections most relevant.

Authors’ abstract

Real-world conversations are rich with pragmatic elements, such as entity mentions, references, and implicatures. Understanding such nuances is a requirement for successful natural communication, and often requires building a local world model which encodes such elements and captures the dynamics of their evolving states. However, it is not well-understood whether language models (LMs) construct or maintain a robust implicit representation of conversations. In this work, we evaluate the ability of LMs to encode and update their internal world model in dyadic conversations and test their malleability under linguistic alterations. To facilitate this, we apply seven minimal linguistic alterations to conversations sourced from popular datasets and construct two benchmarks comprising yes-no questions. We evaluate a wide range of open and closed source LMs and observe that they struggle to maintain robust accuracy. Our analysis unveils that LMs struggle to memorize crucial details, such as tracking entities under linguistic alterations to conversations. We then propose a dual-perspective interpretability framework which identifies transformer layers that are useful or harmful and highlights linguistic alterations most influenced by harmful layers, typically due to encoding spurious signals or relying on shortcuts. Inspired by these insights, we propose two layer-regularization based fine-tuning strategies that suppress the effect of the harmful layers.