Evaluating and Improving LLM Self-Modeling

AI summary

Overview

Research area: Natural language processing, specifically large language model evaluation, post-training with reinforcement learning, and behavioral interpretability (the study of what a model can say about its own behavior).

Technical level: Intermediate. The core question is easy to grasp, but the paper relies on benchmark construction, dummy-predictor baselines, LoRA-based reinforcement learning, and multi-turn agent simulation.

Scope in one sentence: The paper defines "self-modeling" as an LLM's ability to correctly answer verifiable questions about its own input-output behavior, builds a benchmark for it, and tests whether reinforcement learning on synthetic self-modeling data improves that ability without amounting to genuine introspection.

Note: the provided paper content is truncated partway through Appendix B.3 (in the middle of the "Sensitivity to prompt phrasing" subsection), so some later appendix details are not available here and are not reported below.

What This Paper Is About

Language models are increasingly asked questions about themselves: how confident they are, whether an edit to a prompt would change their answer, or what they would output in a hypothetical situation. This paper asks whether such self-reports are actually accurate, by comparing what a model says about its behavior against what it demonstrably does when sampled. It then asks whether this behavioral skill can be improved through training, and whether any improvement reflects real introspective access to the model's internal decision process.

Key Contributions

1. A self-modeling benchmark. The authors aggregate and unify self-modeling tasks from prior work into one suite covering binary, multiple-choice, scalar, and free-text answer formats. The main aggregate uses nine tasks (T = 9), with a tenth task dropped from the leaderboard because proprietary models are included there.
2. A skill metric that corrects for biased behavioral distributions. Because behavioral ground truth is measured from each evaluated model itself, the paper reports "self-modeling skill" — the gap between raw task score and a dummy predictor that knows only the model's overall answer distribution, not the specific example.
3. A scalable synthetic-data and training pipeline. A general task-construction recipe turns behavioral interventions into self-modeling training data, instantiated at scale on roughly 100 single-turn HuggingFace datasets and on multi-turn agentic trajectories from the BLOOM framework over 20 targeted alignment behaviors.
4. Evidence that RL improves self-modeling without establishing introspection. RL post-training improves aggregate skill across three open-weight model families with some transfer to held-out tasks, but cross-model transfer experiments do not show a consistent advantage for a model explaining itself.

Main Findings

• Self-modeling is measurable but far from solved. Across an 18-model leaderboard, the best aggregate skill is +0.147 (DeepSeek-V3.1), and all models remain far from the +1 ceiling.
• Frontier models fail on simple counterfactuals. On a Flip-Decision task built from a simple GSM8K example, GPT-5.5 and Gemini 3.1 Pro both answer the baseline prompt correctly, but after a short misleading factual injection their sampled answers shift to 622 in 9/10 trials for GPT-5.5 and 10/10 for Gemini 3.1 Pro. Asked whether the edit would change their answer, both self-report "No" even though the behavioral ground truth is "Yes" (623 shifts to 622).
• Raw scores can be misleading. If a model's answers flip under perturbation on only 5% of Flip-Decision examples, it can reach a raw score of 0.95 by always predicting "No flip." Skill scores range from -1 to 1, where 0 means no better than the best dummy predictor.
• Rankings are not just capability rankings. In Flip-Decision, smaller or mid-tier models show larger skill gaps than several stronger frontier models, and task difficulty varies substantially by format.
• Strong models still make interpretable errors. Opus 4.7 is used as the illustrative failure case: in one coding Flip-Decision example its majority label moves from False to True under perturbation, and the model predicts "No," scoring 0. In a Feature-Rate example, the true feature rate is 1/5 = 0.20 while the model reports 0.92, yielding an example-level score of 1 − (0.92 − 0.20)² ≈ 0.48.
• RL improves aggregate self-modeling. Multitask RL improves over the no-FT baseline for Llama-3.1-8B, Qwen3-8B, and GPT-OSS-20B on the held-out aggregate benchmark, and the same multitask recipe improves aggregate skill on Qwen3-4B, 8B, and 32B.
• Cross-task transfer is positive but uneven. Single-task RL gains appear on off-diagonal task combinations, but performance still depends strongly on the task type used in training.
• SFT does not transfer. With the same training data, supervised fine-tuning collapses outside its training-query support, and adding RL after SFT does not recover that loss.
• BLOOM-only training is a mixed result. Training only on the harder multi-turn score-prediction track improves GPT-OSS-20B and Llama-3.1-8B on the single-turn aggregate suite, but hurts Qwen3-8B.
• No consistent own-model advantage. When one model explains another model's behavior (Llama explaining Qwen and vice versa, with the explainer fine-tuned on Flip-Rate), the stronger Qwen explainer can outperform Llama even on Llama's own behavioral labels. An own-model advantage does appear for the Qwen3-8B and GPT-OSS-20B pair on Flip-Rate, but the authors conclude successful explanation depends at least as much on the explainer's general capability as on whether the behavior came from that same model.
• The aggregate metric is stable. Sweeps over temperature, resamples per prompt, and examples per seed stay within the reference 95% confidence interval. Dropping the self-modeling system prompt has negligible effect on Llama-3.1-8B-Instruct, and thinking mode moves skill in different directions across model families.
• Precomputed labels are sufficient for training. An ablation that periodically recomputes behavioral labels from the current LoRA checkpoint found no significant difference from using labels precomputed from the initial model; Edit-Proposal is the only task where edits are verified during training.
• Base models score much worse than instruction-tuned ones, but the authors caution this comparison is confounded by format compliance, since the benchmark requires parseable JSON and strict aggregation penalizes parse failures.

Methodology in Plain English

Defining the target. For each task, the researchers first measure what the model actually does. They sample the model's completions on an original prompt and on a perturbed version of the same prompt, then derive a behavioral ground truth — for example, whether the majority answer changed. This label is model-specific: it depends on how that particular model behaves.

Asking the model about itself. Separately, the model is given a self-modeling query asking it to report the property of interest, such as whether the perturbation will flip its answer. The self-report is scored against the behavioral ground truth using task-specific metrics: exact match for binary and multiple-choice, squared error for scalar probability answers, and text similarity or verification for free-text answers. All raw scores are normalized to [0, 1], and unparseable self-reports are counted as worst-case outcomes under strict scoring.

Correcting for easy distributions. Because a model whose answers rarely flip makes "No flip" almost always correct, the paper reports skill as the raw score minus a dummy predictor that uses only the model's overall answer distribution. Aggregate skill is a macro mean over seeds, tasks, and (domain, perturbation) groups.

Building the benchmark. Evaluation examples come from four held-out corpora spanning math, coding, safety, and fairness: GSM8K, HumanEval, WildGuardTest, and BBQ. For each seed, 25 examples per corpus are drawn, with domain-specific perturbations from a manually curated pool and prompt templates from a task-specific pool, then averaged over multiple seeds.

Generating training data. The HF track uses a generate–verify–revise loop: an auxiliary LLM decomposes prompts into editable components and proposes perturbations intended to flip or preserve the model's predefined label; the target model verifies by sampling; failed edits are sent back for revision. The four evaluation corpora are excluded. Data is deduplicated at the level of (input, perturbation) pairs and balanced — for Llama-3.1-8B, this yields a final 42.5/57.5 Yes/No split in Flip-Decision and 1,088–1,275 training examples per task. The BLOOM track uses a fork–verify–revise loop on multi-turn trajectories, where a generator picks a fork point and proposes a perturbation, the target is replayed, and an LLM judge scores the outcome on a 1–10 rubric; examples are kept only when the judged outcome matches the intended effect.

Training. Models are trained with RL using LoRA adapters. The task reward mirrors the raw evaluation metric on a [0, 1] scale, plus a 0.5 bonus for parseable, correctly formatted responses. In Flip-Decision, a correctly JSON-formatted answer gets the 0.5 bonus plus 1.0 task credit if its predicted flip label matches ground truth, or 0.0 otherwise. Multitask training combines examples from all available tasks, with each example routed to its own reward function on a shared scale.

Evaluating. Results are reported under three settings: in-distribution (same task format for training and evaluation), out-of-distribution (cross-task transfer and BLOOM-to-benchmark transfer), and cross-model transfer, where a different model defines the behavioral ground truth.

Why This Matters

Impact on research. The paper turns a vague question —

Authors’ abstract

We study self-modeling: an LLM's ability to answer questions about its own behavior. We focus on verifiable behavioral questions, such as whether a prompt edit would change the model's final answer. To measure this capability, we introduce a benchmark that tests diverse types of self-modeling questions. Current models show non-trivial but limited self-modeling skill, and make systematic mistakes on simple counterfactual questions about their own behavior. To improve self-modeling skill, we develop a scalable synthetic-data pipeline that produces self-modeling training data, and show that reinforcement-learning can improve aggregate self-modeling skill across three open-source model families with some transfer to held-out tasks. These gains, however, do not seem to constitute introspection consistently: improved self-modeling may not arise from privileged access to the model's internal decision process.