A Framework for Context Enrichment and Selective Robustness in LLM Social Agents

AI summary

Overview

Research area: Multi-agent systems and LLM-based social simulation, specifically prompt/context design and behavioral validation of LLM agents.

Technical level: Intermediate. The paper is readable without deep mathematics, but it assumes familiarity with prompting, token probabilities, and basic experimental design (factorial designs, bootstrap intervals, Prisoner's Dilemma payoffs).

Scope (one sentence): The paper proposes a framework for systematically decomposing what "context enrichment" means in LLM social agents, and introduces "selective robustness" as a two-part validation criterion, demonstrated in a single pilot experiment on a Prisoner's-Dilemma-like hospital supply game with two GPT-5.6 models.

What This Paper Is About

Researchers increasingly use LLMs as agents in social simulations, supplying them with roles, relationships, and background information that is supposed to shape their choices. But LLM agents change their behavior substantially when a prompt is reworded in ways that preserve meaning, which undermines confidence in simulation results. The authors ask whether giving agents richer context makes them more robust, and argue that this question cannot be answered until "context richness" and "robustness" are defined in ways that allow controlled comparison — which is what they set out to provide.

Key Contributions

1. A controlled decomposition of context enrichment into five prompt conditions that substitute for one another rather than accumulate: a minimal-context baseline plus neutral padding, generic social background, descriptive decision grounding, and normative decision grounding. The conditions are ordered by the function of the added information, not by length or a vague notion of "richness."

2. "Selective robustness" as a validation criterion, combining (a) surface invariance — stability across alternative representations intended to preserve decision-relevant meaning — and (b) world responsiveness — a prespecified directional shift in behavior when decision-relevant social facts change. This adapts metamorphic testing and CheckList-style behavioral testing (invariance tests vs. directional expectation tests) to the LLM social-agent setting.

3. A grounded pilot study showing the framework in action: a 2 × 5 × 4 factorial design crossing two social worlds, five context conditions, and four surface forms, using the SiliSocS simulation system and two models (GPT-5.6 Sol and Terra).

4. A public codebase and dataset for the pilot experiment, released at the authors' GitHub repository.

Main Findings

• Motivating evidence is large: prior work by Ye et al. [2026] is cited as reporting shifts of up to 76 percentage points in cooperation rates in a Prisoner's Dilemma under context-format changes that preserve similar social meaning.

• Grounded conditions dominated ungrounded ones on world responsiveness for both models: Sol's responsiveness rose from 27.9 (generic background) to 85.9 (descriptive) to 92.0 (normative) percentage points; Terra's went from −2.1 (generic) to 57.2 (descriptive) to 77.1 (normative).

• Sol achieved the preferred combination; Terra did not: under descriptive grounding Sol's mean surface sensitivity fell from 27.9 to 3.7 percentage points while responsiveness rose to 85.9; normative grounding kept sensitivity nearly unchanged at 3.9 and raised responsiveness to 92.0 — placing Sol near the "upper left" selective-robustness region.

• Terra responded strongly to the world but stayed surface-sensitive: descriptive grounding cut Terra's surface sensitivity from 30.6 to 13.2 points and normative grounding gave 15.0 points, with responsiveness of 57.2 and 77.1 respectively.

• Ungrounded conditions failed the joint criterion: some were more surface-sensitive (e.g., Terra's generic background at 30.6 points with responsiveness of −2.1 [−10.8, 7.1]), others relatively stable but weakly world-responsive (Sol's neutral padding: sensitivity 14.2, responsiveness 6.7 [1.2, 12.2]).

• One extreme surface failure: in Terra's local normative condition, alternative prose produced 89.9% transfer versus at most 0.9% for the other three surfaces — echoing the Ye et al. [2026] findings, and matching prior work showing the same perturbation can affect models very differently (Sclar et al., 2024).

• Terra's residual instability is localized: its grounded regional cells stayed near ceiling (97.6% to 100.0%), while the local descriptive condition gave 84.1% under alternative prose versus 19.0% to 36.5% for other surfaces, and the local normative condition gave 89.9% versus 0.2% to 0.9%.

• Sol's grounded conditions were near-deterministic by world: regional transfer ranged from 77.6% to 99.2% across descriptive and normative conditions, while every local value was no more than 0.1%.

• Payoffs were held constant across worlds: the numerical payoffs were identical in the regional and local worlds, yet grounded conditions produced large differences in transfer probability — suggesting the models respond to social and institutional information beyond the payoff table.

Methodology in Plain English

The authors built a single game inside SiliSocS. One focal agent plays the emergency-supply manager of Hospital A and privately chooses whether to transfer its supply batch to Hospital B or retain it, knowing the Hospital B manager faces the same simultaneous choice. The payoff table is a standard Prisoner's Dilemma ordering (T=5 > R=3 > P=1 > S=0).

Two "worlds" were written that differ in a decision-relevant way: a regional world where a shared arrangement covers supply continuity across both hospitals, and a local world where each hospital has its own separate arrangement. The authors prespecified that transfer should be higher in the regional world.

They then crossed those two worlds with five context conditions. All five share the same fixed opening (role), decision task, actions, and payoff table. The four non-minimal conditions each insert exactly one block at the same location, and the blocks replace rather than stack: unrelated background detail (neutral padding), organization-level social facts disconnected from the decision (generic background), relational and procedural facts tied to the decision but stated descriptively (descriptive grounding), or the same facts phrased as a role requirement (normative grounding).

Finally they rendered the social-context block in four surface forms meant to preserve meaning — canonical prose, bullets, a key-value table, and alternative prose (paraphrases) — producing 2 × 5 × 4 = 40 cells per model. They queried gpt-5.6-sol and gpt-5.6-terra ten times per cell (400 calls per model, 800 total) with no conversational history, temperature 0.7, reasoning effort "none," a 64-token output limit, zero retries, and up to 20 candidates. Transfer probability was computed from returned token log-probabilities at the first output position as t/(t+r) over the "TRANSFER" and "RE" (retain) candidates.

Two measures summarize each context: mean surface sensitivity (average absolute difference from canonical prose across the other three surfaces, averaged over the two worlds; lower is more stable) and world responsiveness (regional-minus-local difference averaged across surfaces; positive follows the prespecified direction). Confidence intervals come from resampling the ten calls within each cell 10,000 times.

One measurement caveat the authors report: the provider's default top-p of 0.98 caused one action candidate to be omitted in 339 of the 800 calls; they bounded the transfer probability using the unreturned mass and used the interval midpoint, and a sensitivity analysis showed varying these estimates across their bounds changed aggregate measures by no more than 0.49 percentage points and preserved all context rankings within each model.

Why This Matters

Impact on research: The paper reframes prompt context as an experimental variable rather than a fixed backdrop. It gives social-simulation researchers a vocabulary for distinguishing kinds of added context and a two-part validation test that guards against a specific failure mode: an agent that looks "stable" only because it ignores the social world entirely. It also directly addresses calls in recent reviews and position papers for more standardized, reproducible evaluation of LLM social simulations.

Real-world applications:

• Agent-based policy and institutional modeling — simulating how agents behave under shared regional versus local governance arrangements, where the framework's regional/local contrast is directly analogous.
• Emergency and supply-chain logistics simulations — the pilot's hospital supply-transfer scenario is itself a plausible planning sandbox for resource allocation drills.
• Evaluation and QA of LLM products used in decision support — the invariance/directional-response pair is a reusable red-teaming template for any deployed agent whose output should track substance, not formatting.
• Behavioral benchmarking and model comparison — the surface/grounding heat-map style comparison gives procurement and model-selection teams a way to see which models are brittle to formatting and which are simply insensitive.

Industry relevance: Any organization shipping LLM agents that make or advise on decisions in multi-agent settings has a stake in knowing whether a formatting change (prose → table → bullets) can swing behavior. The paper's Terra result — 89.9% versus 0.9% transfer depending only on paraphrase — is a concrete argument for including surface-perturbation tests in pre-deployment validation.

Future Directions

• Replicate beyond one setting: the pilot covers one decision setting and two models from the same family; the authors state the observed patterns — such as generally greater selective robustness in grounded conditions — require further investigation before generalization.
• Resolve the salience confound: the grounded prompts also elaborate on the world distinction, so the experiment cannot fully separate decision grounding from increased salience of that information. The authors note a further experiment could compare decision grounding against other ways of making the world distinction salient, and acknowledge that any plausible grounded prompt may naturally draw more attention to the relevant world information.
• Independent semantic validation: no independent human raters assessed whether the four surface forms are truly semantically equivalent, or whether the non-normative prompts accidentally carried normative force. The authors propose human raters for both checks.
• Calibrate response magnitude against humans: selective robustness is an internal diagnostic, not a fidelity claim. A model with 100% transfer in the regional world and 0% in the local world moves in the expected direction but may exaggerate a response that humans show only moderately; human decisions in the same settings could establish what magnitude is plausible.
• Reusable tooling: existing or future simulation frameworks could adopt prompt-surface and world checks as reusable evaluation components.

Target Audience

Researchers building or auditing LLM-based social simulations; multi-agent systems practitioners who need to know how sensitive an agent's behavior is to prompt construction; evaluation and safety engineers designing behavioral test suites for deployed agents; and social scientists using LLM agents as stand-ins for human populations, who need internal diagnostics before treating simulated behavior as evidence. Readers wanting a replicable template rather than a definitive empirical claim will get the most from it — the authors explicitly frame the pilot as an illustration of the framework's utility, not a general result about context enrichment.

Authors’ abstract

Researchers use large language models as agents in social simulations, where agents are prompted with observations and background information intended to shape their actions. Yet LLM agents have been shown to substantially change their behavior under meaning-preserving prompt variations, creating a validation problem for social simulation. One plausible hypothesis is that this brittleness reflects a lack of grounding and can be reduced by enriching the context provided to the agents. Whether richer context improves robustness remains unclear, in part because context richness and robustness are rarely defined in ways that support controlled comparisons. We address the first need with controlled context enrichment, a set of prompt substitution rules that systematically enriches distinct types of context: neutral padding, generic social background, descriptive decision grounding, and normative decision grounding. We address the second by introducing selective robustness as a validation criterion: insensitivity to meaning-preserving prompt variation combined with a directional response to decision-relevant changes in the social world. We demonstrate the framework in a Prisoner's Dilemma-like game, showing how it can be used to identify which forms of context enrichment produce more desirable and robust agent behavior. We find that these forms of enrichment produce distinct behavioral patterns that vary between the two models tested.