The Pulse
Meta Researchers Train Agents Against Less Cooperative Users
Meta researchers report that agents trained with MIMESIS, a simulator designed to model varied user behavior, scored higher than agents trained with GPT-5.5 across eight environments and nine unfamiliar user simulators. The study measures t
AI.info Team · Written with AI ·
Meta researchers argue that AI agents can look better in training than they will perform with people when the simulated users are too helpful. Their proposed answer, MIMESIS, is a model trained to imitate varied user behavior and then serve as a practice partner for agents. In a paper submitted to arXiv on October 7, the team reports that agents trained with MIMESIS scored higher than agents trained with GPT-5.5 across eight environments and nine unfamiliar user simulators. The comparison is still confined to simulated users: the authors say they have not tested whether those gains transfer to human interactions.
When helpful simulators make tasks easier
Most simulated users in agent training come from general-purpose assistant models. The MIMESIS authors say those models tend to be cooperative and explicit, unlike real users, who may leave preferences unstated, give incomplete answers or change their requirements. A simulator that supplies information too readily can change the task itself, making it easier for an agent to succeed than a human conversation would.
The paper illustrates the mismatch with a fixed GPT-5.5 agent on the tau-bench benchmark. The agent succeeded in 63.6% of interactions with human users, compared with 82.4% to 84.4% when frontier API models played the user. The researchers say their purpose-built simulators tracked human task difficulty more closely than the off-the-shelf alternatives.
MIMESIS adds 13 patterns of user behavior
The team describes MIMESIS in 4-billion- and 9-billion-parameter versions built on Qwen3.5. Training combines human conversations, reasoning supervision from ThoughtTrace, and reinforcement learning across several tasks. The simulator is also trained to exhibit 13 behavior patterns drawn from user interactions, including withholding evaluation criteria, shifting requirements over time and declining to cooperate with clarification questions.
On the SOUL-Index, which measures several aspects of user simulation, MIMESIS-9B scored 65.7. The strongest listed frontier baseline, Claude-Opus-5, scored 64.9. The paper also reports that MIMESIS improved behavioral fidelity over Claude-Opus-5 by 13.4 points on RealUserSim and reduced its Turing distance by 3.6 points on SimulatorArena; the two measures assess different aspects of how simulated conversations compare with human ones.
Training gains held across nine simulated users
For the agent-training experiment, the researchers froze MIMESIS and used it to generate multi-turn conversations for reinforcement learning. They then evaluated the trained agents with nine user simulators that were not used for training. The paper reports a mean score of 29.54 for agents trained with MIMESIS, against 26.10 for agents trained with GPT-5.5, across eight environments.
The team also tested Coached On-Policy Self-Distillation, or CSD. The method uses a simulator’s private reasoning traces and next user utterance to produce coaching notes during training, adding token-level guidance to task rewards. With CSD, the reported mean rose to 31.09 across the same evaluation users. The deployed agent does not receive those private traces or coaching notes; it acts on the conversation available to it.
The authors leave the human test open
The paper’s results show transfer among language-model simulators, not evidence that MIMESIS-trained agents perform better with people. The authors also caution that different simulators may share behavioral biases, and that a model’s generated thoughts are interpretations rather than observations of human mental states.
“Human evaluation is needed to assess whether the resulting policies remain effective in real interactions.”
Hoang Phan and co-authors, the paper’s authors, write
The October 7 submission is a research paper, not evidence that Meta has released model weights for general use. Its arXiv record links to a GitHub repository for the project. The immediate claim is narrower: in the authors’ tests, training against MIMESIS improved scores when agents later faced other simulated users; whether that carries over to conversations with people remains unmeasured.