Echoing: Identity Failures when LLM Agents Talk to Each Other

AI summary

Overview

Research area: Multi-agent LLM systems — specifically the reliability of autonomous agent-agent (AxA) conversations, as opposed to the human-agent or single-agent settings that current benchmarks target.

Technical level: Intermediate. The paper combines a light formalization (a partially observable stochastic game, a binary echoing metric) with a large empirical study spanning many commercial and open-weight models; no new model training or architecture is proposed.

Scope: A systematic measurement of "echoing" — a specific identity-drift failure in which one LLM agent abandons its assigned role and mirrors its conversational partner — across 66 AxA configurations, 4 domains, 2500+ conversations and over 250,000 LLM inferences.

What This Paper Is About

When two LLM agents talk to each other without a human in the loop, there are no subtle grounding or steering signals to keep them on task, so failures can arise that no single-agent benchmark can predict. The authors identify one such failure, which they call echoing: an agent drops its own identity, objective and perspective and instead adopts those of its conversational partner, undermining the objectives it was given. The paper's goal is to define echoing, measure how prevalent it is across models and domains, and test whether reasoning, prompting or protocol-level interventions can remove it.

Key Contributions

1. Formalization of AxA interactions and the echoing failure. The authors model agent-agent conversation as a partially observable stochastic game in which each agent (A_i) is characterized by an identity (I_i), an objective (O_i), a tool set (T_i), a utility (U_i) and an LLM policy (\pi_i), and they define echoing as a response whose language or decisions align with the partner's identity (I_j) rather than the agent's own (I_i). They also define a domain-specific LLM-based metric, EchoEvalLM(H_T, I_i, I_j) = {σ, a_e, m_e}, that returns a binary identity-inconsistency flag, the inconsistent agent, and the first offending message.
2. A large-scale empirical study of echoing prevalence. Experiments cover 66 AxA configurations, 22 customer-like customer-agent models, 3 seller configurations, 3 transactional domains, 3 prompt variants, and 2500+ conversations (over 250,000 LLM inferences), showing rates from 5% to 70%.
3. Analysis of the dynamics and confounds of echoing. The study shows echoing is not a prompt artifact, arises as conversations grow longer (average onset at turn 7.6, median 8.0), and is masked by standard task-completion metrics.
4. First-reflex mitigation strategies. The authors test prompt design, reasoning effort, and a protocol-level structured-response format that reduces echoing below 10% (reported as 9% in the abstract), and show that none of these eliminates it.

Main Findings

• Echoing is widespread across providers. Observed echoing rates range from 5% to 70% depending on model and domain. Rates vary sharply by model family, domain, and AxA configuration.
• Model ranking is not what scale alone predicts. OpenAI's GPT-5 achieved the lowest echoing rates on average; GPT-4.1 showed moderate consistency; GPT-4o showed significant domain sensitivity; Gemini-2.5-Flash showed consistently high rates; Gemini-2.5-Pro was highly variable; Sonnet-4 was moderate and domain dependent. Open-weight Llama-3.1 models showed low rates (11.5% for 8B, 9.1% for 70B), outperforming many larger proprietary models — which the authors read as evidence that training data and post-training alignment, not architecture or scale, drive the failure.
• Reasoning does not fix it. Reasoning models averaged 32.8% echoing versus 37.7% for non-reasoning variants. Across reasoning effort levels the rates were essentially flat: low 32.7%, medium 32.8%, high 32.9% (summarized in one figure caption as a drop from 37.7% to around 32.6–32.9%).
• Prompt engineering attenuates but does not eliminate echoing. Across the minimal, behavioral, and identity-boundary prompt variants, echoing persisted. Gemini-2.5-Flash stayed high (64–73%) across all variants; OpenAI models responded inconsistently. Even explicit anti-echoing instructions failed in already-susceptible models.
• Echoing is domain sensitive. GPT-4o showed 58% echoing in car sales, 25% in hotel booking, and 17% in supply chain — a 41% variation. Gemini-2.5-Pro and Sonnet-4 showed roughly 43% variation across domains. GPT-4.1 stayed low at 5–13% and GPT-5 at 2–10%. Supply chain showed the lowest echoing across models (except Gemini-2.5-Flash), which the authors attribute to the enterprise nature of the agents in that domain.
• Customer agents echo more than seller agents. Aggregated across configurations, echoing was more prevalent in customer agents, possibly because training data emphasizes "enterprise" roles.
• In a non-transactional advisory domain, echoing is much lower. In doctor-patient medical consultation, echoing appeared only in GPT-5 (2%), Gemini-2.5 (15–18%), and Llama-3.1-8B (30%). The authors hypothesize that the clear authority gradient in doctor-patient relationships protects against role drift.
• Echoing emerges late in conversations. Average onset was turn 7.6 (median 8.0). Per-model: Gemini-2.5-Flash 6.9, GPT-4o 7.6, o3 8.4, Gemini-2.5-Pro 8.9, Sonnet-4 7.8. GPT-5 and GPT-4.1 echoed earlier (around turns 3 and 4) but had the lowest overall rates.
• Task-completion metrics hide the failure. 93.2% of AxA runs completed successfully, and 93% of conversations were considered complete even when identity drift occurred. Echoing does not prevent completion; it degrades the quality and representativeness of the outcome. Conversations with echoing were slightly longer on average (9.6 vs 8.7 turns), so role confusion does not cause early termination.
• Structured responses reduce echoing substantially. Requiring agents to declare their role and separate it from natural-language content each turn reduced echoing to below 10% across the tested GPT and Sonnet configurations (24 configs across the 3 transactional domains); the abstract reports 9%. It does not reach zero.
• LLM-judge detection is reliable. Against 150 manually reviewed conversations (stratified sampling), the judge showed 91.1% agreement, Pearson correlation 0.825, Cohen's κ = 0.822, precision 86.7%, recall 95.1%, and F1 0.907. Cross-model judge agreement ranged from 79.6% to 88.9%.

Methodology in Plain English

The authors build a testbed in which two LLM agents converse in turns to complete a transaction. Each agent gets a private system prompt encoding who it is, what it wants, and an internal utility function it is told to maximize, plus its own tools that the other agent cannot see; only natural-language messages pass between them. The customer agent is the main experimental variable (22 models from OpenAI, Google, Anthropic and Meta), while the seller agent takes one of three configurations (GPT-4o at temperature 0.1, GPT-5 at medium reasoning, Gemini-2.5-Pro at medium reasoning). Non-reasoning models run at temperature 0.1, and reasoning models are tested at low, medium and high effort.

Each interaction runs until task completion or until each agent has responded at least 12 turns, with at most 10 LLM calls per agentic turn, and at least 10 independent runs per configuration. Three domains are transactional (hotel booking, car sales, supply chain procurement) and one is advisory (medical consultation). Three prompt variants are compared, from a minimal role assignment up to an identity-boundary prompt that explicitly tells the agent not to adopt the partner's role.

To detect echoing, a GPT-4o judge with structured output reads the full conversation history and flags persona inconsistency, which agent is inconsistent, and the first inconsistent message. Only successfully completed conversations (93.2% of runs) are analyzed, so behavioral consistency is separated from task success. A human review of 150 conversations validates the judge.

Why This Matters

Impact on research: The paper argues that AxA reliability cannot be inferred from single-agent evaluations, and that multi-agent protocol work — including efforts such as Google A2A, IBM BeeAI and Outshift by Cisco — needs to build in evaluation and mitigation of behavioral failures, not just message passing and authentication. Because echoing persists across reasoning effort, prompt variants and domains, the authors argue it behaves like a hallucination-class limitation specific to agent-agent interaction, rooted in model training and alignment rather than deployment choices. They also call for evaluation frameworks that go beyond completion to measure behavioral consistency and outcome value.

Real-world applications:

• Consumer negotiation agents — e.g. a customer agent booking a hotel room or buying a car on someone's behalf, where echoing leads to completed deals on sub-optimal terms.
• Supply chain and B2B procurement — supplier and buyer agents negotiating price, quality and delivery, a domain where echoing was lowest in this study but still present.
• Healthcare or advisory agents — doctor and patient agents exchanging information during a checkup, where echoing is lower but collaborative knowledge sharing also makes it harder to distinguish legitimate role overlap from drift.
• Agent protocol and platform design — structured turn formats, periodic role reinforcement, and fallback arbitration as safeguards in frameworks that let agents from different vendors interoperate.

Industry relevance: The authors are at Salesforce AI Research, and the framing is explicitly enterprise-focused: the customer agent (the party representing a human or organization) is varied far more than the seller agent, because that is where the failure concentrates. The finding that 93% of conversations "succeed" while identity drifts means deployed systems could look healthy on completion dashboards while quietly producing poor outcomes for the party that the echoing agent was supposed to represent.

Future Directions

• AxA-specific datasets and training. Since prompting, reasoning and protocol scaffolds only attenuate echoing, robust fixes likely require training or data tailored to agent-agent interaction, not surface-level interventions.
• Turn-level mitigations. The authors tried refreshing or summarizing roles every n turns (for example every 3), which produced disordered conversation flow, and they explicitly leave a fuller study of turn-level interventions to future work.
• Mechanistic analysis of echoing. The paper notes it did not analyze activations in open-weight models to understand why echoing happens at a deeper level, and flags this as an open challenge.
• Broader and more nuanced settings and metrics. Open questions include multi-party conversations, collaborative planning, and long-horizon tasks that may reveal additional failure modes; the asymmetric nature of echoing (customer vs seller); alternative detection strategies beyond LLM-based judges; and evaluation criteria that can distinguish legitimate collaboration from identity drift in advisory domains.

Target Audience

Researchers and engineers working on LLM agents, multi-agent orchestration, and agent interoperability protocols; evaluation and safety teams who need to know why task-completion metrics miss behavioral failures; and product teams deploying customer-facing or procurement negotiation agents who want a concrete, measured picture of how role identity can degrade when two models are left to talk to each other.

Authors’ abstract

As large language model (LLM) based agents interact autonomously with one another, a new class of failures emerges that cannot be predicted from single agent performance: behavioral drifts in agent-agent conversations (AxA). Unlike human-agent interactions, where humans ground and steer conversations, AxA lacks such stabilizing signals, making these failures unique. We investigate one such failure, echoing, where agents abandon their assigned roles and instead mirror their conversational partners, undermining their intended objectives. Through experiments across $66$ AxA configurations, $4$ domains (3 transactional, 1 advisory), and $2500+$ conversations (over $250000$ LLM inferences), we show that echoing occurs across major LLM providers, with echoing rates as high as $70\%$ depending on the model and domain. Moreover, we find that echoing is persistent even in advanced reasoning models with substantial rates ($32.8\%$) that are not reduced by reasoning efforts. We analyze prompt, conversation dynamics, showing that echoing arises as interaction grows longer ($7+$ agent turns) and is not merely an artifact of sub-optimal experiment design. Finally, we introduce a protocol-level mitigation where targeted use of structured response reduces echoing to $9\%$.