Latent Collaboration in Multi-Agent Systems

AI summary

Overview

Research area: Large language model multi-agent systems (MAS), latent-space reasoning, and inference efficiency.

Technical level: Advanced. The paper combines a system design with formal theorems on expressiveness, information preservation, and computational complexity, plus large-scale empirical benchmarking.

Scope: The paper introduces LatentMAS, a training-free framework in which LLM agents reason and exchange information entirely through continuous hidden representations rather than text, and evaluates it across 9 benchmarks and 5 backbone models.

What This Paper Is About

Almost all LLM multi-agent systems use natural language as the medium of thought and communication: an agent generates text, and the next agent reads that text. This paper asks whether agents can instead collaborate directly inside the continuous latent space, reasoning through hidden states and passing those states between models. The goal is a system that is more expressive, lossless in its information transfer, and cheaper to run than text-based MAS, without requiring any additional training.

Key Contributions

1. A training-free latent collaboration framework (LatentMAS). Each agent performs auto-regressive latent thoughts generation by appending its own last-layer hidden embeddings back as inputs, replacing token-by-token decoding with continuous hidden-state generation.

2. A shared latent working memory for cross-agent transfer. Each agent's layer-wise KV caches, which capture both the input context and the newly generated latent thoughts, are collected and prepended into the next agent's corresponding transformer layers, so the successor conditions on its predecessor's full internal state without re-encoding.

3. A lightweight input-output alignment operator. Because last-layer hidden states differ statistically from learned token embeddings, the authors introduce a linear projection matrix W_a ≈ W_out^† W_in of size d_h × d_h, computed once and reused, to map output vectors back into the valid input embedding space and prevent representation drift.

4. Theoretical analysis of expressiveness, fidelity, and complexity. Theorem 3.1 bounds the text length needed to losslessly express latent thoughts; Theorem 3.3 establishes information preservation through latent working memory; Theorem 3.4 compares LatentMAS per-agent complexity against text-based MAS achieving the same expressiveness.

Main Findings

• Accuracy gains over baselines: Across all 9 tasks, LatentMAS improves over the single-model baseline by an average of 14.6% and 13.3% under the sequential and hierarchical settings respectively, and by 2.8% and 4.6% over text-based MAS. The abstract reports gains of up to 14.6% higher accuracy.

• Large token savings: LatentMAS reduces token usage by 70.8% and 83.7% relative to sequential and hierarchical TextMAS. Across the broader efficiency analysis, it reduces token usage by 59.4%–87.9% compared with TextMAS and by 15.0%–60.3% compared with single agents.

• Faster end-to-end inference: LatentMAS provides 4× and 4.3× faster inference on average versus sequential and hierarchical text-based MAS. Even after accelerating TextMAS baselines with the vLLM service, LatentMAS still achieves a 2.6×–7× speedup over the vLLM-optimized TextMAS.

• Fewer steps than text CoT: With fewer than 50 latent steps, LatentMAS attains comparable or higher performance on reasoning-intensive tasks such as AIME 24/25, whereas TextMAS typically requires more than 20K output tokens to complete full text-based CoT trajectories.

• Latent thoughts are semantically meaningful: Comparing newly generated last-layer embeddings against token-by-token TextMAS responses on 300 MedQA questions (40 latent steps for LatentMAS, 4096 max-token budget for TextMAS), the paper reports that LatentMAS embeddings occupy nearly the same region of embedding space as TextMAS token embeddings and largely cover their distribution, indicating both semantic consistency and greater expressive capacity.

• Alignment prevents representation drift: Without W_a, newly generated hidden states deviate from the original input embeddings; after applying W_a, the aligned vectors realign with the input embeddings. Applying W_a yields consistent accuracy gains of 2.3%–5.3% across 3 benchmarks.

• Optimal latent step depth: On Qwen3-14B across three tasks, accuracy rises with more latent steps and peaks around 40–80 steps, then plateaus or declines, suggesting excessive latent generation adds redundant information.

• Expressiveness scales with model size: Theorem 3.1 implies latent thoughts generation can be O(d_h / log|V|) times more efficient than text reasoning. For Qwen3-4B / 8B / 14B models, the paper reports latent thoughts generation can be 235.7 / 377.1 / 471.4 times more efficient than text-based reasoning.

• Illustrative benchmark numbers (Table 1, Sequential MAS): On ARC-E with Qwen3-4B, LatentMAS reaches 98.6 accuracy versus 96.4 for TextMAS and 95.4 for the single model, using 581 output tokens versus 2420 for TextMAS. On HumanEval+, Qwen3-14B: 86.5 accuracy for LatentMAS versus 81.1 for TextMAS and 76.8 for the single model, with token usage reduced by 65.6%.

• Results not shown in the main text: Results on Llama backbones are reported in Appendix D.2; additional model analyses in Appendix E.1; case studies in Appendix J.

Methodology in Plain English

The authors build a multi-agent pipeline where no agent ever writes intermediate text. Inside each agent, instead of sampling a token and feeding it back, the model takes its own final-layer hidden vector, projects it through a small precomputed matrix so it looks like a legitimate input embedding, and feeds it back into the model. This repeats for m steps, producing a sequence of "latent thoughts" — the paper tests m ∈ {0, 10, 20, 40, 80}.

To pass information between agents, they exploit the key-value cache that transformer decoders already maintain. After an agent finishes its latent steps, they collect the KV caches from every one of its L transformer layers and prepend them into the corresponding layers of the next agent's cache. This means the next agent inherits the predecessor's full context and reasoning without any re-encoding. Only the final agent decodes an actual text answer.

They test two existing MAS architectures as templates: a sequential chain of planner, critic, refiner, and solver agents, and a hierarchical setup with code, math, and science domain agents feeding a summarizer. They evaluate on 9 benchmarks — GSM8K, AIME24, AIME25, GPQA-Diamond, MedQA for math and science; ARC-Easy and ARC-Challenge for commonsense; MBPP-Plus and HumanEval-Plus for code — using 5 off-the-shelf models from the Qwen3 family (4B, 8B, 14B) and Llama 3 family (3B, 8B). Baselines are single LLM agents, sequential TextMAS, and hierarchical TextMAS.

Implementation uses the HuggingFace past_key_values interface for cache transfer, with an additional vLLM backend integration supporting prefix caching and tensor-parallel inference. Temperature is set to 0.6 and top-p to 0.95 across all baselines and their method; maximum output lengths are 2,048 tokens for ARC-Easy, ARC-Challenge, and GSM8K, 4,096 for MedQA, MBPP+, and HumanEval+, 8,192 for GPQA, and 20,000 for AIME24 and AIME25. Results are reported as the mean over three independent runs, on 8× NVIDIA A100-80G GPUs.

Why This Matters

Impact on research. LatentMAS challenges the assumption that natural language must be the lingua franca of multi-agent systems. Prior latent-collaboration work typically depends on auxiliary alignment mechanisms and additional training; this framework is end-to-end training-free, and the paper offers theorems on expressiveness, information preservation, and complexity that give a formal account of why latent collaboration should dominate text mediation. It also provides a "debug mode" that converts inner latent thoughts into human-readable text, which is a step toward interpretability in an otherwise opaque pipeline.

Real-world applications (domains drawn from the paper's benchmark suite):

• Clinical question answering and medical decision support, per the MedQA evaluations.
• Automated code generation and program synthesis, per MBPP-Plus and HumanEval-Plus.
• Mathematics and competition-style problem solving, per GSM8K, AIME24, and AIME25.
• Scientific and graduate-level reasoning, per GPQA-Diamond.

Industry relevance. The reported combination of higher accuracy with 70.8%–83.7% lower token usage and 4×–4.3× faster inference directly targets the two dominant costs of deploying agent systems at scale: token billing and latency. A training-free method that drops into existing MAS architectures and works with vLLM prefix caching and tensor parallelism lowers the barrier to adoption, since no fine-tuning pipeline is required.

Future Directions

• Generalization beyond the tested architectures. The authors state LatentMAS is agnostic to specific collaboration strategies and applicable to sequential, hierarchical, or other advanced MAS designs; only two architectures are evaluated here, so broader MAS topologies remain untested.

• Extending to more backbone families and scales. The main results cover Qwen3 (4B, 8B, 14B), with Llama results relegated to an appendix; how latent collaboration behaves across other model families, including substantially larger scales, is an open question.

• Understanding the latent step budget. Accuracy peaks around 40–80 latent steps and then plateaus or declines, and the authors attribute the decline to redundant or less useful information. What exactly is being generated in those later steps, and how to adaptively choose m per task, is unresolved.

• Interpretability and auditing. The "debug mode" is presented as an extension for converting latent thoughts to text, with implementation details deferred to appendices. Making latent agent collaboration reliably auditable is an ongoing problem.

Target Audience

Researchers and engineers working on LLM-based multi-agent systems, agentic AI infrastructure, and inference efficiency. The paper is most useful to readers comfortable with transformer internals — KV caches, hidden states, embedding layers — and with multi-agent orchestration patterns. Practitioners building production agent pipelines will find the accuracy, token, and latency comparisons directly relevant, while theory-oriented readers will focus on the expressiveness and complexity theorems. Beginners may find the formalism in Sections 3 and the appendix demanding, though the core idea is stateable simply: let agents think and talk in vectors instead of words.

Authors’ abstract

Multi-agent systems (MAS) extend large language models (LLMs) from independent single-model reasoning to coordinative system-level intelligence. While existing LLM agents depend on text-based mediation for reasoning and communication, we take a step forward by enabling models to collaborate directly within the continuous latent space. We introduce LatentMAS, an end-to-end training-free framework that enables pure latent collaboration among LLM agents. In LatentMAS, each agent first performs auto-regressive latent thoughts generation through last-layer hidden embeddings instead of text. Then, a shared latent working memory preserves and transfers each agent's internal representations and latent thoughts, ensuring lossless information exchange without re-encoding. We provide detailed theoretical analyses showing that LatentMAS achieves higher expressiveness and lossless information preservation with lower overall complexity than standard text-based MAS. In addition, empirical evaluations across 9 comprehensive benchmarks spanning math and science reasoning, commonsense understanding, and code generation show that LatentMAS outperforms advanced single agents and text-based MAS baselines, achieving up to 14.6% higher accuracy, reducing output token usage by 70.8%-83.7%, and providing 4$\times$-4.3$\times$ faster end-to-end inference. Code and data are fully open-sourced at https://github.com/Gen-Verse/LatentMAS.