Exploring Collaboration between a language and a non-language agent

AI summary

Overview

Research area: Natural Language Processing / LLM-agent collaboration, with a chess testbed and an extension to Go. Combines reinforcement learning (DAPO), representation projection (multimodal-style projectors), and interpretability of neural game engines.

Technical level: Advanced. The paper assumes familiarity with LLM post-training (SFT, policy-gradient RL, importance ratios and clipping), tool-calling interfaces, and how neural game engines encode policy and value.

Scope: The paper argues that text is a lossy channel between LLMs and non-language specialist agents, introduces latent state internalization as an alternative, and quantifies the resulting performance gap ("verbalization debt") across six chess tasks and a Go transfer experiment.

What This Paper Is About

LLMs are increasingly used to orchestrate specialized subagents, but that orchestration currently happens in natural language. In domains like chess, robotics, and autonomous driving, the strongest agents are not language models — their expertise lives in continuous latent representations that have no faithful text equivalent. The paper asks whether forcing those representations through a verbalization bottleneck is a structural limitation, and proposes instead to project the subagent's internal activations directly into the LLM's token stream so the LLM can attend over them alongside its own reasoning.

Key Contributions

1. Latent state internalization: A paradigm for LLM–non-language agent collaboration in which a single reasoning trace interleaves three token types — language tokens (the LLM's chain of thought), action tokens (moves that advance the environment), and latent state tokens (the subagent's penultimate-layer activations projected into the LLM's embedding space).
2. LLAMIA (Large Language and Action Models with Internal Agents): A two-stage training framework — supervised projector alignment with the LLM frozen, followed by end-to-end RL with DAPO — yielding a single model that matches or exceeds task-specific specialists, frontier models including GPT-5.1 with tool access, and every baseline on all six benchmark tasks.
3. Verbalization debt: The first empirical quantification of the performance gap between internalized and verbalized integration in heterogeneous LLM-agent collaboration, shown to be consistent across tasks, to widen with interaction depth, and to persist as the backbone scales from 4B to 14B.
4. LLAMIA-Bench: A curated benchmark of six chess tasks spanning behavior cloning, puzzle understanding, commentary, and planning, including three new evaluation targets: Wild BC (three OOD splits: GM-25, Low-Time, ΔElo), interest estimation, and Agadmator-2K (1,900 narrated games with move-aligned transcripts).

Main Findings

• A single 14B model beats specialists and frontier systems: LLAMIA-14B achieves the highest score on all six LLAMIA-Bench tasks, surpassing frontier verbalized systems and remaining competitive with dedicated task-specific finetunes trained on substantially more in-domain data. LLAMIA-8B already leads on all six tasks, and LLAMIA-4B on four of six. The paper states LLAMIA "matches or exceeds task specialists and frontier models including GPT-5.1 with tool access across all benchmark tasks."
• The verbalization debt widens throughout training and does not close with scale: With identical backbone, subagent (Lc0-BT4), data, and DAPO recipe, LLAMIA consistently achieves higher reward than LLAMIA-Verb. The gap reaches 2–3× by convergence, and persists at every scale from 4B to 14B.
• Puzzle Interest is the diagnostic case: Ranking positions by community-derived interestingness has no text surrogate in any engine output. Every verbalized system scores ≤12 regardless of model scale or frontier capability, while LLAMIA-14B reaches 52. LLAMIA-Verb-14B scores 8 on Interest while LLAMIA-4B already reaches 38.
• The gain comes from Lc0's latent content, not from weights or capacity (Table 1, 14B): LLM-Only (RL, no engine) scores 34 BC-M / 19 BC-W / 0.22 difficulty / 0.07 interest / 16.1 BLEU-2 / 0.13 G-eval; LLM-ChessCLIP (RL, raw board encoder in the same 32 slots) scores 39 / 28 / 0.24 / 0.08 / 23.1 / 0.29; untrained Qwen3+Lc0 tool use scores 39 / 33 / 0.28 / 0.05 / 18.8 / 0.15; LLAMIA-Verb scores 45 / 39 / 0.45 / 0.08 / 33.2 / 0.40; LLAMIA-SFT (latent, no RL) scores 51 / 46 / 0.66 / 0.48 / 38.5 / 0.58; and full LLAMIA scores 53 / 49 / 0.71 / 0.52 / 45.8 / 0.75.
• Internalization changes the kind of collaboration that emerges: Classifying 500 episodes per task per system with a GPT-4o judge (κ = 0.78 vs. human raters), LLAMIA adapts its strategy per task — engine-follow dominates gameplay (65%), consult-then-override dominates behavior cloning (48%), counterfactual query dominates commentary (40%). LLAMIA-Verb collapses to engine-follow on every task (62–76%).
• Latent integration is inference-cost neutral: LLAMIA invokes the subagent less often than LLAMIA-Verb (1.9 vs. 2.9 calls per query at 14B). Each latent invocation adds a fixed 32 tokens (182 vs. 150 total), but the lower call count offsets this, so average tokens-per-query and wall-clock latency are comparable or lower. Training cost stays within ~6% of the verbalized pipeline at every scale (4B: 9.1 vs. 8.8 GPU-hours; 8B: 16.8 vs. 15.0; 14B: 22.6 vs. 21.4).
• LLAMIA reproduces human behavioral signatures: A human study with n = 12 skilled players, all rated ≥1700 Elo, found LLAMIA-14B passes as human in 61% of trials (detection 39%, below chance), while LLAMIA-Verb is detected in 72%. Post-game ratings place LLAMIA alongside Maia* on perceived human-likeness, despite LLAMIA receiving no human-move supervision.
• Commentary preference favors latent access: 72.2% of 180 judgments favor LLAMIA, with the Insight gap (1.7 pts) exceeding the Accuracy gap (1.1 pts) — explaining why a move is strong requires representational features that do not survive verbal compression.
• Latent tokens are instruction-modulated: On a fixed back-rank mate position, attention over the latent tokens shifts with the target Elo — concentrating on mating geometry at 2000 Elo and dispersing to material at 1100 Elo — even though the latent representation is identical in both cases.
• Results transfer beyond chess: On Go with KataGo-b18 as the non-language subagent, LLAMIA-Go-14B achieves 48/50 top-1 human move-match at ranks 5k/5d using only 8k positions, matching the rank-calibrated KataGo-HumanSL expert and outperforming the verbalized control by ~10 points.

Methodology in Plain English

The researchers set up a controlled comparison in which the only thing that changes is how the engine's knowledge reaches the LLM. An LLM (Qwen3, tested at 4B, 8B, and 14B) plays and reasons about chess with access to a tool API: functions to read the board and legal moves, to advance the game, and a get_policy function that queries the engine's assessment of any position — either the current one or a hypothetical one reached by a candidate move.

Two versions are trained identically, differing only at the interface. In LLAMIA-Verb, get_policy returns a text summary (top moves with prior probabilities and value estimates). In LLAMIA, the same call additionally injects k = 32 continuous "state tokens" produced by a LatentBridge — a three-layer MLP with GeLU activations that projects the engine's penultimate-layer activations into the LLM's embedding space. The subagent is Lc0-BT4, a 15-layer Transformer encoder with 240M parameters producing a 1024-dimensional latent; the penultimate block (layer 14 of 15) was chosen because its held-out Stage-1 alignment loss was lowest across all blocks. The subagent stays frozen throughout; the LLM decides when to re-invoke it rather than re-encoding on every step.

Training has two stages. Stage 1 trains only the projector while the LLM is frozen, on state–policy pairs from the subagent's self-play, using cross-entropy to generate the correct action — this avoids catastrophic forgetting of language ability. Stage 2 unfreezes both the LLM and the projector and optimizes them jointly with DAPO, a group-relative policy optimization variant with asymmetric clipping. State-token positions are agent-injected and excluded from the policy gradient via gradient masking; gradients therefore shape both what representation the projector presents and how the LLM reasons over it, including when to query.

Evaluation uses six chess tasks with verifiable metrics: move-matching accuracy for behavior cloning, Spearman ρ for difficulty and interest estimation, and G-eval and BLEU-2 for move annotation and game-level commentary. Baselines are GPT-5.1 with verbalized Lc0 tool access, untrained Qwen3-14B with the same verbalized tool and 5-shot prompting, LLAMIA-Verb, and LLAMIA, with SFT and SFT + DAPO checkpoints reported to separate supervised pretraining from RL.

Why This Matters

Impact on research: The paper reframes LLM-subagent integration as an interface problem rather than a capability problem. It shows that verbalization is not merely suboptimal but structurally limiting — the debt widens with interaction depth, does not close with LLM scale from 4B to 14B, and is not fixed by RL budget. It also introduces a benchmark and a measurements framework (the verbalization debt) that other multi-agent and tool-use researchers can apply. Because internalization requires model weights, LLAMIA is positioned as a bridge: it interacts with closed-weight LLMs in natural language while internalizing the subagent, giving those models indirect but faithful access to non-language expertise.

Real-world applications:

• Grounded game commentary: Explaining why a move is strong requires policy gradients, value topology, and look-ahead depth; human evaluators preferred LLAMIA's commentary on strategic insight over the verbalized baseline.
• Opponent-specific preparation: The engine's positional understanding combined with the LLM's ability to reason over human intent and different Elo levels (attention shifts with the target rating) supports preparing against specific opponents.
• Designing interesting puzzles: Interest estimation depends on the engine's policy distribution and value gradients across candidate moves — a signal no verbalized engine output carries.
• Robotics and autonomous driving: The paper names these as domains where the strongest available agents are non-language models whose expertise lives in latent representations, making the same bottleneck relevant.

Industry relevance: The approach adds only ~6% to training cost and is inference-cost neutral (comparable or lower tokens-per-query and wall-clock latency than the verbalized interface), while delivering a model that beats frontier systems on every benchmark task. That cost profile matters for deploying orchestration pipelines that route through specialized non-language components.

Future Directions

• Extending internalization beyond games: The Go experiment (KataGo-b18, 384 trunk channels, 361 board intersections, 8k positions) provides initial evidence of transfer, but the paper frames this as "initial evidence" only; generalizing to robotics, autonomous driving, and other latent-expert domains remains open.
• The closed-weight access constraint: Internalization requires access to model weights and cannot be applied directly to closed-source models. Whether the LLAMIA-as-bridge workaround can be made more direct or more general is unresolved.
• Whether the verbalization debt ever closes: The paper reports that the gap persists as the LLM scales from 4B to 14B and does not close with RL budget within the evaluated range; scaling beyond this range is not reported.
• Understanding the emergent strategies: LLAMIA develops counterfactual queries and multi-step lookahead absent from LLAMIA-Verb, and its strategy distribution shifts by task. How these strategies are acquired, and whether they can be steered or made more reliable, is left as an open question (strategy definitions and per-task heatmaps appear in Figure 4).

Target Audience

Researchers and practitioners working on LLM orchestration, tool use, multi-agent systems, and reinforcement learning post-training will get the most from this paper, particularly those whose pipelines route through specialized non-language components. It is also relevant to game AI and chess/Go researchers interested in neural engine interpretability and human-like play, and to teams building commentary, analysis, or preparation products where the value lies in explaining why rather than what. Readers need working familiarity with policy-gradient RL and multimodal-style projector architectures to follow the methodology in full.

Authors’ abstract

LLMs are increasingly deployed as orchestrators that coordinate specialized subagents to solve complex tasks through natural language. However, in many important domains like game playing and robotics, the strongest available agents are not language models. Integrating non-language agents with LLMs would require \emph{verbalization}: compressing their rich continuous representations into sparse textual summaries at each interaction step. To study whether verbalization constitutes a bottleneck, we introduce \textsc{LLAMIA-Bench}, a suite of six diverse collaborative chess tasks spanning three facets: behavioral imitation, state assessment, and natural-language explanation. Each task instantiates a well-established chess problem that neither the LLM nor the chess engine can solve alone. To solve LLM collaboration with non-language agents, we introduce \emph{latent state internalization}, which projects the subagent's continuous representations directly into the LLM's token stream as learned state tokens, with dynamic re-encoding as actions advance the environment state. Comparing internalization to verbalized integration, our experiments reveal a consistent \emph{verbalization debt}: the performance gap widens throughout training and persists as the LLM scales from 4B to 14B parameters. A single 14B model, \textsc{LLAMIA}, trained with latent state internalization, matches or exceeds task specialists and frontier models including GPT-5.1 with tool access across all benchmark tasks, and generalizes out-of-distribution where task-specific finetunes collapse