Controlling Multimodal Conversational Agents with Coverage-Enhanced Latent Actions

AI summary

Overview

Research area: Natural Language Processing / multimodal conversational agents, reinforcement learning, and vision-language models.

Technical level: Intermediate — the paper assumes familiarity with reinforcement learning (policy optimization, MDPs) and vision-language model fine-tuning, but explains its latent action design from first principles.

Scope: The paper proposes replacing token-level reinforcement learning with a compact latent action space for fine-tuning multimodal conversational agents, and builds that space from both paired image-text data and large-scale text-only data.

What This Paper Is About

Fine-tuning vision-language models as multimodal conversational agents with reinforcement learning is hard because the policy must explore an enormous text token space — with vocabulary size |V| and maximum response length m, the sampling space grows as |V|^m. This paper learns a small learned codebook of latent actions instead (size |C| = 128, versus a token vocabulary of 152K for Qwen2.5-VL), so the agent picks among 128 actions at each step rather than hundreds of thousands of tokens. The central obstacle the authors tackle is that a latent action space needs broad coverage to support effective RL exploration, but paired image-text data is scarce, so they also draw on massive text-only corpora by learning a cross-modal projector.

Key Contributions

1. First application of latent actions to multimodal conversational agent RL fine-tuning. The authors state they are the first to introduce latent actions for fine-tuning MCAs via RL, which significantly reduces the exploration space.
2. A latent action space built from both paired image-text data and text-only data. A cross-modal projector P transforms text embeddings into image-text embeddings, allowing massive unpaired text corpora to contribute to the codebook while avoiding unimodal bias.
3. A novel cycle consistency loss for projector robustness. A reverse projector P' maps image-text embeddings back to text embeddings, and the cycle constraint P'(P(e^T)) ≈ e^T is enforced on text-only data.
4. A two-task, multi-algorithm evaluation showing consistent gains. The method is tested on multimodal role-playing conversation and multimodal personalized conversation across GRPO, Dr.GRPO, DAPO, and BNPO, and the cross-modal projector is shown to be critical for latent action coverage.

Main Findings

• Latent action RL beats token-level RL by about 4% on average. Averaged over all settings in Table 1, the latent action method outperforms token-level RL by 4%.
• Gains are consistent across four RL algorithms. GRPO, Dr.GRPO, DAPO, and BNPO all improve when switched from token-level to latent action optimization.
• Gains hold at both model scales. Results are consistent for Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct, indicating scalability.
• Strongest numbers on MMRole ID. With Qwen2.5-VL-3B-Instruct and GRPO, latent action RL reaches 0.949 on MMRole ID versus 0.838 for token-level GRPO; with Dr.GRPO, 0.953 versus 0.867.
• Largest degradation comes from dropping text-only data. The "Ours w/o text-only data" ablation drops to 0.865 average (from 0.893), with particular weakness in out-of-distribution settings — OOD on MMRole (0.861) and LS2 on PCogAlignBench (0.817).
• Cycle consistency loss matters. Removing it causes an average performance drop of 2.3% (0.870 average versus 0.893).
• Removing the cross-modal projector also hurts. Learning the codebook directly from text-only representations e^T introduces a unimodal bias, where the policy over-relies on textual representations.
• Rollout diversity improves substantially. Latent action RL consistently and significantly outperforms token-level RL in semantic diversity — for example, BNPO on MMRole gives 1.291 ±0.002 with latent actions versus 1.077 ±0.001 with tokens.
• Computational cost is modest. Rollout takes 1.13× longer due to the extra latent action prediction step, but policy updates take only 0.86× the time of the baseline, for a total RL training time of 1.08× token-level RL. The limitations section reports inference latency increases by 1.13×.
• Fine-grained improvements concentrate on harder dimensions. Both the method and baselines do well on basic capabilities like Fluency (Flu) and Conversational Flow (CF), but the method shows substantially more pronounced gains on challenging personalized dimensions such as Tone Consistency (TC) on MMRole.

Methodology in Plain English

The researchers keep the base vision-language model and add three small modules that share it, so overhead is marginal:

1. Language world model — takes the current observations (image x^V and tokens x^T_1:t) plus a latent action a_t, and autoregressively produces the next token. It has an encode module (reusing the original VLM's transformer blocks) and a merge module, where a two-layer MLP combines the context embedding with the latent action embedding from the codebook.
2. Inverse dynamics model — sees future observations including x^T_{t+1} and infers which discrete latent action a_t ∈ {1, …, |C|} produced them. It is used only for training, never at inference.
3. Policy model — sees only current observations and predicts the latent action, mirroring what the inverse dynamics model would have chosen.

Building the latent action space happens in two steps. First, inverse dynamics learning jointly trains the inverse dynamics model, language world model, and codebook on a mixed corpus of paired image-text data plus text-only data, minimizing the negative log-likelihood of reconstructing the next token. Second, policy behavior cloning trains the policy to predict the same latent actions the inverse dynamics model inferred.

Handling the data scarcity problem. For paired examples the image-text embedding comes straight from the VLM encoder. For text-only examples, the authors instead use a cross-modal projector P to map the text embedding into the image-text embedding space — a much cheaper alternative to training a conditional image generator G(V|T), given the high dimensionality of images. The projector is trained in two stages: first initialized on paired image-text data with a Gaussian negative log-likelihood loss (predicting mean and variance of the target embedding), with the reverse projector P' trained symmetrically; then jointly trained on paired plus text-only data with a cycle consistency loss enforcing P'(P(e^T)) ≈ e^T.

RL fine-tuning. At each step the policy samples a latent action and the world model generates the next token, with the world model frozen so only the policy's latent action distribution is optimized. Because the optimized sequence length equals the response length, the method does not lengthen the optimized sequence relative to token-level RL.

Data and evaluation. Paired data totals 14 million images and 1 billion text tokens, drawn from Conceptual-12M (image-caption pairs), N24News (multimodal news), and WikiWeb2M (multimodal Wikipedia). Text-only data comes mainly from SlimPajama-627B, containing 627 billion text tokens. Downstream tasks are MMRole (Comment subset, trained on the in-distribution split and evaluated on ID and OOD test sets) and PCogAlignBench (trained on LS1, evaluated on LS1 and LS2). Evaluation uses the LLM-as-a-Judge metric with scores ranging 1–10, reporting the ratio of the model's average score to the ground-truth response's average score, with mean and standard deviation over three runs. Rollout diversity is measured as semantic diversity using BGE-M3 embeddings, over 5 independent runs.

Why This Matters

Impact on research. The paper addresses a structural bottleneck in applying RL to language and multimodal agents: exploration over a token vocabulary scales exponentially with response length. Demonstrating that a 128-entry latent codebook can outperform token-level RL across four algorithms and two model sizes gives the field a concrete, RL-agnostic recipe. The cycle-consistency trick for projecting text-only corpora into a multimodal embedding space is also a transferable idea for any setting where aligned multimodal data is the constraint.

Real-world applications:

• Multimodal role-playing agents for entertainment, where personality, tone, and knowledge consistency must be maintained across long, image-grounded conversations (the MMRole task).
• Multimodal personalized assistants, where responses must be aware of a user's role set, body behavior, mind/feelings, and conversational context (the PCogAlignBench task).
• Online education and tutoring agents that ground explanations in images while staying consistent with a teaching persona.
• Customer-facing conversational systems that need to reason over user-supplied images and respond in a consistent brand voice.

Industry relevance. The method is compatible with existing RL infrastructure, keeps reward functions unchanged from the baselines, and costs only 1.08× total RL training time — a small price for roughly 4% average gains. The release of code and data at the AlibabaResearch DAMO-ConvAI repository lowers the barrier to adoption.

Future Directions

• Reduce the added latency. The paper notes that the additional latent action prediction step increases RL training time by 1.08× and inference latency by 1.13×, which the authors list as a limitation to address.
• Extend beyond conversational tasks and larger models. Validation on more diverse tasks such as visual mathematical reasoning, and on larger-scale VLMs, is left to future work due to computational resource constraints.
• Make latent actions interpretable. The authors state it remains unclear what semantic concepts the learned latent actions capture or how they relate to human-understandable behaviors, and call for a systematic investigation.
• Mitigate persona and role-set bias. The ethics section flags that role-set bias may arise from incomplete data collection in PCogAlignBench, and encourages practitioners to consider diverse user backgrounds when constructing personas.

Target Audience

Researchers and engineers working on reinforcement learning for large language and vision-language models, multimodal conversational agent design, and latent action or world-model approaches. It will be most useful to readers who already understand policy optimization basics such as GRPO-style objectives and want a concrete method for shrinking the action space in multimodal agent fine-tuning, as well as practitioners deciding whether the training and inference overhead of latent actions is worth the reported accuracy and diversity gains.

Authors’ abstract

Vision-language models are increasingly employed as multimodal conversational agents (MCAs) for diverse conversational tasks. Recently, reinforcement learning (RL) has been widely explored for adapting MCAs to various human-AI interaction scenarios. Despite showing great enhancement in generalization performance, fine-tuning MCAs via RL still faces challenges in handling the extremely large text token space. To address this, we learn a compact latent action space for RL fine-tuning instead. Specifically, we adopt the learning from observation mechanism to construct the codebook for the latent action space, where future observations are leveraged to estimate current latent actions that could further be used to reconstruct future observations. However, the scarcity of paired image-text data hinders learning a codebook with sufficient coverage. Thus, we leverage both paired image-text data and text-only data to construct the latent action space, using a cross-modal projector for transforming text embeddings into image-text embeddings. We initialize the cross-modal projector on paired image-text data, and further train it on massive text-only data with a novel cycle consistency loss to enhance its robustness. We show that our latent action based method outperforms competitive baselines on two conversation tasks across various RL algorithms.