Skip to content
AI.info

Research

Duo: A Reciprocal World Model for Full-Duplex Interaction

Overview Research area: Generative world models, full-duplex multimodal interaction, and streaming joint audio–video generation (computer vision / multimodal ML). Technical level: Advanced.

Duo: A Reciprocal World Model for Full-Duplex Interaction
arXiv
2611.15131
Published
2026-11-11
Authors
Hengyuan Cao, Biao Gong, Guoshan Lu, Qiyuan Zhang, Lin Liu, Yiding Tian, Yilei Jiang, Yuxin Ma, Chengzhang Wu, Kun Chen, Yiqian Xie, Ting Liu, Xin Zheng, Xijun Gu, Tianren Ma, Jiancheng Pan, Chi Zhang, Xuzhe Zheng, Siyi Li, Zhenzhong Lan

AI summary

Overview

Research area: Generative world models, full-duplex multimodal interaction, and streaming joint audio–video generation (computer vision / multimodal ML).

Technical level: Advanced. The report assumes familiarity with diffusion/flow-matching models, autoregressive language models, distillation (DMD, Self Forcing), and attention-mask design.

Scope: A technical report introducing the Reciprocal World Model (RWM) formulation and Duo v1, a modular two-part system (an autoregressive Intent Module plus a causal audiovisual Realization Module) that perceives audio, video and text continuously while generating spoken, visible responses in real time.

What This Paper Is About

Existing world models simulate the physical side of a world — predicting how scenes evolve from past observations and control signals — but place intelligence outside the world, so users steer it only through low-degree-of-freedom inputs such as keyboard, mouse, gamepad or text prompts. The authors argue that intelligence and physical reality are equally fundamental to a world, and propose the Reciprocal World Model, in which a generated world contains beings that perceive the user in real time and decide how to respond, interrupt, or stay silent. Duo v1 is their first modular instantiation: a system that streams audiovisual responses in 17-frame chunks (approximately 0.71 s at 24 fps) while continuously ingesting user audio, video and text.

Key Contributions

  1. The Reciprocal World Model (RWM) formulation. A reframing of world models around two equally fundamental capabilities — Intent (the cognition underlying world interactions) and Realization (the generation of coherent audiovisual behavior) — with interaction formalized as chunk-wise, full-duplex exchange in which neither module accesses future inputs.

  2. Autoregressive intent for interaction rather than instruction encoding. Instead of using a language model only as an instruction encoder for a diffusion model, the Intent Module reasons autoregressively over the ongoing exchange and emits a participation decision at every chunk (<speak> followed by text, or <wait>), with its hidden states — not just decoded text — carrying the decision to the Realization Module.

  3. Causal chunk conditions on a unified timeline. User speech, user video, intent tokens and assistant media are synchronized on one timeline; each chunk receives exactly the conditions available at its decision boundary, and module-specific attention masks enforce causality within and across chunks. Temporal patch construction is confined inside each chunk so future frames cannot leak into a decision through the visual frontend.

  4. Dual rollout distillation. Distribution-matching distillation into a four-step streaming generator in which the frozen Intent Module autoregressively generates the final assistant turn and the Realization Module conditions on the resulting intent features and its own generated audiovisual history, so training conditions match live inference.

Main Findings

  • Chunk-causal design prevents future leakage. Because temporal patches are completed within each chunk (for a native 17-frame chunk the final pair is (16, 16), repeating the last frame rather than pairing with frame 17 of the next chunk), no future frame can enter a decision through the visual frontend, and an attention mask alone is not relied upon for this.

  • Explicit participation decisions decouple timing from content. The <speak> action separates the decision to begin speaking from the words of the response, since appropriate content can still be delivered at the wrong time; <wait> can coexist with ongoing playback and media continuation.

  • Action-aware rebalancing is needed to avoid a collapse into perpetual waiting. Most chunks contain no new assistant speech, so unweighted training can favor waiting; the authors subsample targets and weight the loss using offline supervision categories (idle waits vs. waits during user speech, ordinary vs. interruption openings, openings vs. continuations, and an annotated question in the preceding user turn as a separate opening tag).

  • History perturbations teach recovery. Truncating or repeating text in selected continuation chunks, and replacing selected ordinary openings with <wait> in the history, exposes the model to imperfect earlier responses while keeping supervision on later valid targets.

  • Training/inference mismatch is explicitly closed. Before distillation the Realization Module sees intent features of recorded assistant text and recorded media history, whereas in live exchange both are self-produced; dual rollout replaces both with the system's own outputs during DMD training.

  • A nuance in DMD supervision. The generator synthesizes a latent window causally while both score models evaluate the completed window bidirectionally; the time-dependent DMD factor is replaced by a sample-adaptive scale with a floor to prevent division by a vanishing scale.

  • Camera-rotation drift is addressed as an auxiliary term. An optional reweighting of the video DMD loss uses rotations estimated by frozen VGGT from decoded rollout frames, with a dead zone, gating and rotational path-length terms.

  • Quantitative results are not reported in the provided content. No benchmark names, dataset sizes, evaluation metrics, or comparison numbers appear in the supplied text; claims are presented as design descriptions and capabilities.

Methodology in Plain English

The system runs on a fixed clock. Time is cut into consecutive chunks of 17 video frames, about 0.71 s at 24 fps; a chunk is just a time interval and need not line up with a word, sentence or turn. At each chunk, two modules act in sequence.

The Intent Module is a multimodal language model adapted with low-rank adaptation. At the start of a chunk it looks at user video and speech-to-text from previous chunks plus its own earlier language, and outputs either "speak" with some text or "wait". Text tokens are admitted only after their aligned audio span ends; assistant text is assigned to the chunk containing its speech onset. The first chunk's output is fixed to <wait> and excluded from the loss.

The Realization Module is a causal audiovisual diffusion transformer initialized from pretrained weights, which generates the assistant's video and stereo audio latents for that chunk conditioned on the reference image, a projection of the Intent Module's hidden states, the user's audio/video latents, and the assistant's own earlier latents. The intent hidden states are passed through a linear projection and a refiner network, with a mask restricting them to the global-instruction positions and the current assistant output. Flow matching is used, with clean time τ = 1 and noise time τ = 0. Audio tokens use the two extreme spatial position IDs to distinguish left and right channels, and 3D-RoPE temporal coordinates are tied to real playback time in 25 ms units so training and inference stay aligned even when a sampled window starts mid-conversation.

Compression: audio is compressed to 40 latent vectors per second per channel at 32 dimensions each; video groups of 17 frames (one keyframe plus four packets of four) become 5 latent frames of 24 dimensions with 16×16 spatial reduction, then merged over 2×2 patches into 96-dimensional vectors. Each chunk therefore carries 5 video latents and 2×29 audio latents.

Training has three stages: (1) supervised fine-tuning of the Intent Module on chunk-aligned conversations with the rebalancing and history-perturbation tricks; (2) training the Realization Module for causal joint generation against recorded history with the Intent Module frozen; (3) distilling a four-step generator by distribution matching, with dual rollout and an optional GAN critic that judges joint audiovisual realism while physically excluding text, intent, reference and clean-context rows. Supporting this, the authors built data pipelines mixing scripted synthetic video calls with recorded conversations, aligned on a shared timeline, and a streaming runtime that coordinates intent inference, generation and incremental decoding while continuing to capture user input.

Why This Matters

Impact on research. The paper argues for a shift in what a world model is: not only a simulator whose dynamics are steered from outside, but a world that contains a participant with its own intent and can respond in real time. Framing participation timing (when to speak versus wait) as a first-class modeling problem alongside audiovisual synthesis, and showing how to keep a modular language-model-plus-diffusion system causal and train/inference-consistent, gives the field concrete machinery for interactive rather than merely explorable generated worlds.

Real-world applications (as described or directly implied by the paper):

  • Spontaneous full-duplex conversation with a generated on-screen presence, including interruptions and silences.
  • Showing real objects to the system and having it respond to them, since speech and vision convey tone, timing, expression and visual details that keyboard and mouse commands miss.
  • Reshaping virtual scenes or guiding an assistant through spoken and visual interaction.
  • Playing audiovisual games without mouse or keyboard controls.

Industry relevance. The design targets streaming deployment: a four-step distilled generator, latent compression chosen for real-time response, a chunked runtime with KV-cache maintenance, and initialization from existing pretrained audiovisual weights (MiniMax-H3) — all of which map onto product concerns such as latency, continuity across responses, and reuse of existing model assets. The released homepage, GitHub repository and Hugging Face collection signal an intent to seed an ecosystem around this emerging category.

Future Directions

  • Beyond v1. The authors explicitly describe Duo v1 as a first step with "the next version coming soon," leaving open what a larger or improved instantiation would change.
  • Unified versus modular instantiation. The paper states that Intent and Realization "can be fulfilled by a unified multimodal model or by a modular system"; whether a single unified model outperforms the modular language-model-plus-diffusion design is unresolved here.
  • Long-horizon consistency of self-generated history. Because both intent features and audiovisual history are self-produced at inference, accumulating drift over long interactions — beyond the camera-rotation drift the auxiliary VGGT-based term targets — remains an open problem.
  • Evaluation. The provided content reports no benchmarks, metrics or ablations, so how full-duplex interaction quality, participation timing and audiovisual synchronization should be measured — and how Duo v1 compares to prior streaming and full-duplex systems cited in related work — is left as open ground for broader exploration.

Target Audience

Researchers and engineers working on world models, streaming and full-duplex multimodal systems, joint audio–video generation, and diffusion distillation — particularly those building modular systems that couple a language model to a generative media backbone. It is also useful for practitioners who need a concrete account of causality, chunk conditioning and train/inference alignment in streaming generators. Readers seeking empirical benchmark comparisons will not find them in this report; readers new to diffusion or autoregressive modeling will find the architecture sections dense.

Authors’ abstract

In this report, we introduce the Reciprocal World Model (RWM) and present Duo v1, a full-duplex world model that continuously perceives audiovisual inputs while responding to users in real time. The RWM treats intelligence and physical reality as equally fundamental to what constitutes a world. Existing world models largely focus on simulating the latter, leaving reasoning and decision-making as external controls rather than intrinsic parts of the world itself. In RWM, we…

The full abstract on arXiv
Read the original paper