Multimodal Duplex Interaction Agent

AI summary

Overview

Research area: Real-time multimodal (speech, vision, text) interaction models and LLM-based agents — submitted to arXiv under the audio/speech processing category (eess.AS), arXiv:2609.08977v3.

Technical level: Advanced. The paper assumes familiarity with autoregressive language modeling, streaming speech-language architectures (Thinker-Talker), multimodal encoders, tool calling, and agent orchestration runtimes.

Scope: The paper introduces Gander, a duplex interaction model built on MiniCPM-o 4.5 that couples a real-time, full-duplex front "cerebellum" with an asynchronous, training-free back "brain" for long-horizon agentic tasks.

What This Paper Is About

Conventional AI assistants work in discrete turns: the user finishes speaking, the system processes the whole utterance, then responds. The authors argue this is unnatural compared with human conversation, where participants listen and speak concurrently, interrupt, backchannel, and redirect the interaction. Their goal is a single system that combines continuous, low-latency multimodal interaction (speech, camera, screen) with the longer-horizon reasoning and tool use needed for real workflow tasks — without forcing either capability to compromise the other.

Key Contributions

1. Gander, a duplex interaction model with an asynchronous agent loop. Gander connects real-time interaction to longer-horizon agentic reasoning and supports modular extension of reasoning capability, so different or stronger reasoning backends can be swapped in without retraining the interaction model.

2. A Cerebellum-Brain collaborative framework with tool-call feedback and chunk-based streaming. The framework yields a broad set of interactive behaviors: interruption, proactive interaction, robustness to background interference, multi-party interaction, backchannel communication, reasoning, and tool-assisted task execution.

3. A streaming Thinker-Talker front cerebellum. The model flattens streaming user inputs and its own outputs into an ordered token stream at the chunk level, providing a unified representation for low latency and continuous interaction, with a per-chunk control token that decides whether to listen, speak, or interrupt.

4. Open release of model weights, code, and data to facilitate further research and development of duplex interaction agents.

Main Findings

• Architecture split resolves a design tension: Casual conversation favors immediate responses and continuous contextual adaptation, while complex workflows require longer-horizon reasoning, iterative planning, tool use, and sustained execution. Handling both in one monolithic model creates a tension between responsiveness and reasoning capacity, which the authors address by decoupling the real-time cerebellum from the asynchronous brain.

• Interactivity is modeled intrinsically rather than by pipeline modules: The paper argues that interactivity benefits from being modeled inside the interaction model instead of relying on independently designed modules such as VAD, because real-world scenarios include spontaneous interruptions, agent-initiated engagement, background noise, multi-party conversation, and backchannel behaviors.

• No external voice activity detection is needed: Because the model reaches a decision once per one-second window, proactive behavior arises from the same mechanism and no external VAD module is required to trigger responses.

• Structured tool interface for task control: The cerebellum delegates work through special-token tool calls defining three operations — task_start (creates a background task), task_send (routes follow-up input to an existing task; supports main steering and read-only fork inquiry), and task_resolve (deterministic lifecycle control via cancel, allow_once, allow_session, deny).

• Two runtime control modes: In lean mode the runtime directly executes actions classified by the cerebellum, giving a shorter control path, lower latency, greater determinism, and fewer failure points, but task configuration is largely fixed at deployment time. In coordinator mode an independent control-plane model generates execution directives on reasoning intensity, questioning policy, permission policy, and delivery strategy, at the cost of extra model invocations, latency, computational overhead, and nondeterminism. The coordinator is currently involved primarily in task_start.

• Training-free back brain: The back brain (instantiated by systems such as Codex or Claude Code) requires no additional training within Gander and is designed as a plug-and-play component. It can call three runtime-specific interfaces: context_fetch, memory_search, and share.

• Perception is compressed for real-time budget: Visual frames use any-resolution partitioning with a SigLIP vision transformer and a query-based resampler, giving roughly a 16×16 reduction relative to the raw patch grid — more aggressive than the 4×4 compression adopted in many prior multimodal models — and run at up to 448×448 resolution. Audio is encoded at roughly 50 frames per second, then downsampled 5× by a lightweight MLP projector to roughly 10 tokens per second.

• Bounded context via sliding window: Context is managed with a fixed budget of 128 chunks, roughly a two-minute rolling temporal receptive field, with oldest chunks evicted as new ones arrive.

• Evaluation scope: The authors report results on Full-Duplex-Bench, SpokenQA, and WorldSense, plus internal human evaluations. Internal human evaluations indicate Gander maintains natural and expressive spoken dialogue; benchmark results demonstrate effective turn-taking capability and "encouraging" results on spoken question answering and related understanding tasks. The exact numeric scores are not reported in the available paper content.

• Demonstrated interaction settings: Gander supports background noise interference, multi-party interactions, and backchannel communication, and shows context-sensitive behaviors such as resolving ambiguous references with visual context.

Methodology in Plain English

The authors split the problem in two rather than asking one model to do everything.

The front cerebellum is a real-time multimodal model that continuously watches and listens. It ingests streaming audio and video, chops the interaction into one-second windows, and converts everything in a window — the audio tokens, the visual tokens, a control token, and any text the model chooses to emit — into a single flat sequence of tokens processed by a normal causal language model. That control token is the key trick: the model explicitly predicts listen, speak, or interrupt once per second, so it learns when to stay quiet and when to talk as part of its ordinary next-token prediction. It keeps a rolling memory of the last 128 chunks (roughly two minutes) so inference cost stays bounded.

When a request is too complex for the cerebellum, it emits a structured tool call (task_start, task_send, or task_resolve). A coordinating runtime — with a persistent gateway organizing work into Projects, Tasks, Runs, WorkerEvents, and Deliveries — hands the task to the back brain, a general-purpose coding/agent backend that needs no extra training. The cerebellum sends the transcribed query plus the relevant final video frames. While the back brain works, the user can keep talking to the cerebellum, revise the task, or ask side questions in read-only forks that don't disturb the main execution thread. The back brain can also push verified findings or progress back to the cerebellum via a share interface, and the cerebellum summarizes results into natural spoken replies.

The researchers initialized the interaction model from MiniCPM-o 4.5 and trained on conversational, interactive, multimodal understanding, and agentic data.

Why This Matters

Impact on research. The paper reframes interactivity as an intrinsic modeling capability rather than an external pipeline concern, and it argues against treating speech as a mere interface to an agent. It provides a concrete, open instantiation (weights, code, data) of a decoupled design where real-time responsiveness and long-horizon reasoning don't compete inside one model — a template others can test and extend.

Real-world applications (drawn from settings the paper discusses):

• Real-time voice assistants that can be interrupted mid-sentence and that proactively offer intermediate feedback.
• Workflow-oriented agent scenarios such as code generation, file and computer operations, and multi-step task execution driven by spoken interaction.
• Information retrieval and document processing delegated to a background agent while conversation continues.
• Conversational settings with background noise, multiple speakers, and brief acknowledgments ("uh-huh", "right", "I see") that should not terminate a task.

Industry relevance. The system was developed by authors affiliated with the Hunyuan Speech Team at Tencent, Zhejiang University, Shanghai Jiao Tong University, the Chinese University of Hong Kong, and Nanyang Technological University, and it positions itself alongside product-oriented systems such as GPT-Live with Codex and Claude Voice Mode, as well as Qwen Audio Agent. The plug-and-play brain means a vendor can upgrade reasoning capability without retraining the interaction model — an economically meaningful property for deployed assistants.

Future Directions

• Longer-horizon agent tasks. The authors state that broader long-horizon agent tasks remain a promising direction, since current evaluation focuses on representative conversational and tool-assisted settings.
• More diverse deployment conditions. The paper explicitly flags more diverse deployment conditions as an open area beyond the current evaluation.
• How the cerebellum should pre-process delegated requests. The authors list open design considerations: whether the cerebellum should perform additional reasoning on the transcribed query before passing it on, and whether a trainable back brain could directly receive the user's full multimodal input stream in real time. Gander takes the straightforward path of passing the transcribed query and relevant final video frames.
• Summarization granularity. Whether the cerebellum should summarize the back brain's intermediate execution states or only its final result is described as a data-scaling design choice determined by training data and target scenarios, rather than a settled decision.

Target Audience

Researchers and engineers working on speech-language models, full-duplex and real-time interaction, multimodal foundation models, and LLM-based agents. It is most useful to readers already comfortable with streaming architectures, token-level modeling, and agent orchestration who want to understand how one system can unify real-time conversational behavior with asynchronous task execution. Readers looking for detailed benchmark numbers should note that the specific scores are not reported in the available paper content — the abstract only describes the results qualitatively, with full results appearing on the project page and in the released materials.

Authors’ abstract

In this work, we present Gander, a native multimodal duplex interaction model that builds on MiniCPM-o 4.5 and is further adapted for realtime interaction with an asynchronous agent loop. In contrast to conventional turn based systems, Gander continuously processes streaming user inputs, enabling full-duplex interaction in both everyday conversations and complex workflow agent scenarios. Users can interrupt an ongoing response, while the model can proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) a Cerebellum-Brain collaborative framework, Cerebellum is responsible for realtime interaction while the Brain handles complex reasoning and higher level agentic tasks. The two components interact continuously through tool calling and the agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, where user inputs and model outputs are flattened into an ordered token stream at the chunk level. We evaluate Gander across conversational ability, interactive capability, understanding, and tool assisted task execution. Internal human evaluations show that Gander maintains natural and expressive spoken dialogue, while benchmark results demonstrate effective turn taking capability and encouraging results on spoken question answering and related understanding tasks. Gander also supports a range of challenging interaction settings, including background noise interference, multi-party interactions, and backchannel communication. While our current evaluation focuses on tool assisted settings, broader long horizon agent tasks and more diverse deployment conditions remain promising directions for further study. We release Gander together with its models, code, and data to facilitate further research and development in the community.