Realtime-Venus: A full-duplex interaction system with asynchronous delegation

AI summary

Overview

Research area: Real-time multimodal conversational AI — full-duplex spoken and audio-visual interaction systems with asynchronous tool/agent delegation.

Technical level: Advanced. The paper assumes familiarity with streaming autoregressive multimodal architectures, speech tokenization, turn-taking/full-duplex dialogue, and tool-use orchestration.

Scope (one sentence): The paper describes Realtime-Venus, a system pairing two separately trained 9B streaming models with a shared asynchronous execution harness, and reports understanding, full-duplex, and delegation results against named benchmarks and online baselines.

Note: the supplied paper content is truncated partway through Section 6 (the full-duplex and delegation data pipeline). Results for sections after that point are only available through the abstract, introduction, and contribution statements.

What This Paper Is About

Natural interaction requires a system to keep perceiving while it is speaking, decide when to respond without being asked, and handle requests that take longer than a conversational turn. Existing streaming models handle some of these capabilities, but coordinating live conversation with slow background computation is unsolved — a background task needs a stable record of the request and its evidence, while its answer must be delivered into a conversation that may have changed while the task ran.

Realtime-Venus addresses this by placing continuous perception, conversational control, native speech generation, and private delegation requests on one shared causal timeline, and by adding a separate harness that executes delegated work asynchronously and returns results for the frontend to speak at a time of its choosing.

Key Contributions

1. Two proactive full-duplex models. Realtime-Venus-Omni (audio-visual) and Realtime-Venus-Audio (spoken), two separately trained 9B models that integrate continuous perception, conversational control, native speech generation, and private delegation under a unified streaming formulation. The authors state that Realtime-Venus-Omni is, to their knowledge, the first full-duplex omni model to support asynchronous backend invocation for reasoning and tool execution while maintaining video interaction, and that with memory augmentation it supports hour-scale video understanding.

2. Realtime-Venus-Harness. A shared execution framework that binds each task to the evidence available at the request boundary, executes registered capabilities asynchronously, and returns results to the originating session while the frontend retains control over when and how the response is spoken.

3. A coupled duplex and delegation data pipeline. A pipeline combining scenario planning, speech realization, and temporal alignment to construct trajectories that link conversational events to delegation requests, background results, and response continuations.

4. A unified post-training recipe. A single recipe mixing offline understanding, proactive full-duplex trajectories, and delegation workflows, applied to both models with modality-specific data coverage.

Main Findings

• Video understanding. Among the evaluated online models, Realtime-Venus-Omni achieves the highest scores on six of eight video benchmarks, including StreamingBench (70.2%), OVO-Bench (64.7%), and Daily-Omni (81.3%).

• Audio understanding and spoken QA. Across eight audio understanding and spoken question answering benchmarks, Realtime-Venus-Audio leads the compared models on MMAU (78.0%), MMAU-Pro (63.2%), Llama Questions (83.8%), and Speech CMMLU (67.8%), while matching the best VoiceBench AlpacaEval score of 4.81.

• Full-duplex behavior. On Full-Duplex-Bench v1.5, Realtime-Venus-Audio responds to 75% of user interruptions and achieves continuation rates of 97%, 88%, and 86% under backchannels, other-directed speech, and background speech respectively, exceeding Gemini 3.1 Live and GPT-4o on all three continuation metrics.

• Turn control is semantic, not binary. Because the interaction-control token is predicted from the same semantic representation used for response generation, acoustically similar overlaps can produce different continuations. The paper distinguishes three cases: pauses and background noise trigger <|listen|>; backchannels preserve <|speak|> and the current response plan; interruptions emit <|turn_eos|> and are handled as stop/reject, repair/update, or redirect/follow-up.

• Delegation is evaluated separately from execution. The authors report that complementary tool-use and delegation-decision evaluations distinguish correct routing from successful task completion and identify remaining challenges in executing external work during conversation. Specific scores for those evaluations are not available in the supplied content.

• Training data composition. The models are trained on over 2.8 million samples across nine data categories. Approximately 70% of the corpus is video data (used only by Realtime-Venus-Omni) and approximately 30% is audio data (shared by both models). Approximately 56% is offline understanding, approximately 37% is proactive duplex interaction, and approximately 6% is delegation workflows.

Methodology in Plain English

Base architecture. Both models are adapted from MiniCPM-o 4.5, an open-source 9B model, and inherit its Omni-Flow architecture. Realtime-Venus-Omni encodes aligned visual and audio streams with SigLIP2 and Whisper-Medium; Realtime-Venus-Audio drops the visual branch. Projected features go to a Qwen3-8B language backbone, whose text and hidden states condition discrete S3 speech-token prediction. A streaming flow-matching decoder turns those tokens into waveform chunks using reference audio from the system prompt.

One-second chunks. Streaming is organized into one-second units. At each unit the language model predicts <|listen|> or <|speak|>; for speaking units, response text is generated and conditions aligned S3 speech tokens (roughly 25 tokens per second). <|chunk_eos|> closes a speech chunk and <|turn_eos|> ends an assistant turn. This schedule keeps perception running while the model speaks and keeps the unspoken continuation revisable.

Three interleaved streams. Each chunk aligns a user stream (audio features at about 10 per second, plus visual features in the omni model), an assistant stream (foreground text, speech tokens, and text-only delegate instructions), and a background stream (asynchronous text from the backend). Four control tokens — <delegate>, </delegate>, <backend>, </backend> — are added to the tokenizer, input embeddings, and LM head. Delegate spans are hidden from display and speech; backend results enter as external context at an available chunk boundary rather than as a prediction target.

Playback synchronization. Text emission is adapted to accumulated playback progress: at each chunk the scheduler picks the number of candidate text tokens whose estimated duration best aligns with the current chunk boundary, emitting fewer tokens when speech lags and more when playback capacity is available.

Training-free long-video memory. An external memory module, requiring no training or parameter updates, decides which sampled frames to archive using a gating mechanism based on motion-compensated prediction cost (inspired by AdaCodec), with local-change protection and a maximum consecutive-dropping constraint. Retrieval does fine-grained matching between query tokens and stored frame visual tokens using a MaxSim-style maximum-similarity score, weight-normalized by median absolute deviation, then combines relevance with a visual-novelty term inspired by Maximal Marginal Relevance. Retrieved frames are merged with temporally adjacent audio and assembled chronologically with the recent short-term window and the query.

Prompted-only evaluation of the harness. The harness has three stages — capture, dispatch, return. Capture fixes the evidence snapshot at the request boundary using a bounded session buffer and look-back horizon; dispatch routes the objective to a registered capability (multimodal, general, or a registered skill) using a model-based decision over capability contracts, with schema validation before execution; return polishes the normalized result into spoken text and wraps it in a single private <backend> message. Work items move through Queued → Running → Completed → Delivering → Delivered, with Failed as a terminal state, and a freshness deadline excludes stale results.

Training. A unified post-training pass mixes proactive duplex data, delegation data, and general understanding data. Loss is computed only on response spans, excluding system, user, and media-placeholder tokens; full-duplex supervision covers the per-second listen/speak decision tokens together with spoken text, and long replies are supervised continuously across units. Loss is normalized per sample. Only the Thinker is updated; the acoustic decoder remains fixed and is excluded from the training objective. Samples that fail to decode or exceed the length budget are discarded.

Why This Matters

Impact on research. The paper separates two problems that are usually conflated: what a reply should say, and when it should be said. By giving the harness responsibility for content and wording while the frontend retains conversational timing and speech, it offers a concrete interface for combining latency-sensitive streaming models with slow external computation. The unified stream serialization also treats delegation events and background results as first-class entries on the same causal timeline as user input and model output, rather than as an out-of-band mechanism.

Real-world applications.

• Live video assistants that watch and listen continuously and speak up when something warrants attention, without being prompted.
• Voice agents that handle a spoken question, dispatch a multi-step lookup or tool call in the background, and answer later in the same conversation without stalling.
• Always-on device and meeting companions that distinguish backchannels and background speech from real interruptions, so they keep talking when appropriate and stop when corrected.
• Hour-scale video review or monitoring, where a training-free memory module recovers relevant earlier moments beyond the model's rolling context window.

Industry relevance. The work targets deployment constraints directly: 9B model scale, one-second chunk streaming, a separate capability registry that can be extended with new skills without changing the frontend delegation protocol, and a work lifecycle with explicit failure, staleness, and playback-acknowledgment states. The project page and code repository are linked, and the system is built on an open-source base model.

Future Directions

1. Improve execution of delegated tasks during live conversation: the authors' own tool-use and delegation-decision evaluations reportedly distinguish correct routing from successful completion and identify challenges in executing external work mid-dialogue.
2. Extend the capability registry with new registered skills and executors beyond the two core capabilities (multimodal and general) described.
3. Push long-context video interaction further: the memory module is presented as supporting hour-scale understanding, and its gating, retrieval, and reassembly components are natural targets for refinement.
4. Broaden full-duplex robustness across conversational roles, since continuation behavior was measured under three specific conditions (backchannels, other-directed speech, background speech) and interruption response to 75%.
5. The truncated portion of the paper covers the duplex and delegation data pipeline; the details of scenario planning, speech realization, and temporal alignment are the parts of the method not yet available in this content.

Target Audience

Researchers and engineers building real-time conversational agents — especially streaming audio-visual dialogue, full-duplex turn-taking, and tool-augmented spoken assistants. It is also relevant to product teams weighing the tradeoff between one frontend model and a split of conversational timing (frontend) versus task execution and reply wording (harness). Readers without a background in streaming multimodal architectures will find the model-design and training-recipe sections demanding.

Authors’ abstract

Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and linguistic cues, while video interaction also requires grounding the conversation in evolving visual context. We present Realtime-Venus, a proactive full-duplex interaction system with two separately trained 9B models: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction. Each model serves as a complete conversational frontend, integrating continuous perception, conversational control, and native speech generation through a shared causal timeline for user inputs, model outputs, and delegation events. A dual-loop runtime coordinates live interaction with background reasoning and tool execution. Foreground interaction continues while Realtime-Venus-Harness executes tasks asynchronously and returns results for integration into the ongoing dialogue. Both models follow a common post-training recipe combining offline understanding, proactive full-duplex trajectories, and delegation workflows. Among the evaluated online models, Realtime-Venus-Omni achieves the highest scores on six of eight video benchmarks, including StreamingBench (70.2%), OVO-Bench (64.7%), and Daily-Omni (81.3%). Across eight audio understanding and spoken question answering benchmarks, Realtime-Venus-Audio leads the compared models on MMAU (78.0%), MMAU-Pro (63.2%), Llama Questions (83.8%), and Speech CMMLU (67.8%), while matching the best VoiceBench AlpacaEval score of 4.81. On Full-Duplex-Bench v1.5, Realtime-Venus-Audio responds to 75% of user interruptions and achieves continuation rates of 97%, 88%, and 86% under backchannels, other-directed speech, and background speech, respectively, exceeding Gemini 3.1 Live and GPT-4o on all three continuation metrics.