StreamDuo: A Live Commentator and a Research Agent for Unseen Games

AI summary

Overview

Research area: Streaming video large language models (video LLMs) for automatic live game commentary, combined with retrieval-augmented generation and multi-agent tool use.

Technical level: Intermediate. The paper assumes familiarity with video LLMs, token-level language modeling losses, LoRA adapters, and KL-divergence-based distillation, but the central idea (a commentator that asks a separate agent for missing game knowledge) is easy to grasp without the math.

Scope (one sentence): The paper defines and studies cross-game streaming commentary — training a commentator on a single game and running it on games it has never seen — and introduces StreamDuo, a two-agent system in which a trainable commentator asks a frozen, wiki-connected research agent for entity identities using the raw video frames as the query.

What This Paper Is About

Streaming video models can now commentate live gameplay, but they are trained and evaluated on the same games, so they memorize the game world rather than learn to look things up. When retrained on only Baldur's Gate 3 and pointed at ten unseen games, existing commentators keep good timing (they still speak at the right moments) but lose knowledge: they rarely name what is on screen, and when they do they often substitute names from the training game (for example, calling a Tears of the Kingdom character "Karlach"). StreamDuo's goal is to let the commentator keep its transferable perception and timing skills while fetching game-specific facts at test time from the wiki of whatever game is being played.

Key Contributions

1. A new problem formulation and evaluation protocol. The authors define cross-game streaming commentary, introduce a source-game leakage metric that counts training-game names appearing in commentary on unseen games, and show empirically that current streaming commentators transfer timing but not world knowledge.

2. The StreamDuo architecture. A trainable commentator is paired with a frozen research agent (Qwen3-VL-8B with a BM25 search tool over the game's wiki). The commentator queries with the current frames rather than text, because a model that cannot recognize an entity cannot describe it well enough for a text query; the agent returns up to three entries two seconds later while the stream continues.

3. Information-gain ask labels. Instead of labeling every second where the human caster names an entity, the authors label an ask moment only where supplying an ideal answer measurably increases the model's own likelihood of producing the human's line (gain above γ = 1 nat). The decision to ask is thus model-dependent, not a fixed heuristic.

4. On-policy self-distillation for noisy answers (OPSD). A frozen copy of the model that sees the ideal answer acts as teacher for the student that sees a noisy agent answer, with contrastive amplification of the answer's effect and a reverse KL objective on the student's own samples, so the model learns to use a correct entry and ignore mismatched ones.

Main Findings

• Knowledge does not transfer, timing does. Trained on Baldur's Gate 3, Proact-VL reaches an Ent-R of 0.082 on ten unseen games, and VideoLLM-online names no correct entity at all despite speaking frequently. Zero-shot LiveCC names more entities (Ent-R 0.099) than the retrained specialists, but with a TimeDiff of 3.03 s.

• StreamDuo leads on nearly every metric. Against Proact-VL trained from the same base on the same clips, StreamDuo raises Ent-R from 0.082 to 0.116, Ent-F1 from 0.118 to 0.141, K-strict from 0.085 to 0.127, and improves timing (TimeDiff 1.76 vs 1.84; F1 0.552 vs 0.536). LLM-judge scores rise as well: PAUC 0.055 vs 0.050, LiveU 5.14 vs 4.91, FinalQ 3.53 vs 3.32. The exception is CC (0.120 vs 0.095 for Proact-VL, but 0.312 for the near-silent MMDuet2 trained baseline), which the authors note rewards near-silent models and does not reflect commentary quality.

• Leakage drops sharply. Over all 2,470 test clips and without a word cap, StreamDuo names 3,329 entities with 34 leaked Baldur's Gate 3 names (1.0%), versus Proact-VL with 1,811 mentions and 200 leaked names (9.9%) and StreamingVLM with 17.3%. In League of Legends, 54.6% of Proact-VL's names come from Baldur's Gate 3, against 7.8% for StreamDuo.

• A game dossier alone barely helps. Adding the research agent's game-level dossier to a baseline's context changes Proact-VL's K-strict by only 0.003 (0.085 to 0.088); it lists the game's entities but cannot say which one is on screen.

• Both training stages contribute. In the ablation, random renaming plus the dossier gives Ent-R 0.083 but raises LiveU from 4.91 to 5.03; adding asking (Stage 1) lifts Ent-R to 0.092 and K-strict to 0.094; adding OPSD (Stage 2) reaches 0.116 and 0.127. Overall the three components add 0.034 Ent-R and 0.042 K-strict over the Proact-VL recipe.

• Information-gain labels beat naive ones. Labeling every human entity mention instead of information-gain moments drops Ent-R from 0.092 to 0.079 and FinalQ from 3.48 to 3.38.

• Learned asking beats fixed schedules. Asking every 3 s inflates Ent-R to 0.199 and K-strict to 0.208, but Ent-F1 falls to 0.105 and judge scores drop (FinalQ 3.36), because many named entities do not fit the moment. The learned policy asks 264 times over 2,470 clips and achieves the best Ent-F1 (0.141). Never asking yields Ent-R 0.104 and FinalQ 3.54.

• Questions precede naming and answers are used immediately. Before a question the commentator names an entity in only 1 to 4% of seconds, below the 7% base rate of a random second; when the answer arrives two seconds later the share jumps to 32%, 4.6 times the random rate, stays at 22% one and two seconds after the answer and 12% five seconds later, and returns to baseline within about eight seconds. With asking disabled, those same seconds name an entity only 5% of the time.

• Stage 2's exact recipe matters. Fine-tuning on the research agent's real answers lowers Ent-R to 0.086, below Stage 1's 0.092. The forward KL variant reaches the highest Ent-R (0.131) and K-strict (0.145) but leaks names in 19.3% of mentions and has the lowest LiveU and FinalQ (4.82 and 3.31). Weaker teachers reach only 0.093 or 0.102 Ent-R. Only the full recipe (reverse divergence, α = 1, ideal answer) improves knowledge and judge metrics together without leaking.

• The research agent's size is not the bottleneck. Swapping Qwen3-VL agents from 2B to 30B-A3B at test time keeps Ent-R between 0.101 and 0.116 and judge scores within 0.03, even though the 2B agent answers only 18,103 of 80,529 queried moments versus about 50,000 for the larger models. Stage 2 alone (0.092 to 0.116) matters far more than agent scale.

• The system runs in real time. The research agent answers in 0.405 s on average and at most 0.713 s, comfortably inside the 2 s delivery delay; the commentator needs 0.256 s of compute per second of video, about four times faster than real time. StreamDuo can run on two NVIDIA H100 GPUs, one per agent.

Methodology in Plain English

The setup. The stream is processed one second at a time. Each second, the commentator sees the frames plus a short history of recent spoken lines, and a small head on the model's "flag" state outputs a probability of speaking now. A second head outputs a probability of asking. If it speaks, the language model generates a short line; if not, it stays silent. Speaking is gated at p > 0.27 and asking at q > 0.9, with a higher bar for continuing a run of speech than for starting one, so the model can stop once an event ends.

Why knowledge has to be external. During training, all source-game entity names are randomly renamed from clip to clip, so a name in training data is evidence only within its own clip. This blocks the shortcut of memorizing names but cannot supply names for a new game — hence retrieval at test time.

The two agents. The commentator handles perception and delivery, which transfer across games. The frozen research agent handles world knowledge: it reads the frames, proposes search keys (including any readable on-screen names), runs BM25 over the game's wiki of 190,722 pages, rechecks candidates against the frames, and returns up to three entries with a name, type and one-line description. The answer arrives two seconds later and is placed in the history, available for subsequent seconds — a deliberate asynchrony the commentator must learn to live with.

Stage 1 — learning when to ask. A frozen copy of the model scores the human caster's line twice, once with an ideal answer in the history and once without. The difference is the information gain; seconds where gain exceeds 1 nat are labeled as ask moments (specifically, the moment two seconds before, so the answer would arrive in time). Lines the model can already produce, or lines no answer could help with, get no label. The ask head is trained with a binary loss alongside the standard language-modeling and speak losses, with all parameters updated for one epoch at a learning rate of 10⁻⁵. The same stage exposes the model to answer formats: histories contain the correct entry mixed with distractors, and sometimes miss it entirely.

Stage 2 — learning to use the answer. Supervised training teaches the format of an answer but not how to judge it. Here the student sees the noisy answer exactly as at test time, while the teacher — the same model with its LoRA adapter switched off — sees the ideal answer. The teacher's predictions are contrastively amplified (α = 1) against its predictions with no answer at all, so tokens that become likely because of the answer are boosted. The student samples its own lines and minimizes the reverse KL to the teacher, which both corrects the lines it would actually produce (including wrong-entry repetitions) and forces commitment to a single name rather than spreading probability across all entries. Only rank-32 LoRA weights are updated for 300 steps, preserving the speaking behavior learned earlier.

Evaluation. Metrics cover timing (TimeDiff, F1), knowledge (entity recall, entity F1, and K-strict, which counts only lines requiring game knowledge, all capped at the first 126 words — twice the human average — so talking more cannot inflate recall), and LLM-judge scores from Qwen3-30B (PAUC, CC, LiveU, FinalQ). Baselines are run zero-shot and after one epoch on the same Baldur's Gate 3 clips.

Why This Matters

Research impact. The paper separates two capabilities that prior commentary benchmarks conflated — when to speak and what to say — and shows they behave very differently under domain shift. The source-game leakage metric gives the field a concrete way to measure hallucinated domain knowledge, and the information-gain labeling scheme offers a general recipe for training a model to know what it does not know, which the authors note streaming video models currently lack entirely.

Real-world applications:

• Automated esports and game-stream casting, where new titles, patches, champions and players appear constantly and no single trained model can keep up.
• Accessibility and localization, generating live descriptive commentary for viewers who cannot see the screen or who watch in another language.
• Game-streaming platforms and creator tooling, providing automatic commentary for the long tail of titles that no human caster covers.
• General streaming assistants that must reason about unfamiliar visual domains — live sports with unfamiliar teams, product streams, surveillance — where a wiki or knowledge base exists but the specific entity is novel.

Industry relevance. The system runs on two H100 GPUs at roughly four times real time, with an average research-agent latency of 0.405 s inside a 2 s budget, and the commentator asks only about once every six minutes. That is a practical operating profile, and the finding that research-agent scale from 2B to 30B barely changes outcomes means deployment does not require the largest available model.

Future Directions

• Handling unanswerable or partially answerable moments. The qualitative results show the commentator sometimes reads the wiki entry verbatim instead of commenting ("Rosalind Myers is the third and current president…") and sometimes repeats an unrelated entry, as with the Warden Support Alpha in StarCraft II. Better filtering and a notion of "no good entry exists" remain open.

• Going beyond entity naming. The current knowledge bottleneck is names of characters, items, places and skills. Rules, tactics, objectives and match context are not addressed, and the authors note that tactical remarks receive no ask label because no retrieved answer helps with them.

• Closing the timing gap between asking and needing. The fixed two-second delay is only viable because the agent answers in at most 0.713 s; learning when the answer must arrive, or allowing multi-turn exchanges with the agent, could let the commentator ask earlier for slower lookups.

• Extending beyond games and wikis. The recipe assumes a curated wiki and a game-level dossier. Whether the same ask/verify loop works with noisier sources or open web search — and whether the commentator could learn from its own confident mistakes rather than contrastive teachers — is untested.

Target Audience

Researchers and graduate students working on streaming video LLMs, online/continuous video understanding, and proactive response timing; practitioners building retrieval-augmented or tool-using agents for latency-constrained live settings; and game-industry or streaming-platform engineers interested in automated casting. It will also interest anyone studying domain generalization and hallucination, since the leakage analysis is a clean case study of a model falling back on memorized training-domain facts when its knowledge fails. Readers without background in video LLMs can follow the architecture and results but may need to consult the cited streaming-model papers (LiveCC, Proact-VL) for the underlying turn-by-turn formulation.

Authors’ abstract

Streaming video LLMs can now commentate live gameplay, but they are trained and tested on the same games. We study cross-game streaming commentary, in which a model trained on one game commentates games it has never seen, and find that current commentators transfer their timing but not their knowledge of the game world. We propose StreamDuo, which pairs a trainable commentator with a frozen research agent that reads the wiki of the game being played. When the commentator sees something it cannot name, it sends the current frames, and the research agent returns a few wiki entries two seconds later while the stream continues. We train only the commentator, on a single game, in two stages. Information-gain labels teach it to ask only where an answer would help it say what the human caster said, and on-policy self-distillation from a copy of itself that sees the ideal answer teaches it to use noisy answers. Trained on Baldur's Gate 3 and tested on ten unseen games, StreamDuo raises entity recall from 0.082 to 0.116 over Proact-VL trained on the same data, cuts source-game leakage from 9.9% to 1.0%, and achieves the best timing and the highest judged liveliness and overall quality among streaming models.