DirCast: Live Esports Directing and Commentary with a Single Streaming Video-Language Model

AI summary

Overview

Research area: Computer vision and streaming video-language models, applied to live esports broadcast production (automatic camera directing plus live commentary).

Technical level: Advanced. The paper assumes familiarity with video-language models, token streams, KV caches, LoRA fine-tuning and imitation-learning concepts such as exposure bias.

Scope: The paper introduces the task of "live broadcast production" — choosing which of ten player cameras goes on air each second while simultaneously generating commentary — and presents DirCast, a single streaming video-language model that does both, along with a dataset of 27 professional Dota 2 matches.

What This Paper Is About

Live esports broadcasts normally require two skilled people: a director who picks which camera feed goes on air every second, and a caster who talks over the resulting footage. Existing streaming commentary models assume the directing is already done and only commentate over a finished human-produced broadcast. This paper asks a model to take the raw camera feed of every player and produce both the shot selection and the commentary itself, using only past observations and without knowing what happens next.

Key Contributions

1. A new task and dataset: The authors define live broadcast production and build a dataset of 27 professional Dota 2 matches, with a rendered camera per hero, the human director's on-air choices, transcribed and polished caster commentary, per-hero game state, and a stream of game events.

2. DirCast, a single streaming model for both roles: A causal video-language model that scans all ten feeds at low resolution each second, decides with a hold-or-switch director head whether and where to cut, reads the chosen camera at high resolution, and then speaks over its own choice.

3. Program-as-memory context design: Low-resolution feed views expire after a fixed window, while on-air views, events and spoken words persist, which bounds per-second cost over streams of any length and allows real-time operation on a single GPU.

4. Training and inference recipes for self-directed operation: Perturbed program rollouts, event-anchored directing targets that credit cuts to kill participants, and adaptive thresholds that track the decision rates of human crews, together making the model robust when it runs on its own output.

Main Findings

• Directing agreement: DirCast reaches a Top-1 of 26.95, a tolerance-based Top-1 of 32.0 and an Any-visible score of 41.81, ahead of the strongest baseline, an action-based rule at 25.43 and 36.11, and ahead of every zero-shot vision-language model tested.
• Zero-shot VLMs direct poorly: None of the eight zero-shot models beats the action rule on agreement, scale does not help within a family, and GPT-5 reaches a Top-1 of only 13.53 — lower than the static never-cut rule at 17.1.
• Cut rhythm: DirCast cuts 6.7 times per minute against 7.9 for the human crew, while zero-shot models cut 20 to 56 times per minute with cut precision between 6 and 9 percent.
• Kill coverage: DirCast puts 89.1 percent of kills on air, more than any baseline and more than the human director at 50.3 percent; its fight coverage is 41.1, ahead of the human director's 37.4 but below the action rule's 62.4.
• Commentary win rate: Under the Proact-VL protocol judged by GPT-5, DirCast wins 29.21 percent of comparisons against GPT-4o, versus 19.38 percent for the strongest of eleven streaming commentators (LiveCC-7B-Base), even though every baseline watches the human broadcast while DirCast comments on its own program.
• Speaking timing and delivery: DirCast starts speaking 0.30 seconds after the human casters on average and leads all three live-delivery scores plus PAUC at 0.024 against 0.015 for the best baseline.
• Where DirCast does not lead: Proact-VL obtains the highest speaking F1 at 83.21 by speaking in 99.9 percent of seconds, while DirCast speaks in 86.6 percent and reaches 78.22; Fidelity instead favors very sparse speakers such as MMDuet at 10.1 percent, though among models speaking in more than half of seconds DirCast has the highest fidelity at 4.97 against 4.54 for Proact-VL.
• Joint beats separate: Training one shared model beats two role-specific models on both jobs — Top-1 of 27.00 against 25.17, commentary win rate of 29.21 against 24.36 — while halving memory to 18.9 GB and running at 0.32 s per video second against 0.53 and 0.56.
• Directing improves commentary: On the same human program, the joint model still wins 27.84 percent against 25.67 for a commentary-only model.
• Ablations: Removing the hold-or-switch pointer costs the most, dropping Top-1 from 27.00 to 21.65; removing state tokens drops it to 22.59; removing program rollouts drops it to 22.90 and also lowers commentary win rate to 28.35; removing event-anchored targets keeps Top-1 comparable at 26.10 but raises Any-visible from 41.80 to 43.06.
• Long streams: On eight continuous 288-second windows, keeping every view costs a P95 of 1.05 s per video second and a 92.6k-token cache; a 36-second horizon keeps the cache at 6.9k tokens, memory at 18.9 GB and cost flat at 0.29 s, and raises Top-1 from 23.7 to 25.1 and speaking rate from 67.2 to 76.7.
• Scaling with data: Going from 5 to 10 training matches gives small gains of 0.98 in Any-visible and 0.56 in Top-1, while doubling to 20 gives 5.56 and 6.36, reaching 41.80 and 27.00, with no sign of saturation.

Methodology in Plain English

The authors start from an off-the-shelf video-language model (Qwen3-VL-8B) and fine-tune it with LoRA for three epochs on eight H100 GPUs. Each second is fed to the model as one fixed block of tokens: the game clock and events, ten small camera thumbnails each preceded by a token summarizing that hero's game state, a director token, then the chosen camera again at higher resolution, then a commentary token and the words spoken.

Directing is split into two simpler decisions instead of one ten-way classification. A gate estimates whether to leave the current shot at all, and a pointer ranks the other cameras. Holding is the default, so the model does not flicker between feeds.

Memory is handled by keeping only what a human crew would remember: the shots that actually aired and the words spoken over them. The ten thumbnail views expire after eight seconds, and on long streams any second older than a training clip leaves the cache, so per-second cost stays flat.

To prepare the model for its own mistakes, the researchers deliberately corrupt the on-air camera during training for a few seconds at a time while the targets remain human, so the model learns to cut back and to commentate over whatever is showing. They also ground each kill to the camera of the hero who made it, and accept a cut to a killer or victim as correct even when the human director stayed elsewhere. At inference, the cut and speech thresholds drift based on how often the model has recently been cutting and speaking compared with the measured rates of human crews.

The dataset is built by rendering ten hero-following cameras from each match replay, aligning renders to the official broadcast using the on-screen game clock (residual about 0.3 s), labeling each second with the hero the broadcast is showing, and transcribing the caster audio with Whisper, then polishing it with GPT-5-mini under strict guards. Clips are 36 seconds at one frame per second; 20 matches are used for training, 3 for calibration and 4 for testing.

Why This Matters

Impact on research: The paper moves streaming commentary from a solved-input problem to a coupled production problem, where the model's own directing decisions change what it later sees and describes. It shows that a shared model beats two separately trained specialists on both roles, and it provides a benchmark with all camera views preserved rather than only the aired program.

Real-world applications:

• Broadcasting the long tail of matches, from regional leagues to amateur games, that currently get a fixed camera and no commentary.
• Assisting human production crews with automated shot suggestions and draft commentary during busy moments.
• Automated highlight and clip generation, since the model already identifies which camera shows a kill and names the participants.
• Extending the same directing-plus-narration pattern to other multi-camera settings such as sports, lectures or live events.

Industry relevance: DirCast runs in real time on a single GPU with bounded memory, which is the practical requirement for live deployment. Its higher kill coverage than the human crew (89.1 percent against 50.3 percent) suggests value as a tool that catches action the broadcast missed rather than only as a cost reduction.

Future Directions

• Scaling the dataset: The scaling curve shows no saturation, and new matches need only camera renders and the aired broadcast with no manual labels, so collecting hundreds of matches is the obvious next step.
• Beyond a single game and a single title: The work covers Dota 2 only, with ten fixed hero cameras; whether the hold-or-switch director and program memory transfer to other esports or to physical multi-camera sports is untested.
• Improving commentary metrics and judging: The authors note that F1 rewards speaking in nearly every second and Fidelity rewards near-silence, and that GPT-5.1 was unavailable so GPT-5 judged instead; better evaluation of live commentary quality is an open problem.
• Closing the remaining gaps: Event-anchored targets trade human agreement for event coverage (Top-1 26.10 versus 27.00, Any-visible 43.06 versus 41.80), and the model still trails the action rule on fight coverage (41.1 versus 62.4), so balancing imitation against event-driven cutting remains open.

Target Audience

Researchers and engineers working on streaming video-language models, online video understanding and real-time multimodal systems; esports and sports broadcast technologists interested in automated production; and practitioners studying imitation learning under exposure bias, particularly those applying on-policy rollouts and adaptive inference thresholds to sequential decision tasks.

Authors’ abstract

A live esports broadcast needs a director who chooses which camera goes on air and a caster who talks over that choice, every second and without knowing what comes next. Streaming video language models have learned to commentate, but only over a broadcast that a human director has already produced. We study live broadcast production, in which a model receives the raw camera of every player and must both direct and commentate from the past alone. We present DirCast, a single streaming video language model that does both. Every second it scans all cameras at low resolution, decides with a hold-or-switch director whether and where to cut, and reads the chosen camera at high resolution before it speaks. Its context keeps what went on air and what was said and lets the other views expire, which bounds the cost of a stream of any length. We build a dataset of 27 professional Dota 2 matches with a camera for every hero, the human director's choices and the casters' commentary. DirCast directs better than rule-based and zero-shot directors including GPT-5, and its commentary wins 29.2 percent of comparisons against GPT-4o under the Proact-VL protocol, against 19.4 percent for the strongest of eleven streaming commentators that watch the human broadcast.