The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation

AI summary

Overview

Research area: Joint audio-video generation, specifically script-driven content creation with structured prompts — sitting at the intersection of diffusion/flow-matching generative models, multimodal conditioning, and cinematic post-production tooling.

Technical level: Intermediate to Advanced. The paper is written accessibly and the core idea is simple, but following the method requires familiarity with cross-attention, text encoders, LoRA adaptation, and rotary position encodings. The evaluation section, by contrast, is readable with only basic knowledge of video-generation benchmarks.

Scope (one sentence): The paper diagnoses a temporal misalignment between structured scripts and joint audio-video generators, and proposes Temporal Context Routing (TCR), a parameter-free attention bias that routes each prompt's timing into both the video and audio conditioning pathways, evaluated on 200 test scripts against four open-source joint generators and three specialized multi-shot video generators.

What This Paper Is About

Joint audio-video generators can produce visually convincing clips where sound and picture match each other, but they give weak control over when things happen — when a shot cuts, and when a line of dialogue is spoken. In script-driven generation, where a structured prompt assigns each shot and each dialogue line to a specific time span, this timing currently lives only in the text encoding and is never tied to the actual temporal coordinates of the video or audio latents. The result is a failure mode where video and audio are synchronized with each other yet both drift away from the script's timeline; TCR's goal is to make the structured script itself a third aligned modality.

Key Contributions

1. Identification of the temporal gap in script-driven generation. The authors articulate a specific failure mode: because shot and dialogue timing is encoded only in the prompt's text representation, it remains unaligned with the temporal coordinates of either modality, so video and audio may stay mutually synchronized while jointly deviating from the script timeline. The proposed remedy is to extend temporal alignment beyond video and audio to include the structured script.

2. Temporal Context Routing (TCR). A mechanism that maps script-specified timing onto the shared video-audio temporal axis. For each token's assigned interval, TCR computes a duration-normalized routing score and adds it as a bias to the video-text and audio-text cross-attention logits during both training and inference. Because the design is per-prompt and additive, overlapping shot and dialogue prompts remain independently controllable without modifying text, query, or key representations.

3. A coarse-to-fine data construction pipeline. A three-stage procedure that builds multi-shot training clips with dialogue spanning shot transitions: clip boundaries placed in speech-free regions using speech-band silence detection plus visual shot segmentation; coarse semantic annotation and timestamps from Gemini under a predefined script schema; and refinement using PySceneDetect for shot boundaries and WhisperX for word-level speech alignment, with final annotations rounded to a 0.1 s grid.

4. Evaluation across three complementary protocols. An end-to-end comparison against open-source joint generators, a controlled comparison against alternative temporal operators on the same backbone, and a blinded human study — together with ablations on temporal refinement and prompt sharing across modalities.

Main Findings

• Large reductions in shot-timing error. On 200 test scripts, TCR reaches a Shot Boundary MAE of 0.042 s versus 1.11 s for LTX-2.3, the strongest baseline on that metric — a 96% reduction. The paper notes that at 24 fps this is approximately one output frame.

• Shot structure is matched far more often. Shot IoU rises from 0.532 (LTX-2.3) to 0.957, and exact shot-count accuracy rises from 36.0% to 93.0%.

• Dialogue lands on time far more often. Dialogue Acc@0.5 s increases from 28.3% to 84.1%. Among the audio-generating baselines, OVI scores 8.6% and JoyAI-Echo 23.2% on this metric.

• Quality is not sacrificed. TCR records the highest IQ (0.7032) and Sync-C (2.78) and the lowest WER (8.48%) in the end-to-end table, while remaining competitive on the other quality and synchronization metrics. LTX-2.3 has a higher offset accuracy (31.2% versus 30.5%).

• TCR beats other temporal operators under matched conditions. With the same backbone, timing map, refined training data, and optimization settings, TCR reduces Shot Boundary MAE by more than 60% relative to both Gaussian Interval RoPE (0.113 s) and a hard interval mask (0.108 s), reaching 0.042 s, and also achieves the highest Dialogue Acc@0.5 s at 84.1% versus 83.7% (hard mask) and 82.8% (Gaussian Interval RoPE). The auxiliary reference "Intervals as text" reaches 0.601 s Shot Boundary MAE and 43.8% Dialogue Acc@0.5 s.

• Fine-grained supervision matters a great deal. Removing temporal refinement degrades Shot Boundary MAE from 0.042 s to 0.375 s and drops Dialogue Acc@0.5 s from 84.1% to 37.6%, while IQ and AES remain comparable — indicating the degradation is specific to temporal accuracy.

• Sharing both prompt types across both branches helps. A variant giving Shot prompts only to the video branch and dialogue prompts only to the audio branch ("separate A/V prompts") keeps shot timing close to TCR (0.047 s) but degrades Dialogue Acc@0.5 s, WER, Sync-C, and offset accuracy.

• TCR is also competitive against purpose-built multi-shot video generators. Against MultiShotMaster it improves Shot Boundary MAE from 0.054 s to 0.042 s, Shot IoU from 0.945 to 0.957, and exact shot-count accuracy from 79.0% to 93.0%. CineTrans obtains the highest VBench quality scores, while TCR ranks second on both IQ and AES.

• Temporal advantage is stable across training. TCR achieves the lowest Shot Boundary MAE and highest Shot IoU at every evaluated checkpoint, with the advantage already present at 3,000 steps and stable through 9,000 steps; the main tables report the 7,000-step checkpoint.

• Human raters prefer TCR on all five dimensions. In a blinded pairwise study with 16 randomly sampled cases (eight per comparator) and 28 participants, TCR receives 72.3% of tie-inclusive overall-preference votes against LTX-2.3 and 83.9% against JoyAI-Echo.

Methodology in Plain English

The authors start from an existing 22B-parameter joint audio-video generator, LTX-2.3, which has separate text cross-attention modules feeding its video and audio towers, plus an audio-video cross-attention pathway that keeps the two modalities in sync with each other. They leave the self-attention and the audio-video cross-attention untouched, and modify only the two text cross-attention modules.

The script is organized as a structured prompt with four prompt types — Reference (recurring people, scenes, objects), Shot (shot content and camera attributes), Event (temporally localized audio events, here spoken dialogue), and Global (clip-wide context) — adapted from the MTSS schema, with the Subtitle stream omitted so burned-in text transcriptions do not condition the model. Each prompt is assigned an interval, and all tokens inheriting from that prompt receive that interval.

For each token interval, TCR computes a center and a radius, then a routing score that is zero at the interval's center and reaches −β/2 at each endpoint, where β = 5 (giving an endpoint score of −2.5). Because the score is normalized by the interval radius, prompts of different durations get the same relative routing profile. This score is added to the ordinary semantic cross-attention logit. Mathematically, adding the routing score to the logit is equivalent to multiplying the unnormalized attention weight by an exponential factor — so TCR reweights attention without altering the text, query, or key representations at all. Tokens with no assigned prompt interval receive a routing score of zero. The same routing score is computed separately at the video and audio temporal coordinates, even though the two modalities use different latent grids, because both are expressed in seconds against the same clip timeline.

TCR introduces no learnable parameters, so only LoRA adapters are trained, under the original joint flow-matching objective. Because timing fields are stripped out of the serialized script and supplied separately through a token-level timing map, timing enters the model as an explicit control signal rather than as text the model must infer from.

Training data comes from a three-stage pipeline: clip construction places boundaries in speech-free regions while preserving internal shot transitions; Gemini annotates each clip into the schema with coarse timestamps; and refinement replaces coarse shot boundaries with PySceneDetect-detected cuts (retaining only scripts whose annotated Shot prompt count matches the detector's count) and replaces dialogue timing with WhisperX word-level alignment, with all timestamps rounded to a 0.1 s grid. This yields 57,022 training examples from two short-drama collections.

Evaluation uses 200 test scripts containing 640 Shot prompts and 441 dialogue prompts, with no test script sharing a source-media identifier or caption hash with training. Every model generates at 704 × 1280 resolution and 24 fps without first-frame conditioning, and metrics are averaged over the 200 outputs from each model.

Why This Matters

Impact on research. The paper reframes a problem that is easy to overlook: mutual audio-visual synchronization is not the same as script fidelity. By showing that a structured script can be treated as an additional modality to align against, it opens a direction where the conditioning signal's timing is a first-class object rather than something the text encoder must implicitly encode. The result that a parameter-free additive attention bias outperforms both a hard interval mask and an interval-aware RoPE variant is a useful data point for the broader literature on temporal conditioning. The ablation showing that refinement quality (0.1 s grid annotations) matters more than the choice of operator — 0.042 s versus 0.375 s Shot Boundary MAE — also puts weight on data construction as a research contribution in its own right.

Real-world applications:

• Short-form drama production, which the paper names explicitly as a target application alongside advertising, where a screenplay is converted by an LLM into a structured representation and then generated.
• Advertising and promotional content built to a timed script, also named in the paper.
• Any pipeline where a structured, timed script must be realized as finished audio-video output rather than described loosely in a text prompt.
• Storyboarding-style iteration on multi-shot sequences, where the paper's shot-count accuracy and shot-boundary metrics translate directly into whether a generated clip can be cut where the editor intended.

Industry relevance. The paper is a collaboration across PKU, Qwen Applications, HKUST, CUHK, UChicago, and SJTU, with corresponding authors affiliated with PKU and Qwen Applications, and it builds on the MTSS schema from the Tencent Hunyuan Team. The framing — moving generative video "beyond isolated clip synthesis toward structured content-production workflows" — is aimed at the production-tooling side of the field, where controllability and adherence to a specified timeline matter more than raw visual novelty. Since TCR adds no learnable parameters and modifies only two cross-attention modules, it is architecturally cheap to graft onto an existing joint generator.

Future Directions

• Beyond spoken dialogue. The paper states that its Event prompts describe temporally localized audio events, "with our experiments focusing on spoken dialogue." Extending routing to non-speech events such as sound effects is an obvious open question that the paper leaves unaddressed.
• Harder script structures. The study uses 16 cases for human evaluation and 200 scripts for automatic metrics. Whether the temporal advantage holds for longer clips, more shots, denser overlapping dialogue, or scripts where shot and dialogue boundaries conflict more aggressively is not reported.
• Dependence on the annotation pipeline. The refinement ablation shows large sensitivity to annotation quality, and the pipeline relies on Gemini, PySceneDetect, and WhisperX, with a consistency check that discards scripts whose Shot prompt count disagrees with the detector. How performance degrades on material where that check cannot be satisfied is not reported.
• A principled choice of routing strength. The paper fixes β = 5 throughout and defers sensitivity analysis of this parameter to Appendix G, so the behavior of the routing profile at other strengths — and whether the optimal value is dataset- or duration-dependent — remains a question the work raises rather than settles.

The paper's conclusion does not itself enumerate a future-work agenda, so the directions above are framed as open questions the results raise rather than commitments the authors make.

Target Audience

Researchers and engineers working on generative video, joint audio-video synthesis, and multimodal conditioning — particularly those interested in controllability, temporal grounding, and long-form or multi-shot generation. It will also be useful to practitioners building script-driven production tools who need to know whether a generative model can honor a timeline, and to readers studying attention-level conditioning mechanisms, since TCR is a clean, parameter-free case study of routing a control signal through cross-attention. Readers without any background in diffusion or transformer attention will find the metrics and results sections accessible but the method section steep.

Authors’ abstract

Joint audio-video generation models have made substantial progress in visual quality and audio-visual synchronization. However, they still provide limited control over when shot transitions occur and dialogue is spoken. This limitation constrains their application in script-driven content creation, where timing errors can undermine narrative coherence and the viewing experience. Current joint generators align video and audio representations on a shared temporal axis, yet the precise timing of shots and dialogue specified in a structured prompt is encoded only in the prompt's text representation and remains unaligned with the temporal coordinates of either modality. Consequently, video and audio may remain synchronized with each other while both fail to follow the script timeline. This mismatch motivates us to extend temporal alignment beyond video and audio to include the structured script. We therefore introduce Temporal Context Routing (TCR), which maps the script timing onto the shared temporal axis of video and audio generation and routes each prompt's guidance to the corresponding positions in both modalities. Compared with the baseline on 200 test scripts, TCR reduces Shot Boundary MAE by 96%, from 1.11 s to 0.042 s, and raises Dialogue Acc@0.5 s from 28.3% to 84.1%. TCR achieves these improvements while maintaining visual quality and audio-visual synchronization comparable to those of the baselines. A user study further shows that participants prefer TCR on all five evaluated dimensions.