Behind Video Generation: Observations and Insights on Multimodal Understanding from Kling

AI summary

Overview

Research area: Multimodal understanding (multimodal large language models) as the upstream infrastructure behind controllable video generation — captioning, prompt enhancement, text encoding, reward modeling, and automatic evaluation.

Technical level: Advanced. The report assumes familiarity with diffusion transformers (DiTs), mixture-of-experts architectures, reinforcement learning for model refinement, and video-generation benchmarking.

Scope: A technical report from the Kling team arguing that video generation quality is the projection of multimodal understanding through an entire data–conditioning–alignment–evaluation chain, examined module by module across successive Kling generations.

Note on coverage: the supplied paper text is truncated partway through Section 3.3. Sections on prompt enhancement, text encoding, reward modeling, and task-specific evaluation are therefore summarized here only at the level of detail given in the abstract and introduction.

What This Paper Is About

As video generation has moved from isolated clips to coherent multi-shot audiovisual sequences, the hard problem has shifted from rendering pixels to controlling semantics — deciding what to generate, what to preserve, and how to organize it over time. This report asks which kinds of multimodal understanding actually reach the generator, through what representation, and with what evidence of downstream benefit, rather than simply asking whether a multimodal model scores well on general benchmarks. It traces that question through Kling's own production stack, from the shared Kling-MLLM foundation model to captioning, prompt enhancement, text encoding, reward modeling, and evaluation.

Key Contributions

1. A chain-level framing of generation quality. The paper argues that generation quality is the projection of multimodal understanding through the whole data–conditioning–alignment–evaluation chain, so a visible generation failure is a symptom rather than a diagnosis — the same artifact could come from a caption that omits supervision, a conditioning representation that discards temporal structure, a reward model that favors smooth but static motion, or an evaluator that never tests the requested event.

2. Kling-MLLM, a shared omni-modal foundation model specialized per module. A single autoregressive backbone — a dense 27B variant plus 35B-A3B and 397B-A17B mixture-of-experts variants, initialized from Qwen3.5–Qwen3.8 checkpoints — unifies images, videos, audio, synchronized audio–video, and arbitrarily interleaved media, then is post-trained into captioners, prompt enhancement models, text encoders, reward models, and evaluators.

3. Captioning as a semantic contract, including reference-grounded captioning. Because a DiT is trained on far fewer samples than a language model, the paper treats description density per unit of video duration as the central measure of data efficiency, specifies six information families a generation-oriented caption must carry, and extends captioning so that reference bindings (which subject, which image, which identity) are made explicit.

4. Module-specific engineering lessons and a benchmark suite. Reliable text encoders come from perceptual training rather than stronger language ability; candidates can be screened from text alone before costly end-to-end training; reward modeling pairs a general reward model for explainable localized critiques with a decision reward model for fast multidimensional scoring; and evaluation should be task-specific and decomposed so failures can be traced upstream.

Main Findings

• The bottleneck is shifting from rendering to semantic control. Visual fidelity improvements do not automatically produce correct actions, camera language, reference binding, or narrative continuity. High aggregate VQA scores do not imply reliable understanding of camera language, left–right relations, composition, lighting, shot boundaries, identity binding, or dense temporal change — capabilities the paper describes as weakly represented in standard supervision.

• Data efficiency depends on description density. Both training throughput and the supply of usable video preclude closing the sample gap between a DiT and a language model through scaling, so each training video must carry as much supervision as possible. Dense generation-oriented captions and reference-grounded captions with explicit bindings are designed to maximize density, and a perception-tuned text encoder ensures that supervision actually reaches the generator.

• A mature DiT has a sweet spot in its caption distribution. Trained on dense captions, the generator responds best to conditions written in that distribution. Basic prompt enhancement translates user requests into it and is refined with text-level and generator-in-the-loop reinforcement learning; agentic prompt enhancement divides the work between story refinement and retrieval of captions from compelling videos, which the paper says improves reasonableness, faithfulness, and vividness together where a single model faces conflicting goals.

• Temporal grounding needs explicit timestamps, not implicit token position. The paper reports that prompting experiments with Gemini APIs showed that stating total audio duration markedly improves temporal grounding. Kling-MLLM therefore brackets each audio span with start and end timestamps and inserts absolute-time anchors every five seconds, aligning audio with the shared timestamped timeline used for synchronized audio–video.

• Audio representation is scale-dependent but interface-stable. Smaller variants use a Qwen3-Omni-style tower encoding a 16-kHz monaural waveform as 13 audio tokens per second. Larger variants use a dual-branch Mel–VAE tower fusing a 16-kHz, 128-bin Mel spectrogram at 100 frames per second with a 44.1-kHz waveform for an in-house VAE, projecting to 25 audio tokens per second — the same embedding interface either way.

• A four-stage curriculum, not four disjoint corpora. Stage 1 trains only the audio tower with vision and language frozen; Stage 2 unfreezes all three modules on a mixture dominated by audio and synchronized audio–video; Stage 3 moves to long sequences and shifts toward temporal perception; Stage 4 freezes both modality towers and updates only the LLM on the highest-quality data. The paper also reports that merging the Stage 1 and Stage 4 checkpoints yields a better checkpoint than either alone.

• Linguistic fluency conceals supervision errors. In the paper's example, two captions of a 17.7-second clip from Meridian are compared: the fluent caption mislabels a hard cut as a fade, omits the title's timing, misses pre-appearance dialogue, and leaves visible text untranscribed — errors that lexical-overlap metrics can reward.

• Benchmark results are genuinely mixed, not a clean sweep. Kling-MLLM-397B-A17B records the lowest LibriSpeech clean WER (1.18), the strongest Chinese LATBench results, the lowest video-SALMONN total error, both OmniCap-IF metrics, and 61.05 on IV-Bench and 63.16 on MVU-Eval — within 0.36 and 0.55 points of the best reported results. It also leads DSI-Bench, Event-Bench, MotionBench (57.82), Camera Motion VQA (54.72), and VidCapBench coverage (93.81). Meanwhile Doubao2.0-Lite in thinking mode leads MMBench-EN, PerceptionBench, LongVideoBench, MME-VideoOCR, ShortVid-Bench, all TimeLens scores, and VidCapBench accuracy; the paper states that image, speech recognition, video captioning, and multimodal interleaving results remain mixed across systems.

• Orchestrated pipelines beat scaling a single annotator. The paper argues that explicitly mining previously unobserved dimensions and resolving inter-expert conflicts expands semantic coverage and accuracy far more effectively than scaling a general-purpose annotator, then distilling the multi-expert output into a single-pass student model for deployment.

• Caption quality cannot be inferred from VQA performance. Citing CAPability, the paper notes a "know but cannot tell" gap: models can verify a specified detail on demand while still failing to decide autonomously what a caption should state.

Methodology in Plain English

The team starts from a single large multimodal model rather than a separate model per job. Everything — text, images, video frames, audio, and combined audio–video — is converted into one interleaved token sequence with explicit boundary markers and physical timestamps, so the language model can reason about "when" as well as "what." Videos are sampled at a frame rate and frame count chosen for their duration and the available context; audio is either encoded as a Mel spectrogram or, in bigger models, as a fused Mel-plus-waveform representation; synchronized audio–video is serialized into one interleaved stream in which each audio token sits before the visual block carrying the same timestamp.

That base model is then trained in four stages that gradually widen multimodal supervision and lengthen context: audio first (with vision and language frozen), then joint multimodal training, then long-sequence temporal perception, then a final long-context pass over only the cleanest data with the encoders frozen.

For data, the team runs an automated pipeline that discovers public datasets, filters out sensitive, license-incompatible, or evaluation-overlapping sources, converts each into a common conversation-style schema, and audits the result with an automatic revision loop (question sampling, perplexity checks with Qwen3-Omni, multimodal review with Gemini) before human sign-off. Internal Kling task data and agent/tool-use data are added on top.

For captioning, the team replaces "write a good description" with an explicit contract: a table of six information families (subject and action; space and composition; cinematography and light; time and shots; visible and audible text; aesthetics), each paired with the downstream failure that occurs if it is missing, plus four formatting rules — stable explicit entity naming, monotonic validated shot timestamps, faithful transcription of visible text, and audio described at the point where it occurs. Because producing such captions requires several specialized models plus a conflict-resolution layer, the resulting offline "factory" is distilled into a single-pass student model for production use.

Why This Matters

Impact on research. The paper reframes multimodal understanding as infrastructure rather than a leaderboard target, and it supplies a concrete argument for why general-purpose VQA benchmarks under-measure what video generation needs. Its claim that a text encoder's usefulness comes from perceptual training rather than raw language ability, and that candidates can be screened from text alone before end-to-end training, suggests a cheaper experimental loop for anyone building conditioning stacks.

Real-world applications (as grounded in the paper's stated scope):

• Short-form video and advertising production, where multi-shot sequences with persistent subjects, camera language, dialogue, music, and sound must be specified and controlled rather than merely rendered.
• Avatar and performance content, via Kling AI Avatar, where identity binding and motion fidelity depend on reference-grounded conditioning.
• Motion transfer and precise editing, via Kling Motion Control, where the system must understand what to preserve as well as what to change.
• Automated captioning, QC, and retrieval over video archives, using the density-oriented caption schema and the task-specific evaluation protocols the paper describes.

Industry relevance. The report is an unusually candid account of production trade-offs: why dense captions are a data-efficiency strategy rather than a quality nicety, why agentic decomposition beats a single prompt enhancement model with conflicting objectives, why perceptual bandwidth should be treated as a first-class design parameter in reward models, and why evaluation protocols must be designed to expose scoring shortcuts. Each of these is a decision point that teams building generative video systems face directly.

Future Directions

• Tracing failures to their upstream source at scale. The paper's central premise is that a failure can originate anywhere in the chain. Turning that from a diagnostic principle into a routine instrument — automatically attributing an artifact to captioning, conditioning, reward modeling, or evaluation — remains an open engineering problem.

• Extending reference grounding and multi-shot reasoning. Reference-grounded captioning and explicit reference bindings are presented as the current answer for identity and subject continuity; how far this generalizes to long sequences with many recurring subjects and interacting constraints is not settled.

• Making evaluation resistant to shortcuts. The paper calls for task-specific, decomposed protocols that expose scoring shortcuts, and its own results show a mixed picture across systems and benchmarks. Whether decomposed, directly observable criteria can become standard practice — rather than aggregate scores — is unresolved.

• Preserving factory-level quality in distilled captioners. Distilling a multi-expert pipeline into a single-pass student reduces latency, but the paper does not report how much semantic coverage or conflict resolution is lost in that compression, which is a natural next question.

Target Audience

Researchers and engineers building generative video or image systems who need to design the understanding layer — captioning schemas, prompt enhancement, text encoders, reward models, and evaluation — rather than only the generator. It is also useful for multimodal LLM researchers interested in omni-modal architectures, timestamped audio–video serialization, and multi-stage training curricula, and for evaluation researchers concerned with how benchmark scores diverge from production requirements. Readers looking for a self-contained model card or reproducible ablations will find the report's framing and engineering lessons more useful than its truncated experimental detail.

Authors’ abstract

Video generation has progressed from isolated clips to coherent short-form audiovisual sequences, and the bottleneck is shifting from rendering to semantic control. This report summarizes the observations and insights on multimodal understanding gained from successive generations of Kling. We argue that generation quality is the projection of multimodal understanding through the entire data--conditioning--alignment--evaluation chain, and we examine this chain module by module. Kling-MLLM, a shared multimodal foundation model that unifies image, video, audio, synchronized audio--video, and interleaved inputs, is specialized for each downstream module. Because a DiT is trained on far fewer training samples than a language model, we treat captioning as a semantic contract that maximizes description density, and extend it to reference-grounded captioning that makes reference bindings explicit. Prompt enhancement translates user intent into this caption distribution: basic PE is derived from captions of compelling videos and refined with text-level and generator-in-the-loop reinforcement learning, while agentic PE refines the story and retrieves compelling examples. We show that a reliable text encoder comes from perceptual training rather than from stronger language ability, and that candidates can be screened from text alone before end-to-end training. Reward modeling pairs a general reward model for explainable, localized critiques with a decision reward model for fast multidimensional scoring, treating perceptual bandwidth as a first-class design parameter. Finally, we review task-specific automatic evaluation, focusing on protocols that expose scoring shortcuts and help trace failures to their upstream source. We hope these lessons offer practical guidance for building the understanding stack behind controllable video generation.