When Numbers Speak: Aligning Textual Numerals and Visual Instances in Text-to-Video Diffusion Models

AI summary

Overview

Research area: Text-to-video (T2V) diffusion models, specifically numerical alignment between prompt numerals and the number of visual instances generated.

Technical level: Advanced. The paper works directly with Diffusion Transformer (DiT) internals — multi-head self- and cross-attention maps, PCA projections, attention head scoring, clustering, and inference-time attention modulation.

Scope: The authors introduce NUMINA, a training-free "identify-then-guide" framework that detects count mismatches from a model's own attention maps, corrects the resulting layout, and steers regeneration so the video contains the requested number of objects, evaluated on a new 210-prompt benchmark called CountBench across Wan and CogVideoX models.

What This Paper Is About

Text-to-video diffusion models generate visually strong videos but frequently produce the wrong number of objects when a prompt specifies a count (e.g., asking for three dogs and getting two). The authors probe a representative model, Wan2.1-1.3B, and attribute the failure to two factors: numerals receive diffuse, low-contrast cross-attention compared to nouns, verbs, and adjectives, and the heavily down-sampled spatiotemporal latent space makes individual object instances hard to separate. The goal is to fix counting at generation time without retraining, without input videos, and without external segmentation masks.

Key Contributions

1. An empirical finding about attention: the authors show that attention in T2V models already exposes visual information related to the number of instances, and that instance-separable information is concentrated in a small number of attention heads rather than distributed across all of them.
2. A training-free framework (NUMINA): an identify-then-guide pipeline that selects discriminative self- and cross-attention heads to build a countable latent layout, conservatively refines that layout under count constraints, and modulates cross-attention during regeneration to enforce the corrected count.
3. The CountBench benchmark: 210 prompts covering instance counts from 1 to 8 and compositions of 1 to 3 object categories, created with GPT-5 prompt candidates followed by manual review to remove repetitive or illogical prompts.
4. Demonstrated gains and compatibility: counting accuracy improvements across Wan2.1-1.3B, Wan2.2-5B, Wan2.1-14B and CogVideoX-5B, plus compatibility with seed search, prompt enhancement, and the EasyCache inference-acceleration technique.

Main Findings

• Counting accuracy on Wan models: On CountBench, NUMINA raises CountAcc from 42.3% to 49.7% (+7.4) on Wan2.1-1.3B, from 47.8% to 52.7% (+4.9) on Wan2.2-5B, and from 53.6% to 59.1% (+5.5) on Wan2.1-14B.
• Baselines for comparison: Seed search reaches 45.5% (+3.2) on the 1.3B model, 48.8% (+1.0) on the 5B model, and 56.1% (+2.5) on the 14B model. Prompt enhancement reaches 47.2% (+4.9), 49.0% (+1.2), and 56.9% (+3.3) respectively.
• Small model surpasses larger baselines: NUMINA enables the 1.3B model (49.7%) to exceed the counting accuracy of the much larger Wan2.2-5B baseline (47.8%).
• CLIP alignment improves: CLIP score rises from 33.9 to 35.6 (+1.7) on Wan2.1-1.3B, from 34.3 to 34.7 (+0.4) on Wan2.2-5B, and from 34.2 to 34.4 (+0.2) on Wan2.1-14B.
• Temporal consistency is maintained: TC improves from 81.2 to 83.4 (+2.2) on Wan2.1-1.3B, stays at 85.0 (+0.0) on Wan2.2-5B, and improves from 83.3 to 84.0 (+0.7) on Wan2.1-14B, despite the method actively adding and removing objects.
• Per-numeral breakdown (Wan2.1-1.3B): the baseline already scores 68.7% for two objects, then falls to 44.5% for three objects, where NUMINA achieves a 16.2% improvement. At eight objects, the baseline drops to 11.3% while NUMINA nearly doubles it to 20.7%.
• Attention-derived layout beats a detector-derived layout: using GroundingDINO on the same frames yields 47.5% (+5.2) CountAcc and 82.8% TC, while the attention-derived layout yields 49.7% (+7.4) and 83.4%, a 2.2% advantage attributed to the attention layout being native to the DiT latent.
• Head selection matters: random single-head selection gives 44.1% (+1.8), averaging all heads gives 43.0% (+0.7), Top-3 gives 48.2% (+5.9), Top-2 gives 49.4% (+7.1), and Top-1 gives 49.7% (+7.4), supporting the claim that instance-separable information is a sparse property held by few heads.
• Layout refinement costs are complementary: overlap cost only gives 45.1% (+2.8); overlap plus center cost gives 46.9% (+4.6); overlap plus temporal cost gives 48.9% (+6.6); all three give 49.7% (+7.4).
• Timestep choice: CountAcc rises quickly and reaches 49.7% at reference timestep 20; increasing to timestep 40 yields only a 3.2% relative gain over timestep 20 while doubling pre-generation cost, and beyond 40 accuracy declines.
• Generalization to a different architecture: on CogVideoX-5B, which uses an MMDiT with unified attention rather than a dedicated cross-attention module, CountAcc goes from 40.2% to 44.4% (+4.2), TC from 78.1% to 80.2% (+2.1), and CLIP from 34.8 to 35.4 (+0.6). Seed search and prompt enhancement give only +2.5% and +2.3% CountAcc on this model.
• Combination with existing strategies: combining NUMINA with seed search and prompt enhancement reaches 54.2% CountAcc (+11.9), 83.6% TC (+2.4), and 35.5 CLIP (+1.6), surpassing the standalone NUMINA (49.7%), and exceeding larger baselines including Wan2.2-5B at 47.8% and Wan2.1-14B at 53.6%.
• VBench Subject-Consistency: scores change from 83.1 to 83.6 on Wan2.1-1.3B, 84.3 to 84.7 on Wan2.1-14B, 83.4 to 83.5 on Wan2.2-5B, and 84.6 to 84.6 on CogVideoX-5B.
• Inference overhead: Wan2.1-1.3B runs at 292 s wall-clock with 14.3 GB VRAM and 42.3% CountAcc; NUMINA at 431 s and 16.3 GB with 49.7%; NUMINA plus EasyCache at 355 s and 16.3 GB with 49.4% CountAcc.
• No-reference addition case: when no reference instance exists for a category, the no-intervention baseline achieves only 48.8% accuracy without layout refinements; the paper compares two geometric priors for refinement, beginning with a circle, but the provided content is truncated at this point and the full comparison is not reported in the text available.

Methodology in Plain English

NUMINA runs in two phases and never trains or fine-tunes any network. Inference uses 50 denoising steps on the official Wan T2V series.

Phase one — identify the mismatch. The method watches attention during early denoising (attention is read at timestep 20 and layer 15). Because attention heads behave very differently, it does not average them. It scores each self-attention head for how well it separates instances using three signals: overall intensity spread (foreground versus background), variance across non-overlapping blocks (structural richness that penalizes both over-smoothing and degeneracy), and average Sobel gradient magnitude (edge clarity), combined as S(SA) = S1 + S2 + γS3. The single best-scoring head is kept. For each target noun, it separately picks the cross-attention head whose peak activation is highest, on the reasoning that a more concentrated peak means the token is better aligned to a visual region.

The selected self-attention map is split into contiguous regions by clustering, and the selected cross-attention map is thresholded (values below a 0.1 peak-ratio are suppressed) and grouped by density-based clustering into a focus mask. Each candidate region is kept only if its overlap with the focus mask, computed as intersection area over region area, is at least τ. The surviving disjoint foreground regions form the layout, and the number of such regions is an explicit, countable estimate that can be compared against the numeral in the prompt.

Phase two — correct conservatively. To remove an object, the smallest region of that category is erased and its pixels relabeled as background, on the principle of minimal structural change. To add an object, a template is created: the smallest existing region of that category is copied so the new instance keeps the category's intrinsic scale and shape, or if no instance exists, a circle of radius r is used. The template defines geometry only, not appearance. It is placed by minimizing a heuristic cost over a uniform grid of candidate centers: an overlap term penalizing collisions with the existing layout, a center term pulling placement toward the existing spatial distribution, and a temporal term keeping the instance stable relative to its position in the previous frame, summed as C = C_o + C_c + λC_t.

The refined layout then guides regeneration through training-free modulation of cross-attention: softmax(S_pre + B)V, where S_pre are the pre-softmax scores and B starts at zero. Removal sets B to a large negative constant over the removed region. Addition either sets B to k·δ(t) when the instance came from the manual circle template, or overwrites the pre-softmax scores in the added region with the mean score computed from the reference region, scaled by δ(t). The intensity function δ(t) is monotonically decreasing, so guidance is strong early when layout is decided and weak later when fine detail matters.

Why This Matters

Impact on research. The paper reframes count errors not as a data or scale problem but as a latent-layout problem that can be diagnosed and corrected at inference. It also makes a broader methodological point: structural guidance complements seed search and prompt enhancement rather than replacing them, and the combined pipeline reaches 54.2% CountAcc, notably above any single strategy. The release of CountBench (210 prompts, counts 1–8, 1–3 categories) gives the field a targeted evaluation set where existing text-to-video benchmarks largely focus on visual quality, temporal coherence, or general text alignment instead of precise numerics. The finding that instance separability is concentrated in a small number of heads is reusable by other attention-manipulation work.

Real-world applications (drawn from the settings and motivations the paper describes):

• Instructional and educational visualization, where a prompt such as a specified number of components or steps must match the rendered video.
• Entertainment and content creation, where precise quantities in a script need to appear on screen.
• Any precision-sensitive generation pipeline where a wrong object count invalidates the output rather than merely reducing aesthetic quality.
• Post-hoc correction workflows, since the method works at generation time and can be combined with seed search, prompt enhancement, and acceleration techniques like EasyCache to keep runtime manageable.

Industry relevance. The method is training-free and model-agnostic in practice: it is demonstrated on Wan2.1-1.3B, Wan2.2-5B, Wan2.1-14B, and adapted to CogVideoX-5B's MMDiT by decomposing unified attention into video-to-video (treated as self-attention) and text-to-video (treated as cross-attention) sub-matrices. Its measured overhead is modest — 16.3 GB VRAM on the 1.3B model and 355 s wall-clock when paired with EasyCache versus 292 s for the unmodified baseline — and it improves CLIP alignment while maintaining or improving temporal consistency, which matters for studios and platforms that cannot afford regressions in perceived quality.

Future Directions

• Pushing toward perfect accuracy: the authors state that achieving perfect accuracy across all scenarios remains challenging, so closing the remaining gap on CountBench is an open problem.
• Very dense instances: generating tens or hundreds of instances is explicitly described as unexplored, and the authors call fully numerically precise video generation for any number an important direction for future research.
• Understanding why head separability is sparse: the ablation shows Top-1 selection beats Top-2, Top-3, random, and averaging, but the paper does not explain what determines which heads carry instance information or whether that selection transfers across layers, timesteps, and architectures.
• Broadening architecture coverage: the CogVideoX-5B adaptation shows the idea transfers beyond dedicated cross-attention designs, leaving open how the approach behaves on other backbones and on text-to-image settings, where the related CountGen work is static-image and SDXL-specific.

Target Audience

Researchers and engineers working on diffusion-based video and image generation, particularly those interested in controllability, attention interpretability, and count or layout fidelity. It is also relevant to practitioners who need precise object counts in generated video and who cannot retrain models, and to benchmark designers, since CountBench addresses a gap in numerical evaluation. Readers should be comfortable with DiT architectures, cross-attention, and standard clustering and scoring methods; the paper assumes familiarity with diffusion sampling and flow matching rather than explaining them from scratch.

Authors’ abstract

Text-to-video diffusion models have enabled open-ended video synthesis, but often struggle with generating the correct number of objects specified in a prompt. We introduce NUMINA , a training-free identify-then-guide framework for improved numerical alignment. NUMINA identifies prompt-layout inconsistencies by selecting discriminative self- and cross-attention heads to derive a countable latent layout. It then refines this layout conservatively and modulates cross-attention to guide regeneration. On the introduced CountBench, NUMINA improves counting accuracy by up to 7.4% on Wan2.1-1.3B, and by 4.9% and 5.5% on 5B and 14B models, respectively. Furthermore, CLIP alignment is improved while maintaining temporal consistency. These results demonstrate that structural guidance complements seed search and prompt enhancement, offering a practical path toward count-accurate text-to-video diffusion. The code is available at https://github.com/H-EmbodVis/NUMINA.