TTOM: Test-Time Optimization and Memorization for Compositional Video Generation

AI summary

Overview

Research area: Computer vision and generative modeling, specifically text-to-video (T2V) generation and compositional controllability using Video Foundation Models (VFMs) and Diffusion Transformers (DiTs).

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching sampling, cross-attention maps, LoRA adaptation, and test-time optimization.

Scope: A single paper proposing TTOM, a model-agnostic, training-free framework that optimizes lightweight inserted parameters at inference time against LLM-generated spatiotemporal layouts, and stores the results in a persistent parametric memory for reuse across a stream of prompts.

Affiliations listed: National University of Singapore, University of Science and Technology of China, and Harbin Institute of Technology (Shenzhen). Published under CC BY 4.0 as arXiv:2510.07940v2 [cs.CV] 28 Feb 2026.

What This Paper Is About

Existing video generation models can produce realistic footage but frequently fail at compositional prompts, where several objects, attributes, counts, and relationships must be combined correctly (motion, numeracy, spatial relations, interactions). Prior fixes apply explicit bounding-box layout guidance by directly editing latents or attention maps on a per-sample basis, which can degrade video quality, discards any learning from one prompt to the next, and never improves the underlying model. TTOM instead inserts new parameters into the model, optimizes them during inference to match a planned spatiotemporal layout, and keeps the optimized parameters in a memory so that later, similar prompts can reuse or refine them.

Key Contributions

1. Unsupervised test-time optimization (TTO) for compositional T2V. Given a layout produced by an LLM from the text prompt, the method incorporates and optimizes lightweight parameters per data sample, steering generation toward the prescribed layout without touching the latents directly.

2. A parametric memory mechanism for historical optimization context. Optimized parameters are stored as key-value pairs (keyed by an abstracted prompt pattern) and support insert, read, update, and delete operations, with a fixed capacity and eviction by least-frequently-used policy — enabling a streaming, lifelong-learning style generation setting.

3. A probe study of attention-layout relevance in Diffusion Transformers. The authors measure mIoU overlap between cross-attention maps at each layer and segmentation maps (GroundingDINO + SAM 2) over 200 prompts, finding large variance across layers, which motivates layer selection for optimization.

4. Empirical validation on two benchmarks. Reported relative improvements over CogVideoX-5B and Wan2.1-14B on T2V-CompBench are stated as 34% and 14% in the abstract and Section 1, and as 34.45% and 15.83% in the results table and Section 4.2.

Main Findings

• Compositional gains on T2V-CompBench (overall Avg.): CogVideoX-5B rises from 0.4189 to 0.5632 with TTOM (+34.45%), and Wan2.1-14B rises from 0.5314 to 0.6155 (+15.83%). For comparison, CogVideoX-5B + DyST-XL scores 0.5081 and + LVD scores 0.4739; Wan2.1-14B + LVD scores 0.5439.

• Motion and numeracy show the largest relative improvements: Motion improves 63.69% on CogVideoX-5B (0.2658 to 0.4351) and 82.57% on Wan2.1-14B (0.2696 to 0.4922); numeracy improves 37.10% (0.3706 to 0.5081) and 15.02% (0.5113 to 0.5881) respectively. Other reported per-category gains include spatial relation, consistent attribute, dynamic attribute, action, and interaction.

• Ablation isolates the two components (motion category): baseline 0.2696, TTO alone 0.4321 (a 60.27% improvement), TTO plus memory 0.4922 (an additional 13.91%).

• Continual TTO trades latency for quality: with memory initialization and no per-sample TTO, motion is 0.4754 at 427 s latency; adding continual TTO gives 0.4846 at 627 s, versus 0.2696 at 425 s for the un-initialized baseline. Using Top-10 retrieved entries instead of Top-5 gives 0.4437 (no TTO) and 0.4705 (with TTO), which the authors attribute to richer context possibly introducing noise.

• JSD is the best of the tested alignment losses: on motion and numeracy, JSD scores 0.4321 and 0.5881, versus CoM loss at 0.3626 and 0.4697, and CE loss at 0.2912 and 0.5218.

• Memory size from pseudo-training data scales performance (motion): 50 → 0.3519, 100 → 0.4344, 150 → 0.4751, 200 → 0.4922.

• Guidance timesteps and iterations saturate: motion peaks at 5 optimization timesteps (0.4321) versus 1 (0.2730), 3 (0.3692), and 7 (0.4296); motion peaks at 8 iterations (0.4321) versus 4 (0.3789), 12 (0.4130), and 16 (0.3547). The authors suggest attention maps carry entangled information beyond layout, so excessive alignment optimization disrupts other signals.

• VBench semantic dimensions improve in several areas: object class goes from 0.8342 to 0.9486 (CogVideoX-5B) and 0.8628 to 0.9921 (Wan2.1-14B); multi-object from 0.6728 to 0.7952 and 0.6958 to 0.8216; color from 0.8840 to 0.9246 and 0.8859 to 0.9289; spatial relation from 0.7943 to 0.8215 and 0.7539 to 0.8074. The remaining reported dimensions (scene, appearance style, temporal style, overall consistency) change only modestly and are slightly lower than the backbone in a few cells, such as overall consistency for CogVideoX-5B (0.2742 to 0.2678).

• VBench overall scores improve: total 0.8201 → 0.8318 and quality 0.8272 → 0.8314 for CogVideoX-5B; total 0.8369 → 0.8492 and quality 0.8559 → 0.8573 for Wan2.1-14B. Semantic rises from 0.7917 to 0.8332 and from 0.7611 to 0.8166. TTOM+backbones also beat the LVD-enhanced variants on these totals.

• Qualitative claims: the authors report motion-pattern transfer from memory into the foundation model, plus failure cases in the appendix (Figure 10); the failure case details are not given in the provided content.

Methodology in Plain English

The pipeline runs in stages:

1. Layout planning. A large language model (OpenAI GPT-4o) reads the user prompt and produces a spatiotemporal layout: for each object, an object phrase plus a sequence of 4-dimensional bounding boxes with start and end frame indices. The LLM first writes descriptions of object motion and camera behavior, then generates boxes, then a verification step checks and corrects spatial and temporal consistency.

2. Attention probe. The authors generate videos and extract text-video cross-attention maps for each object phrase, average-pool them over the text dimension, and measure overlap (mIoU) with segmentation maps from GroundingDINO + SAM 2. The result is that only some layers have strong attention-layout relevance, so optimization targets those.

3. Test-time optimization. Each bounding box is smoothed with a Gaussian kernel into a soft mask. The loss is the Jensen–Shannon divergence between the attention map and the soft mask, averaged over objects. Instead of back-propagating into the latent, the method inserts new parameters (LoRA, rank 32, on Query/Key/Value/Output projections of every cross-attention block) and updates only those with AdamW at learning rate 1e-4, for the first five denoising steps. This is claimed to avoid distribution collapse and, crucially, to leave behind reusable parameters.

4. Parametric memory. Each optimized parameter set is stored as a key-value pair, where the key is derived by abstracting the prompt into a template (for example, "<object A> drifts right to left above <object B>."), embedding it, and indexing. Memory supports insert (new prompt, no match), read (load matched parameters and either generate directly or continue optimizing), update (write back refined parameters), and delete (least-frequently-used eviction when capacity is exceeded). Memory can be made user-specific and scaled either by capacity or by the number of parameters per item.

5. Evaluation setup. Experiments integrate TTOM into CogVideoX-5B (49 frames at 8 FPS) and Wan2.1-14B (81 frames at 16 FPS). For fair comparison with per-sample methods, a "test-time independence" setting generates 200 pseudo-training prompts with GPT-4o, builds memory from them, and forbids inserting or updating memory from test samples.

Why This Matters

Impact on research. TTOM reframes compositional video control as a streaming, memory-backed adaptation problem rather than a set of independent inference-time hacks. It argues that test-time optimization results should be retained and reused, and that layout knowledge can be disentangled and transferred — a claim the authors support with reported transferability and generalization. It also provides a layer-wise probe of attention-layout relevance in DiTs, addressing a known gap between UNet-era attention-control methods and modern diffusion transformers.

Real-world applications.

• Advertising and social media production, where prompts commonly specify counts, colors, and object motions that must be rendered exactly.
• Filmmaking and previsualization, where a creator iterates on a sequence of related shots and benefits from a system that remembers prior requests.
• Personalized content creation, since the memory can be maintained per user and track their historical requests and intentions.
• Interactive creative tools and storyboarding, where a continuous stream of prompts arrives and latency matters — memory allows skipping optimization when a prompt closely matches history.

Industry relevance. The method is backbone-agnostic and training-free — it adds LoRA parameters to existing checkpoints rather than retraining them — and is reported to improve both open DiT models (CogVideoX-5B, Wan2.1-14B) and to beat commercial-style baselines including Pika-1.0, Gen-3, Dreamina-1.2, and Kling-1.0 on T2V-CompBench average. It offers an explicit quality-versus-latency dial (skip optimization, or optimize further) that production systems can tune, and it introduces the notion of a user-history memory as a service-level feature. The paper also acknowledges environmental costs of intensive computation in its ethics statement.

Future Directions

1. Better context fusion. The Top-k retrieval experiment shows that more retrieved entries can add noise (Top-10 underperforms Top-5); the authors state that more sophisticated fusion strategies are needed to leverage historical context.

2. Scaling the memory in both dimensions. The authors propose increasing item-level capacity (a larger history window) and increasing the number of parameters per item so each TTO captures more scene patterns — both left as directions.

3. Reducing the cost of continual TTO. Continual optimization improves motion (0.4846 vs 0.4754) but raises latency from 427 s to 627 s; closing this gap without losing quality remains open.

4. Handling excessive alignment. Because attention maps encode entangled information, performance degrades past 5 guidance timesteps and 8 iterations; identifying which signals are disrupted and how to protect them is unresolved. The appendix also reports failure cases (Figure 10) that are not detailed in the provided content.

Target Audience

Researchers and engineers working on controllable or compositional video generation, diffusion transformer internals, and test-time adaptation. It is also relevant to practitioners deploying T2V systems who need per-user personalization and a controllable quality-latency trade-off, and to readers tracking memory-based continual learning applied to generative models. Readers without background in diffusion sampling, attention maps, or parameter-efficient fine-tuning will find the methodology section difficult.

Authors’ abstract

Video Foundation Models (VFMs) exhibit remarkable visual generation performance, but struggle in compositional scenarios (e.g., motion, numeracy, and spatial relation). In this work, we introduce Test-Time Optimization and Memorization (TTOM), a training-free framework that aligns VFM outputs with spatiotemporal layouts during inference for better text-image alignment. Rather than direct intervention to latents or attention per-sample in existing work, we integrate and optimize new parameters guided by a general layout-attention objective. Furthermore, we formulate video generation within a streaming setting, and maintain historical optimization contexts with a parametric memory mechanism that supports flexible operations, such as insert, read, update, and delete. Notably, we found that TTOM disentangles compositional world knowledge, showing powerful transferability and generalization. Experimental results on the T2V-CompBench and Vbench benchmarks establish TTOM as an effective, practical, scalable, and efficient framework to achieve cross-modal alignment for compositional video generation on the fly.