Past as State, Present as Attention: Persistent-State Blockwise Flow Matching for Long-Horizon Co-Speech Motion Generation

AI summary

Overview

Research area: Computer vision and generative motion synthesis—specifically co-speech full-body gesture and facial-expression generation from audio, with a focus on long-horizon (minute-scale) sequence generation.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow matching, transformer attention, state-space and linear-attention models (Gated DeltaNet), classifier-free guidance, and SMPL-X motion parameterization.

Scope: The paper proposes PASPA, a blockwise flow-matching architecture that replaces explicit long history with a compact persistent recurrent state for the past while keeping full bidirectional attention for the current block, and evaluates it on BEAT2 and SHOW.

What This Paper Is About

Generating gesture motion that matches speech over long durations is hard because models that produce motion one segment at a time either condition only on a few recent frames (so style slowly drifts, e.g. hands gradually drop from a raised pose) or keep the entire history as tokens (which grows in cost and is hard to learn from with limited co-speech data). The authors argue that the past and the present play fundamentally different roles—past motion carries slowly varying style, while the current segment carries fast-changing local dynamics—and therefore should be represented differently. Their goal is a generator that stays style-consistent and temporally coherent over long rollouts while keeping per-step cost constant.

Key Contributions

1. A stated design principle. The authors identify an asymmetry between "past" and "present" in co-speech motion generation and formulate the Past-As-State, Present-as-Attention principle: history should be compactly summarized into a persistent state, while the current block should be modeled with full attention.

2. The PASPA architecture. A DiT-style network whose self-attention sublayer is replaced by two sequential sublayers: Blockwise Gated DeltaNet (BGDN), which carries an inter-block persistent state built on Gated DeltaNet with a block-causal read rule, and Blockwise Multi-Head Self-Attention (BMHSA), which performs bidirectional attention strictly within each block.

3. A blockwise flow matching (BFM) training scheme. The model is trained with multi-block supervision—each of the N blocks gets its own noise level and is denoised in parallel in a single forward pass—combined with weakly monotonic timesteps, adaptive per-timestep loss weighting via a small MLP, a stop-gradient, logit-normal timestep sampling, and a boundary loss for cross-block continuity.

4. Tailored inference procedures. Prefill-decode rollout with state caching (prefill writes a clean block into the state without running denoising; decode treats the fixed-size state as read-only, giving O(1) query complexity per step regardless of history length), plus a hybrid classifier-free guidance scheme with a learnable null state that guides over speech/expression conditioning and history dependence separately.

Main Findings

• State-of-the-art FGD on BEAT2. In the 1 Speaker ("Scott") setting, PASPA reaches FGD 3.71 (×10 scaling), BC 0.746, Div 13.38, and face MSE 5.768 (×10⁸), the best FGD and MSE among compared methods and the second-best Div, versus the flow-matching baseline GlobalDiff at FGD 4.78, BC 0.705, Div 13.13, MSE 6.330.

• Strong multi-speaker generalization. In the All Speakers setting, PASPA achieves FGD 2.33, BC 0.645, Div 9.42, MSE 3.793, compared with GlobalDiff at FGD 2.63, BC 0.404, Div 8.24, MSE 4.144.

• Best FGD on SHOW. PASPA obtains FGD 0.00111, BA 0.876, Div 1.043 (ground truth: BA 0.867, Div 0.834). This is a 43.9% FGD reduction relative to GlobalDiff (0.00198), with slightly higher BA than GlobalDiff and the second-best Div.

• Human evaluation favors PASPA. A blind ranking study with 18 participants and 167 valid ranking groups, using 20-second clips and four criteria (motion realism, semantic consistency with speech, motion-speech synchronization, motion diversity), found PASPA had the highest top-1 win rate on all four criteria, 39.8% overall.

• Past-as-state beats longer explicit history. Ablating history handling with a fixed recipe: no history gives FGD 7.065; 8-frame attention history gives 6.051; full (all-token) attention history gives 5.710; PASPA's state-based past with attentive present gives 5.216. Retaining more raw history does not match a compressed state.

• Attention for the present is essential. Replacing current-block attention with state-based recurrent modeling (State + State) degrades FGD to 7.652—worse than even the no-history variant—showing that the current block needs dense bidirectional interaction.

• Multi-block supervision needs stabilizers. Naively moving from single-block to multi-block supervision worsens FGD (7.410 → 8.456) due to unordered block timesteps and imbalanced losses. Adding weakly monotonic timesteps (6.346), adaptive weighting (5.614), stop-gradient (5.216), and logit-normal timestep sampling (4.450) recovers and surpasses the single-block baseline.

• Hybrid guidance helps. With condition dropout only and scale (1.5, –), FGD is 3.883; with condition and history dropout plus hybrid guidance at (1.5, 1.5), FGD is 3.532. Removing the second-order joint term from the composition gives 3.967.

• Long-horizon behavior. The paper reports FGD and Div on the terminal 10-second window for rollouts of T ∈ {10, 30, 60} seconds; in the qualitative 52-second example, PASPA preserves a raised-hands style throughout while short-seed truncation monotonically drifts and long-context token attention deviates regardless of temporal distance. The numeric values of the long-horizon table are beyond the truncation point of the supplied text and are not reproduced here.

• Empirical basis for the asymmetry. An autocorrelation analysis on BEAT2 (averaged across speakers, with 95% confidence intervals) shows style features remain correlated beyond 100 frames while local features decay rapidly.

Methodology in Plain English

The authors start from a motivating observation: gesture style (how high the hands are held, how wide gestures are) persists over many seconds, but the fine details of motion change moment to moment. So they split the job in two.

For the past, they use a recurrent mechanism that compresses everything seen so far into a fixed-size memory (a matrix), rather than storing every frame. This mechanism is Gated DeltaNet, which they adapt in two ways. First, they make it operate block by block and change when the memory is read: every frame in the current block reads from the memory snapshot taken at the start of that block, never from the continuously updated version. This guarantees no information from the future of the block leaks into the block's own predictions. Second, because the update rule is untouched, they can reuse the existing chunkwise-parallel formulation, and they set the chunk size equal to the generation block size so the read becomes a single dense matrix multiplication—implemented as a custom Triton kernel built on FLA.

For the present, they use ordinary bidirectional self-attention, but masked so that tokens only attend within their own block. Instead of building the N·K by N·K mask, they reshape the sequence into blocks and treat the block index as a batch dimension, cutting cost from O((NK)²d) to O(NK²d). Neither module alone suffices: the state gives history but no within-block token interaction; the attention gives rich local modeling but no memory.

The network follows the DiT template, swapping AdaLN for AdaRMSNorm and driving modulation from Fourier-embedded timesteps plus a speaker ID embedding. It reuses the GlobalDiff setup: a frozen audio-to-expression network turns WavLM features into expression coefficients, motion lives in global rotation space, and the body is split into a torso branch (with global translation) and a hands branch, merged at the output.

Training treats the sequence semi-autoregressively: every block gets its own noise level, and the network predicts all clean blocks at once in one forward pass, with ground-truth past blocks fed in (teacher forcing). The loss sums rotation/translation MSE, forward-kinematics joint and skeletal-structure terms, and a boundary loss matching velocity and acceleration across block seams. Timesteps are forced to be weakly increasing so training resembles inference, sampled from a logit-normal distribution, and an MLP learns per-timestep weights to balance the terms (with a stop-gradient protecting it).

At inference, the model does not grow a token cache. It prefills the state with the seed segment, then for each block runs M−1 denoising steps while treating the state as fixed, and after finishing a block writes it into the state. Guidance uses four forward passes—unconditional, condition-only, history-only, full—combined in a multilinear formula with separate, time-dependent scales for conditioning and history that activate only in the mid-noise range [0.1, 0.6]. To make this trainable, the conditional input is dropped with probability 10% and the state is left unchanged, nulled, or interpolated with a null state at 70% / 10% / 20%, giving a knob for history strength at test time.

Why This Matters

Research impact. The paper reframes history propagation in long-horizon motion generation as a representation problem rather than a context-length problem. The demonstration that full token history (FGD 5.710) loses to a compressed recurrent state (5.216), and that more context is not automatically better, is a useful corrective to the prevailing "just attend to more past" instinct. The block-causal read constraint and the prefill-decode state caching are reusable engineering patterns for any blockwise autoregressive diffusion system.

Real-world applications:

• Digital humans and virtual avatars for presentations, customer service, and telepresence, where a speaker must gesture naturally for many minutes without visibly degrading.
• Virtual assistants and conversational agents on screen, where gestures must stay on-brand and consistent with the persona's characteristic style.
• Animated characters and game NPCs, where long cutscenes or live dialogue require continuous, speech-aligned body motion without per-frame memory growth.
• Accessibility and content localization, generating speaker-appropriate gesture tracks for narrated content, dubbed video, or sign-adjacent presentation avatars.

Industry relevance. Constant per-step memory and compute (O(1) query complexity regardless of history length) is what makes streaming deployment practical on fixed hardware budgets; linearly growing caches do not. The blind study showing 39.8% overall top-1 preference gives a human-perception signal beyond FGD, which matters for product teams who need to defend quality claims.

Future Directions

• Pushing past the tested horizon. The long-horizon evaluation covers rollouts of T ∈ {10, 30, 60} seconds, while the qualitative example runs 52 seconds; whether the fixed-size state saturates on multi-minute or hour-long generation is untested.

• State capacity and structure. The persistent state is a single fixed-size matrix per mechanism. Whether hierarchical or multi-timescale states would capture longer-range stylistic structure—and how state size trades against FGD and Div—is an open design question the paper does not ablate.

• Reducing the guidance cost. Hybrid CFG requires four network evaluations per denoising step. Distilling or amortizing the unconditional/condition-only/history-only branches, or learning the guidance composition, would improve deployment economics.

• Generalizing the principle beyond co-speech. The past-as-state, present-as-attention division is argued from an autocorrelation analysis on BEAT2 specifically. Testing it on other long-horizon motion and sequential-generation tasks—and on data scales far larger than the co-speech setting the authors cite as data-limited—would show whether the asymmetry is general or domain-specific.

• Evaluation beyond FGD. The paper relies on FGD as the primary quality indicator, with BC, Div, face MSE, BA, and a user study as complements. Metrics that specifically detect slow style drift over long windows remain a gap.

Target Audience

Researchers and graduate students working on motion synthesis, gesture generation, and audio-driven animation who already understand diffusion/flow matching and transformer architectures. It is also valuable for practitioners building real-time avatar or digital-human systems who care about streaming inference cost and long-session stability, and for anyone interested in hybrid architectures that combine linear-attention/recurrent state models with local attention. Readers new to generative motion modeling will find the high-level intuition (past is slow and compressible, present is fast and needs detail) accessible, but the methodology and ablations require background in the underlying formalisms.

Authors’ abstract

Co-speech motion generation remains challenging over long horizons due to the need for temporal coherence and consistent style. Existing methods typically generate long motions segment by segment, either conditioning on a short seed of recent frames, which gradually drifts, or retaining ever-longer explicit histories, which are costly to maintain and hard to learn from limited data. We trace these limitations to a key asymmetry between past and present: past motion carries slowly varying style that should be compactly summarized, whereas the current segment exhibits fast-changing local dynamics that require fine-grained modeling. This asymmetry motivates a simple Past-As-State, Present-as-Attention principle for long-horizon co-speech motion generation. Following this principle, we propose PASPA, which summarizes past history with an inter-block persistent state and models the current segment with intra-block bidirectional self-attention. We instantiate PASPA within a blockwise flow matching framework, with multi-block supervision for effective training over variable histories, prefill-decode state caching for efficient rollout, and hybrid classifier-free guidance for enhanced history and speech conditioning. Extensive public-benchmark experiments show that PASPA achieves state-of-the-art FGD performance and preserves motion style across long-sequence generation.