Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generation

AI summary

Overview

Research area: Computer vision and motion synthesis, specifically co-speech gesture generation — the task of producing 3D body motion that accompanies speech.

Technical level: Advanced. The paper assumes familiarity with variational autoencoders, diffusion models, transformer cross-attention, SMPL-X body representations, and standard gesture-generation benchmarks.

Scope: The paper introduces Puppeteer, a posture-aware and object-grounded diffusion model for co-speech gesture generation operating in a causal latent primitive space, along with the SceneGes dataset and new evaluation metrics.

Authors: Vida Adeli, Soroush Mehraban, Jacob Rommann, Harrison Sanborn, Cole Clifford, and Babak Taati, affiliated with Pickford AI, the University of Toronto, and the Vector Institute. Project page: https://puppeteer.pickford.ai/. arXiv:2609.00369v1 [cs.CV], 31 Aug 2026, licensed CC BY-NC-SA 4.0.

What This Paper Is About

Existing speech-driven gesture models are scene-agnostic: they align gestures to audio and text but ignore the speaker's posture and the furniture around them. In reality, people keep their hands above a table, rest an arm on an armrest, and reduce gesture amplitude when seated near objects, so ignoring the physical space produces gestures that can intersect furniture and that fail to reflect posture. Puppeteer addresses this by decomposing long gesture sequences into structured primitives, compressing them into temporally ordered (causal) latent tokens, and running conditional diffusion in that latent space while conditioning on speech, motion history, an initial posture reference, and 3D object geometry.

Key Contributions

1. First object-grounded co-speech gesture framework. The paper claims to be the first work to address object-grounded co-speech gesture generation, modeling the coupling between communicative gestures, posture, and surrounding objects. Its Table 1 comparison shows prior methods (EMAGE, SynTalker, DiffSHEG, LOM, SemGes, GestureLSM, GestureHYDRA, ViBES, EchoAvatar, MIBURI) each satisfy only subsets of the properties quantization-free, latent space, compact space, causal latent, diffusion modeling, pose aware, and object aware, while Puppeteer is marked as satisfying all seven.

2. Temporally controlled cross-attention masking. An audio-window constraint and a text-span constraint provide rhythmic and word-level semantic alignment, enabled by adapting causal latent autoregressive diffusion to co-speech gesture generation.

3. Posture-aware training. The model trains on conversational data across diverse pose references, explicitly modeling posture–gesture coupling, and the authors benchmark Embody3D for co-speech gesture generation for the first time across diverse posture conditions.

4. SceneGes dataset and new metrics. The paper releases SceneGes, described as the first curated synthetic 3D dataset of embodied co-speech gestures with corresponding 3D objects, plus new evaluation metrics tailored to this task.

Main Findings

• Best FGD, BC, and Div on BEAT2 (Speaker2). On the BEAT2 (Speaker2) test set, all metrics reported in ×10⁻¹, Puppeteer achieves FGD 3.436, BC 7.693, ΔBC 0.201, and Div 14.131, versus the strongest listed baselines GestureLSM (FGD 3.692, BC 7.547, ΔBC 0.166, Div 9.490) and MIBURI (FGD 4.369, BC 7.685, ΔBC 0.369, Div 9.645). Puppeteer is described as best in FGD, BC, and Div while remaining competitive in ΔBC.

• Diversity and realism together. The authors note that EMAGE uses VQ-VAE masked generation with argmax codebook selection, producing deterministic outputs with no diversity; sampling codebooks from the softmax distribution (EMAGE*) introduces diversity but raises FGD (5.117 to 5.205). Puppeteer is reported to achieve both higher diversity and lower FGD.

• Cross-dataset training and fine-grained posture conditioning help most. In Table 3, training only on BEAT2 gives Embody3D FGD 33.649 and LL1 16.606; adding Embody3D drops Embody3D FGD to 2.458 and LL1 to 5.553. Adding a coarse sitting/standing embedding via diffusion-timestep modulation reaches Embody3D FGD 2.300 and LL1 4.297, and replacing it with posture-reference cross-attention reaches Embody3D FGD 1.676 and LL1 2.446.

• Object module and collision loss reduce penetration. In Table 4 (MeanPen ×10⁻⁴, MaxPen ×10⁻¹), using LoM as a proxy placed in the same structured scenes gives MeanPen 12.282, MaxPen 1.465, and LL1 6.563 — reported as 2.8× higher MeanPen, 2.0× higher MaxPen, and 8.4× higher LL1 than the object-aware model. Removing the object module gives MeanPen 6.476, MaxPen 1.239, LL1 1.019, Div 10.009; adding the object module without the collision loss gives MeanPen 7.302, MaxPen 1.090, LL1 0.760, Div 12.061; adding the collision loss gives MeanPen 4.450, MaxPen 0.713, LL1 0.780, Div 11.393.

• Combined audio-window and text-span masking works best. In Table 5, the full-audio baseline gives FGD 3.917, ΔBC 0.431, Div 10.994. The best FGD alone comes from the audio window at ξ=1 (FGD 3.385, ΔBC 0.346, Div 10.462). Combining the window at ξ=1 with text-span masking gives FGD 3.437, ΔBC 0.201, and Div 14.131 — described as the best overall result, increasing diversity while reducing the synchronization gap.

• CausalVAE beats a standard VAE for reconstruction and generation. In Table 6, training on BEAT2 only and testing on BEAT2, CausalVAE reaches FGD 0.161, MPJPE 7.387, ACCL 5.497 versus VAE FGD 0.242, MPJPE 16.757, ACCL 15.013. Training on both BEAT2 and Embody3D and testing on Embody3D, CausalVAE reaches FGD 0.153, MPJPE 9.381, ACCL 5.522 versus VAE FGD 0.590, MPJPE 18.416, ACCL 8.971. For generation (Table 7), CausalVAE latents yield FGD 3.437, BC 7.693, ΔBC 0.201, Div 14.131 versus VAE latents at FGD 4.221, BC 7.570, ΔBC 0.596, Div 10.849; real motion BC is listed as 7.282.

• Qualitative advantages. The paper reports Puppeteer follows speech timing and emphasis more accurately than LOM and GestureLSM on phrases such as "I wanted to be …" and "the first thing …", producing self-referential hand-to-chest cues and brief index-finger emphasis gestures; the causal latent representation also enables gesture in-betweening and completion, and Stage 3 yields object-grounded behaviors such as placing hands on a table.

Methodology in Plain English

Puppeteer works in three stages.

Stage 1 — Compress gestures into ordered latent codes. Long gestures are broken into fixed-length primitives. A causal variational autoencoder (CausalVAE) encodes each primitive into a short sequence of continuous latent tokens at reduced temporal resolution. "Causal" means each token depends only on past and current frames within its receptive field, so the temporal order is preserved as explicit structure rather than being entangled. The encoder and decoder use 1D causal convolutions with strided downsampling and temporal upsampling, and the model is trained with feature reconstruction, auxiliary temporal consistency, KL regularization, and SMPL-X-based geometric consistency losses. This avoids both the quantization errors of VQ-VAE tokenization and the cost of diffusing in high-dimensional motion space.

Stage 2 — Diffuse in the latent space, one primitive at a time. A conditional diffusion model generates each latent primitive autoregressively from noise, conditioned on the last H frames of motion history, audio and token-level text features, and an initial posture reference that is fixed for the whole sequence and reused across primitives. The denoiser concatenates history embeddings with noisy future latent embeddings, applies positional encoding and a diffusion-timestep embedding, and uses transformer blocks with self-attention over all history-plus-latent tokens. Posture is injected through gated cross-attention with a learnable scalar initialized to zero, so the base denoising path is preserved at initialization. Speech is injected through cross-attention on the future latent tokens only, using a temporally controlled mask: an Audio Window Attention that restricts each latent index to a local audio neighborhood scaled by ξ, and a Text Span Attention that restricts latent indices inside a text token's span to attend primarily to that token. Training combines a motion-space loss — weighted so frames with high normalized motion intensity, ω_f = 1 + λ_I I_f², get greater emphasis — with a latent consistency loss. Sampling uses classifier-free guidance and DDIM.

Stage 3 — Add object awareness without forgetting gesture prior. All Stage 2 components are frozen and a gated object fusion module is inserted between self-attention and speech cross-attention. Scenes and the person are represented with Basis Point Sets: 1024 points sampled inside an ellipsoidal support whose lateral and frontal radii exceed the vertical radius, with minimum distances computed to points sampled from surrounding meshes (B_O) and to upper-body vertices selected via linear blend skinning weights (B_P). The module uses a gated transformer design with learnable scalars initialized to zero, and object-training batches are interleaved with BEAT2 and Embody3D replay samples at a 4:1 ratio, with replay samples using a learned no-object token. The training loss weights are λ_x0 = λ_latent = 1 and λ_collision = 50.

Dataset construction. SceneGes was built by curating chair and table assets, arranging scene layouts, generating scenario-driven dialogue and gesture scripts with Gemini, synthesizing scene-conditioned videos with Veo 3, and recovering and refining SMPL-X motion to be collision-free.

Why This Matters

Impact on research. The paper argues that scene-agnostic gesture models miss the inherent correlation between body gestures and physical space, and that scene-aware motion generation has targeted locomotion or task-oriented interaction rather than communicative gesturing. It positions object awareness as the influence of passive scene geometry on conversational gesture — constraining gesture space, affecting resting hand placement, and shaping posture-dependent motion — rather than action-centric manipulation such as grasping or lifting. The causal latent formulation also enables explicit temporal control tasks such as in-betweening and completion, and the paper offers Embody3D benchmarking and new metrics as infrastructure for the field.

Real-world applications (as supported by the paper's framing):

• Embodied conversational agents and avatars that gesture naturally in furnished environments such as offices and living rooms.
• Virtual humans in seated conversation scenarios, where the model is trained to preserve sitting posture and avoid leg or hand intersections with chairs and tables.
• Animation pipelines that need controllable, temporally editable gesture output, since the causal latent space supports gesture in-betweening and completion.
• Synthetic data and evaluation infrastructure for scene-grounded motion research, via the SceneGes dataset and the proposed diversity, posture, and penetration metrics.

Industry relevance. Teams building avatars, telepresence, or virtual assistants in 3D scenes face exactly the problem of gestures that visually clip through furniture and that ignore whether the user is seated or standing. A model that freezes a strong speech-gesture prior and then adds scene awareness through a lightweight gated module — plus a dataset generation pipeline using Gemini and Veo 3 — suggests a practical path to retrofitting existing gesture systems with spatial grounding.

Future Directions

• Generalization to unseen scenes. The paper reports an evaluation on a held-out object split in Appendix G.7 to show generalization to unseen furniture, but the summary content provided does not report those numbers; extending object grounding beyond the curated SceneGes assets and layouts remains an open direction.

• Limitations and failure modes are not reported in the content provided. The paper text here does not include an explicit limitations section, so the boundary conditions of the method remain an unanswered question from this material.

• Broader posture and interaction coverage. The authors note that BEAT2 alone provides limited posture variation and interactions for training gesture VAEs, and that SceneGes is synthetic. Whether training on larger real-world seated and object-interaction corpora would change performance is not addressed.

• Direct object-aware baselines. Because no prior method supports object-grounded co-speech gesture generation, the comparison relies on LoM as a proxy placed into structured scenes (Appendix G.7 for the held-out split); the field still lacks dedicated object-grounded baselines to benchmark against.

Target Audience

Researchers and practitioners in computer vision, graphics, and human motion synthesis who work on co-speech gesture generation, embodied avatars, or scene-aware motion. It is also relevant to applied teams building conversational agents that must behave plausibly in physical or virtual furnished spaces, and to dataset builders interested in the pipeline that produced SceneGes using Gemini and Veo 3. Readers need a background in generative modeling — VAEs, diffusion, and transformer conditioning — to follow the architecture sections in detail.

Authors’ abstract

Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging. Prior speech-driven gesture models emphasize audio-gesture alignment but do not explicitly account for posture constraints or surrounding objects, failing to capture the inherent correlation between body gestures and the physical space. We present Puppeteer, a posture-aware, object-grounded co-speech gesture diffusion model operating in a causal latent space. We decompose long gestures into structured primitives and learn a causal variational autoencoder that encodes them into temporally ordered latent tokens, each depending only on the past. We then perform conditional diffusion directly in the causal latent space, conditioning on speech signals, motion history, an initial posture reference, and object geometry to synthesize physically consistent gestures. This temporally ordered latent formulation enables explicit temporal control and supports tasks such as gesture in-betweening and gesture completion. To better assess co-speech gesture synthesis beyond existing measures, we introduce new evaluation metrics tailored to this task. We also created SceneGes, the first curated synthetic 3D dataset of embodied co-speech gestures and corresponding 3D objects, enabling object-grounded gesture generation. Experiments show that Puppeteer generates more diverse and temporally synchronized gestures than prior methods, while enabling object-grounded gesture synthesis.