Cepheid: Decoupling Scale from Structure for Cross-Family Motion Completion

AI summary

Overview

• Research area: Computer vision and geometry processing — sparse-to-dense motion completion on 3D meshes, spanning character animation, cloth simulation, and robotics.
• Technical level: Advanced. The paper assumes familiarity with graph Laplacians, Dirichlet problems, flow matching, and message-passing graph networks.
• Scope: The paper proposes a single learned prior that turns a handful of observed vertex displacements into dense per-vertex motion across geometry families it was never trained on, without skeletons, registration, or cross-mesh correspondence.

What This Paper Is About

Given a 3D mesh and the measured displacements of only a small subset of its vertices, the goal is to recover the motion of every remaining vertex. The paper's central problem is that machine-learned motion priors trained on one type of geometry (animals, human bodies, robot arms) often do worse than a simple analytical baseline when applied to a different type of geometry (cloth, legged robots), because the residual motion left over after the analytical baseline varies in physical amplitude by more than an order of magnitude across families and cannot be measured without knowing the answer.

Key Contributions

1. Harmonic extension as a general base and control. The authors formulate motion completion as a graph Dirichlet solve on mesh connectivity with observed vertices as boundary conditions, providing both the base for a learned residual and a parameter-free control. Seven of twelve single-family transfers fail to improve on it.
2. Target-independent amplitude recovery. A dimensionless correction, fitted as a ridge regression on the three training sources and then frozen, estimates physical residual amplitude without access to target motion, bringing cloth completion error within 1–8% of a target-informed oracle across observation densities.
3. A shared dimensionless residual prior. A flow-matching network conditioned on geometry and harmonic extension learns the residual divided by the estimated amplitude, letting one checkpoint transfer zero-shot to families absent from training.
4. Cross-domain application without retraining. The same checkpoint densifies point-tracked animal video (CoTracker3 output on a DAVIS sequence) and completes the motion of a soft body lifted by a Franka arm from eight proprioceptive points with none on the payload.

Main Findings

• Single-family priors transfer poorly. Across three single-family models (DT4D animals, AMASS humans, Franka arm) and one unified model, seven of twelve off-diagonal cells in the transfer matrix sit at or above harmonic extension. On cloth, the AMASS and Franka priors are 25.5% and 7.8% worse than the base.
• Residual scale varies enormously across families. Extent-normalized residual amplitudes vary by 18–39× across the three training families at matched density, and fall by more than an order of magnitude across densities. The dataset-wide mean motion denominators span a 39× factor across columns.
• The gauge collapses scale spread. Dividing by observed displacement dispersion (σ_obs) reduces cross-source scale spread from 37.5× to 3.2× at the sparsest rung across held-out conditions of the three training sources.
• Amplitude recovery approaches an oracle. On held-out cloth, amplitude recovery lands within 1.4–8.1% of the target-informed oracle error across densities, and beats a borrowed lookup table by 0.4% and 0.1% at 0.5% and 1%, and 5.6–26.4% from 2% up. Raw observed scale without correction consistently underperforms harmonic extension.
• Unified model beats the base everywhere. The unified checkpoint is below harmonic extension on all five evaluated families: relative error 0.1243 vs 0.2685 on DT4D, 0.0214 vs 0.0693 on AMASS, 0.0201 vs 0.1011 on Franka, 0.3692 vs 0.4377 on cloth, and 0.2225 vs 0.2474 on legged bodies. This cuts error by 15.6% on cloth and 10.1% on legged bodies, and tops the best specialist (the DT4D prior) by 12.9 and 0.5 percentage points on those two columns.
• Unified training does not win by more exposure. Each specialist processes 8.0 million samples; the unified model processes 4.0 million from DT4D, 2.4 million from AMASS, and 1.6 million from Franka, and sees only 8% of the unique training items available in AMASS.
• Beats NSDP and DOMA on DT4D. On seven held-out identities and five densities (2,100 conditions), the unified model lands 44–81% below harmonic extension. NSDP is above the base at every density (1.6–3.6× the base's error), while DOMA is below it at every density (8% at 0.5% rising to 67% at 50%) but has 1.4–2.5× the error of Cepheid.
• Two components are load-bearing. In the ablation, removing harmonic extension makes the model cross above the base in both held-out columns (by 2.4× on each), and removing amplitude recovery crosses on legged bodies while staying below on cloth. Removing conditioning, flow matching, or multi-sampling never crosses the base. Reading the same weights at one sample instead of eight costs 8–13% on the two held-out families.
• Fine-tuning helps on fixed layouts. Fine-tuning to a fixed three-marker gripper placement more than halves the error of the same checkpoint evaluated zero-shot.

Methodology in Plain English

The approach starts from a classical trick: treat the mesh as a graph and solve for the smoothest motion field that agrees with the observed vertices, which is a closed-form linear solve called harmonic extension. That solve gives a reasonable but blurry answer. The authors then ask a network to predict only what harmonic extension got wrong — the residual.

The obstacle is that residuals have physical units, and their size differs wildly between families and depends on the target motion, which is unknown at test time. So the authors separate the two concerns. They estimate the amplitude from quantities they can actually see: the spread of the observed displacements (which supplies the physical unit) multiplied by a dimensionless correction. That correction is a ridge regression on eleven dimensionless descriptors — things like observation density, graph connectivity to observed vertices, observed motion variation, and properties of the base field. It is fitted once on the three training sources and frozen.

With amplitude factored out, the network learns a dimensionless residual instead, so it never has to guess physical scale. The network is an MGN-T backbone — a Transformer attention block between two message-passing blocks — running on a 12-nearest-neighbor coordinate graph, with mesh connectivity entering only through the harmonic base field. It is trained with optimal-transport conditional flow matching and, at inference, averages eight stochastic sample draws over 30 Euler steps. The final prediction is the harmonic base plus the estimated amplitude times the averaged residual. Training uses random masks only; evaluation covers five observation densities from 0.5% to 50%.

Why This Matters

• Impact on research: The paper reframes an architecture question as a scaling question. Instead of arguing that bigger or more diverse training sets solve cross-family transfer, it shows that much of the failure comes from an unnormalized learning target, and that factoring physical scale out of the residual is what makes one prior work across animals, humans, robots, cloth, and legged bodies. The harmonic-extension control is also a methodological contribution: it gives the field a parameter-free bar that learned priors must clear, and the paper reports honestly that many do not.
• Real-world applications:
  • Robot manipulation with soft payloads: Inferring the deformation of a grasped object from robot proprioception alone, demonstrated on a soft bar gripped by a Franka arm in Isaac Lab.
  • Markerless motion capture from video: Densifying a point tracker's sparse output (CoTracker3) into motion for vertices the camera never saw, demonstrated on a DAVIS sequence with 120 completed far-side points.
  • Character and creature animation: Completing dense motion for articulated animals, humanoids, and stylized characters from a handful of vertex trajectories, without a skeleton or template.
  • Cloth and soft-body simulation pipelines: Transferring motion priors to deformable simulations, demonstrated on the MeshGraphNets flag dataset, which is not used in training.
• Industry relevance: Animation, VFX, and game studios routinely need to drive a mesh from sparse handles or tracker output; a checkpoint that generalizes across asset types without per-asset rigging or retraining reduces setup cost. Robotics teams face a similar problem when the object being manipulated has no model, no skeleton, and no correspondence to any training asset.

Future Directions

• Noisy and imperfect observations. The quantitative evaluation assumes exact vertex displacements free of sensor noise, lost tracks, depth distortion, or scanned geometry. The authors list validation on noisy inputs as future work.
• Training across mask families. Training uses random masks only. The prior transfers zero-shot to a fixed gripper placement, but fixed three-marker layouts gain substantially from fine-tuning, suggesting mask-family coverage is an open question.
• Sparser observation than 0.5%. The unified model crosses above harmonic extension only at 0.5% on legged bodies and outperforms it from 1% up, leaving the sparsest regime as an unresolved edge.
• Extending to genuinely recorded data. Only AMASS and the RH20T episodes are recorded data; DT4D is artist-animated, Franka uses forward kinematics on random joint trajectories, and cloth and legged bodies are simulated.

Target Audience

Researchers and graduate students in computer vision, computer graphics, and robot learning who work on motion completion, mesh deformation, or cross-domain transfer. It is most useful to readers already comfortable with graph Laplacians, flow-matching generative models, and message-passing network architectures. Practitioners in animation, VFX, and robotics manipulation who need to drive dense geometry from sparse inputs will find the practical framing (no skeleton, no registration, no cross-mesh correspondence) directly relevant, though the paper's implementation details live in its appendix.

Authors’ abstract

Sparse-to-dense motion completion recovers dense mesh motion from sparse vertex displacements. Residual priors trained on one geometry family frequently fail to beat parameter-free harmonic extension off-family. Moreover, cross-family training meets a scaling barrier: residual amplitude varies by more than an order of magnitude across geometry families and collapses as observations become denser. Since this scale depends on unknown target motion, it cannot be measured directly at test time. We decouple residual structure from physical scale by learning a dimensionless prior, estimating amplitude from observables and a lightweight correction fitted only on training sources. Our unified model surpasses harmonic extension across all evaluated families, including unseen cloth and legged bodies, without skeletons, registration, or cross-mesh correspondence. The same prior applies without retraining to point-tracked animal video and a robot grasping a soft body. Project page: https://su-terry.github.io/cepheid-website/