Schrödinger's Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics

AI summary

Overview

• Research area: Computer vision — probabilistic motion representation, future scene kinematics prediction, and goal-conditioned motion planning.
• Technical level: Intermediate. The paper uses latent variable models, generative decoders, and density estimation in latent space, but explains the core ideas through the Schrödinger's Cat analogy and reports results in standard planning metrics.
• Scope: The paper introduces GARFIELD (Goal-Aware Representations of Future kInEmatic Latent Distributions), a single structured latent representation of scene motion that supports joint trajectory sampling, deterministic density estimation, and flexible spatio-temporally sparse goal conditioning.

What This Paper Is About

Predicting how a scene will evolve from limited observations is inherently ambiguous, because many futures may be consistent with what is currently visible. Existing methods either generate full pixel-level video (spending capacity on appearance rather than motion), or sample a handful of trajectories without explicitly modeling the distribution of possible motion. This paper learns a spatio-temporally structured latent representation of the distribution over future motion given a start image and optional sparse goal positions, so that the same representation can both produce coherent trajectories and expose the underlying motion probabilities directly.

Key Contributions

1. A structured probabilistic latent representation of motion. The latent is factorized into components $\mathbf{z}\tau^i$, where each component corresponds to the point-wise motion distribution $p(t\tau^i \mid \mathcal{C})$ of scene element $i$ at timestep $\tau$, learned end-to-end with a joint encoder $\mathbf{E}\varphi$ and a point-wise generative decoder $\mathbf{D}^\text{p}\psi$.
2. A deterministic density decoder. A decoder $\mathbf{D}^\text{d}_\omega$ predicts a non-parametric distribution over a grid $\mathcal{G}$ in a single ViT forward pass in less than 0.8 ms, replacing expensive Monte-Carlo estimation.
3. Joint sampling of coherent trajectories. A full decoder $\mathbf{D}_\theta$ conditions on all latents jointly to sample mutually consistent trajectories, avoiding the mode-mixing problem of independent point-wise sampling.
4. Flexible spatio-temporally sparse conditioning. Constraints $\mathcal{C} = (I, \widehat{T})$ can be specified for selected objects and timesteps, allowing users to progressively refine the distribution of possible futures and to explore it interactively via entropy.

Main Findings

• Motion planning outperforms baselines on OpenVid-1M. With 4 goals, GARFIELD achieves EPE 0.020, PCK@10% 0.973, PCK@1% 0.544, FDE 0.026 at 0.40 s latency; with 16 goals, EPE 0.013, PCK@10% 0.989, PCK@1% 0.666, FDE 0.017. Comparable rows for FPT are 0.029/0.936/0.493/0.035 at 0.60 s (4 goals) and 0.026/0.943/0.523/0.032 (16 goals); Motion-I2V is 0.055/0.864/0.150/0.060 at 13.2 s (4 goals) and 0.033/0.957/0.263/0.037 at 16 s (16 goals).
• Large speed advantage in sampling. The paper reports sampling full trajectories 97× faster than video world models. Video baselines in Table 1 list latencies of 178 s (CogVideoX) and 39 s (LTX), versus 0.40 s for GARFIELD.
• Density estimation is orders of magnitude faster. The density decoder predicts heatmaps in a single forward pass in less than 0.8 ms, versus Monte-Carlo sampling from motion generation models — a speedup the paper describes as two orders of magnitude. Measured component latencies: encoder $\mathbf{E}\varphi$ 3.0 ms, density $\mathbf{D}^\text{d}\omega$ 0.8 ms, point-wise $\mathbf{D}^\text{p}\psi$ 37.0 ms, full decoder $\mathbf{D}\theta$ 330.4 ms.
• Better probabilistic calibration. GARFIELD's point-wise and full decoders achieve superior energy score compared to Motion-I2V and FPT (Figure 6a). The density estimator achieves superior discretized energy scores (Figure 6b).
• More conditioning tightens the distribution. Figure 6(c) shows additional conditioning reduces entropy and variance. Figure 3 shows GARFIELD given only $|\widehat{T}| = 4$ matches Motion-I2V using $|\widehat{T}| = 16$ in EPE, and outperforms FPT consistently when using more than 2 goal positions.
• Minimal conditioning suffices. With $|\widehat{T}| = 0$ the model assumes a static scene; with $|\widehat{T}| = 1$ it captures motion but produces inaccurate object relations; with $|\widehat{T}| = 2$ it remains uncertain only about exact speed (Figure 5).
• Entropy-guided conditioning is effective. Entropy-based selection of where to add constraints matches oracle-based conditioning using only half the conditional information, while an error oracle performs worse than random selection (Figure 3c).
• Robotic planning without finetuning. On BridgeData / RT1, GARFIELD 0-shot scores 0.76 / 0.77, compared to Track2Act 0.77 / 0.75 and Flow 0.32 / 0.38, while Track2Act is trained on RT1 and BridgeData splits.
• Pedestrian prediction is competitive but not state of the art. On ETH / HOTEL / SDD (ADE/FDE): zero-shot 1.09/1.79, 0.41/0.71, 36.77/49.86; finetuned (<15k iterations) 0.84/1.44, 0.29/0.56, 19.92/34.98. Trajectron++ reports 0.39/0.83, 0.12/0.21, 8.98/19.02, and IDM reports 0.41/0.62, 0.15/0.25, 7.46/13.83.
• Disentangled latents are essential. Ablations on a held-out validation set give EPE/FDE of 0.479/0.482 for a global token, 0.509/0.510 for entangled representations, and 0.011/0.015 for the factorized representation.
• Smaller latents generalize better out of distribution. At $|\widehat{T}|/T = 1%$ EPE is essentially independent of latent size (0.012 for 16, 64, and 256). At $|\widehat{T}|/T = 90%$, EPE is 0.009 (16), 0.008 (64), and 0.019 (256); the paper uses 64 as a balance.
• Full decoder beats point-wise decoder under uncertainty. With $|\widehat{T}| = 2$, the full decoder reaches EPE 0.023 and FDE 0.029 versus 0.035 and 0.060 for the point-wise decoder; at $|\widehat{T}| = 16$ they are closer (0.013/0.017 versus 0.014/0.024).
• Pretraining decoder choice matters. Pretraining with the point-wise generative decoder gives comparable track prediction to full-decoder pretraining, but full-decoder pretraining yields poor density decoding; pretraining with the density decoder's grid cross-entropy objective degrades both the full and density decoders.

Methodology in Plain English

The authors describe a scene as a set of trajectories $t^i = (t^i_1, t^i_2, \dots)$, where $t_\tau^i \in \mathbb{R}^2$ is the 2D image-plane position of element $i$ at timestep $\tau$. Given a start image $I$ and a sparse set of known positions $\widehat{T}$ that some trajectories must pass through, the model learns the conditional distribution $p(\bar{T} \mid I, \widehat{T})$ over the unknown points.

Because real video only shows one realized future, there is no direct ground truth for the distribution. The authors therefore use generative modeling, training on videos annotated with an off-the-shelf optical tracker (CoTracker3 and TapNext are cited). The encoder produces latent components $\mathbf{z}\tau^i$, each tied to one element at one timestep. A point-wise decoder $\mathbf{D}^\text{p}\psi$ is trained jointly so that the latent is forced to be spatially and temporally localized.

For probability estimation, instead of drawing many samples, a separate density decoder $\mathbf{D}^\text{d}\omega$ predicts, for each bin of a $g \times g$ grid ($g = 20$ plus an out-of-bounds token) over the image, the probability that $t\tau^i$ falls in that bin. It is trained with a grid cross-entropy loss against a one-hot target. Entropy of this distribution measures uncertainty.

For producing concrete trajectories, a full decoder $\mathbf{D}_\theta$ conditions on all latents jointly, which resolves interdependencies that independent point-wise sampling cannot. Architecture uses transformer networks with DINOv2R-B image features (DINO unfrozen), 3D Axial RoPE with the starting position masked in the full decoder, and static camera conditioning from FPT. The point-wise decoder follows MAR and is a 34M parameter MLP. Training covers 20M 224² px videos at 12 FPS; the encoder trains 450k steps with goal sparsity $\widehat{T}/T$ rising linearly from 0.5 to 0.01 over the first 50k steps, then the density estimator (150k steps) and full decoder (200k steps) train with the encoder frozen, using batch size 256, AdamW, learning rate 1e-4 (1e-5 for the density decoder), and bfloat16. The model handles $i \in {1 \dots 64}$ tracks over $\tau \in {1 \dots 32}$ timesteps.

Why This Matters

Impact on research. The paper argues that sampling from a generative motion model is not the same as accessing its distribution, and shows a way to get both from one shared latent. It also argues against dense motion fields and pixel-level video prediction for planning, since capacity spent on static background or appearance details is wasted. The representation is open-set and pretrained at scale, contrasting with domain-specific predictors like Social-LSTM, Social-GAN, and Trajectron++, and with FPT's Gaussian-mixture output limited to a single future timestep.

Real-world applications:

• Robotic manipulation and motion planning, where the model can infer robot arm trajectories from start and goal positions without robotics-specific training.
• Autonomous driving and pedestrian trajectory prediction, where uncertainty-aware forecasts are needed.
• Interactive planning tools, where users insert sparse constraints and watch the distribution of possible futures collapse toward feasible trajectories.
• Uncertainty-aware decision making, where high-entropy regions identify where the future is genuinely undetermined.

Industry relevance. Latency is the headline practical claim: 0.40 s trajectory sampling versus 178 s and 39 s for video models, and sub-millisecond density decoding versus Monte-Carlo estimation. The ability to work from positional goals rather than text prompts or goal RGB frames also matters, since the paper notes those signals are ambiguous or hard to obtain in open-set settings.

Future Directions

• Extending the density formulation beyond the current output form, addressing the limitation the authors attribute to FPT's Gaussian mixture output, and possibly beyond the fixed grid resolution, since the Figure 8 grid-size study is truncated in the provided content and only described as trading quality for latency.
• Closing the remaining gap to domain-specific predictors: GARFIELD is outperformed by recent domain-specific methods on pedestrian trajectory prediction even after finetuning, so further finetuning or adaptation strategies are open.
• Improving out-of-distribution behavior under heavy conditioning, where the paper observes that larger latents (256) degrade relative to smaller ones (16, 64) at $\widehat{T}/T = 90%$, potentially due to over-adaptation.
• Investigating pretraining objectives further: the paper finds the density decoder discretization limits the encoder's learning signal, so alternative ways to combine point-wise, full, and density supervision remain an open question.
• Validating entropy-guided conditioning under noisy user input, which the paper states is evaluated in Appendix F but is not included in the truncated content.

Target Audience

Researchers and practitioners in computer vision, robotics, and autonomous driving who work on motion prediction, trajectory forecasting, or world models. It is most useful to readers interested in probabilistic representations of motion, uncertainty-aware planning, and efficient alternatives to video generation for predicting scene dynamics. Readers need some familiarity with latent variable models and generative decoders to follow the later sections, though the central idea is accessible through the Schrödinger's Cat framing.

Authors’ abstract

Predicting how a scene may evolve from partial observations requires reasoning about multiple possible futures rather than committing to a single trajectory. Existing approaches either generate appearance-dominated video predictions or sample a small number of trajectories without explicitly modeling the distribution of possible motion. We introduce Goal-Aware Representations of Future kInEmatic Latent Distributions (GARFIELD), a probabilistic model of scene kinematics that learns a structured spatio-temporal latent representation of the distribution over possible futures given an image and optional spatio-temporally sparse constraints. The same latent representation enables both joint sampling of all trajectories and direct access to the underlying motion distribution through an efficient deterministic density decoder. As a result, uncertainty about future motion can be localized to specific scene elements and timesteps and progressively refined through additional constraints. Experiments demonstrate strong motion planning performance competitive with large video generation models while sampling trajectories $97\times$ faster. Our method further estimates motion densities two orders of magnitude faster than Monte-Carlo sampling from motion generation models, enabling interactive exploration and uncertainty-aware planning.