Robust Long-term Test-Time Adaptation for 3D Human Pose Estimation through Motion Discretization

AI summary

Overview

• Research area: Computer Vision, specifically 3D human pose estimation and online test-time adaptation (TTA) for streaming video.
• Technical level: Intermediate. The paper assumes familiarity with 3D pose estimators, SMPL body parameters, autoencoders, and vector-quantized codebooks, though the core ideas are described conceptually.
• Scope: The paper proposes a long-term online adaptation framework for 3D human pose estimation that combines motion discretization (a codebook of "anchor motions" learned by unsupervised clustering) with a soft-reset mechanism to combat error accumulation when adapting continuously to streaming videos of a single person.

What This Paper Is About

Pre-trained 3D human pose estimators lose accuracy when the test video differs from their training data. Online test-time adaptation tries to fix this by updating the model on unlabeled streaming inputs using self-supervision, but because those self-supervision signals come from its own imperfect predictions, errors compound over time. The authors tackle this error-accumulation problem so that a model can keep adapting to one person's video for tens of minutes and get better, not worse, by exploiting that person's consistent body shape and habitual motion patterns.

Key Contributions

1. Motion discretization for adaptation. The authors derive a discrete codebook of "anchor motions" by performing unsupervised clustering in the latent space of a motion denoising network. Decoded codebook entries act as regular, low-noise supervision signals for the image-based pose estimator, instead of relying only on the estimator's own noisy 3D outputs.
2. Self-replay enabled by the same discretization. Because the codebook can decode anchor motions, the motion denoising network can be adapted using both incoming test-time estimations and pre-trained replayed anchor motions. This addresses representation drift (the latent codes losing their ability to decode consistent motions) without needing access to the original pre-training data.
3. Efficient soft reset. After adapting to each batch, the pose estimator is reverted partway toward its exponential moving average (decay μ_F), which dampens noisy per-batch updates while retaining traits learned from the person's history.
4. A personalized adaptation protocol. The paper evaluates continuous adaptation to a test domain consisting of streaming videos of a single unseen individual, and shows that the above designs let the model benefit from that person's consistent shape and motion traits.

Main Findings

• Better accuracy than prior online TTA methods. With OpenPose 2D detections on Ego-Exo4D (all scenarios), the method reaches 121.5 mm MPJPE / 68.1 mm MPJPE-PA, versus CycleAdapt at 145.0 / 80.5, BOA at 135.1 / 70.0, and DynaBOA at 153.3 / 71.6. The pre-trained estimator alone gives 205.8 / 116.5.
• Consistent gains per scenario. With OpenPose, on Ego-Exo4D the method reports basketball 141.6 / 71.3, soccer 105.3 / 61.3, and dance 106.4 / 67.0. With ViTPose, the same scenarios report 137.4 / 63.4, 102.2 / 56.8, and 99.8 / 59.5.
• Results on 3DPW include mesh error. With OpenPose, the method reports 83.9 mm MPJPE, 51.6 mm MPJPE-PA, and 100.3 mm MPVPE, compared with CycleAdapt at 141.0 / 79.6 / 155.6 and BOA at 98.2 / 55.8 / 114.2. With ViTPose, it reports 85.0 / 53.3 / 100.4.
• 2D detector choice matters differently per dataset. Using ViTPose detections instead of OpenPose improves Ego-Exo4D results (121.5 / 68.1 to 116.4 / 60.8), but gives limited improvement on 3DPW. The authors attribute this to 3DPW frequently containing overlapping participants that ViTPose struggles to separate.
• Competitive with domain generalization baselines. Compared to methods pre-trained on datasets beyond Human3.6M, the method with ViTPose reports 116.4 / 60.8 on Ego-Exo4D, better than SPIN (137.3 / 71.5), I2L-Mesh (136.6 / 73.8), PyMAF (135.5 / 70.2), VIBE (130.6 / 70.1), TCMR (126.4 / 66.9), GLoT (127.1 / 66.7), and HMR-2.0b with a ViT-H/16 backbone (125.2 / 65.6). On 3DPW the method with ViTPose reports 85.0 / 53.3 / 100.4 versus HMR-2.0b's 81.3 / 54.3 / 94.4.
• Ablation: the components combine synergistically. Removing both motion discretization components and soft reset gives 144.3 / 80.6 MPJPE/MPJPE-PA. Soft reset alone gives 122.9 / 69.2. Anchor loss alone raises error (138.2 / 74.8) because anchors are noisy without self-replay. Using anchor loss, self-replay, and soft reset together gives the best result, 121.5 / 68.1.
• Complementary 3D supervision signals. Using only the denoised motion L_p gives 122.9 / 69.2; using only anchor loss L_ach gives 123.1 / 69.8; using both reaches 121.5 / 68.1. The authors explain that anchor motions preserve regular patterns over long adaptation while denoised motion retains fine details lost through discretization.
• Soft-reset decay must be tuned. Decay values of 0, 0.9, 0.95, and 1.00 yield 138.2 / 74.8, 122.7 / 69.2, 121.5 / 68.1, and 125.3 / 70.2 respectively. A conservative 0.95 is found beneficial.
• Continuous adaptation helps on average. Disabling continuous adaptation (resetting to pre-trained weights before each batch) gives 125.3 / 70.2, versus 121.5 / 68.1 with it. Gains are evident in soccer and dancing; benefits are limited in basketball, where camera setups severely distort some participants and enlarge the domain gap.
• Prior methods degrade under long-term self-supervision. The authors report that DynaBOA and CycleAdapt suffer severe error accumulation, with accuracy decreasing through continuous adaptation, and that BOA/DynaBOA show larger errors especially in MPJPE and MPVPE due to reliance on 2D supervision without adequate 3D guidance.

Methodology in Plain English

The framework has two alternating parts that update each other in cycles: a pose estimator F that turns an image into SMPL pose, shape, and translation (and thus 3D joints), and a motion denoising autoencoder M (an encoder E and decoder D) that looks at a window of poses and cleans up their temporal motion.

Before test time, the authors pre-train M on Human3.6M as a standard denoising autoencoder. In parallel, they cluster the encoder's latent motion representations to build a residual codebook of k = 3 layers, each with N_c = 512 codes of dimension d = 512. The codebook entries are updated by exponential moving average. Decoding a codebook entry produces an "anchor motion," a plausible, regular movement pattern.

At test time, the incoming stream is split into batches of 160 frames. For each batch, the estimator is updated using a combination of detected 2D keypoints (reprojection error), a temporally averaged body shape, the denoised motion from M, and the anchor motion from the codebook. Because the anchor motion comes from a discrete set of regular patterns, it filters out high-frequency noise that would otherwise be fed back into the model.

Then M is updated. The authors build "replay" motions by sampling random codes from the pre-trained codebook and decoding them with the pre-trained decoder, and feed these alongside the test-time motion estimates into M. This self-replay keeps the codebook decoding consistent anchor motions as the latent space evolves, and it avoids needing the original training data.

After each batch, a soft reset applies: the estimator's weights become μ_F · F_pre + (1 − μ_F) · F, an exponential moving average that blurs out noisy per-batch updates. The M codebook is synchronized with replay latents using decay μ_C = 0.999.

Experimental setup. Models are pre-trained on Human3.6M, following prior TTA work, using the same CycleAdapt pose estimator with a ResNet-50 backbone. M processes 16 frames at 15 fps. Adaptation uses Adam with a learning rate of 5×10⁻⁵ annealed to 1×10⁻⁶ via cosine annealing, 12 cycles, mini-batch size 32, loss weights λ1 = 0.001, λ2 = 0.1, λ3 = 0.3, soft-reset decay μ_F = 0.95, and a replay mini-batch size of 4. Evaluation uses Ego-Exo4D (30 participants, basketball/soccer/dance, recordings of 10 to 50 minutes, undistorted exocentric videos) and the 3DPW test split (5 participants, 35,515 frames at 30 fps). Metrics are MPJPE, MPJPE-PA, and MPVPE in millimeters over 17 MS COCO keypoints. Mesh reconstruction was not evaluated on Ego-Exo4D because annotations were unavailable.

Why This Matters

Impact on research. Test-time adaptation methods often assume adaptation is beneficial, but this paper makes error accumulation across long streams an explicit, measured failure mode. It also shows a route to using discrete, learned motion priors as a stabilizer for self-supervised adaptation, and it reframes the test domain as a single person rather than a broad dataset, which is a realistic deployment setting that prior work left largely unexplored.

Real-world applications:

• Fitness and sports coaching, where a system analyzes an athlete's form over long training sessions and should improve as it watches the same person move.
• Physical rehabilitation monitoring, where continuous observation of one patient's habitual motion could sharpen joint angle estimates for progress tracking.
• Human-robot collaboration and immersive interaction, where accurate streaming 3D pose enables responsive systems to react to a specific user.
• Motion capture and animation pipelines that need personalized body shape and movement style captured from a single continuous recording.

Industry relevance. The method reuses an existing public pose estimator and its pre-training, avoids storing pre-training images for exemplar guidance (unlike BOA and DynaBOA, which the authors note increase storage and run-time costs), and reports competitive run-time cost in the appendix. That combination makes it more practical for on-device or privacy-sensitive settings where the original training data cannot be retained.

Future Directions

• Handling distorted or difficult appearances. The authors note limited gains in Ego-Exo4D basketball scenarios, where some participants are severely distorted by camera setups, and list this as a current limitation.
• Stronger backbones and global trajectories. They identify more powerful ViT-based estimators and recent work on recovering global human trajectories from moving cameras as natural extensions of their ResNet-50, local-frame setup.
• Better use of test-time traits under stable adaptation. Adaptive loss weights or adaptive EMA schedules are named as techniques that could exploit personal traits while keeping adaptation stable.
• Extending motion discretization to other self-supervised adaptation problems. The paper only applies the anchor-motion idea to 3D pose estimation, leaving open whether the same discretization-plus-replay strategy transfers to other streaming adaptation tasks.
• Reducing the dependence on 2D detector quality. The differing behavior of OpenPose and ViTPose across datasets, particularly with overlapping people in 3DPW, suggests robustness to 2D detection errors is an unresolved issue.

Target Audience

Researchers and engineers working on 3D human pose estimation, test-time adaptation, or domain adaptation for video, who already understand self-supervised learning and SMPL-style body representations. It is also relevant to practitioners building personalized pose-based products who need models that stay stable over long streaming sessions rather than degrading, and to readers interested in how discrete latent representations can regularize continuous regression tasks.

Authors’ abstract

Online test-time adaptation addresses the train-test domain gap by adapting the model on unlabeled streaming test inputs before making the final prediction. However, online adaptation for 3D human pose estimation suffers from error accumulation when relying on self-supervision with imperfect predictions, leading to degraded performance over time. To mitigate this fundamental challenge, we propose a novel solution that highlights the use of motion discretization. Specifically, we employ unsupervised clustering in the latent motion representation space to derive a set of anchor motions, whose regularity aids in supervising the human pose estimator and enables efficient self-replay. Additionally, we introduce an effective and efficient soft-reset mechanism by reverting the pose estimator to its exponential moving average during continuous adaptation. We examine long-term online adaptation by continuously adapting to out-of-domain streaming test videos of the same individual, which allows for the capture of consistent personal shape and motion traits throughout the streaming observation. By mitigating error accumulation, our solution enables robust exploitation of these personal traits for enhanced accuracy. Experiments demonstrate that our solution outperforms previous online test-time adaptation methods and validate our design choices.