Lifelong Domain Adaptive 3D Human Pose Estimation

AI summary

Overview

Research area: Computer vision — 3D human pose estimation, specifically unsupervised domain adaptation and lifelong (continual) learning.

Technical level: Advanced. The paper assumes familiarity with 2D-to-3D lifting pipelines, generative adversarial networks, diffusion models, and the standard metrics (MPJPE, PA-MPJPE) used in pose estimation.

Scope: The paper introduces and solves a new task called lifelong domain adaptive 3D human pose estimation, where a pose estimator is pretrained on a labeled source domain and then adapted sequentially to a stream of unlabeled target pose domains without access to the source or any earlier target domains.

What This Paper Is About

Existing domain adaptation methods for 3D human pose estimation assume target pose distributions are static, or allow simultaneous access to all target data. In reality, target pose distributions keep shifting — the paper cites autonomous driving as an example, where a system must move from predicting pedestrian intentions outdoors to monitoring passenger safety inside a vehicle. The authors define a new setting in which the estimator is adapted to one target domain at a time, cannot revisit earlier domains, and must still perform well on all previously seen domains while learning the current one.

Key Contributions

1. A new task definition. The paper introduces lifelong domain adaptive 3D HPE, where the estimator is pretrained on a labeled source domain at time t=0 and then adapted sequentially to target domains tg_1, ..., tg_N, with no access to the source or prior targets during any adaptation step. The authors state this is the first work to tackle lifelong domain adaptation in 3D HPE.

2. A GAN-based adaptation framework. The method combines 3D pose generators, a 2D pose discriminator, and a 2D-to-3D lifting pose estimator, using a min-max game to reduce domain shift and align original and augmented poses.

3. A new 3D pose generator paradigm. Each generator integrates three kinds of knowledge: pose-aware encoding (with six body part segments — left hand, right hand, left leg, right leg, torso, and extended torso), temporal-aware encoding, and domain-aware encoding produced by an unconditional 2D pose diffusion sampler. The pose- and temporal-aware parts target current-domain adaptation; the domain-aware part targets catastrophic forgetting.

4. Comprehensive benchmarking. Experiments across H3.6M, MPI-INF-3DHP, and 3DPW in three adaptation configurations, plus ablations of every generator component, every loss, EMA, diffusion sampling steps, generative model choice, 2D pose source (DET and HRNet), and 2D-to-3D backbones.

Main Findings

• Cross-scenario adaptation on H3.6M (S1 → S5 → S6 → S7 → S8, t=4): The method reaches average MPJPE/PA-MPJPE of 44.9/36.9, versus 47.6/39.4 for PoseDA-LL, the strongest baseline listed. The authors report this as an average improvement of 2.7mm/2.5mm over PoseDA-LL. Source-only at t=0 gives 53.5/46.3 on S5, 56.3/47.7 on S6, 50.4/41.9 on S7, and 46.6/35.1 on S8.

• Cross-dataset adaptation on H3.6M → 3DHP (TS1–TS6, t=6): Average MPJPE/PA-MPJPE of 75.3/50.7 versus 80.7/54.5 for PoseDA-LL, described as a 5.4mm/3.4mm improvement. TS1 shows a gain of 6.7mm/5.6mm. Source-only at t=0 ranges from 83.0/59.2 (TS1) to 109.4/64.5 (TS5).

• Multi-dataset adaptation (H3.6M → 3DHP → 3DPW and H3.6M → 3DPW → 3DHP, t=2): The method surpasses PoseDA-LL by 9.8mm/7.4mm on average, with a 13.6mm/11.0mm improvement on 3DHP for the first sequence (75.3/51.1 versus 88.9/62.1).

• Domain-aware encoding is the most critical generator component: Removing DE degrades MPJPE by 8.2mm on 3DHP and 6.3mm on 3DPW for the H3.6M → 3DHP → 3DPW task.

• EMA is essential against forgetting: Removing the exponential moving average causes drops of 5.9mm/6.5mm; removing the discrimination loss L_dis causes increases of 7.2mm/4.1mm. The paper concludes all components (L_2D, L_3D, L_dis, EMA, and the three generator components) are necessary.

• Sampler step count matters: With maximum training steps T=400, sampling only T/10 steps gives the best results (75.3/51.1 on 3DHP, 81.7/45.6 on 3DPW), outperforming step=0 random noise (83.4/57.3 and 83.7/47.6), T/40, T/20, T/5, T/2, and full T sampling.

• GAN beats other generative models in this setting: On H3.6M at t=4, the GAN approach averages 44.9/36.9 versus 47.9/39.8 for DDIM and 48.7/40.5 for VAE. On H3.6M → 3DHP at t=6, it averages 75.3/50.7 versus 79.7/54.3 (DDIM) and 81.4/55.4 (VAE). The authors attribute the choice of GAN for 3D pose generation to interpretability, while using diffusion for the domain-aware priors because diffusion preserves mode coverage and diversity better than GANs.

• Robust to different domain arrival orders: With reversed sequences (S1 → S8 → S7 → S6 → S5 at t=4), the method leads PoseDA-LL by 2.8mm MPJPE and 3.9mm PA-MPJPE on average; for H3.6M → TS6 → … → TS1 at t=6, it leads by 4.2mm MPJPE and 3.4mm PA-MPJPE.

• Works with detected 2D poses: When DET generates the 2D poses, the method still leads PoseDA-LL by 13.2mm MPJPE and 11.2mm PA-MPJPE on 3DHP for the H3.6M → 3DHP → 3DPW task, though performance degrades relative to using ground truth 2D poses.

• Backbone agnosticism: The supplementary reports analysis with PoseFormer and MixSTE as the 2D-to-3D lifting backbones. The PoseFormer results are reported for t=2 on the two multi-dataset tasks; the MixSTE table is truncated in the provided content. Detailed computational complexity and runtime analysis are listed as supplementary sections but their values are not reported in the available text.

Methodology in Plain English

The pipeline has three moving parts.

The pose estimator. A standard 2D-to-3D lifter takes 2D joint coordinates as input and predicts 3D joint positions. It is first trained on labeled source data (time t=0), where source training pairs are written as (x_i^sr, y_i^sr) with M_sr pairs.

The generators and discriminator. At each new target domain, the estimator predicts pseudo-3D poses from that domain's 2D poses. Three generators then augment those poses: one encodes body-part structure (the six part segments), one encodes temporal information from consecutive frames through a temporal weighted convolutional network, and one encodes domain-aware priors. The augmented 3D poses are projected back into 2D using the source camera parameters. A 2D discriminator plays a min-max game with the generators, distinguishing original from augmented 2D poses, which pushes the augmented poses toward the target distribution.

The domain-aware prior. To remember earlier domains without storing their data, the model trains a 2D pose diffusion sampler (DDIM-style, with a U-Net) on 2D poses from prior domains. At adaptation time it draws new 2D poses from this sampler and encodes them as domain-aware embeddings. The sampler converges quickly because pose data is small (typically 16×2 for 16 joints) compared with image inputs (typically 224×224 for a ResNet). At sampling time only T/10 of the training steps are used, since exact reconstruction is not the goal.

Training objective. Three losses drive learning: L_3D aligns predicted 3D poses with augmented 3D poses using MSE plus a feedback term that keeps augmentation meaningful; L_2D compares ground-truth 2D poses with augmented 2D poses using MSE plus a normalized-scale term that balances preserving and normalizing scale; L_dis is a Wasserstein GAN loss with gradient penalty. The generators are updated with L_3D − β·L_dis, and the discriminator plus estimator with L_2D + γ·L_dis. After each domain, an exponential moving average (η = 0.99) blends the pre-adaptation and post-adaptation estimator weights for the next round. The generators and discriminator carry directly over from one timestamp to the next.

Setup details. Learning rates are 1e-4 for generators/discriminator and 5e-5 for the estimator, with α = 0.35 and β = γ = 2.5, Adam for generators/discriminator and AdamW for the estimator, batch size 1024, 27 frames, 40 epochs of source pretraining and 30 epochs per target adaptation. The diffusion sampler uses a U-Net, batch size 64, Adam with lr = 2e-4, 10 training epochs, and sampling steps drawn from Uniform[1, 400]. Evaluation uses a 16-keypoint body model with MPJPE and PA-MPJPE.

Baselines. Since no prior lifelong 3D HPE baselines exist, the authors convert existing methods: AdaptPose, PoseDA, and CycleAdapt become AdaptPose-LL, PoseDA-LL, and CycleAdapt-LL; the lifelong classification methods RMT and CoTTA become RMT-Pose and CoTTA-Pose with classification losses swapped for MSE.

Why This Matters

Research impact. The paper opens a new evaluation protocol for 3D HPE that reflects how pose distributions actually change over time, and it argues that prior source-free and test-time adaptation settings assume static target distributions. By defining the task and providing converted baselines, it gives the community a comparable starting point. The finding that a diffusion-based domain-aware prior plus EMA is the combination that fights forgetting is a reusable design lesson beyond pose estimation.

Real-world applications (drawn from the paper's stated motivations and scenarios):

• Person re-identification, which the paper lists as a downstream application of 3D HPE.
• Action recognition, also listed among the motivating applications.
• Virtual reality, listed as a third motivating application.
• Autonomous driving, described concretely: predicting pedestrian intentions in outdoor environments and monitoring passenger safety inside vehicles, two contexts with very different pose statistics that a deployed system must handle in sequence.

Industry relevance. Any deployed pose system — driver and occupant monitoring, robotics, sports analytics, AR/VR — will encounter environments that change over time and cannot upload or retain every past dataset, whether for privacy, storage, or bandwidth reasons. The lifelong setting matches that constraint, and the paper's use of detected 2D poses (DET, HRNet) rather than only ground truth suggests the approach is intended to sit on top of real perception stacks.

Future Directions

• Reducing dependence on ground-truth 2D poses. The main experiments use ground truth 2D input; the supplementary shows performance degradation with DET and HRNet. Closing that gap, or training the whole stack end-to-end with a 2D detector, is a natural next step.
• Scaling to longer and noisier domain streams. The paper evaluates up to 4 sequential H3.6M subjects and 6 3DHP test sets. Whether the diffusion prior and EMA still hold up across many more domains, or with domains that overlap or recur, is untested here.
• Runtime and deployment cost. Computational complexity and runtime analysis are promised in the supplementary section list but their results are not reported in the available content, leaving efficiency on embedded or automotive hardware an open question.
• Generalizing the recipe beyond pose. The paper asserts this is the first lifelong domain adaptation work in 3D HPE and notes that existing lifelong DA methods focus on classification. Whether the same generator-plus-diffusion-prior structure transfers to other structured regression tasks, and whether one-stage generative alternatives (the VAE and DDIM comparisons) can be made competitive with better design, remain open.

Target Audience

Researchers and graduate students working on 3D human pose estimation, unsupervised domain adaptation, or continual/lifelong learning. It is also relevant to applied engineers building deployed pose systems that must adapt across changing environments without retaining past data. Readers unfamiliar with GANs, diffusion models, or MPJPE will need background reading first, since the method and results sections assume that vocabulary throughout.

Authors’ abstract

3D Human Pose Estimation (3D HPE) is vital in various applications, from person re-identification and action recognition to virtual reality. However, the reliance on annotated 3D data collected in controlled environments poses challenges for generalization to diverse in-the-wild scenarios. Existing domain adaptation (DA) paradigms like general DA and source-free DA for 3D HPE overlook the issues of non-stationary target pose datasets. To address these challenges, we propose a novel task named lifelong domain adaptive 3D HPE. To our knowledge, we are the first to introduce the lifelong domain adaptation to the 3D HPE task. In this lifelong DA setting, the pose estimator is pretrained on the source domain and subsequently adapted to distinct target domains. Moreover, during adaptation to the current target domain, the pose estimator cannot access the source and all the previous target domains. The lifelong DA for 3D HPE involves overcoming challenges in adapting to current domain poses and preserving knowledge from previous domains, particularly combating catastrophic forgetting. We present an innovative Generative Adversarial Network (GAN) framework, which incorporates 3D pose generators, a 2D pose discriminator, and a 3D pose estimator. This framework effectively mitigates domain shifts and aligns original and augmented poses. Moreover, we construct a novel 3D pose generator paradigm, integrating pose-aware, temporal-aware, and domain-aware knowledge to enhance the current domain's adaptation and alleviate catastrophic forgetting on previous domains. Our method demonstrates superior performance through extensive experiments on diverse domain adaptive 3D HPE datasets.