RAPTR: Radar-based 3D Pose Estimation using Transformer

AI summary

Overview

Research area: Radar-based 3D human pose estimation for indoor sensing (computer vision / wireless sensing).

Technical level: Advanced. The paper assumes familiarity with transformer decoders, deformable attention, bipartite set matching, and multi-view sensor geometry.

Scope: A one-paper summary of RAPTR, a transformer pipeline that estimates 3D human poses from multi-view mmWave radar heatmaps using only cheap weak labels (3D bounding boxes and 2D keypoints).

What This Paper Is About

Radar can sense human bodies indoors while preserving privacy and working through occlusion, but existing radar-based 3D pose systems require fine-grained 3D keypoint labels that are expensive and hard to collect, typically using non-portable motion capture rigs such as VICON or LiDAR that still suffer from occlusion. RAPTR replaces those labels with much cheaper supervision: coarse-grained 3D bounding boxes and fine-grained 2D keypoints in the image plane. The goal is to learn geometrically consistent 3D poses from multi-view radar heatmaps under this weak supervision.

Key Contributions

1. First weak-supervision radar 3D pose framework. RAPTR is presented as the first radar-based 3D human pose estimation framework to explicitly use low-cost weak labels (3D BBoxes and 2D keypoints) rather than fine-grained 3D keypoint labels.

2. A structural loss function coupling a two-stage decoder. The loss combines a 3D Template (T3D) loss at the pose decoder, which uses 3D BBox labels, with a combined 3D Gravity (G3D) and 2D Keypoint (K2D) loss at the joint decoder, so the model learns geometrically consistent 3D poses from weak supervision.

3. Pseudo-3D deformable attention. Reference points and sampling offsets are defined in the 3D radar coordinate system and projected onto the two radar views (range-azimuth and range-elevation), eliminating per-view offset estimation and giving better scalability as the number of radar views grows.

4. A new state of the art on two indoor radar datasets. RAPTR reports reduced joint position error of 34.3% on HIBER and 76.9% on MMVR relative to baselines, with code released at https://github.com/merlresearch/radar-pose-transformer.

Main Findings

• HIBER WALK split: RAPTR reaches an overall MPJPE of 22.32 cm, versus 58.25 cm for Person-in-WiFi 3D, 38.20 cm for QRFPose, and 33.96 cm for HRRadarPose. This is a reduction of 61.7%, 41.6%, and 34.3% respectively.

• HIBER MULTI split (multi-person): RAPTR reaches 18.99 cm overall MPJPE, compared with the second-best HRRadarPose at 33.19 cm. This is a reduction of 77.7%, 58.8%, and 42.7% versus Person-in-WiFi 3D, QRFPose, and HRRadarPose. Person-in-WiFi 3D and QRFPose degrade noticeably from WALK to MULTI, while RAPTR stays nearly consistent.

• Challenging joints: On HIBER WALK, HRRadarPose reports a wrist error of 42.33 cm, while RAPTR reports 26.55 cm. RAPTR keeps the gap between its best- and worst-estimated joints within 10 cm.

• Directional accuracy on HIBER: RAPTR's errors along horizontal (h), vertical (v), and depth (d) are 8.41 / 4.85 / 17.73 cm on WALK and 7.80 / 4.38 / 14.54 cm on MULTI, showing much lower horizontal and vertical error than baselines.

• MMVR: Because 3D keypoint labels are unavailable, the paper uses center distance and edge-length error of a fitted 3D BBox. RAPTR achieves a center distance of 31.41 cm, with edge length errors of 22.90 cm (h), 10.66 cm (v), and 50.56 cm (d). Center distance improves by 76.9%, 85.1%, and 80.9% over Person-in-WiFi 3D (136.14 cm), QRFPose (210.75 cm), and HRRadarPose (164.46 cm).

• Loss ablation: Using only the 2D keypoint loss at the joint decoder causes a depth-ambiguity failure with MPJPE of 381.18 ± 0.28 cm (WALK) and 375.73 ± 6.31 cm (MULTI). Removing the T3D loss gives 28.54 ± 4.57 cm and 57.90 ± 9.81 cm; applying K2D+G3D at both decoders gives 27.49 ± 3.40 cm and 23.43 ± 3.44 cm; using T3D with K2D only (no G3D) gives 25.96 ± 4.95 cm and 25.83 ± 3.87 cm. The proposed structural loss achieves 22.32 ± 0.06 cm and 18.99 ± 0.16 cm.

• Attention ablation: Replacing pseudo-3D deformable attention with the decoupled 2D deformable attention from QRFPose (keeping everything else identical) yields 23.25 ± 1.38 cm (WALK) and 19.47 ± 0.95 cm (MULTI), so pseudo-3D attention gives roughly 4% and 2.5% improvements.

• Comparison to 2D-to-3D lifting: A pose-lifting baseline using a pre-trained uplifting model trained on vision datasets scores 43.43 ± 2.66 cm (WALK) and 41.76 ± 6.85 cm (MULTI), well behind RAPTR.

• Qualitative behavior: RAPTR produces structurally consistent poses matching 3D labels in position and orientation, while baselines show misaligned limbs and implausible joint configurations. Other baselines appear to overfit to 2D image-plane alignment, causing structural collapse in 3D space.

Methodology in Plain English

The system takes radar heatmaps from two synchronized arrays — one horizontal (azimuth) and one vertical (elevation) — over T = 4 consecutive frames. Each array's raw pulses are converted by a 3D FFT into a range–Doppler–angle spectrum, integrated along the Doppler axis to boost signal-to-noise ratio, and mapped into Cartesian range-azimuth and range-elevation heatmaps in the 3D radar coordinate system.

A shared ResNet-style backbone extracts multi-scale features from each view. A cross-view encoder with L_enc transformer layers fuses the two views by running shared deformable cross-attention twice per layer — once with horizontal features as key/value and vertical as query, then the reverse — with residual connections preserving view-specific detail.

A two-stage decoder then produces the pose:

1. The pseudo-3D pose decoder starts with N = 10 pose queries, each carrying a reference pose in 3D. Queries and reference poses are refined over L_pose layers, where offsets are predicted in normalized 3D space and applied through a Sigmoid/inverse-Sigmoid update.

2. The pseudo-3D joint decoder takes K joint queries for one subject (K = 14 for HIBER, K = 17 for MMVR**) and refines each joint with the same pseudo-3D deformable attention over L_joint layers.

The pseudo-3D deformable attention is the architectural centerpiece. Each query predicts 3D sampling offsets. A 3D reference point plus offsets defines 3D sampling coordinates, which are projected onto the two 2D radar views for feature sampling. Attention weights over the offsets and the two views are predicted and softmax-normalized, and the sampled features are combined into a single attended feature. Because offsets live in 3D, they are not estimated redundantly per view, which scales better with more radar views. An optional view mask can exclude features from a specific radar view.

Supervision uses only weak labels. A 3D Template loss builds a keypoint template offset by the centroid of the labeled 3D BBox (the 3D gravity center) and measures Euclidean distance to the initial pose estimate at the pose decoder. At the joint decoder, a 3D Gravity loss compares the centroid of the refined pose with the BBox-derived gravity center, and a 2D Keypoint loss transforms the refined radar-space pose into camera space and projects it into the image plane, combining Euclidean image-plane error with an OKS loss. A classification (focal) loss applies to confidence scores. Predictions are matched to labels via bipartite matching, as in set-based detection, and the terms are combined with weighting factors λ1 through λ5.

Evaluation uses MPJPE in centimeters in the world coordinate system, broken down per joint and along the horizontal, vertical, and depth axes. HIBER uses data protocols "MULTI" and "WALK" with views 2 through 10; MMVR uses the single-person open-space split "P1S1". Baselines are Person-in-WiFi 3D, HRRadarPose, and QRFPose — with QRFPose reimplemented from scratch because no public code exists, and verified against the original report using 3D keypoint labels.

Why This Matters

Impact on research: The paper opens a route to radar 3D pose estimation without expensive motion-capture-grade 3D keypoint annotation, which is a major scaling bottleneck for the field. It also shows that naive 2D-only supervision collapses under depth ambiguity (MPJPE jumping to the hundreds of centimeters), and that 3D geometric constraints derived from coarse BBoxes repair this. The pseudo-3D deformable attention offers a view-count-scalable alternative to per-view 2D deformable attention.

Real-world applications:

• Elderly monitoring in homes and care facilities, where cameras raise privacy objections.
• Smart building management, including occupancy and activity sensing.
• Robotic navigation and human-aware robotics indoors.
• Hazardous or non-line-of-sight conditions such as fire or smoke, and through-wall sensing, where cameras fail and radar's penetration capability helps.

Industry relevance: The approach uses commercial mmWave radar, which is low-cost, and the released code plus the use of two public datasets (HIBER, MMVR) make it directly reproducible. The paper's authors are affiliated with Mitsubishi Electric Research Laboratories (MERL), the University of Osaka, and Mitsubishi Electric's Information Technology R&D Center, and the paper notes broader impacts around safety and energy efficiency of indoor systems alongside the need to keep perception results secure and private.

Future Directions

• Resolving front/back depth ambiguity. The paper's stated limitation is that 2D keypoint labels cannot tell whether a person faces toward or away from the camera, so estimated poses may bend joints in the opposite direction in depth from the actual pose. New constraints or labels that encode facing direction could address this.

• Robustness in crowded and interactive scenes. The authors note that occlusion and human-to-human interference degrade performance, and that these effects become more pronounced in crowded or interactive environments. Extending beyond HIBER's WALK and MULTI and MMVR's P1S1 single-person split to harder multi-person settings is a natural next step.

• Reducing dependence on 2D label precision. Since refinement of the template in the joint decoder is supervised by 2D keypoint labels, overall 3D accuracy is highly dependent on their image-plane precision. Alternatives that are less sensitive to label noise would widen applicability.

• Direct 3D evaluation on MMVR. MMVR lacks 3D keypoint labels, so the paper must approximate performance with BBox center distance and edge-length error. Obtaining ground-truth 3D keypoints for such datasets, or validating the approximations, would allow direct MPJPE comparison — and the paper's scalability argument for pseudo-3D attention as the number of radar views grows remains an open avenue to test further.

Target Audience

Researchers and engineers working on radar or RF-based human sensing, multi-view transformer architectures, and weak-supervision 3D pose estimation. It is also relevant to practitioners building privacy-preserving indoor perception products, and to readers interested in deformable attention designs that operate in 3D while sampling from 2D sensor views. The paper presupposes solid background in transformer decoders and pose estimation metrics; beginners would need substantial preliminary reading.

Authors’ abstract

Radar-based indoor 3D human pose estimation typically relied on fine-grained 3D keypoint labels, which are costly to obtain especially in complex indoor settings involving clutter, occlusions, or multiple people. In this paper, we propose \textbf{RAPTR} (RAdar Pose esTimation using tRansformer) under weak supervision, using only 3D BBox and 2D keypoint labels which are considerably easier and more scalable to collect. Our RAPTR is characterized by a two-stage pose decoder architecture with a pseudo-3D deformable attention to enhance (pose/joint) queries with multi-view radar features: a pose decoder estimates initial 3D poses with a 3D template loss designed to utilize the 3D BBox labels and mitigate depth ambiguities; and a joint decoder refines the initial poses with 2D keypoint labels and a 3D gravity loss. Evaluated on two indoor radar datasets, RAPTR outperforms existing methods, reducing joint position error by $34.3\%$ on HIBER and $76.9\%$ on MMVR. Our implementation is available at https://github.com/merlresearch/radar-pose-transformer.