Sound-based Multi-Person 3D Pose Estimation

AI summary

Overview

Research area: Computer vision and active acoustic sensing, specifically non-invasive human 3D pose estimation.

Technical level: Advanced. The paper assumes familiarity with Transformer architectures, DETR-style query-based detection, attention mechanisms, Short-Time Fourier Transform and Mel spectrograms, and standard pose estimation metrics.

Scope: The paper proposes SoundMHPE, the first method to estimate the 3D poses of multiple people from acoustic signals alone, along with a purpose-built 6-hour multi-person acoustic pose dataset (AMP).

What This Paper Is About

Existing sound-based pose estimation systems can only recover the pose of a single person, because when several people move at once their acoustic signatures overlap and reflect off one another, making it hard to tell which signal change belongs to which body. This paper builds a system that accepts overlapping acoustic signals and outputs 3D pose sequences for every person in the scene simultaneously, using only a pair of speakers and a microphone.

Key Contributions

1. First multi-person acoustic pose estimation. The authors present the first attempt to estimate 3D poses of multiple people using only acoustic signals, using an active sensing setup where a speaker emits a signal and a microphone captures the reflections.
2. Acoustic Multi-scale Encoder (AME). An encoder that builds spectrograms at three different temporal window sizes to capture both high-temporal-resolution and high-frequency-resolution information, with dedicated Temporal Self-Attention and Frequency Self-Attention to keep those two kinds of information separate.
3. Temporal Pose Decoder (TPD). A decoder that assigns a separate learnable query to each predicted pose frame for each individual (plus one instance query per person for confidence), and uses Motion Self-Attention for within-person temporal dynamics and Interaction Self-Attention for between-person acoustic interactions.
4. The AMP dataset. A 6-hour Acoustic Multi-person Pose dataset containing approximately 432K synchronized frames of multi-person 3D pose and acoustic data, collected because no existing dataset supports this task.

Main Findings

• SoundMHPE outperforms both baselines on every metric. Against Shibata et al. (adapted single-person acoustic model) and Yan et al. (repurposed WiFi-based multi-person model), SoundMHPE achieves MPJPE 106.5 mm versus 121.7 mm and 119.9 mm, PA-MPJPE 65.0 mm versus 71.5 mm and 69.7 mm, and PCKh@0.5 of 0.43 versus 0.36 for both baselines.
• Both proposed components help, and the decoder matters most. Removing the AME gives MPJPE 115.2 mm, PA-MPJPE 67.5 mm and PCKh@0.5 0.38; removing the TPD gives 116.5 mm, 69.0 mm and 0.38; the full model gives 106.5 mm, 65.0 mm and 0.43.
• Multi-person performance degrades only modestly relative to single-person. On single-person data, SoundMHPE reaches MPJPE 95.0 mm, PA-MPJPE 58.9 mm and PCKh@0.5 0.47, versus 111.3/65.8/0.39 for Shibata et al. and 108.7/64.1/0.40 for Yan et al. On the hardest triple-person setting it reaches 111.2 mm, 68.1 mm and 0.44, versus 124.5/73.3/0.39 and 122.4/71.2/0.38.
• Decoupling attention axes beats standard self-attention. Using standard attention in the encoder with MSA+ISA in the decoder yields MPJPE 111.7 mm, PA-MPJPE 65.1 mm and PCKh@0.5 0.40; using TSA+FSA in the encoder with a standard decoder yields 114.4 mm, 67.9 mm and 0.39; using both proposed schemes yields 106.5 mm, 65.0 mm and 0.43.
• Wider STFT windows help more than narrower ones. Window choices (L/4, L/2, L) give MPJPE 118.5 mm, PA-MPJPE 69.6 mm and PCKh@0.5 0.37; (L/2, L, 2L) give 117.0 mm, 68.9 mm and 0.37; (L, 2L) gives 114.5 mm, 67.9 mm and 0.38; the chosen (L, 2L, 4L) gives 106.5 mm, 65.0 mm and 0.43. The authors conclude high frequency resolution is critical for resolving subtle motion-induced acoustic variations.
• Qualitative behaviors. In double-person scenes the baselines struggle to track "twisting" motion while SoundMHPE reconstructs it; in triple-person scenes SoundMHPE estimates "raising both arms" even while another person walks. The paper's Figure 7 marks one pose-estimation failure with a yellow arrow.
• Generalization to an unseen reflective environment is demonstrated only qualitatively. With black partitions placed in the room to change reflection properties, SoundMHPE still estimated coarse poses, but no numerical results are reported for this experiment.
• Cross-modal applicability to WiFi. On the Person-in-WiFi 3D (PiW) benchmark, the authors' method achieves MPJPE 122.6 mm, PA-MPJPE 69.1 mm and PCKh@0.5 0.31, compared with PiW (Single) at 127.4/71.5/0.13 and PiW (Multi) at 161.0/82.7/0.06. Because WiFi channel state information is not a waveform, the waveform-specific Multi-Scale STFT was omitted and AME self-attention plus the full TPD were applied directly to CSI features.

Methodology in Plain English

Sensing setup. A pair of loudspeakers emits a repeating time stretched pulse (TSP) signal. An ambisonics microphone records the reflections in four channels: W (omnidirectional) plus X, Y and Z (directional), which lets the system capture three-dimensional spatial information. Pose ground truth comes from a Motive motion capture system (OptiTrack) with 16 cameras.

Turning sound into a picture of motion. The received waveform is cut into fixed-length segments and each is converted into a log-Mel spectrogram. To get both sharp timing and fine frequency detail at once, the authors run this process three times with different STFT window sizes (L, 2L, 4L) paired with Mel filter banks (B, 2B, 4B), producing three spectrograms they call high-time, mid and high-freq. These are concatenated along time and projected so their frequency dimensions match.

Encoder. The resulting tokens pass through three stacked AME blocks. Rather than one big attention operation over everything, Temporal Self-Attention looks only within each window-size spectrogram (capturing timing), while Frequency Self-Attention looks across the different window sizes for the same stretch of audio (capturing frequency structure).

Decoder. The model predicts N_out consecutive pose frames from the audio covering N_out + N_prev frames. Each person gets N_out + 1 learnable queries: one per predicted frame plus one instance query whose output feeds a linear layer to produce a confidence score. With M = 15 potential people and N_out = 8, the decoder holds 135 queries. Motion Self-Attention restricts attention to a single person's own queries, and Interaction Self-Attention lets that person attend to the other people's queries, so temporal smoothness and inter-person effects are modeled separately.

Training. Predictions are matched to ground-truth people with Hungarian matching, as in DETR. The loss combines mean squared error on pose with a binary cross-entropy loss on the confidence score, weighted by lambda = 0.5. Settings: N_out = 8, N_prev = 16, Mel filter banks B = 128, embedding dimension E = 256, three encoder blocks, two decoder blocks, AdamW optimizer with weight decay 1e-4, 500 epochs and learning rate 5e-5.

Evaluation. Metrics are MPJPE, PA-MPJPE (Procrustes-aligned) and PCKh@0.5, where the threshold is half the head-to-neck distance. Evaluation is cross-subject: the three subject groups are rotated in a three-fold cross-validation, training on two groups (ten subjects) and testing on the unseen group (five subjects), then averaging.

Dataset. AMP contains 15 subjects (12 male, 3 female) with heights from 150 cm to 181 cm, divided into three groups with randomized pairings and positions. Participants performed walking, twisting, and raising both hands at random speeds and orders. Poses use a 21-joint skeleton. Each group contributed 72 minutes of single-person data and 24 minutes each of double-person and triple-person data, recorded at 20 fps for roughly 432K frames total, in a room with background noise and reverberation.

Why This Matters

Research impact. This work opens a task that had no prior formulation, dataset, or baseline: multi-person 3D pose estimation from sound. It supplies both a task definition and an accompanying dataset, and its cross-modal experiment suggests the architectural ideas transfer to WiFi sensing, which is a useful signal for the broader non-visual sensing community.

Real-world applications.

• Monitoring systems in indoor facilities, workplaces, or public spaces where cameras are undesirable.
• Sports analysis, where the setup can remain unobtrusive and unaffected by lighting.
• Disaster relief efforts, where smoke, dust or darkness defeats camera-based pose estimation.
• Environments where wireless communication is restricted, such as medical facilities or aircraft, are explicitly named as motivation for using acoustics instead of RF/WiFi.

Industry relevance. Acoustic sensing uses cheap commodity hardware: a pair of loudspeakers and a single ambisonics microphone. Unlike RGB methods it does not depend on lighting and is less exposed to the privacy concerns the paper attributes to camera-based approaches; unlike wireless methods it can be deployed where wireless transmission is restricted, and the paper argues acoustic signals have the potential to work behind obstacles because of their relatively long wavelengths. The ceiling is that all present acoustic pose methods, including this one, assume a soundproof room, and the paper notes that real-world deployment remains a general challenge because large-scale comprehensive datasets have not yet been constructed.

Future Directions

• Evaluate across multiple experimental environments. The authors state they plan to advance sound-based pose estimation by testing SoundMHPE in more environments; the current unseen-environment test with partitions is qualitative only, so quantitative generalization numbers are an open item.
• Improve robustness and generalization. The conclusion explicitly lists improving robustness and generalization capability as future work.
• Scale up data collection. The paper identifies the absence of large-scale, comprehensive datasets as a reason real-world deployment is difficult, framing its own AMP contribution as a step toward fixing that.
• Move beyond the soundproof-room assumption. Both existing acoustic methods and this one assume a soundproof room; handling background noise and reverberation outside controlled settings remains unresolved.

Target Audience

Researchers and graduate students working on human pose estimation, acoustic or wireless sensing, and ubiquitous computing; practitioners building non-camera monitoring, sports analysis, or search-and-rescue sensing systems; and anyone interested in Transformer architectures for spatio-temporal signal-to-pose problems, including readers of the DETR-based multi-person pose estimation literature that this work builds on.

Authors’ abstract

Can we recover the 3D poses of multiple people using only sound? This paper presents the first attempt to estimate multi-person 3D poses solely from acoustic signals. Estimating the poses of multiple individuals using acoustic signals is inherently challenging due to the superposition of motion-dependent signal variations. Unlike single-person scenarios, the presence of multiple subjects leads to overlapping acoustic signatures, making it difficult to attribute specific signal changes to an individual's pose. Furthermore, the complexity is compounded by inter-person reflections, which introduce intricate propagation delays that obscure the temporal motion-acoustic relationship. To address these issues, we propose SoundMHPE (Sound-based Multi-person Human Pose Estimator), a novel encoder-decoder framework consisting of two key components. First, the Acoustic Multi-scale Encoder captures diverse temporal and fine-grained frequency features to isolate subtle acoustic signatures from complex, overlapping signals. Second, the Temporal Pose Decoder employs an attention mechanism to disentangle multi-person information across successive frames. By jointly accounting for temporal dynamics and inter-person dependencies, this component precisely reconstructs frame-wise individual poses. To validate our approach, we constructed the 6-hour Acoustic Multi-person Pose (AMP) dataset consisting of 432K synchronized frames of multi-person pose and acoustic data, and demonstrated that our SoundMHPE outperforms baseline models. Project page: https://oumi03.github.io/sound-mhpe/