Generalized Audio-Driven Synthesis of Precise Drummer Motion

AI summary

Overview

Research area: Computer vision and graphics — audio-driven character animation and generative motion synthesis, specifically full-body drumming performance from raw audio.

Technical level: Advanced. The paper assumes familiarity with diffusion models, motion capture representations, forward kinematics, and audio feature extraction.

Scope: One sentence — this paper introduces a diffusion-based framework, a dual-objective loss, a motion capture dataset with augmentation, and two new evaluation metrics to synthesize spatially precise and temporally aligned drummer motion directly from audio.

What This Paper Is About

Synthesizing realistic drumming motion from sound is hard because drum audio is noisy and acoustically ambiguous, the motion involves extreme accelerations (stick velocities often exceeding 10 m/s), and the same drum hit can be played by either hand. Existing approaches rely on motion matching or MIDI input and struggle to generalize to uncurated, real-world audio, while the field also lacks metrics that can tell precise drumming apart from jittery motion. The authors build a generative system that maps raw audio to full-body drumming motion with centimeter-level stick precision, and they propose metrics to measure that precision rigorously.

Key Contributions

1. High-fidelity dataset and augmentation strategy. A motion capture dataset of 3 h 30 min 53 s (1,518,450 frames) of professional drumming recorded at a high frame rate across fundamentals, grooves, and songs, plus a two-stage augmentation that generates m = 50 audio variants per sample using randomly selected drum kits and 13 randomized audio effects.

2. Dual-objective loss for high-frequency precision. A loss that decouples skeletal integrity from drumstick precision: body joints and stick orientations are supervised as rotations (6D representation), while stick tips are directly supervised in Cartesian coordinates, enabling centimeter-level impact accuracy without smoothing out natural body dynamics.

3. New metrics for percussive motion. The Percussive Alignment Score (PAS), a parameterized Gaussian kernel measuring audio-motion onset correspondence that approaches unity for precise performance and near-zero for random motion, and Impact Point Deviation (IPD), which quantifies stick-tip hitting fidelity per drum component.

4. Proof-of-concept motion-to-MIDI transcription. The audio-to-motion pipeline is repurposed as a drum transcriber, reported to outperform specialized state-of-the-art transcribers on the authors' evaluation set without task-specific training.

Main Findings

• Spatial precision improves dramatically. Average impact point deviation is 8.4 cm for the rotations-only baseline versus 1.9 cm for the dual-objective model. Per-component, snare deviation drops from 14.5 cm to 0.5 cm, ride cymbal from 12.7 cm to 0.8 cm, and crash cymbal (L) from 15.2 cm to 2.2 cm. Standard errors are below 0.01 cm for the rotations-only model and below 0.006 cm for the dual-objective model. For context, each drum component is around 25 cm in diameter, and a stick tip may travel up to 8 cm between consecutive frames at 120 fps.

• Temporal alignment approaches ground truth. Ground truth achieves an average PAS of 0.91. Adding Gaussian noise to motion onsets drops PAS to 0.80 at σ = 25 ms and 0.59 at σ = 50 ms, showing the metric discriminates alignment quality. The authors' model scores 0.82, outperforming the rotations-only baseline at 0.68 and scoring closer to ground truth than even the 25 ms noise condition.

• Users cannot reliably distinguish generated motion from capture. In a two-alternative forced choice study with 22 users rating 15 pairs of 10-second clips, ground truth was preferred over the rotations-only model 92.9% of the time (p < 0.001), and the authors' model was preferred over the rotations-only model 92.8% of the time (p < 0.001). Against ground truth, ground truth was preferred only 58.7% of the time (the authors' model achieved a 41.3% preference rate), which a binomial test found not statistically significant (p = 0.08).

• Augmentation is what enables generalization. On 100 randomly chosen audio tracks from the 13.6-hour Groove dataset, the augmented model scored consistently higher PAS than the non-augmented counterpart, raising the average from 0.7 to 0.84.

• Emergent intensity control. Because the dataset contains takes with varying tempo and energy, the model learned to modulate motion amplitude with input gain: 0 dB audio produces larger motion arcs, while -15 dB audio produces gentler trajectories with smaller arcs.

• Transcription results are preliminary and qualitative. The paper reports that polyphonic tools such as Omnizart and Spotify's Basic-Pitch struggled with percussive transients, while specialized models like ADTOF performed more strongly; motion-to-MIDI is described as more faithfully reproducing ground truth in preliminary tests. Quantitative transcription results are stated as ongoing work and are not reported.

• Non-curated audio evaluation is qualitative only. The authors state that aggregated quantitative metrics for non-curated recordings were excluded because of the lack of a standardized dataset and the high variance among tracks.

Methodology in Plain English

The authors recorded a professional jazz drummer playing a standard 10-component drum kit (snare, kick, hi-hat, hi-hat pedal, two high toms, floor tom, ride cymbal, two crash cymbals) using nine OptiTrack cameras at 120 Hz over two days, with audio captured at 44.1 kHz and MIDI stored for evaluation only. Motion is represented as sequences of poses: 27 body joints encoded as 6D rotations, two drumstick orientations as 6D rotations, and the two stick tips as Cartesian coordinates. Altogether a pose is a 180-dimensional vector, and the drumsticks are treated as rigid bodies attached to the hands. Cross-armed playing was deliberately excluded based on the professional drummer's advice.

To make the model robust to different kits and rooms, each captured sample is paired with m = 50 augmented audio variants: audio is resynthesized from MIDI with a randomly chosen acoustic or electronic drum kit, then passed through 13 randomized effects (reverb, noise, equalization, compression, pitch shift), each with strength sampled uniformly from [0, 1]. One variant is selected at random per epoch during training.

Audio is converted into 44-dimensional per-frame features synchronized to 120 Hz: binary onset detection, binary beat tracking, amplitude envelope, spectral centroid, and 40 MFCCs. The authors deliberately use interpretable handcrafted features instead of a pretrained encoder like Jukebox. Input must be a drums-only track; for polyphonic music, a source separator such as Demucs or Spleeter is applied first.

The generative backbone adapts the EDGE diffusion architecture, a Transformer-based noise predictor conditioned on audio through cross-attention. Training uses 120-frame (one-second) windows, 25,000+ motion sequences with 0.5 s overlap, and about 3,000 held-out test sequences, optimized with Adam (learning rate 3·10⁻⁴, batch size 128) for 6000 epochs on one NVIDIA GeForce RTX 3090 over approximately 48 hours. At inference, sampling starts from pure noise over T = 1000 steps but uses the DDIM sampler so only about 5 steps are needed. Long performances are generated with a sliding window at 50% overlap, blending windows using spherical linear interpolation for rotations and linear interpolation for positions.

The key design choice is the dual-objective loss: a weighted sum of rotation error (λ_r = 0.5) and stick-tip position error (λ_p = 1). Rotation supervision keeps bone lengths fixed and poses natural, while direct Cartesian supervision of the tips avoids the error accumulation that forward kinematics introduces at the end of the kinematic chain. The authors omit the velocity, forward-kinematics, and foot-contact losses used in the original EDGE work, reporting they did not noticeably improve drumming motion quality.

For evaluation, IPD extracts stick-tip positions at audio onsets and uses density-based clustering to discard detections from the non-striking hand, then compares per-component mean impact points to ground truth by Euclidean distance. PAS computes, for each audio onset, the soft correspondence to the nearest motion onset (detected as peaks in stick linear acceleration) using a generalized Gaussian kernel with α = 40 ms and β = 7, so that small delays score high and perceptually asynchronous delays score low.

Why This Matters

This work establishes what the authors describe as a first successful baseline for generalized audio-driven drumming motion, moving beyond MIDI-driven and motion-matching systems toward end-to-end audio input that works on real recordings. It also supplies the field with standardized precision metrics (IPD and PAS) that the authors argue were previously missing, enabling systematic benchmarking of percussive motion synthesis.

Real-world applications:

• Entertainment and games: scalable generation of realistic drummer performances without manual animation or motion capture, including arbitrary-length performances.
• Interactive music education: visualizing how a performance should be played, with motion synchronized precisely to the audio.
• Music production and analysis: the motion-to-MIDI pipeline offers an alternative route to drum transcription, which the authors note is valuable for fine-grained editing of instrumental tracks.
• Virtual performance and avatars: driving digital performers from in-the-wild audio tracks retrieved from arbitrary sources.

Industry relevance: the framework targets deployment on non-curated audio by combining source separation with a drum-only feature extractor, addressing a practical barrier for studios that lack MIDI or whose MIDI is temporally misaligned with the audio.

Future Directions

• Finger-level and facial detail. Adding finger motion and facial dynamics would improve performer realism. The authors note that finger-stick dynamics involve complex sliding and multi-point pivoting, so explicit stick-tip supervision would likely remain necessary even with high-quality finger motion.

• Stylistic nuance. Capturing stylistic expression such as high-intensity punk energy or a laid-back jazz feel through style transfer or expanded datasets remains open.

• Manual authoring controls. Tools for overriding arm assignment and applying temporal offsets would give creators fine-grained control for professional post-production.

• Two-stage architectures and flexible drum kits. The authors suggest investigating a two-stage model, for example first predicting stick-tip positions from audio and then synthesizing motion from those targets, and note that generalizing beyond a fixed spatial drum configuration (via additional capture or geometric conditioning) is left as future work.

• Rigorous transcription evaluation. Extensive quantitative evaluation of the motion-to-MIDI pipeline against existing state-of-the-art transcribers is described as ongoing.

Target Audience

Researchers and practitioners in computer graphics, character animation, and generative modeling who work on music- or audio-conditioned motion synthesis. It is also relevant to music information retrieval researchers interested in drum transcription, to animation and game studios seeking scalable performance generation from audio rather than MIDI, and to anyone needing evaluation metrics for spatial and temporal precision in percussive motion. The paper's heavy use of diffusion terminology, motion representations, and kinematic reasoning makes it best suited to readers with an intermediate-to-advanced background in these areas.

Authors’ abstract

Music-driven character animation enables and enhances transformative applications in entertainment and interactive education. However, synthesizing realistic drumming motion from audio remains challenging due to the inherent tension between high-acceleration dynamics and the need for extreme spatial-temporal precision. Existing approaches, often reliant on motion matching or MIDI input, struggle with generalizing to diverse real-world audio. Moreover, the field lacks standardized evaluation metrics capable of distinguishing precise drumming from noisy motion. In this paper, we introduce a generative diffusion framework featuring a dual-objective loss function that decouples skeletal integrity from drumstick precision, thus enabling centimeter-level stick precision without sacrificing natural body dynamics. Additionally, leveraging our own dataset and data augmentation strategy, the model generalizes to non-curated, in-the-wild audio. To rigorously evaluate performance, we propose two novel metrics: an impact-to-target distance to quantify spatial precision and an audio-motion correlation score to assess temporal alignment. Our quantitative analysis and user studies demonstrate that our system generates high-quality motion that is often indistinguishable from ground-truth performances.