Towards Characterizing Knowledge Distillation of PPG Heart Rate Estimation Models

AI summary

Overview

• Research area: Machine learning for wearable physiological sensing; specifically, knowledge distillation of deep models that estimate heart rate from photoplethysmography (PPG) signals.
• Technical level: Intermediate — assumes familiarity with neural networks, model compression, and standard evaluation metrics, but the core ideas are explained concretely.
• Scope: An initial, characterization-focused study that sweeps teacher and student model capacities across four distillation strategies to describe how distilled PPG heart-rate models scale in accuracy and compute cost.

What This Paper Is About

Deep learning models can estimate heart rate from PPG signals collected by wearables, but the large models that perform well are too memory- and latency-heavy to run on a watch or fitness tracker. This paper asks how those large models can be compressed into smaller "student" models through knowledge distillation, and whether the resulting accuracy is predictable from the size of the teacher and student. Rather than proposing a new distillation algorithm, the authors map out the behavior of existing methods across a wide range of model sizes.

Key Contributions

1. A systematic sweep of distillation for PPG heart rate estimation, varying both teacher sizes (2, 3, 4, 5, 6, 8, 10, 12 residual blocks) and student sizes (1, 2, 3, 4, 5, 6, 8, 10 residual blocks).
2. A head-to-head evaluation of four distillation strategies — hard distillation, soft distillation, decoupled knowledge distillation (DKD), and feature distillation — under a fixed 12-block teacher.
3. A characterization of distillation scaling behavior, showing distilled student performance follows a predictable exponential curve against model size, with saturation appearing around 6 residual blocks (139K parameters).
4. A system-level benchmark linking model size to GPU inference time and peak memory, quantifying the compute-performance trade-off of distillation.

Main Findings

• Distillation beats training from scratch. Across configurations, distilled student models achieved lower Mean Absolute Error (MAE, in beats per minute) than models of the same size trained from scratch. The from-scratch baseline was consistent with prior work, with the 8-block target model yielding an MAE similar to that reported by Meier et al. using the same model.
• DKD performed best of the four strategies. With the teacher fixed at 12 blocks, DKD achieved the lowest MAE at every student size: 8.899 (1 block), 6.772 (2), 6.689 (3), 6.849 (4), 6.522 (5), 6.291 (6), 5.959 (8), and 5.759 (10).
• Ranking of the remaining strategies. Feature distillation came next (9.397 at 1 block down to 6.409 at 10 blocks), followed by soft distillation (10.380 to 6.327), and hard distillation last (11.734 to 6.493). Hard distillation performed worst because its discrete labels carry little information; soft distillation was marginally better.
• Hyperparameters that worked best for DKD. Through hyperparameter search, alpha = 1, beta = 8, temperature tau = 2, and cross-entropy loss weight CE = 1 gave the best results. Under these settings, non-target class distillation (NCKD) probabilities were weighted 8 times more than target class distillation (TCKD) probabilities.
• Predictable exponential scaling. Distilled student performance followed a predictable exponential curve in its relationship to student model size, consistent with prior scaling-law work on language model distillation. Performance appeared to saturate at student models of 6 residual blocks (139K parameters). DKD, feature distillation, and soft distillation all adhered to these curves, while hard distillation produced a much sharper saturation at a smaller model size.
• Architecture matters. Comparing ResNet and multilayer perceptron (MLP) students distilled from a ResNet-like teacher, MLP students also scaled predictably but consistently underperformed ResNet students, which showed better scaling efficiency and a lower error floor. The authors attribute this to the inductive biases of convolutional layers and residual connections.
• Teacher size effects. Larger teachers generally performed better, but the authors hypothesize that too-large models may overfit easily, degrading results.
• Substantial compute savings. Benchmarked on an Nvidia RTX 2080-Ti GPU, inference time rose from 0.512 s (1 block) to 4.758 s (12 blocks), and memory usage from 9.468 MB to 23.483 MB. Distilling the 12-block model to a 1-block model produced nearly a 90% decrease in inference time and a 60% decrease in memory usage, with only a 30% reduction in MAE.

Methodology in Plain English

The authors use a 1D-ResNet backbone (the variant used by Meier et al.) that classifies instantaneous heart rate into 180 classes covering 30 to 210 BPM, treating the problem as classification rather than regression. Model capacity is varied by changing the number of residual blocks, which produces an approximately exponential sweep of parameter counts — from 23,292 parameters (1 block) up to 863,676 (12 blocks).

Four distillation recipes are tested. Hard distillation trains the student on the teacher's discrete argmax predictions. Soft distillation trains the student on the teacher's full output probability distribution. Decoupled knowledge distillation (DKD) splits the teacher's outputs into target-class and non-target-class components that can be weighted separately. Feature distillation goes inside the network, training the student to match the teacher's intermediate feature maps. Every distilled student is compared against a same-size model trained from scratch.

All models were trained for 300 epochs at a learning rate of 5×10⁻⁴ with cross-entropy loss. The data came from three free-living PPG datasets — WildPPG, PPG-DaLiA, and GalaxyPPG — totaling 107 hours of signals. Only the green PPG channel was used, resampled to 25 Hz and segmented into 8-second windows with 2-second strides, with heart rate ground truth derived from ECG. Splits were participant-independent (80% of participants for training, 20% for evaluation), with 2-fold cross validation across all experiments.

Why This Matters

Impact on research. The paper extends distillation scaling laws — previously studied mainly for language models — into physiological sensing. It gives other researchers a baseline characterization to build on and shows that choices like architecture and distillation strategy shift the scaling curve, not just the absolute error.

Real-world applications:

• Continuous heart rate tracking on smartwatches and fitness trackers, where the model must run within tight memory and latency budgets.
• Real-time exercise feedback, such as monitoring heart rate response to exercise intensity as it happens.
• Passive screening for conditions such as hypertension, where on-device inference also keeps sensitive physiological data private.
• General compression of other wearable sensing models (for example, the audio and accelerometer distillation work cited in the paper), where the same scaling methodology could apply.

Industry relevance. Wearable and consumer health companies need models that fit on low-power microprocessors. This work quantifies what they can expect from distillation — specifically, that a much smaller model can recover most of the accuracy while cutting inference time and memory dramatically — and identifies DKD as the strongest of the tested strategies.

Future Directions

• Cross-dataset generalization. The current evaluation uses a naive cross-validation scheme with shuffled samples from three datasets. The authors want to study generalization by training on one dataset and testing on another, citing work such as Kasnesis et al.
• Richer teacher models. Experiments so far use a straightforward supervised ResNet as the teacher. The authors propose using larger models trained with recent contrastive or self-supervised approaches — noting that the model from Saha et al. (2025) will be open source soon — to see whether richer features distill better.
• New distillation strategies. The four methods here are all drawn from existing literature. The authors intend to design distillation methods specifically suited to this class of physiological sensing tasks.
• On-device validation. The compute benchmarks were run on an Nvidia RTX 2080-Ti GPU, which the authors explicitly state is not representative of the target scenario (microprocessors in wearable devices). Testing on actual edge hardware remains open.

Target Audience

Researchers and engineers working on on-device machine learning for wearables and mobile health, especially those interested in model compression, knowledge distillation, or scaling laws. It is also useful for practitioners who need a concrete, benchmarked sense of the accuracy-versus-compute trade-off when shrinking physiological sensing models, and for students entering the intersection of time-series health sensing and efficient deep learning.

Authors’ abstract

Heart rate estimation from photoplethysmography (PPG) signals generated by wearable devices such as smartwatches and fitness trackers has significant implications for the health and well-being of individuals. Although prior work has demonstrated deep learning models with strong performance in the heart rate estimation task, in order to deploy these models on wearable devices, these models must also adhere to strict memory and latency constraints. In this work, we explore and characterize how large pre-trained PPG models may be distilled to smaller models appropriate for real-time inference on the edge. We evaluate four distillation strategies through comprehensive sweeps of teacher and student model capacities: (1) hard distillation, (2) soft distillation, (3) decoupled knowledge distillation (DKD), and (4) feature distillation. We present a characterization of the resulting scaling laws describing the relationship between model size and performance. This early investigation lays the groundwork for practical and predictable methods for building edge-deployable models for physiological sensing.