CanonicalPhys: Pose-Robust Remote Photoplethysmography via Canonical-Space Priors

AI summary

Overview

• Research area: Computer vision applied to remote photoplethysmography (rPPG) — estimating the blood volume pulse and heart rate from ordinary facial video without contact.
• Technical level: Advanced. The paper assumes familiarity with deep rPPG backbones, homography estimation, the dichromatic reflection model, and the classical POS/CHROM signal-processing pipelines.
• Scope: The paper diagnoses head pose as a coordinate-structural nuisance for deep rPPG and proposes CanonicalPhys, a parameter-free canonical-space wrapper (four-point homography plus three canonical-space priors) around an unmodified FactorizePhys backbone, evaluated on UBFC-rPPG, PURE, OBF, and MMPD.

Note: the provided paper content is truncated partway through Section 4.7 (Qualitative Analysis); the conclusion is not included, and some dataset-size details are not reported in the available text.

What This Paper Is About

Deep rPPG models achieve sub-bpm heart-rate error on frontal, stationary faces but degrade sharply when the head turns: on MMPD, the state-of-the-art FactorizePhys backbone's MAE grows 1.60 times from frontal frames (|yaw| < 15°) to large-yaw frames (|yaw| ≥ 45°). The authors argue this is not a data-augmentation problem but a coordinate-structural one — in image coordinates, the same pixel maps to different anatomy at different poses, which blocks three priors otherwise natural for rPPG: the dichromatic reflection model, pulse-phase invariance across skin regions, and the POS/CHROM chromaticity projection. Their goal is to restore a pose-stable anatomy-to-pixel mapping so those priors become usable inside a deep network at no added parameter cost.

Key Contributions

1. A differentiable four-point homography with no trainable parameters that maps each frame into a canonical face coordinate system, absorbing the dominant rigid component of head pose before any learned processing and approximately restoring a pose-stable anatomy-to-pixel mapping.
2. Three canonical-space components enabled by that change of coordinates, none adding trainable parameters: a physics-informed per-pixel Lambertian weight from the dichromatic reflection model, a cross-ROI temporal consistency loss enforcing pulse-phase invariance, and a knowledge distillation loss from windowed POS applied to canonical forehead pixels.
3. A matched-seed pose-stratified evaluation on MMPD that isolates pose from other distribution shifts, plus a full cross-dataset transfer across UBFC-rPPG, PURE, OBF, and MMPD reporting the complete metric suite (MAE, RMSE, MAPE, Pearson r, SNR).
4. A failure-mode analysis delineating regimes where the claim does not yet hold: moderate yaw, large yaw, extreme yaw, landmark fallback, and pitch/roll.

Main Findings

• Pose degradation is reduced but not eliminated: On MMPD, CanonicalPhys reduces the frontal-to-large-yaw MAE degradation ratio from 1.60 times (FactorizePhys) to 1.33 times for canonicalization alone, and flattens the mild-yaw bin from 1.32 times to 1.07 times for the warp + Lambertian + consistency variant (across CanonicalPhys variants).
• Pose-stratified MAE (MMPD, Table 1): FactorizePhys: frontal 11.95, mild 15.76, moderate 16.21, large 19.08. Canonicalization only: frontal 11.21, mild 12.23, moderate 17.36, large 14.90. Warp + Lambertian + consistency: frontal 11.50, mild 12.27, moderate 15.65, large 16.57.
• Canonicalization alone carries most of the gain: the warp-only model reaches 11.67 bpm on MMPD versus FactorizePhys's 13.69, closing roughly three quarters of the gap to the full model, with no auxiliary loss or Lambertian weight — which localizes the mechanism to the change of coordinates rather than to regularization.
• Consistency loss is the largest auxiliary effect: −0.50 bpm from warp-only on MMPD, ahead of the Lambertian weight (−0.22). Distillation is individually second-order and slightly negative on aggregate MMPD MAE (+0.42), yet the three components are super-additive: the full model (10.95) improves on the sum of individual deltas (11.37), and distillation still tightens seed variance (UBFC within-dataset std falls to ±.09 for the full model).
• Cross-dataset gains track pose, not sensor: Seed-matched UBFC-trained comparisons show MAE of 1.18 → 0.96 on UBFC, 2.97 → 2.01 on PURE (−32%), 0.69 → 0.65 on OBF, and 13.69 → 10.91 on MMPD (−20%). Pearson correlation on MMPD rises from 0.18 to 0.33, and PURE seed variance is more than halved.
• Pose-richness correlates with absolute gain: Non-frontal frames (|yaw| ≥ 15°) make up 0% of UBFC, approximately 0% of OBF, 7.9% of PURE, and 14.3% of MMPD. Absolute MAE reduction rises accordingly: −0.22 bpm on near-frontal UBFC, −0.96 on PURE, −2.78 on MMPD. The relative percentages invert (−32% on PURE versus −20% on MMPD) only because PURE's baseline error is far smaller.
• Comparison to handcrafted and deep baselines on UBFC → MMPD: Under identical preprocessing, CHROM and POS reach 13.61 and 13.53 bpm MAE, matching FactorizePhys's 13.69, which suggests residual deep-learning error on this target is dominated by pose- and motion-induced nuisance rather than pulse-modeling capacity. PhysNet and EfficientPhys reach 11.52 and 19.09 bpm, both above CanonicalPhys's 10.91.
• Full 4×4 transfer matrix: CanonicalPhys wins 13 of the 16 cells against matched FactorizePhys. Largest reductions fall on pose-rich transfers to non-laboratory targets: 23% on PURE → MMPD and 18% on MMPD → UBFC, and by more than half on OBF-trained transfers. FactorizePhys wins the two near-frontal PURE-trained laboratory cells (PURE → UBFC and PURE → PURE, both below 1.3 bpm and within cross-seed noise) and MMPD → PURE (4.98 vs 5.76 bpm).
• The auxiliary losses act as designed, not as generic regularizers: Reading the forehead and two cheek signals from backbone voxel embeddings on UBFC, mean pairwise phase coherence rises from 0.19 without the consistency loss to 0.68 with it. The Lambertian weight without supervision repairs precisely the 30° to 45° yaw band.
• POS teacher sanity check: On synthetic pulses, POS applied to canonical forehead pixels recovers the injected pulse with Pearson r ≈ 0.96.
• Cost: CanonicalPhys has the same 52K parameter count as the FactorizePhys backbone (no added trainable parameters) and under 10% wall-clock overhead.

Methodology in Plain English

The researchers take an off-the-shelf FactorizePhys rPPG backbone and change what the network sees, rather than what it learns.

First, they pick four facial anchors — the outer left eye, outer right eye, left mouth corner, and right mouth corner — and define fixed canonical target positions for them: (0.30, 0.40), (0.70, 0.40), (0.38, 0.72), and (0.62, 0.72). For every frame they solve an 8-equation DLT linear system in closed form (batched torch.linalg.lstsq) to get the homography that moves the observed anchors onto those targets, then warp the frame with bilinear grid sampling. The operation is differentiable and has no learnable parameters. Frames with degenerate anchor quadrilaterals fall back to the identity warp; to suppress landmark jitter, they compute the temporal median homography within a chunk and use it for all frames in the chunk. The authors note the warp is deliberately low-capacity: a 2D homography is exact only for in-plane rotations, but MediaPipe analysis shows more than 95% of MMPD frames have |yaw| < 45°, so the rigid in-plane component dominates.

Second, they add a per-pixel weight based on the dichromatic reflection model, whose diffuse pulse-carrying component scales as the cosine between the skin normal and the camera axis. In canonical coordinates each pixel maps to a fixed anatomical landmark with a fixed canonical 3D normal; the per-frame MediaPipe rotation brings that normal into camera space, and its z-component is passed through a sigmoid with sharpness α = 8 and bias β = 0.55. This weight multiplies the canonical frame before temporal differencing and adds no learnable parameters.

Third, they add two training-only losses. The cross-ROI consistency loss places three Gaussian soft masks at the backbone's 13×13 feature resolution — forehead at (0.50, 0.20) with σ = 0.10, and left and right cheeks at (0.28, 0.58) and (0.72, 0.58) with σ = 0.07 — extracts a scalar per ROI, temporally differences and normalizes it, and penalizes cosine distance across the three unordered ROI pairs. The POS distillation loss extracts a spatially averaged RGB trace from the canonical forehead and applies POS in its original windowed form with window length L = ⌈1.6 s × FPS⌉ = 48 frames at 30 fps, then distills with a negative Pearson loss against the stop-gradient pseudo-BVP. Weights are λ_cons = 0.1 and λ_dist = 0.2, fixed across all experiments; both losses are disabled at inference.

Landmarks (MediaPipe Face Mesh, subsampled to K = 68 dlib-compatible indices) and 4×4 facial transforms are extracted once per video and cached in LMDB sidecars. All models are trained with the negative Pearson loss using Adam at learning rate 10⁻³, batch size 4, for 30 epochs, on 72×72 RetinaFace-cropped frames chunked into 128-frame clips, with an 80/20 subject split and results on the official held-out test split following rPPG-Toolbox configurations. Every comparison is re-run over 3 seeds under an identical pipeline.

Why This Matters

The work reframes pose robustness in rPPG from a capacity/augmentation problem to a geometry problem, and shows a zero-parameter, architecturally portable fix. Because the canonicalization module plugs in ahead of any backbone — and the three priors are expressed as a weight map and two losses — the recipe is complementary to existing cross-dataset adaptation methods (the authors note FreqPhys and HOT take an orthogonal route via feature-space alignment) and could in principle be combined with them. It also returns classical physical priors (POS, CHROM, the dichromatic model) to a usable role inside deep networks, after they had become largely sidelined.

Real-world applications:

• Telemedicine and remote patient monitoring, where subjects are not instructed to hold still and non-frontal frames are routine.
• Neonatal and infant care, where contact sensors are problematic and the face is frequently turned.
• Driver-state monitoring, where the camera is fixed but the head moves constantly.
• Any deployment where pose-induced degradation directly bounds practical utility, since the authors argue realistic deployments routinely contain non-frontal frames.

Industry relevance: rPPG is attractive precisely because it needs only an ordinary camera, so improving robustness on existing webcam-grade hardware without adding parameters or significant runtime (under 10% wall-clock overhead, 52K parameters) is directly relevant to consumer health sensing, fleet/vehicle safety systems, and camera-based vital-sign products that cannot rely on controlled frontal acquisition.

Future Directions

• Visibility-gated cheek consistency: The consistency-loss variant is the weakest pose-robust variant at |yaw| ≥ 45° because the far cheek is self-occluded and the single-phase loss demands agreement with a partially missing region; weighting each ROI by per-frame landmark visibility is proposed as the natural fix and left to future work.
• 3D mesh warping for extreme yaw: Beyond 60°, the four-anchor quadrilateral is no longer detectable for a non-trivial fraction of frames and those frames fall back to identity, matching the image-space baseline by construction. The authors state they make no pose-robustness claim there and call a 3D mesh warp the natural extension. This connects to the UV-texture-map approach of Cantrill et al., which the authors describe as more expressive than their planar warp but requiring per-frame 3D mesh fitting.
• Pitch and roll: The contribution list names pitch/roll as one of five regimes where the claim does not yet hold; the truncated Section 4.6 text does not contain that subsection, so the specific finding is not reported in the available content.
• Combining with feature-space adaptation: The authors describe their geometric prior as complementary in principle to distributional-alignment methods such as FreqPhys and HOT, leaving combination unexplored.

Target Audience

Researchers and engineers working on camera-based vital-sign sensing and deep rPPG, particularly those deploying models outside frontal, stationary laboratory conditions. It is also relevant to computer-vision practitioners interested in geometry-aware priors, facial-landmark-based canonicalization, or inserting classical signal-processing teachers into deep pipelines. Readers need background in deep rPPG benchmarks and metrics (MAE, RMSE, MAPE, Pearson r, SNR) and in classical POS/CHROM physiology modeling; the paper is not an introductory treatment.

Authors’ abstract

Deep remote photoplethysmography (rPPG) attains sub-bpm heart-rate error on frontal, stationary faces yet degrades sharply under head pose: on MMPD, the state-of-the-art FactorizePhys backbone's MAE grows $1.60\times$ from frontal ($|\text{yaw}|{&lt;}15^\circ$) to large-yaw ($|\text{yaw}|{\geq}45^\circ$) frames. We argue that pose is a \emph{coordinate-structural} nuisance rather than a data-augmentation problem: in image coordinates the same pixel maps to different anatomy at different poses, blocking three priors otherwise natural for rPPG, namely the dichromatic reflection model, pulse-phase invariance across skin regions, and the POS/CHROM chromaticity projection, each of which presumes a stable anatomy-to-pixel mapping. We introduce \textbf{CanonicalPhys}, which prepends a differentiable four-point homography that fixes four facial anchors at canonical positions; in this canonical frame the three priors become expressible as a per-pixel Lambertian weight, a cross-ROI temporal consistency loss, and knowledge distillation from windowed POS, none of which adds trainable parameters over the backbone. At an identical parameter count, CanonicalPhys reduces MMPD's frontal-to-large-yaw MAE degradation from $1.60\times$ to $1.33\times$ and flattens the mild-yaw bin from $1.32\times$ to $1.07\times$ (across CanonicalPhys variants), with matched cross-dataset MAE reductions of up to $32\%$ on pose-rich targets. Code: https://github.com/infraface/CanonicalPhys