Densemarks: Learning Canonical Embeddings for Human Heads Images via Point Tracks

AI summary

Overview

• Research area: Computer vision — dense correspondence learning, human head modeling, and canonical-space representations for faces and heads.
• Technical level: Intermediate. The paper assumes familiarity with Vision Transformers, contrastive learning, trilinear interpolation, and 3D Morphable Models, though the core idea is intuitive.
• Scope: The paper introduces DenseMarks, a learned per-pixel 3D canonical embedding for human head images trained from point tracks, and benchmarks it on point matching, dense warping, monocular head tracking, and stereo reconstruction.

What This Paper Is About

Tracking human heads typically relies on sparse facial landmarks or parametric 3D meshes, which cover only stable facial features and break down under occlusion, extreme pose, or for regions like hair and accessories. DenseMarks aims to replace that sparse supervision with a dense, per-pixel representation: a Vision Transformer predicts a 3D coordinate inside a shared "canonical unit cube" for every pixel of a head image, so the same semantic point on any person lands at the same cube location. The goal is complete, pose-robust head correspondence that includes hair and accessories and supports nearest-neighbor matching, region querying, and downstream tracking.

Key Contributions

1. A 3D canonical embedding space for heads. Instead of a UV surface parameterization, DenseMarks uses a unit cube in 3D, which the authors argue handles complete heads including hair and accessories without the seams that UV space introduces. Each voxel of a discretized grid of resolution N_d × N_d × N_d carries a D-dimensional latent feature, forming a learnable matrix E_raw in R^((N_d)^3 × D), smoothed by a 3D Gaussian filter with strength σ.

2. Supervision from off-the-shelf point tracks rather than dense ground truth. The network is trained on pairwise matches obtained by running CoTracker3 over in-the-wild talking-head videos, with a CLIP-style contrastive loss that pushes matched features together and unmatched features apart. The loss requires the pairwise cosine-distance matrix of normalized features to approach the identity matrix.

3. Multi-task structuring of the canonical space. Landmark anchoring to 300W-format predefined cube locations and a face-parsing segmentation head (a single conv1x1 layer) are added on top of the contrastive objective, making the space interpretable and queryable by region. The overall loss combines the contrastive term with λ_lmks = 50 and λ_segm = 1 weights.

4. A compact 3-dimensional embedding that beats far larger features. The resulting embeddings use only 3 dimensions yet outperform baselines whose embeddings range from 16 to 1280 dimensions on geometry-aware matching benchmarks.

Main Findings

• State-of-the-art correspondence quality on Nersemble. On same-person image pairs, DenseMarks achieves MAE 3.68, RMSE 5.90, and PCK@r=0.05 of 0.90. The strongest generic baseline, DINOv3 (768-dim), reaches MAE 7.60, RMSE 12.69, PCK 0.72; CSE (16-dim) reaches MAE 11.22, RMSE 17.92, PCK 0.55; Sapiens (1280-dim) reaches MAE 14.88, RMSE 24.12, PCK 0.56; FaRL (768-dim) reaches MAE 21.38, RMSE 34.41, PCK 0.46; Fit3D (768-dim) reaches MAE 12.75, RMSE 21.83, PCK 0.57; Hyperfeatures (384-dim) reaches MAE 8.26, RMSE 13.29, PCK 0.72.

• Cross-person consistency. For identity preservation on cross-person pairs, DenseMarks obtains ArcFace 0.384 and Met3R 0.388, versus 0.359 and 0.490 for CSE, 0.329 and 0.454 for Hyperfeatures, 0.266 and 0.460 for DINOv3, 0.167 and 0.595 for Sapiens, 0.166 and 0.632 for FaRL, and 0.236 and 0.558 for Fit3D.

• The canonical space itself is doing real work. Removing it (training on head data without the canonical bottleneck) worsens every metric: MAE 6.35, RMSE 10.20, PCK 0.85, ArcFace 0.348, Met3R 0.455, compared to 3.68, 5.90, 0.90, 0.384, and 0.388 with it.

• Robustness to noisy pseudo-ground-truth. Dropping tracks to 10% of the original still yields MAE 4.52; keeping 20% gives 4.17; 40% gives 3.87; 80% gives 3.86; the full-track run (at 50% of training steps) gives 3.85. Adding Gaussian noise is more damaging: σ = 16 px yields MAE 6.29, σ = 8 gives 5.14, σ = 4 gives 4.26, σ = 2 gives 3.86, and σ = 0 gives 3.85. Even at σ = 16 px the method is reported to still outperform all baselines.

• Landmark and segmentation losses matter for interpretability, but over-weighting them hurts. With λ_segm = 0, MAE improves slightly to 3.68 and RMSE to 5.86, but ArcFace drops to 0.353 and Met3R rises to 0.420; λ_segm = 0.2 gives MAE 3.72, RMSE 5.95, ArcFace 0.400, Met3R 0.374; λ_segm = 5 gives 3.98, 6.39, 0.384, 0.400. Lowering λ_lmks to 10 degrades performance (MAE 5.35, RMSE 8.68), while raising it to 250 gives 3.92, 6.29, 0.393, 0.387.

• Qualitative failure modes of baselines. Dense warping visualizations show Hyperfeatures and especially Sapiens matching by color rather than semantics, while DINOv3 and CSE produce semantically meaningful but artifact-prone correspondences.

• Downstream gains in monocular tracking. Adding a DenseMarks-based photometric loss on a FLAME UV texture to the off-the-shelf VHAP tracker improves robustness most for extreme poses and in challenging regions such as the neck and ears.

Methodology in Plain English

The researchers start from a practical problem: dense head correspondence needs supervision, but no dataset of dense per-pixel head matches exists. Their workaround is to mine that supervision from video. They take CelebV-HQ, run GroundedSAM2 with the text prompt "person" on the first frame of each clip to isolate the foreground, sample points uniformly in that region, and let CoTracker3 track those points through the video. Videos with fewer than 80 surviving tracks, or with failed foreground segmentation, are discarded, leaving 32K of the original 35K videos, with no more than 400 tracks per video. Points that leave the frame after augmentation are dropped from training.

Training then works in a Siamese fashion. Two frames from the same video are passed through the embedder independently. The embedder is a DINOv3 backbone with a DPT head that outputs a 3-channel image at the input resolution (512 × 512) — these three channels are the predicted cube coordinates. Those coordinates index into the latent voxel grid E via trilinear interpolation, producing high-dimensional semantic features. The contrastive loss is applied to features at the tracked point locations: matched points should have similar features, unmatched ones should not. Interpreting the cube grid as keys and values in an attention-like scheme, the interpolation step is effectively a soft lookup.

Two extra signals force the cube to be structured rather than arbitrary. Landmark predictions (300W format, anchored to predefined cube coordinates, with the loss summed over 68 landmarks) pin well-known facial features to fixed locations, and a segmentation head applied to the extracted semantic features predicts face-parsing masks trained against FaRL masks refined with a face-parsing model. Smoothness comes for free from filtering the latent grid with a 3D Gaussian, which is a deliberate analogue of smoothing function bases with a Laplace-Beltrami operator in functional-map literature.

Optimization uses AdamW with a learning rate of 5·10⁻⁵ for the backbone, 10⁻⁴ for the DPT head, and 10⁻³ for the latent features, with cosine annealing over 140K steps and 2,800 warmup steps. Eight image pairs per batch fit on a single NVIDIA RTX 3090 Ti, taking 1.5 days. At inference, correspondence is just nearest-neighbor search in the canonical space, and because the space is a cube, a user can click a region once and apply it elsewhere.

Why This Matters

• Research impact. The paper shows that a 3-dimensional learned embedding can beat 768- to 1280-dimensional foundation-model features at geometry-aware matching for a specific category, and that dense supervision can be bootstrapped from point trackers instead of hand-annotated correspondences. It also demonstrates that a canonical-space bottleneck enforces cross-identity and cross-pose consistency, extending ideas from NOCS-style object coordinate learning into the head domain where 3D model collections are scarce.

• Real-world applications.

  • AR/VR and telepresence: stable head and face alignment for avatars, including hair and accessories that landmark-based trackers ignore.
  • Video conferencing and live streaming: robust face/head tracking under occlusion, extreme rotation, and motion blur.
  • Film and game production: dense, queryable head regions for digital doubles, texture transfer, and retexturing pipelines.
  • Multi-view capture: stereo and multi-view reconstruction from embeddings, demonstrated by triangulating 2-view and 3-view correspondences in Nersemble using known camera parameters.

• Industry relevance. The method is designed as a complement to existing 3DMM tracking rather than a replacement — the monocular tracking experiment shows better FLAME fitting when a DenseMarks photometric term is added. This makes it directly pluggable into production head-tracker stacks. The code and model checkpoint are stated to be released publicly.

Future Directions

• Extending beyond heads. The authors explicitly name full-body and other domains as future work, noting that progress depends on publicly available high-resolution data collections. The robustness results on degraded tracks are presented as evidence the approach might transfer to settings where accurate tracking is hard, such as full-body capture or highly dynamic non-rigid objects.

• Handling sequences absent from the training data. The stated limitation is that CelebV-HQ consists mainly of interview-style and film-shooting footage, so backside head views, extreme head rotations, and very rapid motion are rare. Quality is expected to drop on those sequences, especially for monocular tracking, which needs most per-pixel correspondences to be reliable.

• Improving robustness to noisy trackers. Since Gaussian noise on tracks degrades accuracy more than track omission, reducing sensitivity to positional track error is a natural next step.

• Reducing reliance on pseudo-ground-truth tools. Landmarks, segmentation masks, and point tracks all come from off-the-shelf models, so the learned space inherits their failure modes. The paper reports stronger robustness to landmark and segmentation errors than to track noise, but does not report training or inference latency, memory footprint, or scaling behavior with cube resolution N.

Target Audience

Researchers and engineers working on human head and face modeling, dense correspondence, or video-based 3D reconstruction will get the most from this paper, along with practitioners building avatar, telepresence, or motion-capture pipelines who need correspondence that covers hair and accessories. Readers interested in canonical-space learning, functional maps, or bootstrapping supervision from foundation-model pseudo-labels will also find the training recipe useful. Some background in Vision Transformers and metric evaluation (PCK, MAE, ArcFace) is helpful but not required to follow the main argument.

Authors’ abstract

We propose DenseMarks - a new learned representation for human heads, enabling high-quality dense correspondences of human head images. For a 2D image of a human head, a Vision Transformer network predicts a 3D embedding for each pixel, which corresponds to a location in a 3D canonical unit cube. In order to train our network, we collect a dataset of pairwise point matches, estimated by a state-of-the-art point tracker over a collection of diverse in-the-wild talking heads videos, and guide the mapping via a contrastive loss, encouraging matched points to have close embeddings. We further employ multi-task learning with face landmarks and segmentation constraints, as well as imposing spatial continuity of embeddings through latent cube features, which results in an interpretable and queryable canonical space. The representation can be used for finding common semantic parts, face/head tracking, and stereo reconstruction. Due to the strong supervision, our method is robust to pose variations and covers the entire head, including hair. Additionally, the canonical space bottleneck makes sure the obtained representations are consistent across diverse poses and individuals. We demonstrate state-of-the-art results in geometry-aware point matching and monocular head tracking with 3D Morphable Models. The code and the model checkpoint will be made available to the public.