Registration is a Representation Problem: Solving Whole-Body PET/CT Registration

AI summary

Overview

Research area: Medical image registration, with a focus on representation learning and foundation models for biomedical imaging, applied to longitudinal whole-body PSMA PET/CT alignment.

Technical level: Advanced. The paper assumes familiarity with deformable registration, diffeomorphic deformation models, image similarity losses such as local normalized cross correlation, and segmentation-based losses.

Scope: The paper argues that the bottleneck in registration is the representation used to establish correspondence rather than the optimizer, and demonstrates this by combining foundation-model features, hand-crafted descriptors, and segmentation maps inside a two-stage GPU-accelerated solver that won the Learn2Reg 2026 PSMAReg Challenge without any task-specific training.

What This Paper Is About

Registration methods must decide which voxels in one scan correspond to which voxels in another, and raw image intensities are a poor guide for that decision. Segmentation labels help by supplying semantic identity, but they assign only coarse identities to fine-grained anatomy and often cover just a subset of structures, which is especially limiting in whole-body scans. The authors' goal is to build a whole-body PET/CT registration pipeline that draws its correspondence signal from complementary representations of anatomy, and to do so without training or fine-tuning anything on the challenge's training data.

Key Contributions

1. Reframing registration as a representation problem. The paper argues that fast iterative solvers are limited not by their optimization methods but by the representation used to match images, and shows that swapping in foundation-model features produces large gains.

2. A two-stage hybrid registration framework built on complementary representations. Stage 1 stacks CT, PET, 32-dimensional per-voxel anatomix features, and hand-crafted MIND-SSC descriptors into a multi-channel volume and optimizes a body-masked squared local normalized cross correlation loss plus a lesion-volume regularizer using the GPU-accelerated FireANTs solver. Stage 2 refines the deformation with segmentation-guided losses, including generalized Dice, a one-sided Chamfer term, a region-based loss for poorly overlapping organs, anatomix-feature matching, and the volume regularizer.

3. Lesion-aware, component-wise volume preservation. The authors construct a recall-biased lesion mask (organ-filtered PET SUV thresholding at SUV > 3, subtracting 16 high-uptake organs dilated by 10 mm, unioned with the LION prediction), penalize local volume change through a Jacobian-determinant regularizer, and modify an existing FFT-based volume projection so that it only corrects lesion components whose warped voxel count deviates beyond a threshold.

4. A first-place challenge result with an engineered, resource-constrained pipeline. The system ranked first on the PSMAReg test leaderboard despite never being trained on the challenge dataset, and the implementation was optimized to fit the challenge's roughly 90-second and 80 GB H100 limits.

Main Findings

• First place on the PSMAReg test set. The paper reports CT Dice 0.830, CT HD95 4.6, PET Dice 0.610, PET HD95 19.4, MTV error 1.78%, TLG error 1.39%, and NDV 0.0008%, ranking first ahead of heyufan1995 (CT Dice 0.832, PET Dice 0.628, MTV error 5.37%, TLG error 5.35%, NDV 0.0032%) and tinymilky (CT Dice 0.832, PET Dice 0.598, MTV error 3.41%, TLG error 3.24%, NDV 0.0009%). The authors note that standard deviations were not provided. Notably, the winning entry's CT Dice is slightly lower than both runners-up; the ranking is a weighted geometric mean of significance scores for registration accuracy (40%), PET biomarker preservation (40%), and field regularity (20%).

• Representations, not just solver tuning, drive the gains. On 65 ground-truth training pairs from 50 held-out subjects, tuning FireANTs improved Stage 1 CT Dice from 0.586 to 0.686 and CT HD95 from 20.07 to 11.09. Adding anatomix and MIND-SSC features pushed CT Dice to 0.775 and HD95 to 6.54, and cut MTV error from 15.56 to 6.31.

• Volume preservation acts progressively and dominates PET biomarker metrics. Within Stage 1, adding the volume penalty reduced MTV error from 6.31 to 4.95% and the projection step reduced it to 3.41%. In Stage 2, the full configuration with volume projection and smoothing reached MTV error 2.99% and TLG error 3.85%, alongside CT Dice 0.812 and NDV 0.00014%.

• Segmentation guidance alone hurts unsupervised tissue alignment. Stage 2 with labels only achieved CT Dice 0.808 but raised CT MAE to 78.3 from 57.3 in the tuned Stage 1 baseline. Adding anatomix supervision restored CT MAE to 62.0 and slightly improved CT Dice to 0.809.

• Removing the representation stack degrades the whole pipeline. Running the full pipeline without anatomix and MIND-SSC features yielded CT Dice 0.797, CT HD95 6.61, MTV error 3.60%, and TLG error 4.57%, worse than the full configuration.

• Field folds occur almost entirely outside the patient. On training pairs that breach the %NDV threshold, 99.5 to 100% of folded voxels lay outside the body mask, motivating a background-only fold repair step restricted to regions outside the dilated fixed-image body mask.

• Throughput engineering was necessary to meet challenge constraints. A direct serial implementation would have exceeded the limits; the authors report reducing the pipeline from 83 GB and 812 seconds per pair to 72 GB and 79 seconds through region restriction, concurrent feature extraction and segmentation, FP16 precision, gradient checkpointing, removal of a redundant segmentor, and lower-resolution segmentation models.

• The main failure mode is topological change. The single diffeomorphic model cannot fit articulated limb motion, for example when arms are crossed at baseline and above the head and outside the field-of-view at follow-up.

Methodology in Plain English

The authors treat registration as a matching problem and ask what should be matched. Instead of comparing raw CT and PET intensities, they build a stack of complementary descriptions of each scan. One description comes from anatomix, a 3D biomedical foundation model pretrained on large amounts of synthetic data, which outputs a 32-dimensional feature vector at every voxel that is intended to be stable across appearance variation while still moving with anatomy. A second description is MIND-SSC, a hand-crafted local self-similarity descriptor. A third is coarse and semantic: TotalSegmentator produces anatomical labels from CT and LION produces PSMA lesion predictions from PET.

These descriptions feed a two-stage solver. Stage 1 combines the CT and PET images with the anatomix and MIND-SSC features into a multi-channel volume for both the fixed and moving scans, then runs a fast GPU-accelerated solver (FireANTs) to remove large deformations, matching with a body-masked local normalized cross correlation computed per channel and adding a penalty that discourages lesion volume from changing. Stage 2 starts from that result and refines it with structure-aware losses: a generalized Dice term for label matching, a one-sided Chamfer term measuring how far warped structures are from their targets, a region-based term that samples up to 4,000 points per fixed organ and penalizes mapped points that fall outside the corresponding moving organ by their distance in millimetres, plus anatomix-feature matching to stabilize tissue not covered by labels and another lesion-volume penalty.

After optimization, the deformation is post-processed in two ways. A volume projection step, which solves a Poisson system with FFTs in under 0.5 seconds on a GPU, is applied selectively: lesion components whose warped voxel count stays within a log-ratio threshold of 0.03 are left alone, while others receive up to four local correction passes and then eight component-wise passes that scale each lesion's volume back toward its original count. Separately, folds in the deformation field are repaired only outside a protected body region dilated by a 5-mm safety margin.

Why This Matters

Impact on research. The paper challenges the prevailing split between learned registration networks and classical optimization solvers. It shows that a solver with adaptable objectives and regularizers, fed by representations from a foundation model pretrained entirely on synthetic anatomy, can beat custom-trained approaches on a benchmark without any dataset-specific training. This reframes the practical question from "which network should I train?" to "which representation should I match?"

Real-world applications:

• Longitudinal oncology imaging. Tracking tumor burden across PSMA PET/CT follow-up scans requires that registration not fabricate or erase lesion volume, since MTV and TLG biomarkers are only meaningful if the deformation preserves ground-truth lesion volume.
• Quantitative imaging biomarkers in trials. The reported MTV error of 1.78% and TLG error of 1.39% on the hidden test set indicates that registration-induced biomarker distortion can be reduced to low single digits.
• Label transfer and organ-level analysis. Because alignment is measured with TotalSegmentator labels and improved sharply in the abdomen, the approach supports transferring anatomical or lesion segmentations between timepoints.
• Rapid deployment to new datasets. The conclusion argues practitioners can align their own datasets without training a network for each new task or modality, provided suitable pretrained representations exist.

Industry relevance. The pipeline's constraints (under roughly 90 seconds per pair on an 80 GB H100, with a peak footprint of 72 GB) are the kinds of limits that determine whether a method is usable in production imaging software. The work also illustrates a broader trend: value increasingly comes from curating and orchestrating several pretrained models in parallel rather than training a single large model from scratch.

Future Directions

• Articulated and topology-changing motion. The authors state that a single diffe

Authors’ abstract

Raw image intensities are a poor surrogate for similarity across images in correspondence problems. Further, while auxiliary segmentation labels can provide semantic identification and improve matching, they can assign only coarse identities to fine-grained anatomy and often cover only a subset of structures, particularly in whole-body settings. We present an approach that instead establishes correspondence using complementary representations of anatomy and demonstrate its utility for longitudinal whole-body PET/CT registration in the Learn2Reg 2026 PSMAReg Challenge. Our 3D foundation model, anatomix, is used to derive fine-grained, per-voxel semantic representations that are consistent across volumes, and we further employ off-the-shelf segmentation models to obtain coarse anatomical identities. Together, these representations drive a GPU-accelerated iterative registration solver (FireANTs) in a hybrid multistage framework: the first stage estimates alignment from dense multi-channel features, after which a second refines the deformation using both anatomix- and segmentation-based losses. Lastly, because registration should not introduce non-biological tumor volume changes across time, we develop lesion-specific component-wise volume preservation schemes. Without any task-specific training, our system won first place in the PSMAReg Challenge leaderboard. Code and interactive visualizations are available at http://www.neeldey.com/reps-for-reg.