Position-Aware Scene-Appearance Disentanglement for Bidirectional Photoacoustic Microscopy Registration

AI summary

Overview

• Research area: Computer vision — cross-domain image registration, scene-appearance disentanglement, and temporal attention.
• Technical level: Advanced (assumes familiarity with registration, instance normalization, AdaIN, and attention mechanisms).
• Scope: The paper proposes GPEReg-Net, a deformation-free registration framework that factorizes images into domain-invariant scene features and global appearance codes, recombines them via Adaptive Instance Normalization, and enriches them with position-encoded cross-frame attention.
• Title note: The full paper content provided is titled "Deformation-Free Cross-Domain Image Registration via Position-Encoded Temporal Attention." The content does not discuss photoacoustic microscopy or bidirectional registration, so those topics are not summarized here because they are not reported.

What This Paper Is About

Standard image registration methods assume that corresponding pixels have similar brightness, which breaks down when the moving and fixed images come from different acquisition conditions (different intensity distributions, or "domains"). GPEReg-Net instead splits each image into a spatial "scene" part and a global "appearance" part, then registers by pairing the moving image's scene with the fixed image's appearance — so alignment happens without ever estimating a deformation field. The paper also adds a temporal attention module to exploit coherence across sequential frames.

Key Contributions

1. Scene-appearance factorization. The authors formalize cross-domain registration as a latent factorization problem and build an encoder-decoder that decomposes images into domain-invariant scene features 𝒔 ∈ ℝ^{64×H×W} and global appearance codes 𝒂 ∈ ℝ^{32}, recombined by AdaIN so that no deformation field is estimated at all.
2. Position-encoded temporal attention. A Global Position Encoding (GPE) module fuses learnable position embeddings, sinusoidal encoding, and multi-head cross-frame attention over a sliding window of k neighboring frames, giving the scene representation inter-frame context.
3. Comprehensive cross-domain evaluation. Validation on two benchmarks — FIRE-Reg-256 (retinal fundus, semi-rigid) and HPatches-Reg-256 (synthetic patches, affine) — reports state-of-the-art results on both with 69 FPS throughput.
4. Efficiency at competitive scale. With 3.40M parameters, the model runs 1.87× faster than SAS-Net (3.35M parameters, 37 FPS) at higher registration quality.

Main Findings

• FIRE-Reg-256 results: GPEReg-Net reaches NCC 0.851, SSIM 0.928, and PSNR 33.47 dB on 973 test patches, the top score on all three metrics. Baselines include Unregistered (NCC 0.762, SSIM 0.494, PSNR 22.36), SIFT (0.449, 0.463, 16.39), Demons (0.672, 0.528, 17.45), Optical Flow (0.552, 0.506, 16.77), SyN (0.549, 0.521, 15.76), VoxelMorph (0.820, 0.916, 25.42), TransMorph (0.832, 0.876, 25.51), and SAS-Net (0.748, 0.855, 32.21).
• Traditional methods fall below the unregistered baseline: On FIRE-Reg-256, the unregistered NCC of 0.762 reflects highly overlapping, well-aligned pairs; all traditional warping methods score below it.
• HPatches-Reg-256 results: GPEReg-Net achieves NCC 0.536, SSIM 0.450, and PSNR 21.01 dB on 500 test pairs, versus Unregistered (0.312, 0.241, 14.87), VoxelMorph (0.448, 0.376, 18.53), TransMorph (0.471, 0.395, 19.24), and SAS-Net (0.502, 0.421, 20.15). That is a 0.86 dB improvement over the strongest baseline, SAS-Net.
• Deformation-based methods struggle with large affine transforms: VoxelMorph and TransMorph perform worse on HPatches-Reg-256 because the ±15° rotation and ±20 px translation exceed the capacity of their deformation fields.
• Temporal context helps: Against SAS-Net, the GPE module contributes a 1.26 dB PSNR gain on FIRE-Reg-256 by improving consistency across sequential frames.
• Speed-quality trade-off: GPEReg-Net runs at 69 FPS with 14.52 ms latency on an RTX 5090. VoxelMorph (0.10M parameters) runs at 327 FPS and 3.06 ms; TransMorph (0.17M parameters) at 351 FPS and 2.85 ms; SAS-Net at 37 FPS and 27.21 ms. The deformation-based methods are faster but score far lower on registration quality.
• Orthogonal factorization rationale: Instance normalization in the SceneEncoder removes per-channel first- and second-order statistics (μ_c, σ_c) while retaining spatial structure; global average pooling in the AppearanceEncoder discards spatial information while retaining channel-wise statistics. The 32-dimensional appearance code is argued to have sufficient capacity for shifts that are predominantly global rather than spatially varying.

Methodology in Plain English

The framework splits the registration model into four learned modules. A SceneEncoder (a U-Net with residual connections and Instance Normalization, four encoding levels with channels C_l ∈ {32, 64, 128, 256}, bilinear upsampling decoder and skip connections) reads the moving image and produces a 64-channel feature map that describes structure but not intensity. An AppearanceEncoder reads the fixed image and compresses it into a single 32-dimensional vector using four stride-2 convolutions, global average pooling, and two fully connected layers — this vector carries "how the image looks" but no spatial layout. A Global Position Encoding module then adds temporal context to the scene features, and an ImageDecoder uses AdaIN to rescale the normalized scene features with affine parameters γ(𝒂) and β(𝒂) derived from the appearance code: AdaIN(𝒔, 𝒂) = γ(𝒂)·(𝒔 − μ(𝒔))/σ(𝒔) + β(𝒂). Three AdaIN-Conv blocks with 64, 32, and 16 channels progressively reconstruct the output, so the registered image is produced without any spatial deformation field.

The GPE module, for a frame index t ∈ {0, …, N−1}, combines a learnable embedding 𝒆_t ∈ ℝ^{64} from a table 𝑬 ∈ ℝ^{N×64}, a fixed sinusoidal encoding 𝒑_t ∈ ℝ^{64}, and cross-frame multi-head attention with H = 4 heads that queries the current frame's spatially-averaged scene features against a running cache of k = 2 neighboring frames. A two-layer MLP fuses the concatenated embeddings into 𝒈_t, which is injected into the scene features with scaling α = 0.1: 𝒔̃ = 𝒔 + α·W_proj(𝒈_t + 𝒄_t), where 𝒄_t is the cross-frame attention output.

Training minimizes a bi-objective loss ℒ = ℒ_recon + λ·ℒ_scene, where ℒ_recon = ‖Î_r − I_f‖₁ enforces pixel fidelity and ℒ_scene = ‖𝒮(I_m) − 𝒮(I_f)‖₂² encourages both images to map to the same scene code regardless of domain. The weight λ = 10.0 was set via multi-task balancing. The model has 3.40M parameters and is trained with Adam (lr = 10⁻⁴, weight decay 10⁻⁵), cosine annealing over 30 epochs, batch size 8, gradient clipping at max norm 1.0, and mixed precision (AMP) on a single NVIDIA RTX 5090. Results are reported with NCC, SSIM, and PSNR.

Why This Matters

• Impact on research: The paper reframes registration as a latent factorization problem rather than a displacement-field estimation problem, and shows the same architecture transfers to a different imaging domain without architectural modification. It argues that scene-appearance disentanglement captures structural alignment at least as effectively as deformation-based approaches, while AdaIN-based appearance transfer improves SSIM and PSNR.
• Real-world applications:
  • Retinal fundus imaging, where subject motion creates the misalignment that FIRE-Reg-256 models with semi-rigid deformations.
  • Cross-domain medical imaging pipelines where the moving and fixed images come from different intensity distributions.
  • Sequential frame acquisition systems, since 69 FPS exceeds typical acquisition rates and enables real-time processing.
  • Natural-image matching and local descriptor work, since HPatches-Reg-256 derives from the HPatches descriptor benchmark and includes viewpoint-driven affine transformations.
• Industry relevance: Deformation-free decoding avoids the domain-specific tuning that deformation-based methods require, and GPEReg-Net reaches higher quality than SAS-Net at roughly a third of its latency while using a comparable parameter budget (3.40M vs 3.35M). Released code is available at https://github.com/JiahaoQin/GPEReg-Net.

Future Directions

• Spatially-varying appearance: The current appearance model 𝒂 ∈ ℝ^{32} captures only global intensity statistics; local illumination gradients may require a spatially-conditioned appearance map 𝒂(𝒙), which the authors leave open.
• Longer sequences: The fixed embedding table 𝑬 ∈ ℝ^{N×64} limits generalization to sequences longer than N frames; adaptive or continuous position encodings are suggested as remedies.
• Compression integration: The authors propose integrating energy-based reconstruction frameworks for learned feature compression.
• Deformation capacity question: Because deformation-based methods (VoxelMorph, TransMorph) underperform under large affine transforms on HPatches-Reg-256, how the factorization behaves under very large geometric misalignment remains an open direction.

Target Audience

Researchers and practitioners in medical image analysis and computer vision who work on registration, domain adaptation, or generative alignment; engineers building real-time clinical or sequential-imaging pipelines; and readers interested in disentangled representation learning and attention-based temporal modeling. Readers need a working knowledge of normalization techniques, AdaIN, and transformer-style attention to follow the method sections.

Authors’ abstract

High-speed optical-resolution photoacoustic microscopy (OR-PAM) with bidirectional raster scanning doubles imaging speed but introduces coupled domain shift and geometric misalignment between forward and backward scan lines. Existing registration methods, constrained by brightness constancy assumptions, achieve limited alignment quality, while recent generative approaches address domain shift through complex architectures that lack temporal awareness across frames. We propose GPEReg-Net, a scene-appearance disentanglement framework that separates domain-invariant scene features from domain-specific appearance codes via Adaptive Instance Normalization (AdaIN), enabling direct image-to-image registration without explicit deformation field estimation. To exploit temporal structure in sequential acquisitions, we introduce a Global Position Encoding (GPE) module that combines learnable position embeddings with sinusoidal encoding and cross-frame attention, allowing the network to leverage context from neighboring frames for improved temporal coherence. On the OR-PAM-Reg-4K benchmark (432 test samples), GPEReg-Net achieves NCC of 0.953, SSIM of 0.932, and PSNR of 34.49dB, surpassing the state-of-the-art by 3.8% in SSIM and 1.99dB in PSNR while maintaining competitive NCC. Code is available at https://github.com/JiahaoQin/GPEReg-Net.