SSMB: Self-Supervised Local Feature Detection under Motion Blur

AI summary

Overview

Research area: Computer vision, specifically local feature (keypoint) detection and matching for images degraded by motion blur.

Technical level: Advanced. The paper assumes familiarity with convolutional and MLP-based network design, self-supervised learning, homography-based pseudo-labeling, and benchmarking protocols for matching, relative pose estimation, and visual localization.

Scope: The paper proposes SSMB, a deblur-free, self-supervised sparse keypoint detector trained without any handcrafted detector or external pseudo-labels, and evaluates it on keypoint repeatability, image matching, relative pose estimation, and visual localization under motion blur.

What This Paper Is About

Motion blur smears local image structure and degrades how repeatably keypoints can be localized, which harms the correspondences that downstream vision tasks depend on. Existing solutions either run a computationally expensive deblur-then-detect pipeline that can propagate restoration artifacts, or train a detector to regress the positions of handcrafted keypoints (SIFT) found on sharp images, which teaches the network the preferences of that handcrafted detector rather than what is genuinely repeatable under blur. The goal of SSMB is to detect keypoints directly on blurred images, learning what to detect from geometry and from sharp-blur image pairs alone, with no external supervision.

Key Contributions

1. Local Discriminability Enhancement (LDE) module. A module that restores fine-grained local discriminability after global feature mixing, added with only about 50K parameters of overhead. The authors present this as a necessary condition for self-supervised training to succeed under blur.
2. Two-stage training pipeline. Geometric pretraining on synthetic geometric shapes, where corner labels come directly from the rendered geometry, followed by blur-aware training on real sharp-blur image pairs. The paper shows geometric pretraining is an essential prerequisite for bootstrapping self-supervised learning under blur.
3. Multi-component self-supervised loss. A combination of homographic adaptation, blur consistency, position consistency, and spatial diversity terms, with spatial diversity identified as the key term that prevents the detector from collapsing to a degenerate solution.
4. State-of-the-art sparse detection results under blur. Evaluations on keypoint detection, image matching, relative pose estimation, and visual localization under motion blur show SSMB outperforming supervised and self-supervised sparse baselines, and surpassing detector-free matching methods at low pixel thresholds.

Main Findings

• Repeatability on Blur-HPatches (blur-to-sharp). SSMB reaches 77.24%, 77.18%, and 77.16% on Easy, Hard, and Tough, beating BALF at every difficulty level (74.12%, 74.45%, 71.84%). BALF drops from 74.12% to 71.84% as blur worsens, while SSMB stays nearly flat.
• Repeatability on Blur-HPatches (blur-to-blur). SSMB achieves 74.20%, 74.33%, and 74.48% on Easy, Hard, and Tough, outperforming all methods at all levels. Unlike most detectors, SSMB improves slightly as blur severity increases, which the authors attribute to the blur consistency loss.
• Deblurring does not close the gap. On Deblur-HPatches, in the Tough deblur-to-sharp setting the best competing deblur-then-detect result (DeblurGAN-v2 with SuperPoint) reaches 58.22%, while SSMB on the blurred image without deblurring reaches 77.16%. The paper concludes that deblurring artifacts limit downstream detection and that deblur-free detection is more effective.
• Matching accuracy. Figure 1 reports that SSMB attains the highest Mean Matching Accuracy at a 3-pixel threshold on the Blur-HPatches Tough split among sparse methods.
• Clean-image performance. Even though SSMB is designed for blurred images, the paper reports it remains competitive with methods optimized for sharp images on the original all-sharp HPatches dataset; the full results are placed in the Appendix.
• Ablation: removing LDE causes collapse. Without LDE, the predicted probability map collapses toward near-zero almost everywhere.
• Ablation: spatial diversity is critical. Despite a weight of only 0.005, the spatial diversity loss is the most critical component; without it, detection collapses onto a handful of salient locations, typically image centers or high-contrast edges, even after geometric pretraining.
• Ablation: pretraining prevents score map collapse. Without geometric pretraining, all spatial locations receive nearly identical scores (Pearson correlation of approximately 1.0 across different images), making the subsequent blur-aware training ineffective.
• Tasks covered but numbers truncated. The paper states extensive evaluations on image matching, relative pose estimation, and visual localization, with SSMB claimed to outperform sparse baselines across all tasks; the specific values for the truncated tables (MMA beyond the overall column, pose AUC, and localization accuracy) are not contained in the provided content.

Methodology in Plain English

SSMB removes the dependency on SIFT-style pseudo-labels entirely and trains on two self-generated signals.

The network is an MLP-based encoder with four blocks, adapted from the multi-axis gated MLP design used in BALF and built on the MAXIM architecture. It downsamples from the input resolution to one-eighth resolution while expanding channels through 32, 64, 128, and 256. Each block contains a Residual Split Head Multi-Axis gMLP layer (RSMA) that splits features into two branches: a grid branch for global spatial mixing and a block branch for fine local mixing, followed by a Residual Channel Attention Block. The problem is that global mixing can wash out the fine local detail needed to pinpoint a keypoint, so the LDE module is inserted between the two branches. It applies layer normalization, runs a depthwise convolution to capture local gradient structure, gates that with a small learned channel-attention bottleneck, multiplies the two, and adds the result back through residual connections. A detector head then produces a full-resolution keypoint probability map (via a dustbin-augmented softmax over 8x8 pixel positions per cell and a pixel-shuffle) and a sub-pixel offset map (via a sigmoid).

Training happens in two stages. In stage one, geometric pretraining, 25,000 synthetic images are generated on the fly by composing checkerboards, line segments, polygons, ellipses, and star patterns on grayscale backgrounds. Because the geometry is controlled, corner labels are derived directly from the rendering, with each 8x8 cell assigned the index of the corner-occupied pixel or a dustbin class. Training uses a masked cross-entropy loss for 10 epochs with batch size 8, 320x320 crops, and the Adam optimizer at an initial learning rate of 1e-5 with 4 epochs of linear warmup. This stage exists because without it the score map collapses.

In stage two, blur-aware training, the model trains on real sharp-blur pairs from the GoPro dataset (2,912 pairs from 30 sequences) for 36 epochs, with batch size 8, 320x320 random crops, 100 homographies per image for online homographic adaptation, an 8x8 spatial diversity grid, rotation augmentation up to 90 degrees, and the same Adam optimizer with the learning rate decayed by a factor of 0.1 at 60% and 80% of total epochs. A single network with shared weights makes two forward passes per iteration, one on the sharp image and one on the blurred image. Four losses combine with weights 0.5, 0.5, 0.1, and 0.005 for the homographic adaptation, blur consistency, position consistency, and spatial diversity terms respectively. The sharp pass is supervised by homographic adaptation, which aggregates probability maps from warped versions of the image back into the original frame (discarding homographies covering less than 50% of the image) to produce stable pseudo-labels. The blurred pass is supervised by a hard cross-entropy toward the detached sharp-image argmax, an L2 regression pulling blurred sub-pixel offsets toward sharp ones, and the diversity term that penalizes any grid cell whose maximum response is low, keeping detections spread across the image. All training runs on a single NVIDIA GeForce RTX 3090.

Why This Matters

Impact on research. The paper argues against two entrenched assumptions at once: that blur must be removed before detection, and that supervised keypoint detectors should imitate handcrafted detectors. It shows a self-supervised model can beat both, and it identifies which specific components (local discriminability, geometric bootstrapping, spatial diversity) keep such training from collapsing. Its reported stability across blur levels, rather than degradation, is a distinctive result.

Real-world applications (applications named in the paper):

• Simultaneous Localization and Mapping (SLAM) on robots or vehicles moving fast enough to blur their imagery.
• Structure-from-Motion reconstruction from handheld or moving-camera photos.
• Visual localization in retrieval-based pipelines, where reusable keypoint maps matter and blur occurred during capture (day and night query images against a reference map).
• Camera calibration and image retrieval, both listed as motivating applications.

Industry relevance. Detector-free matching methods, such as LoFTR, MatchFormer, ASpanFormer, and RoMa v2, deliver strong correspondences but cannot produce reusable keypoints or descriptors, so they suit retrieval-based localization pipelines poorly and cost more at inference because they process the full image pair. A sparse, real-time-oriented detector that holds up under blur is more compatible with deployed localization and mapping systems, which is the gap SSMB targets.

Future Directions

1. Pairing SSMB with learned matchers. The paper evaluates SuperPoint, DISK, and ALIKED additionally with LightGlue, but evaluates SSMB only with HardNet or HyNet descriptors and mutual nearest neighbor matching. Whether SSMB benefits comparably from a learned matcher is not reported.
2. Generalization beyond the training blur. Stage two trains exclusively on GoPro sharp-blur pairs, while evaluation uses synthetically convolved blur on HPatches-derived and Aachen data plus the ArchViz simulation. Performance on real blur distributions outside the GoPro-style training data is not established by the reported results.
3. Releasing code, models, and datasets. The paper states these will be made publicly available upon acceptance, which is a prerequisite for independent replication of the ablations and the claimed state-of-the-art ranking.
4. Reducing the remaining dependencies. SSMB removes SIFT supervision but still relies on online homographic adaptation pseudo-labels and on the GoPro paired sharp-blur dataset. Whether the pipeline can be trained with fewer or no paired sharp references remains an open question.

Target Audience

Researchers and engineers working on local feature detection and matching, image registration, SLAM, SfM, and visual localization, particularly those dealing with degraded or motion-blurred imagery. It is also relevant to practitioners building perception systems where running a full deblurring network before feature extraction is too costly, and to readers interested in self-supervised training strategies that avoid handcrafted pseudo-labels.

Authors’ abstract

Keypoint detection under motion blur remains a significant challenge, as blur distorts local image structure and degrades the repeatability of feature localization. Existing approaches either rely on computationally expensive deblur-then-detect pipelines that may introduce restoration artifacts, or learn to regress the image positions of handcrafted keypoints extracted on sharp images, which reflects the assumptions of the handcrafted detector rather than what is truly repeatable under blur. We present SSMB, a deblur-free, self-supervised keypoint detector for motion-blurred images that requires neither handcrafted detectors nor external pseudo-labels. SSMB introduces the Local Discriminability Enhancement (LDE) module, which restores fine-grained local discriminability after global feature mixing. Training is performed in two stages. First, geometric pretraining on synthetic shapes bootstraps spatially discriminative keypoint detection without any external detector, just from the rendered geometry. Second, blur-aware training on real sharp-blur image pairs learns blur-invariant detection through a multi-component self-supervised objective that enforces cross-domain consistency, geometric alignment, and spatial coverage. Extensive evaluations on keypoint detection, image matching, relative pose estimation, and visual localization under motion blur demonstrate that SSMB establishes a new state-of-the-art among sparse keypoint detectors, consistently outperforming both supervised and self-supervised baselines across all tasks. Code, models, and datasets will be publicly available upon paper acceptance.