IR275K: A Benchmark for Infrared Multi-Frame Super-Resolution Toward Efficient Remote Sensing

AI summary

Overview

• Research area: Infrared remote sensing, multi-frame super-resolution (MFSR), benchmark datasets, and efficient state-space model (SSM) architectures for computer vision.
• Technical level: Intermediate. The benchmark construction, statistics, and evaluation protocol are accessible to a general computer-vision reader; the CGMamba architecture sections (Structured State Space Duality, Center-Guided CrossMamba, 2D rotary position encoding) require background in state-space models.
• One-sentence scope: The paper introduces IR275K, a curated 594-sequence / 275,196-frame infrared video benchmark with sequence-level splits and a fixed x4 evaluation protocol, and uses it to evaluate CGMamba, a 10.90M-parameter SSM-based multi-frame super-resolution reference model.

What This Paper Is About

Infrared remote sensing platforms (satellites, aircraft, low-altitude platforms) capture large volumes of video under constrained detector resolution, power budgets, and downlink bandwidth, and multi-frame super-resolution offers a software route to recovering finer spatial detail from consecutive frames. However, infrared MFSR evaluation has been fragmented across private datasets and ad-hoc protocols, and widely used suites such as REDS, Vimeo-90K, and DIV2K were built for visible-light or single-frame settings and do not capture weak thermal contrast, pervasive sensor noise, sparse texture, and platform-induced frame-to-frame variation. The paper's goal is to provide a curated, reproducible benchmark for infrared MFSR that supports joint accuracy-and-efficiency evaluation, and to probe one lightweight architecture on it.

Key Contributions

1. IR275K benchmark. A curated benchmark of 594 infrared video sequences totaling 275,196 frames, drawn from three public infrared sources, with standardized sequence-level train/validation/test splits and a reproducible x4 evaluation protocol. The dataset is publicly available at https://github.com/InfraRecon7/IR275K.
2. Initial single-frame vs. multi-frame reference comparison. Two infrared single-image super-resolution methods (GPSMamba, IRSRMamba) and a non-learnable bicubic baseline are trained and evaluated on the same infrared testbed as the multi-frame CGMamba, establishing a reproducible starting point for future infrared MFSR work.
3. Benchmark-driven architectural finding. An ablation within CGMamba shows that removing 2D rotary position encoding from Center-Guided CrossMamba reduces PSNR by 2.23 dB and produces severe grid-like artifacts, indicating that explicit spatial anchoring is important for stabilizing SSM-based cross-frame gating in this setting.
4. Split characterization. The fixed partitions are characterized in terms of thermal contrast, temporal noise, and inter-frame displacement, documenting how these infrared-specific factors vary across the splits rather than describing the benchmark through a single averaged imaging condition.

Main Findings

• Dataset scale and composition. IR275K contains 594 sequences and 275,196 frames, with sequence lengths ranging from 16 to 5,625 frames. High-resolution reference frames are 640 x 512 pixels and low-resolution inputs are 160 x 128 pixels, generated by bicubic downsampling at x4 scale.
• Sequence-level partitioning. Training receives 261 sequences with 192,776 frames; validation receives 111 sequences with 27,576 frames; test receives 222 sequences with 54,844 frames. Means are 738.6, 248.4, and 247.0 frames respectively; medians are 548, 300, and 300. Across the full benchmark the mean is 463.3 and the median is 400. No frame from the same sequence appears in more than one partition, and source identity was not used to define the partitions.
• Thermal contrast. Global thermal contrast spans more than one order of magnitude, from 0.009 to 0.32. Split means are 0.105 (train), 0.116 (validation), and 0.122 (test), placing the three splits in the same overall thermal-contrast regime. Fifty percent of training sequences fall within a contrast IQR of [0.034, 0.170].
• Temporal noise. Per-pixel temporal noise ranges from 0.004 to 0.26 across IR275K, with split means of 0.074 (train), 0.077 (validation), and 0.081 (test). The test split has a heavier upper range: its upper quartile reaches 0.149 versus 0.116 and 0.117 for training and validation. Training noise IQR is [0.025, 0.116].
• Inter-frame displacement. The training split concentrates in a moderate-motion regime: mean 0.91 px, median 0.36 px, with 95% of sequences below 4.4 px. Validation and test cover broader distributions with means of 2.57 and 2.86 px and maxima of 25.0 and 26.4 px. The train-to-test median displacement shifts from 0.36 px to 0.56 px.
• Structural ablation (Table II). Base (neither component) reaches 32.49 dB PSNR / 0.8571 SSIM; adding 2D RoPE alone gives 32.47 dB / 0.8561; adding CGCM without 2D RoPE drops to 30.96 dB / 0.8052; the full structure reaches 33.19 dB / 0.8721. The full model gains 0.70 dB over the base, and neither component alone is sufficient.
• 2D RoPE dependency. Removing 2D RoPE from CGCM reduces PSNR by 2.23 dB and produces severe grid-like artifacts along structural boundaries. Adding 2D RoPE alone produces almost no change (32.47 vs. 32.49 dB), indicating a dependency between spatial encoding and cross-frame gating rather than a simple additive contribution.
• Depth scaling (Table III). Performance improves from (1,1,1,1,1) at 29.57 dB (5.84M params, 59.13G FLOPs) through (2,2,2,2,2) at 31.70 dB (8.37M, 85.64G) to (4,4,4,4,4) at 33.53 dB (13.43M, 138.64G). The (3,3,3,3,3) configuration reaches 33.19 dB with 10.90M parameters and 112.14G FLOPs. The (2,2,2,2,2) configuration loses 1.49 dB relative to (3,3,3,3,3); (4,4,4,4,4) yields an additional 0.34 dB at 23.63% higher FLOPs; (5,5,5,5,5) decreases to 32.84 dB under the same training budget.
• Reference comparison (Table IV). Bicubic interpolation achieves 31.50 dB / 0.8182 at 0.229 ms. GPSMamba reaches 32.84 dB / 0.8482 with 36.9M parameters and 429.39G FLOPs at 1165 ms. IRSRMamba reaches 32.67 dB / 0.8447 with 26.40M parameters and 206.35G FLOPs at 618 ms. CGMamba reaches 33.19 dB / 0.8721 with 10.90M parameters and 112.14G FLOPs at 340 ms, exceeding the center-frame infrared SISR references by 0.35–0.52 dB with substantially fewer FLOPs.
• Comparison caveat. The SISR references use only the center frame whereas CGMamba also uses neighboring frames, so the authors frame this as an initial reference comparison under the shared protocol rather than a like-for-like architectural contest.
• Qualitative behavior. CGMamba preserves continuous thermal boundaries more faithfully than the SISR references, which occasionally produce over-smoothed or aliased intensity transitions; error maps show high-error regions concentrated where spatial correspondence matters most.

Methodology in Plain English

Building the benchmark. The authors pooled infrared video from three public sources: the UAV-TSR++ dataset introduced with AnyTSR++, plus two resources released through Science Data Bank. They filtered out fully static scenes (no sub-pixel variation to exploit), severely blurred sequences (noise dominating signal), and sequences with extreme frame-to-frame displacement such as scene cuts (which break the assumption that neighbors share recoverable spatial information). After filtering, 594 sequences remained. Sequences were shuffled whole and assigned to train, validation, or test, so no sequence is split across partitions. The benchmark covers four scene categories: land, ocean, urban, and low-altitude urban sensing.

Evaluation protocol. The task is x4 upscaling of the center frame from consecutive low-resolution infrared inputs. Single-image methods get one input frame; multi-frame methods get three. All low-resolution inputs are bicubic downsampled. PSNR and SSIM are computed on single-channel reconstructions after cropping 10 pixels from the image boundary. Runtime is measured at batch size 1 on an NVIDIA RTX 4090 GPU and averaged over full test-set inference. Reporting runtime, parameter count, and FLOPs alongside PSNR and SSIM is required.

The reference model. CGMamba reconstructs the high-resolution center frame in three stages: a convolutional layer plus a selective scan 4D module extracts intra-frame spatial features; several Center-Guided CrossMamba Groups refine neighboring-frame features under guidance of the central-frame feature; lightweight channel attention blocks refine the fused representation and a pixel-shuffle layer upsamples it.

The two design ideas probed. First, instead of estimating motion and warping frames (the dominant alignment-first strategy), CGMamba fuses implicitly: the center frame supplies the query and each neighboring frame supplies keys, values, and step sizes, so the central frame gates what information is pulled from its neighbors without an explicit motion field. Second, because Mamba flattens 2D images into 1D scan sequences and weakens spatial adjacency, the model adds 2D rotary position encoding: per-pixel phase angles along X and Y are computed from two frequency vectors, concatenated into a 2D angle matrix, and blended with the learned content-dependent positional signal via a learnable mixing coefficient before entering the Mamba-3 kernel's rotation mechanism.

Training. All models were trained on four NVIDIA RTX 4090 GPUs in PyTorch with the Charbonnier loss (epsilon = 1e-3) and no perceptual, adversarial, or frequency-domain auxiliary losses, to keep the protocol controlled. CGMamba used three consecutive 128 x 128 low-resolution patches; the SISR references used only the matching center-frame patch. CGMamba used depths (3,3,3,3,3), feature dimension 128, and SSM state dimension 32, with AdamW (beta1 = 0.9, beta2 = 0.95, weight decay 5e-6), gradient clipping at 1.0, EMA decay 0.999, a linear learning-rate warmup from 0 to 1e-4 over 20,000 iterations, cosine annealing with three restart periods (60,000, 30,000, and 10,000 iterations; restart weights 1.0, 0.7, 0.5), a total of 120,000 iterations, and a minimum learning rate of 5e-6. Effective batch size was 256 (per-GPU 64 across 4 GPUs).

Why This Matters

Impact on research. Infrared MFSR results have been scattered across private datasets and ad-hoc protocols, which makes fair comparison and generalization analysis difficult. IR275K supplies fixed sequence-level partitions, a single reproducible protocol, and explicit reporting of efficiency metrics, giving the community a common testbed for joint accuracy-and-efficiency evaluation. The architectural ablation also produces a concrete, transferable design signal: in this setting, implicit cross-frame gating in an SSM becomes spatially disordered without explicit positional anchoring.

Real-world applications (contexts named or implied in the paper):

• Maritime surveillance using infrared payloads on satellites, aircraft, and low-altitude platforms.
• Emergency response, where timely observation under bandwidth limits is a priority.
• Land, ocean, urban, and low-altitude urban remote sensing, each stressing different reconstruction behavior (artifact suppression in thermally smooth scenes versus boundary preservation across frames in scenes with sharp thermal edges and fast platform motion).
• Spaceborne infrared instrument operation, illustrated by the paper's reference to the Lunar Mineral Spectrometer on Chang'e-6, whose infrared channels showed signal-to-noise degradation requiring temperature-compensated calibration.

Industry relevance. Satellite constellations increasingly produce more observation data than can be downlinked. Onboard or edge processing that improves spatial detail from existing detectors, without detector upgrades or extra downlink bandwidth, is directly relevant to payload designers, remote-sensing service providers, and edge-computing hardware teams. The emphasis on parameter count, FLOPs, and runtime alongside reconstruction quality connects the benchmark to practical power- and throughput-constrained deployments.

Future Directions

• Real-sensor degradation. The current benchmark uses x4 bicubic downsampling as its sole degradation model and excludes detector-specific blur kernels, non-uniform noise fields, and compression artifacts. The authors state that future versions of IR275K will incorporate at least one real-sensor degradation variant to measure the gap between synthetic and realistic evaluation on the same data foundation.
• Longer temporal windows and frame ordering. CGMamba has been evaluated only under the three-frame input setting; the authors note that its behavior under longer temporal windows and its sensitivity to input frame ordering remain uncharacterized.
• Stress-stratified evaluation. The present evaluation reports aggregate test-set performance. Evaluating performance stratified by contrast, noise, and motion is listed as an important next step.
• Broader baselines and metric scope. The quantitative comparison is an initial reference evaluation and does not yet include a broad set of MFSR/VSR baselines; extending metrics beyond PSNR and SSIM to downstream task performance is a planned next step. A separate open question is whether RGB innovations such as propagation, alignment, and temporal attention require infrared-specific reformulation, or whether the main bottleneck is data scale and pre-training strategy.

Target Audience

Researchers and engineers working on infrared remote sensing, thermal imaging, and multi-frame or video super-resolution who need a shared benchmark and protocol. It is also relevant to practitioners designing efficient models for resource-constrained or onboard/edge payloads, since the paper couples reconstruction quality with parameters, FLOPs, and runtime. Readers interested in state-space models will find the ablation on spatial anchoring directly useful, while those primarily seeking dataset-construction methodology will find the split characterization (thermal contrast, temporal noise, inter-frame displacement) instructive. Readers without SSM background can still follow the benchmark sections; the CGMamba architecture sections assume familiarity with Mamba and attention-style formulations.

Authors’ abstract

Efficient processing is becoming increasingly important in infrared remote sensing, where satellite constellations produce large volumes of observations under constrained detector resolution, power, and downlink bandwidth. Multi-frame super-resolution (MFSR) offers a software-based route to spatial enhancement, but its evaluation in infrared sensing remains fragmented across private datasets and ad-hoc protocols. Existing benchmarks do not explicitly capture the thermal contrast, sensor noise, weak texture, and platform-induced frame-to-frame variation that characterize infrared video. We introduce IR275K, a curated benchmark containing 594 infrared video sequences and 275,196 frames. It provides sequence-level train/validation/test splits and a reproducible X4 evaluation protocol. As an initial architectural probe, we further evaluate CGMamba, a lightweight state-space model with 10.90M parameters and 112.14G FLOPs. CGMamba combines 2D rotary position encoding (2D~RoPE) with center-guided cross-Mamba (CGCM) fusion for implicit multi-frame reconstruction. It achieves 33.19dB PSNR, outperforming infrared single-image super-resolution references by 0.35--0.52~dB at substantially lower computational cost. Ablation results show that removing 2D~RoPE from CGCM causes a 1.53dB drop and severe grid-like artifacts. This indicates that explicit spatial anchoring is critical for stabilizing SSM-based cross-frame gating under infrared conditions. IR275K provides a reproducible foundation for accuracy--efficiency evaluation of infrared MFSR methods, while the architectural analysis offers a concrete starting point for spatially aware SSM design under resource-constrained infrared sensing. Dataset and evaluation resources are available at: https://github.com/InfraRecon7/IR275K.