Strip-Fusion: Spatiotemporal Fusion for Multispectral Pedestrian Detection

AI summary

Overview

Research area: Computer vision for robotics and autonomous driving — specifically multispectral (visible light plus thermal) pedestrian detection using spatiotemporal sensor fusion.

Technical level: Advanced. The paper assumes familiarity with object detection architectures, MLP-Mixer-style mixing modules, KL divergence, and standard detection benchmark metrics.

Scope: The paper introduces Strip-Fusion, a spatial-temporal fusion network for detecting pedestrians from sequences of aligned-or-misaligned visible–thermal image pairs, and evaluates it on the KAIST and CVC-14 benchmarks.

Authors: Asiegbu Miracle Kanu-Asiegbu (Department of Mechanical Engineering, University of Michigan), Nitin Jotwani (Electrical Engineering and Computer Science, University of Michigan), and Xiaoxiao Du (Robotics Department, University of Michigan). Code is available at https://github.com/akanuasiegbu/stripfusion. The work was supported by the National Science Foundation under Grant IIS-2153171-CRII.

What This Paper Is About

Pedestrian detection matters for robot perception and autonomous driving, and pairing a normal camera (RGB) with a thermal camera helps because each modality covers the other's weaknesses — thermal sees pedestrian body heat in the dark, RGB has clearer texture in daylight. Existing multispectral methods mostly fuse a single static pair of RGB and thermal images, ignoring how pedestrians move over time, and they often break down when the two camera views are not perfectly pixel-aligned or when a pedestrian is heavily occluded. The goal of this paper is a fusion network that uses several frames of visible–thermal image pairs at once, handles misalignment, and rebalances the two modalities when one dominates the other.

Key Contributions

1. A spatial-temporal fusion method, Strip-Fusion. The pipeline combines temporally adaptive convolutions (TAdaConv) and an MLP-based temporal fusion module to learn spatial-temporal features from image pair sequences, and extends Strip Mixing Modules for cross-spectral (visible–thermal) fusion.

2. A novel KL-Divergence loss for modality imbalance. The loss encourages alignment of the feature distributions of the RGB and thermal modalities, transferring information from the more reliable modality to the less reliable one.

3. A novel post-processing strategy at the feature-map scale. It matches visible and thermal detection pairs between modalities and reduces false positives.

4. Experiments on two benchmarks. Results on KAIST and CVC-14 are reported as competitive with state of the art, with the authors reporting the best performance on heavy occlusion (KAIST) and strong results on spatially misaligned CVC-14, plus extensive ablation studies.

Main Findings

• KAIST, competitive but not leading overall. On the "Reasonable" setting, the authors report their best KAIST MR-All of 7.44 (MR-Day 9.11, MR-Night 4.34) at F, S_T = 1, 1 — behind MS-DETR (6.13 MR-All) and MambaST (7.26 MR-All), but ahead of AR-CNN (9.34), MBNet (8.13), and MLPD (7.58).

• Heavy occlusion is the standout KAIST result. Under the KAIST "Heavy" occlusion setting, the authors report the best MR-All of 41.05 at F, S_T = 3, 3, compared with MS-DETR at 48.71 and MambaST at 44.68. The paper states Strip-Fusion is more effective than MS-DETR for heavily occluded pedestrians.

• Best CVC-14 MR-All and MR-Day. On CVC-14, the authors report state-of-the-art MR-All of 16.34 and MR-Day of 19.30 at F, S_T = 5, 3, ahead of MS-DETR (17.15 MR-All, 21.25 MR-Day), CMM (17.13, 27.81), Kim et al. (18.7, 23.87), and MambaST (18.20, 21.91).

• Large-scale CVC-14 pedestrians favor Strip-Fusion; far pedestrians favor MS-DETR. Table III reports Ours (5, 3) at Near 9.72 and Medium 25.24 versus MS-DETR at 11.30 and 26.50, but worse on Far pedestrians (60.63 vs 56.06 for MS-DETR, and 57.61 for MambaST).

• Far pedestrians on KAIST are a second-best result. The paper attributes this to avoiding down-sampling in the fusion block, which preserves fine-grained features for small-scale pedestrians; MS-DETR reports a lower KAIST Far miss rate of 32.41 versus 35.76 for Ours (3, 3).

• TAdaConv helps, dramatically on CVC-14. Table VIII shows CVC-14 Algo. 1 MR-All improving from 25.85 without TAdaConv to 16.47 with it, and KAIST from 8.59 to 8.02.

• KL divergence helps mainly at short temporal windows. In Table VI (KAIST, F, S_T = 1, 1) the Algo. 1 MR-All is 7.53 with beta = 2 versus 9.69 with beta = 0. On CVC-14 (Table VII), the authors state that models without KL divergence achieved better performance for longer time windows (F/S_T = 7/3 and 7/5).

• Longer sequences do not monotonically help. The authors report that increasing the temporal window does not consistently improve miss rate, which they attribute in part to ROI features being extracted using the current frame's bounding box and to increased movement of pedestrians and the ego vehicle.

• Freezing base weights improves results but widens the gap between heads. Table IV shows freezing base weights helps performance, but tends to increase the performance gap between the VIS and IR columns, making the post-processing strategy necessary.

• Thermal reliability can contradict miss rate. Table IX reports a thermal reliability of 61.38% on KAIST at F, S_T = 1, 1, beta = 2, while the VIS miss rate (7.93) is lower than the IR miss rate (8.82). The authors attribute this contrast to a few well-aligned, high-confidence visible detections.

• Inference speed. The paper reports Strip-Fusion running at 2.7 fps for shorter (3-frame) sequences and 1.6 fps for longer (7-frame) sequences.

Methodology in Plain English

The method trains in two stages. In Stage 1, the authors train separate single-modality (unimodal) models — YOLOv5 networks, one on RGB images and one on thermal images. These learned weights become "base weights" that initialize the multimodal model.

In Stage 2, the model takes a sequence of visible–thermal image pairs and extracts features from each frame with modality-specific backbones. Temporally adaptive convolutions (TAdaConv) are inserted into the backbones and detection branches; these extend the frozen spatial base weights with learnable "calibration weights" so the network can build spatial-temporal representations.

The novel strip fusion module does two things. Spatially, features from the two modalities are processed with a depth-wise separable convolution and an MLP, then concatenated along the feature height so rows alternate between visible and thermal features. The concatenated features are split in half along the channel dimension: one half goes to a Cascade Group Strip Fusion Mixing Module (CGSFMM) for long-range interactions, the other to a Local Strip Fusion Mixing Module (LSFMM) for local interactions. Their outputs are concatenated, passed through a fully connected layer, split again, and added back to the original visible and thermal features via residual connections, followed by a channel mixing block. The fusion happens at 80×80, 40×40, and 20×20 scales, matching the scales shown in the high-level overview figure.

Temporally, the feature maps are divided into s × s patches, the two modalities are concatenated along the patch feature dimension, and the frame and patch dimensions are merged so MLP 2 can mix information across time. The fused representation is then split back into visible and thermal components and added to residual features.

To fight modality imbalance, the authors add a Kullback–Leibler (KL) divergence loss. Instead of estimating uncertainty with Monte Carlo Dropout as prior work did, they measure each modality's reliability directly using Complete Intersection over Union (CIoU). They filter to the top N = 300 bounding boxes and average the top CIoU scores per modality. Whichever modality is more reliable on the final frame (where ground truth exists) becomes the reference, and KL divergence pushes the weaker modality's feature relation matrix toward the stronger one. The total loss combines visible and thermal regression losses, visible and thermal objectness losses, and the KL term scaled by a hyperparameter beta.

Finally, a post-processing algorithm runs independently at each feature scale. It filters detections by confidence, computes IoU between visible and thermal boxes, matches pairs above an IoU threshold, forms the merged box from the convex hull of each pair, and sets the fused confidence to the average (rather than the maximum) of the two scores.

The researchers evaluate on KAIST and CVC-14 using log-average miss rate (MR), reporting MR-Day, MR-Night, and MR-All, plus Near (>115 pixels), Medium (45–115 pixels), and Far (<45 pixels) ranges and occlusion categories. Training used a batch size of six, two NVIDIA A100 GPUs, SGD with momentum 0.843, and a Cosine Learning Rate Schedule decaying from 1.0 to 0.12, with a two-epoch warm-up, for 25 epochs on KAIST and 40 on CVC-14.

Why This Matters

The work targets a practical gap: most multispectral pedestrian detectors treat fusion as a purely spatial, single-frame problem, while real driving scenes involve moving pedestrians and loosely calibrated camera pairs. Showing that temporal modeling plus distribution alignment improves heavy-occlusion and misalignment results points the field toward sequence-based, robustness-focused multimodal perception rather than one-shot fusion.

Real-world applications:

• Autonomous driving and advanced driver assistance, where reliable pedestrian detection under poor lighting and partial occlusion affects path planning and collision avoidance.
• Mobile robots and delivery platforms operating in cluttered urban environments, where robot perception must handle mixed lighting across day and night.
• Nighttime or low-light surveillance and security monitoring, using thermal imagery where visible cameras fail.
• Automotive sensor-fusion systems that must tolerate imperfect calibration and different camera fields of view between visible and thermal sensors.

Industry relevance: the approach builds on YOLOv5 backbones, which are widely used in production detection pipelines, and the paper reports runtime figures (2.7 fps for 3-frame sequences and 1.6 fps for 7-frame sequences) that matter for embedded automotive deployment decisions. The misalignment robustness is relevant to any vehicle that cannot guarantee precise factory calibration over its lifetime. The work was supported by the National Science Foundation, indicating continued public research investment in explainable multi-source data integration.

Future Directions

• Reduce the reliability of ROI features on stale bounding boxes. The authors note that ROI features across a sequence were extracted using the current frame's bounding box, which likely limits gains at longer temporal windows. Propagating or aligning boxes across frames is a natural next step.

• Resolve the reliability-versus-miss-rate contradiction. The thermal reliability percentage (61.38% on KAIST at F, S_T = 1, 1, beta = 2) conflicts with the lower VIS miss rate, and the authors attribute this only to a few high-confidence visible detections. A deeper reliability measure could make the KL loss more principled.

• Improve far/small pedestrian performance. Strip-Fusion is second best on KAIST Far pedestrians and behind MS-DETR on CVC-14 Far pedestrians (60.63 vs 56.06), which the authors attribute to MS-DETR's deformable Transformer fusion aligning small-scale features better.

• Accelerate inference. The reported 2.7 fps for 3-frame sequences and 1.6 fps for 7-frame sequences leave a substantial gap to real-time driving requirements, raising the question of whether the fusion module can be made cheaper without losing the occlusion gains.

Target Audience

This paper is best suited for graduate students and researchers working on multimodal perception, sensor fusion, or pedestrian detection; robotics and autonomous driving engineers evaluating temporal fusion strategies against Transformer- and Mamba-based alternatives; and practitioners interested in concrete recipes (TAdaConv, KL divergence with a CIoU-based reliability signal, feature-scale post-processing) that they can adapt to other multispectral detection tasks. Readers need prior exposure to object detection architectures and benchmark evaluation conventions to follow the experiments and ablations.

Authors’ abstract

Pedestrian detection is a critical task in robot perception. Multispectral modalities (visible light and thermal) can boost pedestrian detection performance by providing complementary visual information. Several gaps remain with multispectral pedestrian detection methods. First, existing approaches primarily focus on spatial fusion and often neglect temporal information. Second, RGB and thermal image pairs in multispectral benchmarks may not always be perfectly aligned. Pedestrians are also challenging to detect due to varying lighting conditions, occlusion, etc. This work proposes Strip-Fusion, a spatial-temporal fusion network that is robust to misalignment in input images, as well as varying lighting conditions and heavy occlusions. The Strip-Fusion pipeline integrates temporally adaptive convolutions to dynamically weigh spatial-temporal features, enabling our model to better capture pedestrian motion and context over time. A novel Kullback-Leibler divergence loss was designed to mitigate modality imbalance between visible and thermal inputs, guiding feature alignment toward the more informative modality during training. Furthermore, a novel post-processing algorithm was developed to reduce false positives. Extensive experimental results show that our method performs competitively for both the KAIST and the CVC-14 benchmarks. We also observed significant improvements compared to previous state-of-the-art on challenging conditions such as heavy occlusion and misalignment.