Re-coding for Uncertainties: Edge-awareness Semantic Concordance for Resilient Event-RGB Segmentation

AI summary

Overview

Research area: Multi-modality computer vision, specifically event-camera plus RGB semantic segmentation under extreme imaging conditions (low light, heavy motion, occlusion).

Technical level: Advanced. The paper assumes familiarity with semantic segmentation architectures, vector-quantised latent spaces (VQ-VAE), attention mechanisms, and event-camera data representations.

Scope: The paper proposes an edge-aware latent re-coding framework (ESC) that uses a discrete "edge dictionary" as a shared semantic bridge between heterogeneous event and RGB features, and validates it on two newly built datasets plus two existing ones.

Note: the provided paper content is truncated. The appendix sections (dataset details, statistical process, dictionary training, training settings) and the final rows of the occlusion ablation table are cut off, so those specific values are not reported here.

What This Paper Is About

RGB semantic segmentation collapses in extreme conditions such as insufficient light or fierce camera motion, because RGB imagery loses information that a high-dynamic-range event camera can still capture. The problem is that event and RGB data are naturally heterogeneous, so naive feature fusion produces feature-level mismatch and poor optimisation, particularly when one modality becomes imbalanced or fails. The paper's goal is to find a common structure in both modalities — semantic edges — and use it to realign them into a single unified semantic space for more resilient fusion.

Key Contributions

1. The ESC framework. Edge-awareness Semantic Concordance, a multi-modality framework that supervises a re-coded edge distribution to realign heterogeneous event and RGB features into a unified semantic space, and jointly optimises them using uncertainties derived from modality distributions.

2. Three modules. Edge-awareness Latent Re-coding (ELR), which re-codes features and distributions bi-directionally; Re-coded Consolidation (RC) and Uncertainty Optimization (UO), which use the re-coded features and uncertainty indicators for resilient fusion.

3. New evaluation datasets. Two synthetic and one real-world event-RGB semantic segmentation dataset — DERS-XS, DERS-XR, and DSEC-Xtrm — built to correct what the authors argue is unreliable evaluation based on RGB-derived pseudo-labels.

4. A new resilience test. The authors state this is the first work to assess model resilience via spatial occlusion evaluation without any fine-tuning.

Main Findings

• Events cluster on semantic edges. Using 200 randomly selected event sequences from true-labeled DERS-XS and real-world DSEC-Semantic, the authors counted the ratio of edge pixels to all pixels and the ratio of events falling on edge pixels to all events. As the edge region expands, the event ratio is always greater than the boundary ratio in both datasets, supporting semantic edge as a bridge between the modalities.

• Best mIoU on the true-labeled synthetic DERS-XS. ESC reaches 75.26 mACC / 67.10 mIoU, against CMNeXt at 73.30 / 64.55, CMX at 71.86 / 63.12, TokenFusion at 64.88 / 56.22, and the RGB-only SegFormer at 62.45 / 53.21. The paper reports this as a 2.55% mIoU improvement over the state of the art on DERS-XS.

• Best mIoU on the real-world DERS-XR. ESC reaches 70.75 mACC / 65.22 mIoU, against EISNet at 66.18 / 61.81, CMNeXt at 66.57 / 60.95, CMX at 64.51 / 59.22, and SegFormer at 55.37 / 51.09 — a reported 3.41% mIoU gain over EISNet.

• Best mIoU on non-extreme DSEC-Semantic. ESC reaches 78.61 mACC / 71.04 mIoU, versus CMNeXt at 77.50 / 69.03, CMX at 76.18 / 68.10, and TokenFusion at 74.60 / 67.39 — a reported 2.01% mIoU gain over CMNeXt. EISNet performs slightly worse here, which the authors attribute to sensitivity to the input cropping strategy.

• Best mIoU under degraded conditions on DSEC-Xtrm. ESC reaches 59.45 mACC / 50.87 mIoU, versus EISNet at 56.77 / 48.76, TokenFusion at 53.04 / 45.41, CMNeXt at 52.12 / 45.16, CMX at 51.29 / 43.95, and SegFormer at 41.74 / 33.88 — a reported 2.11% mIoU gain over EISNet, with less performance degradation under degraded inputs.

• Event-only models remain weak. Event-only SegFormer scores 47.85 / 37.32 on DERS-XS and 47.38 / 38.59 on DSEC-Semantic; EvSegFormer scores 41.48 / 31.85 and 44.72 / 37.13 respectively. Notably, on DSEC-Xtrm event-only SegFormer (48.52 / 37.72) outperforms RGB-only SegFormer (41.74 / 33.88), consistent with the degradation applied to the RGB frames.

• Spatial occlusion robustness (partial table). On DERS-XS, masking RGB drops TokenFusion from 56.22 to 48.44 mIoU, CMX from 63.12 to 54.13, CMNeXt from 64.55 to 54.15, and EISNet from 60.68 to 55.33. Masking events barely changes results (TokenFusion 55.79, CMX 62.70, CMNeXt 64.07), while masking both modalities drops scores further (48.00, 53.73, 53.70). The remaining rows, including the authors' own method, are truncated in the provided content and are not reported here.

• Qualitative behaviour. The authors report that ESC is more stable and robust at segmentation edges, particularly for moving vehicles and pedestrians, and preserves more complete boundaries on DSEC-Xtrm; on real-world DERS-XR they report that ESC segments vehicles where other methods fail.

• Synthetic-to-real transfer. Models trained on synthetic DERS-XS adapt to real-world data with minimal fine-tuning, which the authors present as validation of DERS-XS.

Methodology in Plain English

The starting intuition is that events fire where brightness changes, and brightness changes concentrate at object boundaries — so semantic edges are the shared structure between event streams and RGB images. The authors turn that intuition into an architecture in three steps.

First, they build an edge dictionary. Using a VQ-VAE-style architecture, they take semantic boundary maps derived from ground-truth masks (a mean filter plus an indicator function), encode them into embeddings, quantise those embeddings against a fixed set of learnable dictionary items, and train the tokenizer/detokenizer to reconstruct the boundary map faithfully. The dictionary has 128 items of dimension 256, and is trained as a separate stage with reconstruction, embedding, and commitment losses (commitment weight 0.25).

Second, they re-code in both directions. A known semantic edge can be converted into a one-hot "prior" distribution over dictionary items by nearest-neighbour lookup. Conversely, each modality's own features can be pushed through edge encoders and MLP classification heads to produce a modality-specific probability distribution over the same dictionary items. Taking the arg-max of that distribution gives a key map, which queries the dictionary to produce re-coded edge features for that modality. A cross-entropy loss (weight 0.1) pulls both modalities' predicted distributions toward the prior derived from the ground-truth edge, forcing image and events into the same semantic space.

Third, they consolidate and optimise. The Re-coded Consolidation module uses attention, with learnable noise embeddings added to keys and queries, to let the RGB feature absorb the re-coded edge features from both modalities. The Uncertainty Optimization module turns each modality's maximum probability into a confidence (uncertainty being one minus confidence) and uses that confidence to weight a normalised combination of the two modalities' attention-refined edge features, so that a trustworthy modality dominates at each spatial location. The two resulting features are concatenated and passed to an MLP head for the semantic mask, trained with cross-entropy plus the weighted edge loss.

To evaluate fairly, the authors argue that DSEC-Semantic labels are pseudo-labels derived from RGB and therefore implicitly assume an RGB-only model is optimal. They build DERS-XS in the CARLA simulator (270 sequences, 23 label categories, 1200 frames each, 640×360, with v2e-generated noisy events at a shot noise parameter of 5.0 Hz and low-light frames simulated in the RAW domain; sampled every 100 frames; split into 168 training, 12 validation and 90 test sequences), DERS-XR captured with a DAVIS346 and manually annotated (240 frames, 120 for fine-tuning and 120 for testing), and DSEC-Xtrm by applying the same degradations to DSEC-Semantic.

Implementation uses PyTorch with a MiT-B2 backbone for RGB and MiT-B1 for events, 11 categories, 5 voxel-grid bins, crops of 256×256, random resize scales from 0.5 to 2.0, and test-time upsampling to dimensions divisible by 32. Event-only baselines use 6-channel image representations; multi-modality baselines use 3-bin voxel grids.

Why This Matters

Impact on research. The paper reframes event-RGB fusion as a representation-alignment problem rather than a feature-concatenation problem, and imports discrete shared-latent-space ideas from vision-language modelling into a low-level sensor-fusion setting. Its second, arguably sharper contribution is methodological: the argument that RGB-derived pseudo-labels structurally disadvantage event information, which matters for how the whole event-camera segmentation subfield evaluates itself. The datasets and the fine-tuning-free occlusion protocol give the community concrete alternatives to measure resilience.

Real-world applications.

• Autonomous driving and advanced driver assistance in low light, tunnels, or at night, where event cameras retain motion edges that RGB loses.
• Robotics and drones operating under fast motion or in dark environments, where low latency and high dynamic range are safety-relevant.
• Industrial inspection and surveillance, where cameras may face glare, poor illumination, or unpredictable motion blur.
• Medical imaging and geospatial analysis, listed by the authors as general semantic segmentation domains where corrupted single-modality input is a risk.

Industry relevance. Event cameras are commercially available and attractive for their low power consumption and high temporal resolution, but adopting them in products requires fusion methods that degrade gracefully when one sensor fails or is occluded. This work's explicit uncertainty indicators and occlusion testing speak directly to sensor-failure tolerance, a practical requirement for deployment. Public code and datasets at the project repository lower the barrier to reproduction and benchmarking.

Future Directions

• Scaling the edge dictionary and evaluating its size. The paper fixes the dictionary at 128 items of dimension 256 and frames the input-resolution choice as something to be ablated; how dictionary capacity and latent resolution trade off against accuracy is left for further study.
• Extending resilience evaluation. Spatial occlusion is tested without fine-tuning on DERS-XS, but the reported table is incomplete and the paper does not report full temporal sensor dropout, long-duration failure, or adversarial degradation regimes.
• Broadening beyond driving scenes. DERS-XS is CARLA-simulated and DERS-XR uses 240 manually annotated DAVIS346 frames, with the evaluation categories fixed at 11. Whether the edge-dictionary bridge transfers to indoor, medical, or industrial domains with different edge statistics is untested.
• Understanding the noise-embedding mechanism. The authors give a theoretical explanation for why learnable noise embeddings prevent a query from over-attending to its own modality, but the effect is not isolated in the reported ablations, leaving the mechanism's contribution an open question.

Target Audience

Researchers and graduate students working on multi-modality semantic segmentation, event-camera vision, and sensor fusion; engineers building perception systems that must remain functional when imaging conditions or individual sensors degrade; and benchmark builders interested in the critique of pseudo-labelled evaluation and in the three new datasets released with this work.

Authors’ abstract

Semantic segmentation has achieved great success in ideal conditions. However, when facing extreme conditions (e.g., insufficient light, fierce camera motion), most existing methods suffer from significant information loss of RGB, severely damaging segmentation results. Several researches exploit the high-speed and high-dynamic event modality as a complement, but event and RGB are naturally heterogeneous, which leads to feature-level mismatch and inferior optimization of existing multi-modality methods. Different from these researches, we delve into the edge secret of both modalities for resilient fusion and propose a novel Edge-awareness Semantic Concordance framework to unify the multi-modality heterogeneous features with latent edge cues. In this framework, we first propose Edge-awareness Latent Re-coding, which obtains uncertainty indicators while realigning event-RGB features into unified semantic space guided by re-coded distribution, and transfers event-RGB distributions into re-coded features by utilizing a pre-established edge dictionary as clues. We then propose Re-coded Consolidation and Uncertainty Optimization, which utilize re-coded edge features and uncertainty indicators to solve the heterogeneous event-RGB fusion issues under extreme conditions. We establish two synthetic and one real-world event-RGB semantic segmentation datasets for extreme scenario comparisons. Experimental results show that our method outperforms the state-of-the-art by a 2.55% mIoU on our proposed DERS-XS, and possesses superior resilience under spatial occlusion. Our code and datasets are publicly available at https://github.com/iCVTEAM/ESC.