ELVIS: Enhance Low-Light for Video Instance Segmentation in the Dark

AI summary

Overview

Research area: Computer vision — low-light video understanding, specifically video instance segmentation (VIS), domain adaptation, and physics-based synthetic data generation.

Technical level: Advanced. The paper assumes familiarity with instance segmentation metrics (AP, AR), transformer-based segmentation architectures (Mask2Former), and camera sensor noise models (read, shot, quantization, banding noise).

Scope: The paper proposes ELVIS, a framework that adapts existing state-of-the-art video instance segmentation models to low-light videos using an unsupervised physics-based low-light video synthesis pipeline plus a calibration-free degradation estimation network.

What This Paper Is About

Video instance segmentation — detecting, classifying, tracking and pixel-masking objects across video frames — breaks down badly in dark, noisy, blurry footage, and no large annotated low-light VIS dataset exists to train on. Existing fixes either pre-process frames with a low-light enhancement model (two-stage) or retrain on synthetic low-light data, but most synthetic pipelines were designed for still images and ignore the motion blur caused by the long shutter speeds used in the dark. This paper builds a synthetic low-light video pipeline that models illumination, blur and sensor noise jointly, and injects an enhancement decoder into existing VIS networks so they can separate degradation from scene content. arXiv:2512.01495v2.

Key Contributions

1. ELVIS, described as the first low-light Video Instance Segmentation framework. It integrates an enhancement decoder head into existing VIS architectures so that degradation is disentangled from scene content, improving performance on low-light videos.
2. A physics-based degradation model for synthesizing low-light videos, which the authors describe as the first of its kind, incorporating illumination adjustment, blur degradation and camera noise.
3. VDP-Net (Video Degradation Profiler Network), trained in an unsupervised manner, which estimates degradation profiles so that low-light video synthesis requires no manual camera calibration.
4. Low-light VIS evaluation resources. The paper releases code and dataset, and constructs two evaluation sets: ELVIS-S, built from 5 real low-light videos captured on a RED V-Raptor camera (1080p crops, 50 densely annotated frames per video, 250 labelled frames total), and LMOT-S, a larger pseudo-ground-truth set built from the LMOT validation split using the Segment Anything Model (SAM).

Main Findings

• Synthetic benchmark gains: On the synthetic low-light YouTube-VIS 2019 validation set, adding ELVIS improves every tested method. With a ResNet-50 backbone, MinVIS rises from 36.4 to 37.2 AP, GenVIS from 39.1 to 41.0 AP, and DVIS++ from 38.8 to 42.5 AP — the last being the +3.7 AP headline gain reported in the abstract. With larger backbones, MinVIS (SWIN-L) rises 51.8 to 54.2 AP, GenVIS (SWIN-L) 53.7 to 55.3 AP, and DVIS++ (ViT-L) 55.2 to 56.9 AP.
• Real low-light gains: On LMOT-S, ELVIS raises MinVIS (R50) from 4.1 to 4.9 AP, GenVIS (R50) from 6.6 to 6.7 AP, DVIS++ (R50) from 7.0 to 7.3 AP, and DVIS++ (ViT-L) from 10.0 to 10.5 AP.
• ELVIS beats two-stage baselines: On ELVIS-S, ELVIS reaches 58.0 AP versus 57.3 for StableLLVE, 55.9 for DarkIR and 46.7 for SDSD-Net. On LMOT-S, ELVIS reaches 6.7 AP versus 3.9 (StableLLVE), 3.8 (DarkIR) and 2.5 (SDSD-Net) — the "at least +2.8 AP" margin over two-stage baselines. The separate AR100 comparison (Table S3) gives ELVIS 14.0 versus 8.1, 8.0 and 5.4 respectively.
• ELVIS beats other synthetic pipelines: On ELVIS-S, ELVIS-trained GenVIS scores 54.5 AP versus 53.5 for Lv et al., 51.1 for Cui et al., 35.1 for Lin et al., and 39.9 for ELVIS's own pipeline with randomly sampled parameters. On LMOT-S the corresponding figures are 6.6, 5.1, 5.7, 2.7 and 4.7 AP.
• Calibration-free estimation matters: Replacing VDP-Net's structured degradation profiles with random sampling consistently drops performance in both synthetic-pipeline comparisons (ELVIS-S: 54.5 to 39.9 AP; LMOT-S: 6.6 to 4.7 AP), which the authors attribute to learning a realistic degradation distribution rather than an arbitrary one.
• VDP-Net ablation: Measured against the Real-LOL-Blur dataset, VDP-Net with neither component achieves KLD 0.573 and FID 85.369; with cosine angular loss only, KLD improves to 0.485 but FID worsens to 91.100; with both cosine angular loss and the temporal fusion block, KLD is 0.469 and FID is 85.156. Both components matter.
• Blur model is a close approximation: The maximum intensity difference between the proposed multivariate Gaussian blur and the traditional linear motion blur plus Gaussian blur combination is less than 4%.
• Pipeline also helps low-light video enhancement: Training SDSD-Net on pipelines built from SDSD and DID, ELVIS's pipeline achieves the best scores on DID (PSNR 14.266, SSIM 0.676, LPIPS 0.164) and the best LPIPS on SDSD (0.198, with PSNR 16.946 and SSIM 0.611, second-best behind Lin et al.'s 21.970 PSNR and 0.664 SSIM).
• LMOT-S is a much harder benchmark: LMOT-S contains between 9 and 75 instances per video (mean 31.5), whereas the YouTube-VIS 2019 validation set contains between 1 and 6 (mean 1.7). The authors attribute the low absolute LMOT-S scores partly to this, and report that pre-trained GenVIS with off-the-shelf weights scores 11.7 AP on the paired normal-light LMOT-S frames versus 6.6 AP low-light and 6.7 AP with ELVIS.
• Two-stage enhancement produces artifacts: Qualitative comparison shows SDSD-Net heavily oversegmented; StableLLVE produced a perceptually realistic output that still caused false detections (a dog, extra people); DarkIR produced the highest-quality enhanced output but still misclassified a building as part of a truck.

Methodology in Plain English

The authors take clean, normally lit videos and deliberately degrade them to look like night footage, then train segmentation models on the result. Degradation happens in three steps. First, the video is darkened by a power-of-two exposure factor, matching how camera stops work. Second, blur is applied using a single multivariate Gaussian kernel with three learnable parameters (two spread values and a rotation angle), which stands in for the combined effect of motion blur and defocus blur — the authors argue this is more controllable and cheaper than averaging consecutive frames, and it is almost visually identical. Third, physics-based noise is added: Gaussian read noise, Poisson shot noise, uniform quantization noise, and banding noise modelled as horizontal or vertical Gaussian stripes (both orientations, unlike prior work).

The parameters of all these degradations form a vector called a degradation profile. Rather than relying on camera metadata, the authors train VDP-Net to infer that profile directly from real low-light video. VDP-Net uses a lightweight pre-trained ResNet-18 backbone, a temporal fusion block (average pooling, one-dimensional convolution, batch normalisation and ReLU) and two separate multi-layer perceptrons — one for the global exposure and noise parameters, one for the local blur parameters. Because real footage has no ground-truth degradation labels, VDP-Net is trained by sampling parameter values uniformly within bounds set with input from domain experts, synthesising the corresponding degraded videos, and asking the network to recover the values. The loss combines an L1 term with a cosine angular term, the latter handling the circular nature of blur angle.

For segmentation, the authors use Mask2Former-based VIS methods (MinVIS, GenVIS, DVIS++) and add an enhancement decoder head — a multi-scale deformable attention pixel decoder with ten transformer decoder layers and bilinear upsampling — which reconstructs the original normal-light frames. An L1 loss between the reconstruction and the clean video pushes the network to separate content from degradation. During VIS training, degradation profile values are not sampled randomly but drawn from a pre-collected set estimated by VDP-Net across four real low-light video datasets (SDSD, DID, BVI-RLV and LMOT), so training reflects realistic degradation.

Why This Matters

Impact on research. The paper is positioned as the first domain-adaptation framework for video instance segmentation in the dark, and the first physics-based low-light video synthesis model. It targets a gap the authors state explicitly: no publicly available benchmark is designed specifically for low-light VIS, and most synthetic pipelines model only spatial degradations, neglecting the temporal blur caused by long shutter speeds. The work also supplies ELVIS-S and LMOT-S as evaluation resources for a field that lacks them.

Real-world applications (drawn from those listed in the paper):

• Autonomous driving, where night-time perception must remain reliable.
• Wildlife conservation, where footage is often captured in low light.
• Surveillance, a domain dominated by poorly lit scenes.
• Post-production, where dark footage must be processed; the paper also names robotics.

Industry relevance. Because VDP-Net estimates degradation profiles without camera metadata or calibration, the framework requires no RAW data, event-camera data or capture-side instrumentation — only ordinary sRGB video. The paper reports that ELVIS can be dropped into several existing SOTA VIS architectures with ResNet-50, SWIN-L and ViT-L backbones, meaning it is a bolt-on rather than a replacement architecture. That lowers the barrier to improving dark-video segmentation in deployed systems.

Future Directions

• Properly annotated low-light VIS benchmarks. The authors name this explicitly as critical future work, since their real-world evaluation relies on a small hand-annotated set (ELVIS-S) and a pseudo-ground-truth set (LMOT-S).
• Better handling of crowded scenes and identity switches. The paper reports that LMOT-S's high instance counts and object cross-over cause frequent identity switches and failures to form new tracklets, and that ELVIS reduces but does not eliminate these.
• Optional modelling of camera-specific degradations. The authors deliberately exclude spatially correlated artifacts from compression, demosaicing and in-camera denoising because they are ISP-specific, but note their pipeline could incorporate them if desired.
• Broadening applicability beyond Mask2Former-style VIS. The paper does not compare against a RAW-based pipeline (Chen et al.) because its outputs are unsuitable for the sRGB VIS methods used here, leaving open how such data could be integrated.

Target Audience

Researchers and graduate students working on low-light computer vision, video instance segmentation, domain adaptation and synthetic data generation. It is also relevant to practitioners building perception pipelines for night-time or poorly lit conditions — autonomous driving, surveillance, wildlife monitoring and video post-production — who need to improve existing segmentation models without collecting and annotating large low-light datasets. Readers wanting implementation details will find the code and dataset released at the project page.

Authors’ abstract

Video instance segmentation (VIS) for low-light content remains highly challenging for both humans and machines alike, due to noise, blur and other adverse conditions. The lack of large-scale annotated datasets and the limitations of current synthetic pipelines, particularly in modeling temporal degradations, further hinder progress. Moreover, existing VIS methods are not robust to the degradations found in low-light videos and, consequently, perform poorly even after finetuning. In this paper, we introduce \textbf{ELVIS} (\textbf{E}nhance \textbf{L}ow-Light for \textbf{V}ideo \textbf{I}nstance \textbf{S}egmentation), a framework that enables domain adaptation of state-of-the-art VIS models to low-light scenarios. ELVIS is comprised of an unsupervised synthetic low-light video pipeline that models both spatial and temporal degradations, a calibration-free degradation profile estimation network (VDP-Net) and an enhancement decoder head that disentangles degradations from content features. ELVIS improves performances by up to \textbf{+3.7AP} on the synthetic low-light YouTube-VIS 2019 dataset and beats two-stage baselines by at least \textbf{+2.8AP} on real low-light videos. Code and dataset available at: \href{https://joannelin168.github.io/research/ELVIS}{https://joannelin168.github.io/research/ELVIS}