Rethinking Surgical Smoke: A Smoke-Type-Aware Laparoscopic Video Desmoking Method and Dataset

AI summary

Overview

• Research area: Computer vision for medical video — specifically laparoscopic (keyhole surgery) video restoration, sitting at the intersection of image dehazing/desmoking, video transformer architectures, and surgical scene understanding.
• Technical level: Advanced. The paper assumes familiarity with transformer attention mechanisms, deformable and dilated convolutions, mask-based segmentation heads, and no-reference perceptual image-quality metrics.
• Scope: The paper proposes STANet, the first desmoking network that explicitly distinguishes two types of surgical smoke, together with the first large-scale synthetic laparoscopic video desmoking dataset (STSVD) carrying smoke-type annotations.

What This Paper Is About

Surgical smoke produced by electrocautery or lasers obscures the view in laparoscopic video, making it harder for surgeons to see anatomy and for computer-vision systems to work reliably. The authors observe that this smoke is not one uniform phenomenon: it appears in at least two distinct motion patterns, and existing desmoking and dehazing methods treat all smoke identically. The goal is to build a method that first recognizes which type of smoke is present, then removes each type using a removal strategy matched to its behavior.

Key Contributions

1. A taxonomy of surgical smoke. The authors classify smoke into Diffusion Smoke (local and directional, appearing in the early, pre-collision stage of cauterization) and Ambient Smoke (global and directionless, appearing in the later, post-collision stage), and propose STANet — described as the first smoke-type-aware laparoscopic video desmoking method — to perform type-specific removal.
2. A coarse-to-fine disentanglement module (C2FDM). Because multiple cauterizations occur during surgery, Ambient Smoke from an earlier cauterization can become entangled with Diffusion Smoke from a later one; C2FDM uses smoke-type-aware cross attention between non-entangled and entangled regions to produce cleaner separated masks.
3. A new dataset. STSVD (Smoke-Type-Specific Video Desmoking) is presented as the first large-scale synthetic video desmoking dataset with smoke-type labels: 120 paired clean/smoky videos of 100 frames each at 720 × 1080 resolution, spanning 28 surgery types, with ground-truth smoke masks and coverage of diffusion, ambient, and entangled scenarios.
4. Demonstrated transfer to downstream tasks. Beyond quality metrics, the method is evaluated as a preprocessing step for polyp detection and surgical instrument segmentation, where it outperforms the compared baselines.

Main Findings

• Best overall restoration quality. On the synthetic STSVD test set, STANet reaches PSNR 33.5345, SSIM 0.9733, and LPIPS 0.0402, versus the best desmoking baseline SelfSVD at PSNR 29.5132, SSIM 0.9588, LPIPS 0.0842, and the best dehazing baselines SGDN (PSNR 32.6625, SSIM 0.9674, LPIPS 0.0483) and DehazeFormer (PSNR 32.5400, SSIM 0.9517, LPIPS 0.0463).
• Average gains over the strongest dehazing methods. Against SGDN and DehazeFormer, STANet improves reference metrics by an average of 0.7770 dB PSNR, 0.0156 SSIM, and 0.0120 LPIPS, and no-reference metrics TOPIQ, Q-align, MANIQA, and MUSIQ by 0.0202, 0.0445, 0.0050, and 0.3564 respectively.
• Average gains over the strongest desmoking method. Against SelfSVD, STANet improves reference metrics by an average of 2.8649 dB PSNR, 0.0291 SSIM, and 0.0487 LPIPS, and no-reference metrics by 0.0117, 0.0133, 0.0130, and 1.7906 respectively.
• Real-world generalization. On the paired real-world Vivo dataset STANet records PSNR 23.8427, SSIM 0.8813, LPIPS 0.1446; on the unpaired real-world STSVD-R it records TOPIQ 0.3271, Q-align 3.1328, MANIQA 0.1914, and MUSIQ 40.2063 — the best of all compared methods on these datasets.
• Every component contributes. In the ablation study, training on STSVD instead of PSv2rs (M1 to M2) improves all reference and no-reference metrics. Adding the semantic soft segmentation module (M3) improves PSNR, SSIM, LPIPS by an average of 0.3791 dB, 0.0088, 0.0082 and TOPIQ, Q-align, MANIQA, MUSIQ by 0.0015, 0.0339, 0.0027, 1.1203. Adding C2FDM together with the SHWL loss (M5, relative to M3) adds an average of 0.7836 dB, 0.0059, 0.0064 and 0.0130, 0.0387, 0.0042, 0.5270. Adding the smokeless video reconstruction sub-network (M6) yields further gains across all metrics, reaching PSNR 33.5345.
• Downstream task benefit. On polyp detection datasets CVC-ClinicDB, CVC-ColonDB, and Kvasir-SEG, STANet-desmoked inputs achieve DSC/IoU of 0.9024/0.8394, 0.7138/0.6331, and 0.8857/0.8240; on instrument segmentation (EndoVis18) it reaches IoU 74.814 and mcIoU 51.899. It surpasses the second-best method by 0.0122 in DSC and 0.0118 in IoU on detection, and by 0.587 in IoU and 0.104 in mcIoU on segmentation.
• Competitive complexity. STANet uses 25.15M parameters, 174.13G FLOPs, and 0.21s runtime, beating SGDN on time cost (0.84s), DehazeFormer on parameters (28.98M), and SelfSVD on FLOPs (996.00G). A dynamic activation strategy lets it skip the unused branch when only one smoke type is present.
• Baseline failure modes. The authors report that the best dehazing methods leave residual haze and lose organ color detail because they ignore smoke motion patterns, while SelfSVD struggles to recover textures in dense, entangled smoke because it has no smoke-type awareness.

Methodology in Plain English

STANet is built from three cooperating parts.

First, a feature perception stage takes a sequence of smoky frames and extracts multi-scale spatial features using a ResNet-18 backbone encoder. Because surgical smoke drifts and deforms rather than moving rigidly, the network applies a lightweight version of non-rigid trajectory attention (inspired by SODA) — sharing projection layers, restricting attention to windows, and cutting the number of attention heads and vector dimensions to keep computation manageable.

Second, a mask segmentation stage decides where each kind of smoke is. It uses a set-prediction design with 100 learnable queries, each acting as a local expert that outputs both a small local smoke mask and a smoke-type label. These local masks are combined into two whole-frame masks (one for diffusion, one for ambient) using attention-weighted aggregation, where each local mask gets a learned weight and contributes only to its own type's global mask. A coarse-to-fine disentanglement module then handles the hard case where the two smoke types overlap: it splits the coarse masks into three mutually exclusive regions (diffusion-only, ambient-only, entangled) using binarization and set operations, then uses cross attention to pull the entangled region apart into the two types, and finally runs an iterative refinement block with four self-attention heads.

Third, a reconstruction stage removes smoke in two parallel branches matched to the physics of each type. The diffusion branch concatenates masks from neighboring frames into a temporal composite mask, passes it through a CoordConv layer (with 2 added coordinate channels) to predict an 18-channel offset field, and uses a 3 × 3 deformable convolution to sample features along the smoke's diffusion path. The ambient branch predicts adaptive dilation sampling position maps for K = 3 dilation rates (set to 1, 2, and 3) and fuses three parallel 3 × 3 dilated convolutions, suited to globally spread smoke. A U-Net decoder merges both branches' outputs into the clean frame.

Training uses a multi-task loss for mask segmentation, smoke-type classification, and reconstruction, plus a novel Smoke High-frequency Wing Loss that compares high-frequency components of predicted and ground-truth masks through a 3 × 3 high-pass filter. An exponential modulation factor φ = 1 + λ_g(e^{M_GT} − 1) with λ_g = 2.0 penalizes errors more in dense smoke and less in sparse regions. Optimization uses Adam with learning rate 1 × 10⁻⁴, weight decay 0.05, gradient clipping 0.01, polynomial decay over 90K iterations, and batch size 4, implemented in PyTorch on RTX 3090 GPUs.

For the dataset, the team started from smoke-free 720 × 1080 videos from Cholec80, M2CAI16, and Hamlyn. Smoke is synthesized by a physics-based engine, with a CNN-based surgical tool tip detector locating where the smoke source should be, then blended into the clean frames using pixel-wise soft compositing with adaptive transparency plus motion blur, following an atmospheric-light formulation.

Why This Matters

• Impact on research: The paper reframes desmoking as a type-aware problem rather than generic dehazing, and releases the first video desmoking dataset with smoke-type annotations, giving the field a new benchmark and a new problem formulation to build on. It also shows that video desmoking quality propagates into measurable gains on unrelated downstream tasks (detection and segmentation).
• Real-world applications:
  • Surgical guidance displays, where removing smoke in real time restores visibility of anatomical structures during cauterization.
  • AI-assisted surgical systems, including polyp detection and instrument segmentation pipelines that degrade badly under smoky conditions.
  • Surgical training and video review, where archived laparoscopic footage is degraded by smoke and needs restoration for teaching or audit.
  • Robot-assisted surgery, where automated tool tracking and scene understanding depend on clean video input.
• Industry relevance: The method stays within a practical compute envelope (25.15M parameters, 174.13G FLOPs, 0.21s runtime) and can skip the unused branch when only one smoke type is present, which matters for latency-sensitive operating-room hardware. The smoke-type distinction also maps naturally onto surgical workflow understanding, since diffusion smoke signals an early cauterization stage and ambient smoke a later one.

Future Directions

• Extending the taxonomy. The paper settles on two smoke types plus their entanglement; whether finer or clinically derived categories exist is left open, and Table 4 lists the dataset as covering three smoke types (diffusion, ambient, entangled).
• Closing the synthetic-to-real gap. STSVD is synthetic and Vivo is described as the only existing real-world paired desmoking dataset; expanding real paired data would test whether the type-aware gains hold under genuine imaging conditions.
• Real-time clinical deployment. The reported 0.21s runtime is per-frame on RTX 3090 hardware; whether this meets live surgical video requirements and how it behaves under intraoperative variability is not reported.
• Broader downstream validation. Only polyp detection and instrument segmentation are evaluated; other tasks such as depth estimation, phase recognition, or surgical skill assessment are untested.

Target Audience

Researchers and engineers working on medical image/video restoration, surgical computer vision, and video dehazing; practitioners building AI-assisted surgical systems who need clean laparoscopic input; and dataset/benchmark builders interested in physically grounded synthetic data generation with semantic annotations. Clinicians and clinical-AI collaborators may benefit from the smoke-type taxonomy, but the method sections require a computer-vision background.

Authors’ abstract

Electrocautery or lasers will inevitably generate surgical smoke, which hinders the visual guidance of laparoscopic videos for surgical procedures. The surgical smoke can be classified into different types based on its motion patterns, leading to distinctive spatio-temporal characteristics across smoky laparoscopic videos. However, existing desmoking methods fail to account for such smoke-type-specific distinctions. Therefore, we propose the first Smoke-Type-Aware Laparoscopic Video Desmoking Network (STANet) by introducing two smoke types: Diffusion Smoke and Ambient Smoke. Specifically, a smoke mask segmentation sub-network is designed to jointly conduct smoke mask and smoke type predictions based on the attention-weighted mask aggregation, while a smokeless video reconstruction sub-network is proposed to perform specially desmoking on smoky features guided by two types of smoke mask. To address the entanglement challenges of two smoke types, we further embed a coarse-to-fine disentanglement module into the mask segmentation sub-network, which yields more accurate disentangled masks through the smoke-type-aware cross attention between non-entangled and entangled regions. In addition, we also construct the first large-scale synthetic video desmoking dataset with smoke type annotations. Extensive experiments demonstrate that our method not only outperforms state-of-the-art approaches in quality evaluations, but also exhibits superior generalization across multiple downstream surgical tasks.