CurConMix+: A Unified Spatio-Temporal Framework for Hierarchical Surgical Workflow Understanding

AI summary

Overview

Research area: Computer vision for surgical video analysis, specifically surgical action triplet recognition and hierarchical surgical workflow understanding.

Technical level: Advanced. The paper assumes familiarity with supervised contrastive learning, knowledge distillation, mixup, Swin Transformers, temporal transformers, and multi-label classification metrics such as Average Precision (AP).

Scope: The paper presents a unified spatial plus temporal framework (CurConMix+) for recognizing fine-grained surgical action triplets and transferring them to higher-level workflow tasks, together with a new hierarchically annotated benchmark (LLS48) for laparoscopic left lateral sectionectomy.

Note on completeness: The provided content is truncated. The Results section ends mid-sentence ("This strong spatia"), and results for the hierarchical workflow understanding tasks, ablations, and minority-class analyses referenced in Fig. 3 are not fully reported in the available text.

What This Paper Is About

Surgical action triplet recognition represents each surgical activity as a structured triple ⟨instrument, verb, target⟩, which is far more informative than simply labeling a video frame with a phase name. However, this task is hard because surgical datasets are severely class-imbalanced, many activities look visually nearly identical, and the three components of a triplet are semantically interdependent.

The goal of this work is to address all three of these challenges together rather than one at a time. The authors build on their prior spatial framework CurConMix, extend it with a temporal module (the Multi-Resolution Temporal Transformer), and introduce a new benchmark dataset for laparoscopic left lateral sectionectomy so that results can be evaluated across multiple levels of surgical workflow hierarchy.

Key Contributions

1. CurConMix+: A unified framework that extends the frame-level CurConMix spatial representation method by adding a Multi-Resolution Temporal Transformer (MRTT), which learns temporally coherent action representations through parallel temporal pathways, learnable resolution-fusion weights (γ_k), and a learnable spatial-vs-temporal balance (β).

2. LLS48 benchmark: A new dataset of 48 laparoscopic left lateral sectionectomy procedures with hierarchical annotations at three levels (step, task, action), comprising 5 steps, 18 tasks, and 195 unique action triplets derived from 17 instruments, 16 verbs, and 38 targets.

3. State-of-the-art triplet recognition: The framework reportedly achieves 43.4% AP_IVT on the public CholecT45 dataset and 59.0% AP_IVT on LLS48, outperforming the compared prior methods on both benchmarks.

4. Cross-level generalization: Fine-grained action-level features transfer to higher-level workflow tasks, reportedly reaching 93.3% phase accuracy on Cholec80 and 85.8% step accuracy on LLS48.

Main Findings

• CholecT45 triplet recognition: The full ensemble CurConMix-Ens+ (SwinB+SwinL+MRTT) reaches 43.4% ± 2.2 AP_IVT, compared with 41.5% ± 2.8 for Zou et al. and 41.4% ± 0.0 for TOD-Large. Intermediate variants score 41.6% ± 2.8 (CurConMix-B+, SwinB(224)+MRTT), 41.7% ± 2.4 (SwinB(384)+MRTT), and 42.2% ± 1.8 (CurConMix-L+, SwinL(224)+MRTT).

• LLS48 triplet recognition: The full CurConMix-Ens+ reaches 59.0% ± 2.8 AP_IVT, versus 45.9% ± 2.5 for the strongest listed TERL-B (SwinB(384)+MSTCN) variant, 48.0% ± 2.7 for SelfD-B (SwinB(224)), 27.7% ± 4.4 for RDV, and 27.9% ± 1.8 for RiT.

• Spatial method beats baselines even without temporal modeling: CurConMix-T attains 37.7% AP_IVT on CholecT45, described as outperforming MT4MTL-KD (37.1% ± 0.5). CurConMix-B(224) improves over SelfD from 37.1% to 38.8% on CholecT45 and from 48.0% to 49.7% on LLS48. CurConMix is reported to surpass TERL across all backbones, with gains up to +3.2 AP_IVT on CholecT45 and +5.7 on LLS48.

• Temporal extension adds the largest gains on LLS48: CurConMix-L+ achieves 56.3% AP_IVT on LLS48, exceeding its non-temporal counterpart CurConMix-L (50.5%) by +5.8 points; the paper attributes this to temporal context.

• Transfer to higher-level workflow tasks: Using a frozen triplet-pretrained backbone with only the temporal model trained, the framework reports 93.3% phase accuracy on Cholec80 and 85.8% step accuracy on LLS48.

• Long-tailed structure in LLS48: All annotation levels exhibit a long-tailed distribution, which the authors describe as reflecting real-world imbalance and procedural diversity.

• Efficiency claim: Despite the added contrastive and temporal stages, the authors state the setup adds less than 10% training time over SelfD.

• Qualitative evidence: Fig. 2 shows Grad-CAM visualizations of CurConMix on CholecT45 and LLS48, and Fig. 3 presents a comparison of minority-class performance between TERL and CurConMix+ on both datasets.

Methodology in Plain English

Spatial pretraining (CurConMix). The model first learns what individual frames look like using a three-stage curriculum of contrastive learning. In stage one, two frames are treated as similar if they share a target. In stage two, they must share instrument and target. In stage three, they must share the full triplet of instrument, verb, and target. This coarse-to-fine schedule is designed to avoid fragmenting the embedding space too early in a long-tailed setting.

Choosing hard pairs. Instead of using random pairs, the method ranks frames by cosine similarity and deliberately picks the least similar positives (BottomK) and the most similar negatives (TopN), then samples uniformly from these pools. This forces the model to focus on genuinely ambiguous cases.

Synthetic hard negatives. Because hard negatives are scarce in long-tailed data, the method mixes two hard-negative feature vectors with a Beta-distributed interpolation weight to create virtual negatives that lie between existing ones. These are added to the contrastive objective.

Handling imbalance (self-distillation with mixup). A teacher model trains on hard labels with binary cross-entropy. A student with the same architecture then learns from the teacher's soft predictions. Both teacher and student see input-level mixup, where images and labels are linearly interpolated with a Beta-distributed coefficient. Soft labels are intended to reduce overconfidence on dominant classes.

Temporal modeling (MRTT). Frame embeddings from the student backbone are passed through several parallel pathways, each applying average pooling at a different stride (the paper uses strides {4, 5, 6}) to produce different temporal resolutions. Each pathway has its own transformer encoder with fixed sinusoidal positional embeddings, followed by upsampling back to the original sequence length, and its own classification head. Learnable scalar weights γ_k (a softmax over learned parameters) combine the pathway outputs at the sequence level rather than per frame. Separately, a spatial head produces per-frame logits from the raw embeddings, and a learnable scalar β blends the spatial and temporal predictions.

Training setup. All experiments run on a single NVIDIA GeForce RTX 4090 GPU. Backbones are Swin Transformers (Tiny, Base, Large) pretrained on ImageNet via the timm library. Optimization uses AdamW with learning rate 2×10⁻⁴ decayed to 2×10⁻⁵, weight decay 1×10⁻⁶, and batch size 64. Mixup uses α=0.4 and the contrastive temperature is τ=0.1. Training is 3 epochs of curriculum contrastive pretraining, then 20 teacher epochs and 40 student epochs. The MRTT has a 3-layer encoder, 4 attention heads, feedforward dimension 2048, dropout 0.1, and fixed sinusoidal positional encoding.

LLS48 construction. The dataset covers 48 complete left lateral sectionectomy procedures by four hepatobiliary surgeons at Samsung Medical Center between February 2020 and March 2022, approved under IRB SMC-2023-04-128. Video was recorded at 1920×1080@60 FPS and 1356×728@30 FPS, totaling approximately 52 hours. Five-second clips were extracted every 90 seconds using ffmpeg; after excluding out-of-body scenes, the final set is 1,912 annotated clips (9,560 frames, 2 h 39 min 20 s). Five experts (two hepatobiliary surgeons and three scopists with more than ten years of minimally invasive surgery experience) annotated the clips, using a Streamlit-based dashboard for consensus and traceability.

Evaluation. All experiments use 5-fold cross-validation. Triplet recognition is measured by AP_I, AP_V, AP_T, AP_IV, AP_IT, and the primary metric AP_IVT. For hierarchical transfer, the backbone is frozen and only the temporal model is trained; because the backbone was trained on CholecT45 (45 videos from Cholec80), fine-tuning uses only those 45 videos in 5-fold cross-validation while the remaining 35 non-overlapping videos serve as a fixed test set, with performance averaged over five models. Transfer metrics include accuracy, precision, recall, and Jaccard index, plus the relaxed boundary metric from the MICCAI 2016 M2CAI challenge for phase recognition. Baseline temporal aggregators are TeCNO, Trans-SVNet, and Not End-to-End.

Why This Matters

Impact on research. Most prior work attacks only one part of the triplet problem: distillation methods (SelfD, TOD-Large) target imbalance, multi-task methods (TERL, MT4-MTL, LAM) improve component prediction but use separate branches that limit compositional reasoning, and graph-based methods depend on dataset priors that are themselves biased by imbalance. This paper argues for and demonstrates a single framework that addresses imbalance, subtle visual variation, and component interdependence together, and pairs it with a benchmark that has aligned annotations at three hierarchy levels.

Real-world applications.

• Automated operative documentation that records not just what phase a procedure is in, but which instrument is used on which anatomy with which action.
• Surgical skill assessment and trainee feedback, since fine-grained triplet-level and step-level understanding can support objective performance review.
• Computer-assisted surgery and intelligent operating rooms that need real-time situational awareness.
• Structured, searchable surgical video archives that can be indexed by step, task, and action for quality improvement and research.

Industry relevance. The work targets minimally invasive surgery, where video recording is already routine, making it a natural data source for surgical AI products. The claim that the approach adds less than 10% training time over SelfD suggests the accuracy gains may be achievable without a large compute penalty. The introduction of a hepatic (liver) procedure benchmark also broadens the field beyond the cholecystectomy-dominated evaluation landscape.

Future Directions

• Extending the hierarchy to phase level. LLS48 annotates three of the four SAGES levels (action, task, step) but not phase. Adding phase-level labels would allow a complete four-level evaluation of workflow understanding.

• Broadening beyond two procedure types. Evaluation currently covers CholecT45 (laparoscopic cholecystectomy) and LLS48 (laparoscopic left lateral sectionectomy). Testing whether the unified framework generalizes to other procedure families and to the two different recording resolutions noted in LLS48 would be a natural next step.

• Real-world deployment constraints. The current setup runs on a single RTX 4090 with sequence-level fusion weights designed for stability rather than per-frame responsiveness. Whether the framework can operate at live intraoperative speed and maintain accuracy under artifact-heavy footage remains an open question.

• Public release and reproducibility. The authors state that the code and dataset will be publicly released on GitHub, which would enable independent replication of the reported AP_IVT and transfer results and support further work on the same benchmark.

Target Audience

This paper is most useful for computer vision and medical image analysis researchers working on surgical video understanding, especially those interested in multi-label action recognition, contrastive representation learning, and temporal transformers. It is also relevant to surgical AI engineers and clinical informatics teams building operative documentation or skill-assessment tools, and to surgeons and clinical researchers who need a hierarchically annotated benchmark for laparoscopic liver procedures. Readers without a background in contrastive learning or transformer architectures will find the method sections demanding.

Authors’ abstract

Surgical action triplet recognition aims to understand fine-grained surgical behaviors by modeling the interactions among instruments, actions, and anatomical targets. Despite its clinical importance for workflow analysis and skill assessment, progress has been hindered by severe class imbalance, subtle visual variations, and the semantic interdependence among triplet components. Existing approaches often address only a subset of these challenges rather than tackling them jointly, which limits their ability to form a holistic understanding. This study builds upon CurConMix, a spatial representation framework. At its core, a curriculum-guided contrastive learning strategy learns discriminative and progressively correlated features, further enhanced by structured hard-pair sampling and feature-level mixup. Its temporal extension, CurConMix+, integrates a Multi-Resolution Temporal Transformer (MRTT) that achieves robust, context-aware understanding by adaptively fusing multi-scale temporal features and dynamically balancing spatio-temporal cues. Furthermore, we introduce LLS48, a new, hierarchically annotated benchmark for complex laparoscopic left lateral sectionectomy, providing step-, task-, and action-level annotations. Extensive experiments on CholecT45 and LLS48 demonstrate that CurConMix+ not only outperforms state-of-the-art approaches in triplet recognition, but also exhibits strong cross-level generalization, as its fine-grained features effectively transfer to higher-level phase and step recognition tasks. Together, the framework and dataset provide a unified foundation for hierarchy-aware, reproducible, and interpretable surgical workflow understanding. The code and dataset will be publicly released on GitHub to facilitate reproducibility and further research.