Referring Camouflaged Object Detection With Multi-Context Overlapped Windows Cross-Attention

AI summary

Overview

Research area: Computer vision — referring camouflaged object detection (Ref-COD), a multimodal segmentation task that combines camouflaged-object segmentation with reference cues (reference salient-object images or text descriptions).

Technical level: Advanced. The paper assumes familiarity with transformer attention, encoder–decoder segmentation architectures, cross-attention, multi-head attention, and standard COD evaluation metrics (S_alpha, E, F_beta^omega, M).

Scope: The paper proposes RFMNet, a two-branch network that fuses multi-stage reference salient-object features (and/or CLIP-encoded text features) with camouflaged-image features via an overlapped-windows cross-attention mechanism and a progressive referring feature aggregation decoder, evaluated on the R2C7K benchmark.

What This Paper Is About

Camouflaged object detection (COD) is hard because the target looks like its background — the paper lists medical image polyp segmentation, agricultural pest detection, and industrial defect detection among the applications that motivate it. Ref-COD adds a hint, in the form of reference images of salient objects from the same category as the camouflaged target, or a text description, and the goal is to use that hint to segment the hidden object more accurately.

The authors argue that prior Ref-COD work compresses reference images into a low-dimensional one-dimensional prompt, which discards much of the reference detail and often needs many reference images to work well. Their goal is a network that exploits the full multi-stage reference features and reaches strong performance with fewer reference images, which they frame as more user-friendly for human–machine interaction.

Key Contributions

1. RFMNet, a two-branch network that integrates multi-context reference features into camouflaged feature maps at the corresponding encoding stages, rather than reducing references to a single low-dimensional prompt, and that the authors report achieves optimal performance with fewer images.
2. An overlapped windows cross-attention mechanism (used in the reference-image fusion path, RIF-s) that divides the camouflaged feature map into overlapping windows, performs cross-attention from those windows to reference features, and folds the results back, so the module focuses on local feature matching. It is combined with a learnable parameter alpha in a residual-style fusion.
3. A referring feature aggregation (RFA) module as the decoder, which progressively aggregates adjacent-layer fused features and produces the segmentation predictions layer by layer.
4. A text semantics-guided referring object enhancement module (used in the reference-text path, RIF-t), which computes sentence-level attention from camouflaged features and CLIP text features to generate object-enhancement vectors.

Main Findings

• Full-benchmark results: On R2C7K, RFMNet-S with ResNet-50 and 3 reference images reaches Overall S_alpha 0.829, E 0.903, F_beta^omega 0.719, M 0.030. The SwinTransformer-S variant with 3 reference images reaches S_alpha 0.875, E 0.933, F_beta^omega 0.796, M 0.021.
• Gains over the second-best image-referring model: Compared with DGNet-S, RFMNet-S (ResNet-50 backbones) shows a 6.25% average performance gain in M and a 3.30% improvement in F_beta^omega.
• Text-referring comparison: RFMNet-T outperforms the second-best referring text model by 3.13% on M.
• Transformer-backbone comparison: The SwinTransformer-S-based RFMNet-S surpasses the second-best model, RPMA-S, by 1.5% in F_beta^omega — and RFMNet-S requires only three reference images, versus ten for RPMA-S.
• Fewer references needed: Ablating reference-image count (N from 0 to 5) shows the best scores at N=3 (S_alpha 0.829, E 0.903, F_beta^omega 0.719, M 0.030), whereas the paper states R2CNet and UAT use ICON to extract reference features and need five images for their best results. Performance at N=5 was slightly lower than at N=3.
• Ablation of components: Baseline ResNet-50 + FPN gives 0.773/0.835/0.596/0.045 (S_alpha/E/F_beta^omega/M). Adding RIF_t gives 0.812/0.873/0.677/0.036; adding RIF_s gives 0.798/0.886/0.664/0.037; adding RFA gives 0.822/0.895/0.705/0.032; combining RIF_t+RFA gives 0.827/0.899/0.718/0.031 and RIF_s+RFA gives 0.829/0.903/0.719/0.030. With SwinTransformer-S, RFA alone gives 0.87/0.928/0.785/0.023 and RIF_s+RFA gives 0.876/0.933/0.797/0.021 — an improvement of 0.69%, 0.54%, 1.53% and 8.70% in S_alpha, E, F_beta^omega and M respectively.
• Window configuration matters: The best window setting is the one where windows are partitioned from high to low into progressively larger regions (H_2, H_3/2, H_4/4) with step size half the window size, giving 0.829/0.903/0.719/0.030, versus uniform full-size windows with full step (0.827/0.903/0.719/0.031).
• Where fusion is applied matters: Every referring fusion layer helps; fusing at f_4 in particular raises F_beta^omega from 0.705 to 0.717, and combining layers also improves performance.
• Qualitative behavior: Visualization of intermediate features indicates the overlapped-windows cross-attention suppresses irrelevant regions and locates camouflaged objects, and produces more uniform attention that lets the RFA module segment object regions more completely. Qualitative comparisons cover five challenging scenarios: small object, large object, multiple objects, occlusion, and uncertainty.
• Cost of the gains: Table 6 reports 24.72M params, 27.48G MACs and 46.79 FPS for RFMNet-S (ResNet-50, 512 input) versus 25.1M params, 11.68G MACs and 176.73 FPS for R2CNet at 352 input; the paper states R2CNet's computational speed is 3.7 times faster than RFMNet. For Swin-S at 512 input, RFMNet-S reports 34.1M params, 35.66G MACs and 31.44 FPS, compared with UAT (PVTv2) at 98.05M params, 99.15G MACs and 71.50 FPS at 352 input.

Methodology in Plain English

The network has two branches. A side branch encodes reference images (or text) and a main branch encodes the camouflaged image, both using a shared-style backbone that produces four stages of features. Reference image features from the multiple encoding stages are concatenated across the K reference images and passed through a 1×1 convolution block, giving multi-level reference features for stages 2, 3 and 4 — the stage-1 features are deliberately left out of fusion, following prior work and to keep computational cost down.

Fusion happens in Referring Information Fusion (RIF) modules. For reference images, the camouflaged feature map at a stage is cut into overlapping windows of size k with a sliding step of k/2. Each window is linearly projected and split into multiple heads to form queries, while the reference features form keys and values, and cross-attention computes a softmax-weighted similarity that is folded back to the original map size, averaging values in overlapping regions. A residual blend using a learnable alpha (alpha·fused + (1−alpha)·camouflage) plus a 1×1 convolution block preserves the original camouflaged features. The paper notes one difference from the work it draws inspiration from: here the interaction is between reference features and camouflage features, not between high-level and low-level features. For text, the text-semantics module instead multiplies camouflaged features with CLIP text features to weight sentences and produce enhancement vectors, then concatenates and convolves.

A Referring Feature Aggregation decoder then works from the deepest level upward. Each level's features pass through stacked 3×3 convolution blocks; intermediate features are formed by concatenating the current features with foreground features enhanced from the previous (deeper) layer via bilinear interpolation and pixel-wise multiplication, and a 1×1 convolution with one output channel produces the final prediction at each level.

Training is two-stage. First a baseline backbone plus FPN decoder is trained on the training set at batch size 32 for 45 epochs, and a same-type baseline is trained on the DUTS dataset for 45 epochs to strengthen salient-object feature extraction; CLIP is used pre-trained for text features. That trained backbone becomes the frozen side branch. Second, RFMNet is trained for 500 epochs with backbone parameters frozen, using Adam with polynomial decay, momentum 0.9, initial learning rate 1.5e-4 and power 0.9, at 512×512 input, on a single NVIDIA GeForce RTX 4090. Supervision combines weighted IoU loss and weighted BCE loss at each of the four prediction levels, with the total loss weighting the highest-resolution prediction most heavily: 7L1 + (4L2 + 3L3 + 2L4).

Evaluation uses four metrics: Structure-measure (S_alpha), adaptive E-measure (E), weighted F-measure (F_beta^omega) and mean absolute error (M). Data is R2C7K: a Camo-subset of 5,015 camouflaged images from 64 categories and a Ref-subset of salient-object reference images from 64 categories at 25 images each, totalling 1,600 images; text descriptions of the Camo-subset images were generated with the large model mPLUG-Owl2.

Why This Matters

The paper argues that the low-dimensional reference-prompt approach does not exploit all representative reference features and often needs more images to reach its best performance, which the authors describe as unfriendly for human–machine interaction. RFMNet's reported ability to reach its best results with three reference images, against five for models using ICON-based reference extraction and ten for RPMA-S, is presented as a practical advantage in both reference information sufficiency and the computational load of the reference feature extraction stage.

Real-world applications the paper cites for this line of research:

• Medical image polyp segmentation.
• Agricultural pest detection.
• Industrial defect detection.
• Multimodal human–machine interaction in more complex scenarios.

Industry relevance: the author list includes affiliations with Zeekr (Geely) and the Research and Design Center of Shanghai Institute of Computer Technology Company, alongside Shanghai Polytechnic University and Fudan University, and the work is supported by National Nature Science Foundation of China Grant No. 62203291. Training is reported on a single RTX 4090, and the efficiency table compares RFMNet variants against R2CNet and UAT on parameters, MACs and FPS at 352, 384 and 512 input sizes — the kind of trade-off information a deployment team would need, since the paper also reports the speed penalty relative to R2CNet.

Future Directions

• Speed and efficiency. The authors state that R2CNet runs 3.7 times faster than RFMNet; closing that MACs and FPS gap while keeping accuracy is an explicit open problem.
• Resolution flexibility. The paper notes RFMNet-S suits particular input sizes: at 352 input the stage-3 and stage-4 features are 22 and 11 pixels, so window pixel size and step can only be 2 and 1, otherwise edge padding is required for larger stride settings, and this input size leads to lower computational speed. Making the windows adapt to arbitrary input sizes remains open.
• Reference-count trade-off. Since performance peaks at three reference images and slightly declines at four and five, understanding why more references stop helping — and how to select the most informative references — is a natural next question.
• Extending the fusion design. The paper reports RIF-t and RIF-s as separate fusion paths for text and images; whether a unified module can handle both modalities, and how the window configuration should be set when the number of feature stages or the backbone changes, are not resolved by the reported ablations.

Target Audience

Researchers and graduate students working on camouflaged object detection, referring segmentation, few-shot/support-set segmentation, and multimodal fusion; practitioners building segmentation systems for medical imaging, agriculture or industrial inspection who need to know the accuracy-versus-speed trade-offs of a reference-guided model; and readers interested in how transformer attention variants (window-based, overlapped, cross-attention) can be adapted to align two visually dissimilar feature sources.

Authors’ abstract

Referring camouflaged object detection (Ref-COD) aims to identify hidden objects by incorporating reference information such as images and text descriptions. Previous research has transformed reference images with salient objects into one-dimensional prompts, yielding significant results. We explore ways to enhance performance through multi-context fusion of rich salient image features and camouflaged object features. Therefore, we propose RFMNet, which utilizes features from multiple encoding stages of the reference salient images and performs interactive fusion with the camouflage features at the corresponding encoding stages. Given that the features in salient object images contain abundant object-related detail information, performing feature fusion within local areas is more beneficial for detecting camouflaged objects. Therefore, we propose an Overlapped Windows Cross-attention mechanism to enable the model to focus more attention on the local information matching based on reference features. Besides, we propose the Referring Feature Aggregation (RFA) module to decode and segment the camouflaged objects progressively. Extensive experiments on the Ref-COD benchmark demonstrate that our method achieves state-of-the-art performance.