SGDiff: Scene Graph Guided Diffusion Model for Image Collaborative SegCaptioning

AI summary

Overview

Research area: Computer vision, specifically controllable image semantic understanding — the intersection of image captioning and image segmentation under user prompts, using diffusion models and scene graphs.

Technical level: Advanced. The paper assumes familiarity with diffusion models, transformer architectures, scene graphs, contrastive learning (InfoNCE), and segmentation frameworks such as Mask2Former and SAM.

Scope: The paper defines and tackles a new task, "Image Collaborative Segmentation and Captioning" (SegCaptioning), and proposes a single model, SGDiff, that predicts multiple aligned (caption, mask) pairs from one simple bounding-box prompt.

What This Paper Is About

Existing controllable image understanding systems either require expensive user input (a full sentence, multiple boxes, clicks, or scribbles) or return only one result, and systems that produce both a caption and masks do so sequentially, which causes error propagation. This paper proposes a new task, SegCaptioning, in which a user supplies only a simple prompt such as a bounding box around one object and the model returns multiple diverse but semantically aligned (caption, masks) pairs that the user can choose from. The goal is to reduce user input cost while enriching the output with richer bimodal semantics.

Key Contributions

1. A new task: Image Collaborative Segmentation and Captioning (SegCaptioning). Users input a straightforward prompt (e.g., a bounding box) and the model predicts semantically aligned caption and mask pairs in parallel, rather than one unique outcome or a sequential caption-then-mask pipeline.

2. SGDiff (Scene Graph Guided Diffusion Model). A model that simultaneously generates (caption, masks) pairs from a simple prompt, translating that prompt into a structured scene graph that captures user intention and guides generation.

3. PSGA (Prompt-Centric Scene Graph Adaptor). A module that converts a user's prompt into a subgraph of a global scene graph, filtering irrelevant nodes and reordering them to be consistent with the caption's object order, so the model can capture user intention.

4. SGBTrans (Scene Graph Guided Bimodal Transformer) and MECL (Multi-Entities Contrastive Learning Loss). SGBTrans explores correlations between scene graph and caption features throughout the denoising process using a cross-training scheme; MECL explicitly aligns each mask with its corresponding caption words at both intra-sample and inter-sample levels.

Main Findings

• MSCOCO results: SGDiff achieves SPICE 26.1, CIDEr 137.4, mIoU 66.3, and mAP 47.2, outperforming all compared methods including GLaMM (SPICE 23.3, CIDEr 130.9, mIoU 36.2, mAP 14.8), SCA (SPICE 18.8, CIDEr 85.0, mIoU 32.4, mAP 11.3), and CAT (SPICE 17.9, CIDEr 90.1, mIoU 24.9, mAP 8.2).

• Flickr30k Entities results: SGDiff reaches Bleu-4 38.2, METEOR 27.4, SPICE 24.5, and CIDEr 98.6, exceeding GROUNDHOG (36.7 / 26.5 / 20.4 / 91.3) and GLaMM (35.4 / 26.4 / 18.8 / 95.3).

• PSGA components both help: Removing both filtering and ranking gives SPICE 24.2 and CIDEr 127.4; adding filtering alone gives 24.9 / 133.7; adding ranking alone gives 25.4 / 135.8; using both gives 26.1 / 137.4.

• Cross-attention in SGBTrans matters: Without cross-attention the model scores SPICE 24.5, CIDEr 133.7, mIoU 64.5, mAP 46.0; with cross-attention it improves to 26.1 / 137.4 / 66.3 / 47.2.

• Collaborative training beats single-task training: Training with only the captioning loss gives SPICE 25.2 / CIDEr 135.8 (no mask metrics); training with only the mask loss gives mIoU 63.7 / mAP 44.5 (no caption metrics); joint training gives 26.1 / 137.4 / 66.3 / 47.2.

• MECL's two components are complementary: No MECL gives SPICE 24.8, CIDEr 134.8, mIoU 64.2, mAP 42.7; intra-sample only gives 25.4 / 136.4 / 65.6 / 46.2; inter-sample only gives 25.6 / 136.7 / 65.3 / 46.0; both give 26.1 / 137.4 / 66.3 / 47.2.

• Loss weights have optima: For the SGadaptor weight λ₁, values of 0, 1, 2, 5, and 10 give SPICE of 24.2, 24.9, 26.1, 25.8, and 25.3 respectively, with the best at λ₁ = 2. For the MECL weight λ₂, values of 0, 1, 2, 5, and 10 give SPICE of 24.8, 26.1, 25.6, 25.4, and 25.0 respectively, with the best at λ₂ = 1.

• Qualitative comparison: On MSCOCO, SCA is limited to masks confined within a bounding box and cannot capture semantics outside it, and GLaMM's SAM struggles to predict masks guided by text, while SGDiff produces multiple diverse yet precise results from a simple prompt.

Methodology in Plain English

Given an image and a bounding box around one object, the system needs to produce several (caption, masks) pairs. It works in three pieces.

First, a Prompt-Centric Scene Graph Adaptor (PSGA) builds a global scene graph for the image (following Zellers et al. 2018), then searches for a subgraph centered on the prompted object. Because that coarse subgraph contains too many nodes and edges, an adaptor with self-attention blocks and a mapping layer scores each node's relevance to the eventual caption-mask pair, trained with a binary cross-entropy loss against ground truth built from Intersection over Union (IOU) with a threshold. A separate multi-layer network predicts a permutation matrix that reorders the graph nodes so their order matches the order of objects in the caption, trained with a cross-entropy ranking loss. Irrelevant nodes below a threshold are dropped, and the result is fused with edge features into a graph feature.

Second, a diffusion model with a Scene Graph Guided Bimodal Transformer (SGBTrans) adds Gaussian noise to a caption and denoises it, using the graph feature to guide the process. SGBTrans has 6 blocks, each with self-attention, cross-attention, and a feed-forward layer. It uses a cross-training scheme: caption features query the graph features to refine the caption, while graph features query the caption features to produce visual features for masks. The denoised caption feature predicts words; the mask branch feeds a global visual feature (extracted by Swin-Transformer) and the graph output into Mask2Former to produce class-agnostic binary region masks, trained with Dice and BCE losses.

Third, a Multi-Entities Contrastive Learning Loss (MECL) aligns masks and caption words. Since the relationship is not strictly one-to-one (a word like "dog" may match multiple masks and vice versa), the loss uses InfoNCE-style terms: an intra-sample term that pulls each mask embedding toward its matched word embeddings and away from others (and symmetrically for words), and an inter-sample term that scores whole mask-caption pairs within a minibatch and pulls matched pairs closer.

Training happens in two stages. Stage one trains the captioning branch with the mask loss and MECL removed, using the Adam optimizer at a learning rate of 0.0001 with weight decay 0.05 on two A6000 GPUs for 60 epochs at batch size 16. Stage two jointly trains masks and captions with the complete loss for another 60 epochs at batch size 16. Supplementary details: subgraphs use node size 36 and edge size 64, filtering threshold θ = 0.5, denoising timesteps T = 50, maximum caption length 20 words, and final weights λ₁ = 2 and λ₂ = 1.

Datasets: MSCOCO (123,287 images, five captions each, 133 categories, Karpathy split of 113,287 train / 5,000 validation / 5,000 test) and Flickr30k Entities (31,000 images, five captions each, Karpathy split of 29,000 train / 1,000 validation / 1,000 test), where SAM converts each ground-truth box into an approximate binary mask. Because the raw datasets lack mask-word pair annotations, the authors automatically annotate them: on COCO they match each noun in a caption to the text label of masks; on Flickr30k Entities they transfer boxes to masks with SAM and map masks to caption words the same way. Metrics are Bleu-4, METEOR, SPICE, and CIDEr for captions; Mask IoU for masks; and mAP of mask classification for objects in the caption for matching. Since the method produces multiple results, the top 5 results are selected and the highest captioning score is used for comparison.

Why This Matters

Impact on research: This is described as the first study to support generating multiple caption-mask pairs, opening a new task category that sits between controllable captioning and controllable segmentation. It also challenges the sequential caption-then-mask paradigm used by methods such as CAT, SCA, and GLaMM, showing that parallel prediction with cross-modal alignment can reduce error propagation and improve both caption and mask quality. The ablation results give concrete evidence that scene graph filtering, node-ranking, cross-attention, and explicit mask-word contrastive alignment each contribute measurable gains.

Real-world applications (implied by the task definition):

• Accessibility tools that generate both textual descriptions and object outlines from a single tap or box on an image.
• Photo editing or content creation tools where a user selects one object and receives several labeled, masked variants to choose from.
• Robotics or autonomous systems that need both a natural-language description and precise region masks for a prompted object.
• Image search and annotation pipelines where lightweight prompts produce rich, structured labels without expensive manual labeling.

Industry relevance: The method is built on widely used open components (Swin-Transformer, Mask2Former, SAM-style prompt interfaces, GPT-era multi-modal pipelines), so it is directly comparable to production-grade grounding and captioning systems. Its core selling point for industry is cost reduction — a bounding box is far cheaper for a user or annotator to produce than a full sentence or multiple boxes, while the output is richer. Reported mAP improvements over GLaMM on MSCOCO (47.2 vs. 14.8) and mIoU improvements (66.3 vs. 36.2) are large enough to matter for downstream tasks that depend on precise mask-word grounding.

Future Directions

• Since this is the first exploration of SegCaptioning, the paper states it provides "valuable insights and guidance for future research" but does not list specific follow-up work; natural extensions include scaling the task to more prompt types beyond bounding boxes.
• Extending the automatic mask-word annotation pipeline, which currently relies on COCO's existing mask labels and SAM's box-to-mask conversion on Flickr30k Entities, to datasets without either type of annotation.
• Investigating what happens with larger numbers of generated (caption, masks) pairs and how users would select among them, since results are currently evaluated by taking the top 5 and keeping the highest captioning score.
• Exploring whether the PSGA and SGBTrans design transfers to other prompt modalities (clicks, scribbles, free-form text) that the related work discusses but this paper does not evaluate.

Target Audience

Researchers and graduate students in computer vision and multi-modal machine learning, particularly those working on controllable image captioning, referring segmentation, grounded captioning, or diffusion-based generation. Practitioners building image understanding systems that need joint text and mask outputs from lightweight user input will also find the architecture and ablation results useful. Readers without a background in transformers, diffusion models, or scene graphs will find this paper difficult, as it assumes fluency in all three.

Authors’ abstract

Controllable image semantic understanding tasks, such as captioning or segmentation, necessitate users to input a prompt (e.g., text or bounding boxes) to predict a unique outcome, presenting challenges such as high-cost prompt input or limited information output. This paper introduces a new task ``Image Collaborative Segmentation and Captioning'' (SegCaptioning), which aims to translate a straightforward prompt, like a bounding box around an object, into diverse semantic interpretations represented by (caption, masks) pairs, allowing flexible result selection by users. This task poses significant challenges, including accurately capturing a user's intention from a minimal prompt while simultaneously predicting multiple semantically aligned caption words and masks. Technically, we propose a novel Scene Graph Guided Diffusion Model that leverages structured scene graph features for correlated mask-caption prediction. Initially, we introduce a Prompt-Centric Scene Graph Adaptor to map a user's prompt to a scene graph, effectively capturing his intention. Subsequently, we employ a diffusion process incorporating a Scene Graph Guided Bimodal Transformer to predict correlated caption-mask pairs by uncovering intricate correlations between them. To ensure accurate alignment, we design a Multi-Entities Contrastive Learning loss to explicitly align visual and textual entities by considering inter-modal similarity, resulting in well-aligned caption-mask pairs. Extensive experiments conducted on two datasets demonstrate that SGDiff achieves superior performance in SegCaptioning, yielding promising results for both captioning and segmentation tasks with minimal prompt input.