Synthetic Object Compositions for Scalable and Accurate Learning in Detection, Segmentation, and Grounding

AI summary

Overview

Research area: Computer vision data synthesis for object detection, instance segmentation, and visual grounding (referring expression comprehension). The paper also introduces a new diagnostic benchmark for intra-class referring.

Technical level: Intermediate. The core idea (composing scenes from object segments) is intuitive, but the pipeline involves 3D geometric layout modeling, perspective projection, diffusion-based relighting, and evaluation across open-vocabulary benchmarks.

Scope: The paper presents SOC, a pipeline that generates 20M synthetic object segments and composes them into annotated images with pixel-perfect masks, boxes, and referring expressions, and shows these data outperform or complement much larger real and synthetic datasets.

What This Paper Is About

Building datasets for detection, segmentation, and grounding is expensive and biased: the paper notes that COCO required 2.2M worker hours to annotate only 100K images across 80 categories. Existing synthetic alternatives either render rigid simulated domains with limited object variety, or use models to pseudo-label images and inherit annotation noise. SOC's goal is a data synthesis pipeline that composes high-quality synthetic object segments into new images, so that accurate region annotations are produced by construction rather than by post-hoc labeling, and can be scaled and controlled on demand.

Key Contributions

1. SOC dataset and pipeline. The authors release a large-scale synthetic resource of 20M object segments and 2M composed images (Table 1 lists the current composed scale as 2.4M) with pixel-perfect annotations, described as the first large synthetic dataset to surpass real datasets across diverse models and tasks.
2. Broad empirical validation. SOC is shown to outperform existing synthetic pipelines (Copy-Paste, X-Paste, SynGround, SegGen) and to complement real datasets (GRIT, V3Det, LVIS, COCO) across object detection, instance segmentation, and visual grounding benchmarks.
3. A new diagnostic benchmark. The paper proposes an intra-class referring expression task with human annotations, drawn from COCO and OpenImages V7 (~100 images containing multiple instances of the same category with distinct attributes), and two metrics: Average Gap and Positive Gap Ratio.
4. Controllable data synthesis. The pipeline's controllability is used to generate targeted data for intra-class referring, which the paper shows large generic datasets (GRIT, V3Det) fail to improve.

Main Findings

• Small synthetic data beats much larger real data. With only 50K SOC images, LVIS AP rises from 20.1 to 29.8 (+9.7) and rare-class AP from 10.1 to 23.5 (+13.4), exceeding the +7.0 gain from adding 20M GRIT examples and matching the +10.5 gain from 200K human-annotated V3Det images, on MM-Grounding-DINO starting from an Object365+GoldG baseline of 1.4M examples.
• The advantage holds at 100K. The abstract reports that models trained on just 100K SOC images outperform those trained on GRIT (20M) and V3Det (200K) and on other synthetic pipelines by +24–36%, achieving +10.9 AP on LVIS and +8.4 N Acc on gRefCOCO.
• SOC scales. Going from 50K to 100K raises LVIS AP from 29.8 to 31.0 and rare AP from 23.5 to 26.3; scaling to 400K pushes LVIS AP to 31.4 (+1.6 over 100K) and OdinW-35 mAP to 22.8 (+1.8).
• SOC is complementary, not redundant. Adding 100K SOC images on top of Object365+GoldG+GRIT+V3Det (21.6M examples) raises LVIS AP from 31.9 to 33.2, rare-class AP from 23.6 to 29.8 (+6.2), and improves OdinW-35 average AP by 2.8 points over the baseline.
• Grounding improves sharply on no-target and long-description cases. On gRefCOCO, SOC-FC-50K raises no-target accuracy by +4.6 and DoD FULL mAP by +1.0; SOC-FC-100K raises no-target accuracy by +8.4 (to 97.7) and DoD FULL mAP by +3.8 (to 19.4). By contrast, V3Det adds only +0.5 gRefCOCO P@1 and no no-target improvement, and GRIT adds +1.4 DoD FULL mAP despite 400× more examples.
• Instance segmentation gains concentrate on rare classes. With APE pre-trained on LVIS, adding 50K SOC images raises AP_rare from 40.87 to 44.70 (+3.83), overall AP from 46.96 to 48.48 (+1.52), and AP_frequent by +0.31.
• Limited-data regimes benefit most. With Mask2Former-ResNet-50 on COCO, mixing SOC segments gives roughly a 3% boost on average across data scales and a +6.59 AP gain when only 1% of COCO images are used.
• Ablations isolate the important components. In zero-shot COCO instance segmentation with Mask2Former trained from scratch on 10K SOC images: 3D geometric layout augmentation gives 10.03 AP versus 9.07 (random 2D layout), 8.60 (COCO layout), and 8.82 (LayoutGPT); camera configuration augmentation adds +5.5% (10.03 to 10.58); generative harmonization is the largest factor (6.28 without it, 10.58 with inpainting and relighting, 12.79 with blending, +103.7%); real segments alone give 7.03 AP versus 12.79 for real + SOC synthetic segments (+81.9%).
• Blending preserves small-object detail. Mask-area-weighted re-blending after IC-Light harmonization gives a 2.3% AP boost on LVIS-mini-val (36.3 to 38.6).
• Targeted synthesis wins on the new diagnostic task. Adding large-scale GRIT, V3Det, or both yields negligible or negative Average Gap changes (down to 34.6–36.7); fine-tuning on 100K targeted SOC-SFC-50K + SOC-SGC-50K samples raises Average Gap by 3.1 points to 40.6 and pushes Positive Gap Ratio to 90%.

Methodology in Plain English

The pipeline builds scenes from the ground up rather than annotating finished images.

Segment creation. The authors collect over 46,000 object categories and use Qwen 2.5-32B to write text descriptions for each. Those prompts go to FLUX-1-dev to render single objects on a uniform white background at randomly sampled viewpoints, and DIS extracts each object segment. Generating objects in isolation produces cleaner mask boundaries than generating them in cluttered scenes. The result is 20M segments: 10M frequent-category segments covering 1.6K categories from LVIS, COCO, and ADE20K (200 prompts per category), and 10M general-category segments covering 40K categories from LAION, GQA, and Flickr30K (10 prompts per category), with three segments synthesized per prompt using different random seeds.

Layout with 3D geometry. To stop models from learning shortcuts like "cars are large and near the bottom of the image," depth and 3D position are sampled independently of object category. Each category gets a commonsense physical size range from Qwen2.5-32B. The pipeline samples a focal length, defines close, middle, and far depth bands (following the depth distribution observed in COCO/SA-1B: 40% close, 35% middle, 25% far), samples physical size and 3D position, and projects objects to 2D with perspective projection. Invalid placements (objects too small, too large, or almost completely occluding another, IoU ≥ 0.9) are resampled. Each image contains 5–20 segments sampled with balanced category sampling.

Harmonization. Naive pasting leaves sharp edge artifacts that segmentation models can exploit instead of learning semantics. IC-Light performs background inpainting and global relighting, but can distort small objects or shift colors. The authors re-blend original segments with the harmonized image using a per-object weight, giving smaller objects higher weight to preserve their appearance, then apply a soft matting step to smooth object boundaries.

Camera augmentation. Random scaling (factor 1.0–4.0) followed by cropping simulates zoom, and depth-of-field blur is applied using the circle-of-confusion formula with a randomly sampled focal plane and f-number between 1.4 and 16.

Annotation generation. Boxes and masks are computed by subtracting occluded pixels from each object's original mask. For grounding, QwQ-32B is prompted with each object's box, mask, category, and generation prompt to write 3–6 attribute-based and spatial-based referring expressions per type, yielding at least 9 dense expressions per image. The segments are composed into named datasets: SOC-FC and SOC-GC (frequent and general categories), SOC-LVIS-Category, SOC-SFC and SOC-SGC (single-category images with multiple attribute-varied instances), and SOC-COCO-Mix.

Why This Matters

Research impact. The paper challenges the assumption that annotation scale must come from human labor or noisy model labeling, showing that a composition-based synthetic pipeline can match or exceed orders-of-magnitude-larger real datasets, especially on rare and long-tail categories. Table 1 positions SOC as uniquely combining accurate annotation, controllability, multi-object composition, and open-vocabulary coverage at scale, while also releasing the dataset, pipeline outputs, and a new benchmark.

Real-world applications (as cited in the paper):

• Robotic perception
• Autonomous driving
• Photo editing
• Assistive technologies
• Surveillance cameras, where only a small category set is needed and annotation budgets are limited

Industry relevance. The closed-vocabulary and limited-data results point to practical value: teams with narrow category needs and small labeling budgets can augment real segments with synthetic ones, with the largest gains exactly where real data is scarcest (1% COCO data, +6.59 AP). Because annotation is generated by construction, the cost of adding categories or attributes is controllable rather than tied to human labeling hours.

Future Directions

• Closing the synthetic–real domain gap. The authors note remaining differences in surface texture, material response, and global illumination, and suggest integrating stronger diffusion models to reduce them.
• Improving 3D coherence. Objects are treated as standalone 2D segments without explicit 3D constraints, which the authors say limits tasks such as depth estimation or novel-view synthesis; 3D geometry priors or 3D assets could be added.
• Modeling object interactions. The pipeline treats objects independently and does not model inter-object relations, which could matter for relation recognition tasks.
• Panoptic coverage. SOC produces object-level "things" segments but does not annotate amorphous "stuff" regions or enforce all-pixel partitioning; the authors propose integrating off-the-shelf stuff predictors or extending to panoptic labels.

Target Audience

Researchers and engineers working on data-centric computer vision, particularly those building or fine-tuning detection, instance segmentation, and visual grounding models, and those studying synthetic data generation, copy-paste augmentation, and image harmonization. It is also relevant to practitioners facing long-tail category imbalance or very small annotation budgets, and to dataset builders interested in controllable, benchmark-targeted data synthesis. The intra-class referring benchmark adds value for researchers interested in fine-grained attribute discrimination and diagnosing shortcut behavior in vision-language grounding models.

Authors’ abstract

Visual grouping -- operationalized through tasks such as instance segmentation, visual grounding, and object detection -- enables applications ranging from robotic perception to photo editing. These fundamental problems in computer vision are powered by large-scale, painstakingly annotated datasets. Despite their impact, these datasets are costly to build, biased in coverage, and difficult to scale. Synthetic datasets offer a promising alternative but struggle with flexibility, accuracy, and compositional diversity. We introduce Synthetic Object Compositions (SOC), an accurate and scalable data synthesis pipeline via a novel object-centric composition strategy. It composes high-quality synthetic object segments into new images using 3D geometric layout augmentation and camera configuration augmentation with generative harmonization and mask-area-weighted blending, yielding accurate and diverse masks, boxes, and referring expressions. Models trained on just 100K of our synthetic images outperform those trained on larger real datasets (GRIT 20M, V3Det 200K) and synthetic pipelines (Copy-Paste, X-Paste, SynGround, SegGen) by +24-36% -- achieving +10.9 AP on LVIS and +8.4 NAcc on gRefCOCO. Beyond the general open-vocabulary setup, SOC also enables controllable dataset construction for different use cases and boosts performance in both low-data and closed-vocabulary scenarios. Augmenting LVIS and COCO with synthetic object segments delivers strong performance across different real-data scales and yields even greater improvements under extremely limited real-data conditions, including +6.59 AP on a 1% COCO data setup. Furthermore, this controllability enables targeted data generation for intra-class referring, a diagnostic grounding task we propose that requires fine-grained attribute discrimination.