ReCon: Region-Controllable Data Augmentation with Rectification and Alignment for Object Detection

AI summary

Overview

Research area: Computer vision — generative data augmentation for object detection, combining controllable diffusion models (Stable Diffusion, ControlNet) with zero-shot perception models (Grounded-SAM). Technical level: Intermediate. The paper assumes familiarity with diffusion sampling, cross-attention, and standard detection benchmarks, but the core ideas are explained conceptually. Scope: A training-free, plug-and-play framework that adds region-wise rectification and region-aligned cross-attention to existing structure-controllable generative models in order to produce better synthetic training data for object detectors.

What This Paper Is About

Training robust object detectors requires large, accurately annotated datasets, and creating those annotations is expensive — the paper cites that annotating a single Cityscapes image can take up to 60 minutes. Generative models can synthesize training images, but existing approaches either need complex post-processing or expensive fine-tuning on massive datasets, and they still produce content–position mismatches (objects in the wrong place or of the wrong count) and semantic leakage (regions taking on content from unrelated text prompts). ReCon addresses both failure modes directly inside the diffusion sampling loop, without any additional training.

Key Contributions

1. ReCon, a region-controllable data augmentation method that enhances the regional control of existing structure-controllable generative models without requiring any additional training, and is plug-and-play with those models.
2. Region-Guided Rectification (RGR): a strategy that detects mis-generated regions by comparing sampled content against ground-truth annotations with an off-the-shelf grounding model, then replaces those regions with noisy real data sampled from the original image.
3. Region-Aligned Cross-Attention (RACA): a mechanism that enforces spatial–semantic alignment between region-specific visual tokens and their corresponding textual cues, mitigating semantic leakage across categories.
4. Extensive experiments showing that ReCon improves both the quality (FID) and the trainability (mAP) of generated data across multiple datasets, backbone architectures, and data scales, outperforming both traditional augmentation and current generative approaches.

Main Findings

• COCO main results: Adding ReCon to ControlNet raises mAP to 35.5 (AP50 56.2, AP75 38.4, APm 39.0, APl 46.0), compared with 34.9 for ControlNet alone and 34.5 for the real-data-only baseline. The paper states this surpasses GeoDiffusion's 34.8 and is competitive with models specifically fine-tuned on COCO. DetDiffusion is listed at 35.4, Instance Diffusion at 35.0, GLIGEN at 34.6, ReCo at 33.6, Layout Diffusion at 34.0, Background-inpainting at 35.1, and ControlNet-XS at 35.1.
• Plug-and-play gains on fine-tuned models: Instance Diffusion + ReCon reaches 35.6 mAP; GLIGEN + ReCon is listed in the results table at 35.3 mAP (the text describes GLIGEN's improvement as going from 34.6 to 35.5).
• Data-scarce regimes: On randomly sampled 1%, 5%, and 10% subsets of COCO (each doubled by augmentation), ReCon achieves mAP of 3.9, 16.7, and 21.7 respectively, versus 0.3, 13.0, and 18.5 with real data only, and 2.5, 15.9, and 21.2 with ControlNet. With only 10% of the data, mAP rises from 18.5 to 21.7. ReCon combined with RandAugment reaches 4.2, 17.1, and 22.0, showing compatibility with standard augmentation pipelines.
• Few-shot setting: In a 30-shot training setting on YOLOX-S with COCO, ReCon increases mAP from 5.4 to 6.7 and AP50 from 10.3 to 12.3, following the few-shot split protocol of prior work.
• PASCAL VOC: ReCon reaches 78.5 mAP, compared with 77.1 for the real-only baseline, 76.2 for simply duplicating the dataset (which the paper attributes to overfitting), 77.7 for RandAugment, and 77.8 for ControlNet. Training combines VOC 2007 and VOC 2012 training sets, with evaluation on the VOC 2007 test set (4,952 images).
• Data scaling behavior: Repeating real training data improves results up to 3×, with mAP increasing from 13.0 to 17.1 on the 5% subset and from 18.5 to 21.1 on the 10% subset, but further duplication (5×, 7×) saturates and degrades. ReCon-generated data keeps improving without overfitting; the abstract states that tripling the dataset with ReCon outperforms a sevenfold increase achieved by the baseline.
• Ablation — each component helps: Baseline (no RGR, no RACA) gives FID 13.82 and 34.9 mAP; adding RGR gives FID 13.21 and 35.3 mAP; adding both RGR and RACA gives FID 12.85 and 35.5 mAP.
• Perception target matters: Using x_t as the perception target yields 35.0 mAP, x_{0|t} yields 35.3, and x_{0|(t−N)} obtained via cache-based fast sampling yields the best result at 35.5 mAP.
• Detector generality: On DEIM-D-FINE-N, ReCon reaches 39.8 mAP, 56.6 AP50, 42.5 AP75, and 61.0 mAR, versus 38.5/55.2/41.5/60.4 for real data only and 39.1/55.8/42.1/60.6 for ControlNet. Controls are also evaluated with RetinaNet, ATSS, FCOS, and YOLO-X.
• Stronger perception helps: Using Grounded-SAM with a Swin-Base backbone yields 35.6 mAP versus 35.5 with Swin-Tiny, indicating the method benefits from stronger foundation models.
• Qualitative fixes: The paper reports that ReCon removes an extraneous zebra produced by GLIGEN outside the target bounding box, eliminates a superfluous sheep outside the region of interest, and restores a person that ControlNet fails to generate.

Methodology in Plain English

The researchers do not train a new generative model. They start from a frozen Stable Diffusion v1.5 with a 25-step DDIM sampler and an edge-conditioned ControlNet, plus an off-the-shelf grounding model (Grounded-SAM). Two additions are applied during the sampling loop:

Region-Guided Rectification. Detecting objects in the noisy intermediate latent is impractical, so the method uses a cache-based diffusion acceleration technique to jump ahead N steps (N = 5 by default), predicts a clean image from the advanced latent, and runs a detector on it. Detected boxes are matched against the ground-truth boxes and labels by IoU (threshold 0.5) to find false positives and false negatives. These produce a binary mask M, and the corresponding regions of the latent are replaced with a noisy version of the original real image, while everything else is left untouched. This relies on the "overridability" property of diffusion sampling — intermediate regions can be swapped with content from the same distribution without breaking inference. Rectification is applied at T_r = 4 timesteps, corresponding to the early (0.75T), middle (0.50T), latter (0.25T), and final (0.10T) stages of diffusion, so that layout errors, semantic errors, and low-quality regions are corrected at the stages where each is most fixable. An "exclusive dilation" step (kernel = 7) prevents mask overlap, and false-positive regions are suppressed in the control map.

Region-Aligned Cross-Attention. Because encoding all prompt tokens together causes interference between categories, the method encodes C separate text features for C target categories using prompts in the format [CLASS], plus a global context description for the background. Region-specific visual features are cropped from the latent using the predefined regions, cross-attended only with their own category's text features, and then concatenated back. This ensures region-specific text influences only its own region. The paper notes that a masked-attention alternative still suffers from semantic leakage due to entangled text encodings, and that Instance Diffusion's instance-masked attention requires extra modules and retraining — ReCon needs neither.

Evaluation setup. Training and evaluation use the MMDetection framework. The default detector is Faster R-CNN with an R-50-FPN backbone trained for six epochs. Images containing 3 to 8 objects were selected for generation, producing a dataset of 47,200 images with 227,406 objects. Metrics are mAP, mAR, and FID.

Why This Matters

The work shows that meaningful gains in synthetic training data can come from smarter inference-time control rather than from bigger fine-tuning runs — a practical result for anyone who cannot afford to retrain a diffusion model. Because the method requires no training and integrates with existing structure-controllable generators, it lowers the barrier to producing annotation-consistent synthetic data.

Real-world applications:

• Autonomous driving and robotics perception, where collecting and labelling sensor imagery is slow and expensive.
• Industrial inspection and surveillance, where rare or dangerous object categories are hard to capture in sufficient quantity.
• Medical and scientific imaging, where annotated expert data is scarce and privacy constraints limit data sharing.
• Any organization with small, domain-specific datasets that wants to expand training data without a large generative-model training budget.

Industry relevance: The affiliation list (Tsinghua University, Li Auto Inc., BNRist) points to a direct automotive-perception use case, where large-scale annotated driving data is the primary bottleneck. The data-scarce results (1%, 5%, 10% of COCO) and the few-shot results are the numbers most relevant to teams with limited labelled data.

Future Directions

• Efficiency at scale: The paper's own limitations section states that ReCon may increase computation time as data volume grows and that the extra perception model raises development costs. It suggests integrating a fast sampler and a lightweight perception model as a path forward.
• Better disentangled text conditioning: The authors observe that semantic leakage persists in masked-attention alternatives because text features from different categories are not disentangled during encoding, suggesting improved text encoding as an open problem.
• Stronger foundation models: Results with Swin-Tiny versus Swin-Base perception backbones suggest gains scale with perception quality, raising the question of how far this transfers to future grounding and detection models.
• Bias and misuse mitigation: The societal-impacts section notes that pre-trained generative models may inherit social biases, and that region-wise rectification and alignment may help reduce them; the paper frames responsible-regulation and best-practice questions as unresolved community issues.

Target Audience

Researchers and practitioners working on generative data augmentation, controllable diffusion models, and object detection or instance segmentation — particularly those operating in low-data or few-shot regimes where fine-tuning a generative model is impractical. It is also relevant to engineers in automotive and industrial perception who need to expand annotated datasets at low cost, and to readers interested in inference-time control mechanisms for diffusion models.

Authors’ abstract

The scale and quality of datasets are crucial for training robust perception models. However, obtaining large-scale annotated data is both costly and time-consuming. Generative models have emerged as a powerful tool for data augmentation by synthesizing samples that adhere to desired distributions. However, current generative approaches often rely on complex post-processing or extensive fine-tuning on massive datasets to achieve satisfactory results, and they remain prone to content-position mismatches and semantic leakage. To overcome these limitations, we introduce ReCon, a novel augmentation framework that enhances the capacity of structure-controllable generative models for object detection. ReCon integrates region-guided rectification into the diffusion sampling process, using feedback from a pre-trained perception model to rectify misgenerated regions within diffusion sampling process. We further propose region-aligned cross-attention to enforce spatial-semantic alignment between image regions and their textual cues, thereby improving both semantic consistency and overall image fidelity. Extensive experiments demonstrate that ReCon substantially improve the quality and trainability of generated data, achieving consistent performance gains across various datasets, backbone architectures, and data scales. Our code is available at https://github.com/haoweiz23/ReCon .