Diffuse the object, keep its label: curating detector training data from a few unlabeled photographs via VLM-built 3D vegetation scenes

AI summary

Overview

Research area: Computer vision, specifically synthetic training-data curation for object detection, with an application to humanitarian demining (surface-laid landmine detection in vegetated terrain).

Technical level: Advanced. The paper assumes familiarity with diffusion models (Stable Diffusion XL, SDEdit), LoRA fine-tuning, 3D rendering (Blender), VLM/LLM scene generation, and standard detection metrics (mF1, mAP50).

One-sentence scope: The paper presents a preliminary study of a pipeline that turns a handful of unlabeled photographs of a deployment site plus a small library of 3D object meshes into a labeled detector training set, and compares it against conventional reuse of labels collected at a different site.

What This Paper Is About

Detectors for small or camouflaged objects hidden in vegetation generalize poorly across sites, and labeled images of such objects are scarce, expensive to collect, and sometimes physically hazardous to acquire. Instead of reusing labels from a previously surveyed site, the authors generate labeled training images from a few unlabeled photographs of the deployment site itself: a vision-language model builds a coarse 3D vegetation scene, 3D object meshes are placed inside it, and a LoRA-adapted diffusion pass re-textures the render while a graded mask-lock controls how much diffusion is allowed to touch the object. The paper's central question is which curation choice actually governs cross-site generalization: how much data is generated, or how the object itself is processed.

Key Contributions

1. A render-and-refine pipeline for curating detector training data in vegetated terrain. A VLM generates a coarse Blender scene from one reference photograph, and a LoRA trained on unlabeled crops of the same site adapts the diffusion prior. A single graded mask-lock parameter α ∈ [0, 1] acts as a curation dial setting how much diffusion may touch the object, while mask-locked compositing preserves the placed 3D meshes and their labels.

2. Labels derived from 3D scene geometry rather than manual annotation. Placing object meshes in the VLM-built scene yields bounding boxes, segmentation masks, and per-instance occlusion directly from the scene graph, including a visible-surface (post-occlusion) mask so that occluding vegetation stays in front of the object.

3. A preliminary cross-site generalization study on surface-laid landmine detection. The synthetic-data detector is compared against the same detector trained on a far larger labeled dataset of real images, evaluated once on the SULAND USA split (4,436 frames) that no model sees during training.

4. The finding that the mask-lock grade dominated the data-budget axes in their runs, shifting generalization more than any photograph, patch, or image-count choice they swept.

Main Findings

• Synthetic data matched or exceeded the real labeled baseline on cross-site generalization. The reference synthetic configuration (M2, α = 0.10) reached mF1 = 0.584 ± 0.025 against 0.393 ± 0.021 for the real labeled baseline R1 (YOLOv11-L trained on the ITA split). The authors state the difference is roughly 5.8 combined seed standard deviations, and describe this as preliminary evidence rather than a general claim.

• The mask-lock grade was the most influential curation choice, with an inverted-U shape across the four sampled values. M0 (hard lock, α = 0) scored mF1 0.441 ± 0.051; M1 (α = 0.05) scored 0.503 ± 0.060; M2 (α = 0.10) scored 0.584 ± 0.025; M3 (unlocked, α = 1) scored 0.466 ± 0.030. The paper notes that a denser grid would be needed to trace the curve between these four values.

• Lightly diffusing the object improved minority-class recall; fully protecting it did not. PMA-2 recall roughly doubled from 0.20 (M0) to 0.42 (M2), while PFM-1 recall stayed flat at 0.57 to 0.58 across M0–M2 (the table bolds M0's 0.579 and M2's 0.577 together within a Δ < 0.01 tie band that should not be read as ordered). The authors speculate that the PFM-1's distinctive silhouette already suffices for detection, whereas the low-contrast PMA-2 disc depends on surface appearance.

• Fully unlocking the mask dissolved the object. At α = 1 the diffusion absorbed objects into the background prior, most completely where occlusion was heaviest, and the label no longer marks a landmine. M3 also served as an unprotected-diffusion control analogous to full-regeneration pipelines.

• Raw 3D renders without diffusion were far worse and far more unstable. R2 (raw CG, no diffusion) reached only mF1 0.194 ± 0.111, with seed-to-seed variance an order of magnitude larger than at any α > 0.

• Budget axes were largely flat within error. Sweeping reference photographs (1, 5, 10), patches per photograph (2, 5, 14), and generated image count (100, 250, 500, 1,000), the photograph and patch axes showed flat results within error. A single reference photograph at the reference lock reached 0.562 ± 0.011 mF1, within one standard deviation of the ten-photograph configuration, and was described as the most seed-stable cell in the study.

• The image-count axis was the exception but remained secondary. At the reference lock the curve rose by approximately 0.07 mF1 from 100 to 1,000 images before flattening, still smaller than the +0.14 shift attributed to the mask-lock axis itself.

• The sensor-noise pass left mF1 unchanged within seed variance at both lock settings (at α = 0: 0.441 vs. 0.448; at α = 0.10: 0.584 ± 0.025 vs. 0.571 ± 0.048), but was kept for photometric fidelity of the generated images.

• In-domain accuracy did not predict cross-site performance. Every synthetic configuration reached mAP50 ≥ 0.99 on a held-out validation split of the generated images (except unlocked M3, which plateaued at 0.84 to 0.87, and one 100-image seed at 0.91), yet real-domain mF1 spanned 0.05 to 0.62 across configurations and seeds.

• The graded lock also stabilized results. Moving any configuration from the hard lock to the reference lock roughly halved its seed variance (for example, 0.070 → 0.011 for the one-photograph cell).

• The comparison carries an access asymmetry, acknowledged by the authors. R1 sees no data from the test site, whereas the pipeline consumes a handful of unlabeled test-site photographs, so the comparison is source-site labels alone versus unsupervised adaptation to the target site. Raw frame counts also overstate the supervision gap: of the 22,756 real training frames, 5,234 contain landmine instances, against roughly 900 of the 1,000 synthetic images, an effective ratio closer to 6:1 than to 23:1.

Methodology in Plain English

The pipeline has five stages, and the real-data budget is deliberately tiny.

Stage 1 — VLM scene primitives. One reference photograph is given to a commercial vision-language model, which directly emits Python code that instantiates the terrain in Blender as simple primitives: ground palette, vegetation cards, and clutter such as rocks, sticks, and bare soil. No pre-owned asset library is involved. The camera is sampled near the initial camera pose estimated by the VLM. A scripted spawner then places 3D meshes of the target ordnance — several variants of the PFM-1 butterfly landmine and the PMA-2 anti-personnel landmine — with randomized yaw, position, and partial burial, under a balanced class draw. Because everything is generated in 3D, bounding boxes, instance masks, and per-instance occlusion come out of the scene graph for free. In their experiments, 1,000 scenes were rendered once and reused for every ablation so that scene variance would not be a confound.

Stage 2 — Adaptation pre-phase. This runs once per site. Random square crops are taken from the handful of photographs, with no bounding boxes, masks, or class labels of any kind, and a LoRA adapter is fine-tuned on a frozen Stable Diffusion XL backbone using a fixed short caption describing the target terrain. The adapter is intentionally small so it captures the terrain's surface statistics rather than individual images.

Stage 3 — Mask-locked appearance transfer. Each crude render is passed through SDEdit, which re-noises it into the diffusion latent space and denoises it under the LoRA-conditioned prior, transferring surface appearance while the 3D layout constrains structure. To keep labels valid, the rendered object pixels are pasted back through the visible-surface mask exported by Blender. The graded parameter α controls how much diffusion may touch the object: at α = 0 the object is pixel-protected, at α = 1 the diffusion output is kept everywhere, and intermediate values feather the mask so a thin, controllable share of the object's appearance is inherited from diffusion while geometry and label stay fixed.

Stage 4 — Sensor-noise pass. The composited image goes through a light camera model: the sensor-noise level of the reference photographs is estimated and noise is injected to approximate the measured statistic, followed by a fixed compression pass.

Stage 5 — Detector training. An off-the-shelf YOLOv11-L, initialized from its COCO-pretrained checkpoint, is trained on the generated dataset following the SULAND benchmark recipe one-to-one, so the synthetic-versus-real comparison isolates the training data as the only variable.

Evaluation uses the SULAND benchmark, which contains ground-level RGB images of PFM-1 and PMA-2 surface-laid landmines across ITA and USA sites. The real labeled baseline (R1) is trained on the ITA split (22,756 frames) and everything is evaluated once on the USA split (4,436 frames). Metrics are macro-F1 over the two landmine classes, per-class recall, and mAP50, reported as mean ± standard deviation over five training seeds per configuration. Generation constants include Blender Cycles rendering at 960×960 px with 48 samples, 1–3 landmines per scene with a stratified 1:1 class draw, 10% landmine-free negative frames, SDEdit strength 0.42 with 35 steps and guidance 6.0, LoRA rank 12 with 1,000 steps, and a sensor-noise injection of σ = 15.8 against a measured target of σ = 12.3 on the 0–255 scale. All rendering, LoRA fine-tuning, diffusion, and detector training ran on a single NVIDIA RTX 5090 (32 GB).

Why This Matters

Impact on research. The paper challenges a common assumption in render-then-refine pipelines: that the diffusion model can plausibly re-texture the target object. That assumption holds for mushrooms, vehicles, and human bodies, which are well represented in generative training corpora, but becomes fragile for objects that are rare or absent from those corpora. The authors' answer is to preserve the object explicitly rather than trusting the generative prior, and their results suggest that a small, curated synthetic set can be competitive for cross-site generalization under this recipe — which the authors explicitly state should not be read as a general claim about synthetic data superiority. The paper also reports that in-domain validation was uninformative about generalization in their runs, which is a caution for anyone selecting synthetic data by in-domain metrics.

Real-world applications (as discussed or implied by the paper's framing):

• Humanitarian landmine clearance, where annotated data is scarce, expensive, and hazardous to collect, and where occlusion by vegetation is described as a primary driver of detection failure.
• Invasive species monitoring in natural vegetation.
• Precision agriculture, for example detecting crop and weed instances in the field.
• Wilderness search and rescue, where small objects must be found against cluttered natural backgrounds.

Industry relevance. The pipeline targets a deployment setting where the operator already knows the terrain type and the family of ordnance likely to be encountered from non-technical surveys and historical clearance records. If the findings hold, the per-site data question shifts from collecting and labeling new frames to photographing the site and regenerating the training set, which is a much cheaper operation and can be repeated for each new site. The paper also notes that generative augmentation approaches remain effective under model quantization constraints on edge hardware, and the whole pipeline was run on a single consumer-grade GPU.

Future Directions

• Richer 3D curation. The VLM currently emits only minimal scene primitives. The authors see substantial work remaining on capturing a site's initial 3D distribution inside the simulator, and suggest that supporting the capture with range sensing (for example a LiDAR back-support) could standardize scene geometry and remove dependence on any particular VLM, which they state they cannot meaningfully ablate here.

• Better blending. The mask-lock stage currently paints the rendered object back through a binary visible-surface mask. Soft harmonization and learned compositing could remove the small residual seam that remains at low α and might absorb the appearance gap the graded lock currently transfers.

• Higher-fidelity meshes. Object realism is upper-bounded by the mesh textures, especially for weathering, dirt, and paint wear absent from their meshes. Capturing physically-based materials from inert ordnance would move the object closer to the target distribution without further diffusion pressure.

• Broader evaluation. A second detector family, additional terrain pairs (forest, sand, mixed), a denser mask-lock grid, and a cheap and calibrated predictor of generalization are named as natural next steps. The paper further notes that real sites may mix vegetation with asphalt or bare regions, which would call for semantic-region analysis and eventually a region-conditioned diffusion pass, and that the camera distance to the ground is only approximately recovered while object scale matters for the graded lock.

Target Audience

Researchers and practitioners working on synthetic training-data generation, domain adaptation, and detection under domain shift, particularly those in data-scarce or safety-critical domains. It is also relevant to engineers building per-site detector training pipelines for robotics and remote sensing who need to decide where to spend annotation effort, and to readers interested in how generative models should be constrained when the target object has no meaningful generative prior. The paper is written at an advanced technical level and is explicitly framed as a preliminary study covering one target family, one terrain type, one evaluation split, and one detector architecture, with five stated limitations bounding its findings.

Authors’ abstract

Labeled images of small objects hidden in vegetation are scarce, and detectors trained on them generalize poorly across sites. Rather than reusing labels collected at another site, we synthesize labeled training images from a handful of unlabeled photographs of the deployment site itself. A vision--language model generates a coarse 3D vegetation scene from one photograph; placing 3D object meshes in the scene yields bounding boxes, segmentation masks, and per-instance occlusion directly from the scene geometry, without manual annotation. A lightweight adapter fine-tuned on the photographs conditions a diffusion pass that re-textures the renders, and a graded mask-lock sets how much diffusion may touch the object itself. In our runs this grade was the most influential curation choice: lightly diffusing the object improves minority-class recall over fully protecting its pixels, while unrestricted diffusion dissolves it. Trained on these images, a standard detector matched or exceeded its counterpart trained on a larger labeled dataset of real images from a different site, consistently across seeds on a humanitarian-demining benchmark; the comparison is thus unsupervised site adaptation from a handful of photographs against conventional cross-site label reuse. In our ablations the gains were largely insensitive to the photograph and crop budgets, and in-domain accuracy did not predict cross-site performance.