Action-guided generation of 3D functionality segmentation data

AI summary

Overview

Research area: Computer vision and embodied AI, specifically 3D functionality segmentation, 3D scene synthesis, and synthetic training-data generation for vision-language models (VLMs).

Technical level: Advanced. The paper assumes familiarity with point clouds, part-level 3D segmentation masks, vision-language models, and retrieval-based scene synthesis.

Scope: The paper introduces SynthFun3D, a pipeline that turns a free-form action description into a synthetic 3D scene with automatically generated ground-truth masks for the functional element needed to perform that action, and shows that training on this data improves 3D functionality segmentation on real-world data.

What This Paper Is About

3D functionality segmentation asks a model to find the specific object part required to carry out a natural-language action in a 3D scene, such as the handle you would grab to "open the second drawer of the cabinet near the bed." Progress on this task is limited because the only public real-world benchmark, SceneFun3D, contains just 230 indoor scenes and 3041 functional masks, and collecting more data requires expensive sensors and long annotation procedures. The paper's goal is to generate such training data automatically and cheaply, directly from action descriptions, without any manual labeling.

Key Contributions

1. SynthFun3D, described as the first method capable of generating synthetic visual data paired with functional segmentation masks directly from high-level action descriptions.
2. A metadata-driven 3D object retrieval mechanism that enforces strict alignment between the input prompt and the functional elements present on retrieved objects, using PartNet-Mobility's part masks and semantic labels.
3. An automatic mask generation and material augmentation pipeline that produces pixel-accurate part-level masks across multiple rendered views, plus 200 randomly generated materials for scene variety.
4. Empirical validation showing that augmenting real training data with the generated synthetic data improves 3D functionality segmentation on the SceneFun3D benchmark by +2.2 mAP, +6.3 mAR, and +5.7 mIoU over real-only training.

Main Findings

• Synthetic-only training is comparable to real-only training: Training Gemma3-4B on SynthFun3D data alone (split S) reached 1.23 mIoU versus 1.18 mIoU for SceneFun3D real data alone (split R). The authors interpret this as evidence that correct spatial relationships matter more than visual realism for this task.
• Point accuracy nearly doubles with synthetic data: P-acc was 0.170 for real-only training versus 0.118 for synthetic-only training, and adding material augmentation (S + A) raised it to 0.176 with 2.25 mIoU.
• Mixing real and synthetic data gives the largest gains: R + S reached 4.40 mIoU with 28.1k training images, compared to 2.25 mIoU with 30.3k images for S + A.
• Best result comes from combining all three splits: R + S + A, totaling 46k image frames, achieved 2.56 mAP, 5.17 AP50, 12.81 AP25, 26.54 mAR, 33.26 AR50, 39.10 AR25, 6.91 mIoU, and 0.384 P-acc.
• Zero-shot pointing is nearly impossible for the base model: Untrained Gemma3-4B scored 0.07 mIoU and 0.003 P-acc on SceneFun3D validation.
• There is a large headroom: Replacing VLM pointing with ground-truth points in the Fun3DU pipeline gives an upper bound of 17.17 mAP and 29.26 mIoU.
• Class-level behavior varies: Improvements over real-only training were largest on Furniture, Door, and Outlet categories; Outlets improved roughly ten-fold in mIoU (0.61 vs 6.5). Light switches remained difficult at 0.79 mIoU due to their small size. Windows and Appliances improved less, which the authors attribute to limited asset coverage and overfitting; windows appeared in 2.8% of generated data versus 8.1% of the SceneFun3D training split.
• Generation is far cheaper than annotation: For SynthFun3D the authors estimate an upper bound cost of 1 USD per scene, based on 0.34 USD per scene for LLM inference (12 scenes per hour on an NVIDIA A100 with 80 Gb at about 4.10 USD per hour), compared to the SceneFun3D authors' estimate of 25k USD for 230 annotated scenes.
• Near-perfect pointing does not guarantee perfect masks: The paper notes that segmentation failures can still occur because of limitations of the SegmentAnything model used downstream.

Methodology in Plain English

The pipeline takes an action description such as "open the top left drawer of the cabinet" and works in six stages.

1. Understand the action. An LLM (GPT-OSS-20B served with Ollama) reads the action description, identifies the target object and the functional part to be segmented, and produces a room layout description with spatial constraints, adding plausible extra objects consistent with the inferred room type.
2. Retrieve the scene objects. Non-target objects are retrieved from Objaverse (818k objects, meshes only) using text-text similarity with Sequence-Bert ensembled with text-image similarity from CLIP.
3. Retrieve the target object and its mask. Target objects come from PartNet-Mobility (2k articulated object models across 46 household categories, with part masks and articulation parameters), using PerceptionEncoder instead of CLIP. Candidates above a similarity threshold are kept rather than a single best match. The LLM then filters them: it names the required functional element and states a numeric requirement on how many of them the object must have (for example, handle = 1 for a single-door fridge, handle >= 3 for "the third drawer"). Finally, part masks are reduced to 2D centroids (X and Y normalized, Z discarded), label names are enriched with hierarchy metadata to produce names like "door handle," and the LLM judges whether the arrangement of parts matches the prompt's spatial constraints, outputting the correct part mask ID.
4. Place the objects. Layout constraints are translated into clauses with fixed spatial relations, and a Depth-First-Search algorithm finds placements satisfying all constraints; one random valid solution is chosen.
5. Render video. Camera trajectories orbit the target object with the functional element as the focal point, producing RGB frames plus pixel-accurate ground-truth masks obtained by simply rendering the retrieved part mask. Frames are sampled every three frames for the downstream training data.
6. Augment. For each original video, a second version is rendered with walls and target-object materials swapped for one of 200 randomly generated materials (for example Metal, Matte, Plastic, Glass), which preserves mask validity and scales data at negligible cost.

For validation, the resulting frames are converted into VLM conversations and used to fine-tune Gemma3-4B with LoRA to point at functional elements. That model replaces Molmo in the Fun3DU pipeline, which parses the action, segments the target object in all views, selects the best views, points at the functional element, prompts SAM for 2D masks, and lifts those masks onto the 3D point cloud. Evaluation is on the SceneFun3D validation split, reported as AP25, AP50, mAP, AR25, AR50, mAR, mIoU, and P-acc (a point counts as correct if it is within 50 pixels of ground truth).

Why This Matters

Impact on research. The work targets the data bottleneck that has constrained 3D functionality segmentation, where SceneFun3D has been the only public real-world benchmark. It shows that prompt-conditioned synthetic data, generated without any manual annotation, is a usable and complementary supervision signal for training pointing-capable VLMs, and it introduces an LLM-driven retrieval strategy over part-level asset metadata that goes beyond object-level scene synthesis methods such as Holodeck, which retrieve objects only.

Real-world applications:

• Home and service robots that must follow instructions like "open the second drawer of the cabinet near the bed" and need to know which part to manipulate.
• Assistive and augmented-reality systems that highlight the correct control, switch, or handle for a user in a physical environment.
• Simulation-to-policy training for embodied agents, where cheap task-specific 3D data can be generated on demand for arbitrary actions.
• Scaling data for interaction-aware perception models without laser scanners or paid annotation of small elements such as knobs, handles, and light switches.

Industry relevance. The pipeline is training-free with respect to model weights (it relies on existing LLMs, CLIP, PerceptionEncoder, and asset repositories), runs on a single NVIDIA A100 with 80 Gb of memory, and is estimated to cost at most 1 USD per scene against 25k USD for 230 annotated real scenes. That cost profile makes task-specific data generation practical for teams that cannot fund large-scale 3D annotation.

Future Directions

• Addressing the limitations that arise from the size and coverage of the available 3D asset repositories, Objaverse and PartNet-Mobility.
• Generating more complex 3D scenes in order to provide data for methods that predict functionality segmentation masks directly in 3D.
• Exploring additional techniques for data variety, including augmented prompt generation and appearance augmentations applied specifically to target objects.
• Improving reliability in the pipeline's weak spots, such as window placement failures in the layout strategy and downstream segmentation errors caused by the SegmentAnything model even when pointing is nearly correct.

Target Audience

Researchers and practitioners working on embodied AI, 3D scene understanding, robotic manipulation, and vision-language models, particularly those interested in synthetic data generation, retrieval-based 3D scene synthesis, or training-free data pipelines. It is also relevant to engineers who need cheap, task-specific training data for interaction-aware perception and to readers tracking how LLMs are being used to orchestrate 3D asset retrieval and layout.

Authors’ abstract

3D functionality segmentation aims to identify the interactive element in a 3D scene required to perform an action described in free-form language (e.g., the handle to ``Open the second drawer of the cabinet near the bed''). Progress has been constrained by the scarcity of annotated real-world data, as collecting and labeling fine-grained 3D masks is prohibitively expensive. To address this limitation, we introduce SynthFun3D, the first method for generating 3D functionality segmentation data directly from action descriptions. Given an action description, SynthFun3D constructs a plausible 3D scene by retrieving objects with part-level annotations from a large-scale asset repository and arranging them under spatial and semantic constraints. SynthFun3D renders multi-view images and automatically identifies the target functional element, producing precise ground-truth masks without manual annotation. We demonstrate the effectiveness of the generated data by training a VLM-based 3D functionality segmentation model. Augmenting real-world data with our synthetic data consistently improves performance, with gains of +2.2 mAP, +6.3 mAR, and +5.7 mIoU over real-only training. This shows that action-guided synthetic data generation provides a scalable and effective complement to manual annotation for 3D functionality understanding. Project page: tev-fbk.github.io/synthfun3d.