ElasticFit: Fit-Aware 3D Object Insertion via VLM Reasoning and Generative Adaptation

AI summary

Overview

Research area: 3D computer vision and graphics — language-guided object insertion into pre-existing 3D scenes, combining Vision-Language Models (VLMs), generative asset creation, and geometry- and physics-aware optimization.

Technical level: Advanced. The paper assumes familiarity with VLMs, text-to-image diffusion models, image-to-3D reconstruction, mesh-level collision/support objectives, and rigid-body simulation.

Scope: ElasticFit is a three-stage framework that converts a natural-language instruction plus rendered scene observations into structured 3D fitting constraints, generates a scene-conditioned object, and adapts it to local free space through rigid, uniform, or elastic fitting.

What This Paper Is About

Inserting an object into an existing 3D scene is not just a matter of picking a location — the object also has to fit the local geometry while remaining semantically appropriate and physically plausible. Existing VLM and generative methods mainly decide where to place a fixed, already-available asset, and are limited to pose adjustment or global scaling rather than changing the object's shape. ElasticFit addresses this by treating insertion as constrained fitting, in which the inserted object itself is generated and then adapted to the target region.

Key Contributions

1. Fit-aware object insertion formulation: The authors formulate insertion into pre-existing 3D scenes as fit-aware adaptation, jointly considering semantic intent, local geometric compatibility, and object-shape adaptation.
2. Scene-grounded fitting and adaptation representation: They introduce a structured grounding representation G that converts VLM-inferred cues into explicit 3D constraints (support point, ghost-box center and dimensions, orientation, deformation mode, adaptation scale vector, and semantic object description) for object generation and for rigid, uniform, or elastic fitting.
3. A three-stage pipeline: Semantic-Spatial Grounding (Stage 1), Scene-Conditioned Object Prior (Stage 2), and Geometry- and Physics-Aware Fitting (Stage 3), connected through the shared representation G.
4. Two task sets and evaluation: They evaluate on a 60-task Common Placement Set (S001–S060) covering common spatial relations, and a 60-task Fit-Critical Set (F001–F060) split into pose-and-contact tasks and gap-fitting tasks, with cross-dataset validation on ReplicaCAD and Replica.

Main Findings

• Baseline comparison (Common Placement Set): ElasticFit reaches 69.7% Spatial Relation Success Rate (SRS), 91.7% Collision-Free Rate (CFR), 91.7% Support Success Rate (SupR), and a Contextual Plausibility Score (CPS) of 4.18. This improves SRS from the strongest baseline's 50.8% to 69.7% and SupR from the best baseline's 48.3% to 91.7%.
• Baselines: LayoutGPT scores 50.8% SRS, 73.3% CFR, 48.3% SupR, 3.72 CPS. Holodeck scores 41.5% SRS, 95.0% CFR, 41.7% SupR, 3.63 CPS. LayoutVLM scores 49.0% SRS, 93.0% CFR, 43.9% SupR, 3.82 CPS.
• Collision trade-off: Holodeck and LayoutVLM obtain marginally higher CFR (95.0% and 93.0%) than ElasticFit (91.7%), which the authors attribute to a conservative strategy of placing objects far from cluttered geometry, producing lower SRS and SupR.
• Cross-dataset robustness: On a matched subset of the Common Placement Set, ElasticFit achieves 70.4% SRS, 91.4% CFR, 92.7% SupR, and 4.58 CPS on ReplicaCAD, versus 69.7% SRS, 91.7% CFR, 91.7% SupR, and 4.18 CPS on Replica.
• Fit-Critical Set performance: Across all tasks, ElasticFit achieves 60.8% SRS, 71.7% CFR, 83.3% SupR, and 4.57 CPS. Pose-and-contact tasks reach 73.3% SRS, 76.7% CFR, 80.0% SupR, 4.50 CPS. Gap-fitting tasks reach 48.3% SRS, 66.7% CFR, 86.7% SupR, 4.63 CPS.
• Gap-fitting is the hard case: The authors attribute the lower SRS (48.3%) and CFR (66.7%) on gap-fitting to sensitivity to minor geometric misalignment in highly constrained spaces, while noting SupR (86.7%) and CPS (4.63) remain high.
• SRS/CPS divergence: Of the 25 gap-fitting cases with SRS < 1, 24 (96.0%) still received CPS ≥ 4. Nineteen of those 25 (76.0%) contained an unsatisfied parallel_to relation, indicating orientation constraints are a common source of strict SRS misses.
• Ablation (gap-fitting, separate controlled run): Full ElasticFit gives 47.8% SRS, 76.7% CFR, 83.3% SupR, 4.63 CPS, and 1.08% penetrating vertex ratio. Removing only physics settling gives 46.7% SRS, 76.7% CFR, 80.0% SupR, 4.60 CPS, 1.45% penetration. Removing geometry- and physics-aware fitting gives 47.8% SRS, 36.7% CFR, 83.3% SupR, 4.23 CPS, 6.09% penetration.
• Failure analysis (Fit-Critical Set, 60 tasks): Directional fitting error accounts for 2 tasks (3.3%), generation/adaptation mismatch for 6 (10.0%), and concave support ambiguity for 1 (1.7%).
• Qualitative observation: Baselines struggle with viewpoint-dependent relations such as in front of, behind, left of, and right of, and with bounding-box-based refinement on irregular support geometry. ElasticFit aligns VLM reasoning with the user's current view.

Methodology in Plain English

Stage 1 — Semantic-Spatial Grounding. The scene is rendered from the viewpoint shown to the user. From that view plus dataset-provided semantic-instance annotations, the method builds an XYZ buffer (per-pixel 3D coordinates) and an ID buffer (per-pixel semantic instance ID). A Set-of-Mark image overlays visible object IDs and summarizes candidate anchors. A VLM (Gemini-3.1 Pro Preview is used for all reasoning in both ElasticFit and the baselines) receives the unannotated view, the Set-of-Mark image, and the instruction, and returns structured JSON: an anchor object, a target placement pixel and support object, optional orientation cues, snapping behavior, and generation fields. The predicted pixel is validated against the ID buffer and corrected to the nearest visible pixel on the intended support object if needed, then mapped through the XYZ buffer to a 3D support point. Orientation comes from the optional cues plus pose-specific priors, producing yaw and pitch with roll fixed to zero under an upright-object assumption. A "ghost box" — an estimated maximum fitting volume, not a tight bounding box — is placed with its bottom-face center at the support point.

Stage 2 — Scene-Conditioned Object Prior. A text-to-image generator (an SDXL-style model) generates a 2D object image on a canvas whose aspect ratio follows the projected width-to-height proportion of the fitting region, guided by the object prompt. An image-to-3D model (TRELLIS, used through a wrapper) lifts this image into an initial 3D mesh hypothesis.

Stage 3 — Geometry- and Physics-Aware Fitting. Depending on the deformation mode, the object is fitted to the ghost box: rigid mode preserves geometry and solves for a grounded, collision-aware pose via broad search then narrow mesh-level refinement; uniform mode optimizes one global scale factor plus translation while balancing collision penalties, box constraints, support/gravity terms, and scale maximization; elastic mode anisotropically scales along box-local axes toward the ghost-box dimensions, with nearby scene samples penalizing collisions and maintaining support. After fitting, a lightweight rigid-body simulation in PyBullet settles the object under gravity and contact dynamics, starting from a slightly lifted non-penetrating pose, followed by a conservative correction using nearby upward-facing support points.

Evaluation. Four metrics are used: Spatial Relation Success Rate (SRS), Collision-Free Rate (CFR), Support Success Rate (SupR), and Contextual Plausibility Score (CPS), the last scored on a 1–5 scale by GPT-5.4-mini on anonymized user-view screenshots and the original prompt. For the baseline comparison on the Common Placement Set, ElasticFit is run in a placement-only variant where generation and deformation-aware adaptation are disabled so all methods use the same fixed target assets.

Why This Matters

Impact on research. The paper shifts the object-insertion problem from "where does a fixed asset go?" to "how should the object itself adapt to the space?" It shows that high-level VLM reasoning can be converted into explicit 3D constraints that condition generative models, and that geometry- and physics-aware refinement can substantially raise support and spatial-relation success (SupR 48.3% to 91.7%) without collapsing collision-free rates. It also documents a measurable gap between strict predicate scores and perceived plausibility in constrained spaces, which is useful for how the field interprets its own metrics.

Real-world applications.

• Augmented and virtual reality, where users insert virtual furniture or props into a room and expect them to fit the available space.
• Automated interior design, where an object must satisfy both a design instruction and local geometry (shelves, corners, gaps).
• Embodied AI, where an agent must place a generated object into an already occupied environment without collisions and with proper support.
• 3D content creation and scene editing tools for artists and non-expert users working from natural-language prompts.

Industry relevance. The method chains together off-the-shelf components (SDXL-style generation, TRELLIS lifting, PyBullet simulation, a VLM backbone), suggesting the pipeline is practical to reproduce. The "make-it-fit" capability targets the common consumer scenario of wanting an object that matches a specific niche rather than picking from a fixed catalog.

Future Directions

• True non-rigid deformation. The current elastic mode relies on anisotropic scaling; the authors note that cloth draping over a chair or a pillow compressed into a corner remains an open challenge.
• Explicit containment constraints. Because of proximity constraints, objects can satisfy support but hang partially over a surface edge; adding containment constraints could ensure objects stay fully within the supporting surface.
• Stronger spatial conditioning for generation. The paper attributes generation–adaptation mismatch failures (6 of 60 Fit-Critical tasks, 10.0%) to soft object-prior control, and points toward ControlNet-style or mask/box-conditioned generation plus a deformation-tolerance prior indicating which object categories can safely undergo elastic adaptation.
• Multi-object insertion. Future work could account for multiple simultaneous object placements.
• Robustness and reproducibility gaps. The authors note the task suite was developed alongside the method, repeated-run variability was not quantified, and robustness to errors in predicted semantic instance annotations remains untested.

Target Audience

Researchers and practitioners in 3D computer vision, graphics, and embodied AI who work on scene synthesis, language-guided placement, or generative 3D asset creation. It is also relevant to AR/VR and interior-design engineers who need objects to fit constrained real-world spaces, and to readers interested in how VLM reasoning can be turned into explicit geometric constraints for generative pipelines.

Authors’ abstract

Inserting objects into existing 3D scenes requires more than selecting a plausible location: the inserted object must also fit local geometry while preserving semantic intent and physical plausibility. Although recent Vision-Language Models (VLMs) and generative models enable semantic reasoning and visual content creation, they offer limited 3D grounding and geometric control when an inserted object must fit into constrained local spaces. We introduce \textbf{ElasticFit}, a VLM-guided framework for fit-aware object insertion centered on a novel scene-grounded representation. Given a language instruction and rendered scene observations, ElasticFit infers structured fitting cues that specify where the object should be grounded, what volume it should occupy, how it should be oriented, and its adaptation mode (rigid placement, uniform scaling, or elastic fitting). These cues convert high-level VLM reasoning into explicit 3D constraints that condition object generation and guide downstream geometric fitting. ElasticFit then generates a scene-conditioned object prior, reconstructs it in 3D, and refines the mesh through mode-specific fitting while enforcing collision avoidance, contact consistency, and physical grounding. In fixed-asset baseline comparisons, ElasticFit improves spatial relation success from 50.8\% to 69.7\% and support success from 48.3\% to 91.7\% over the strongest baseline, while providing novel support for generative "make-it-fit" insertions in complex scenarios.