Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling

AI summary

Overview

Research area: Computer vision and 3D scene understanding, specifically composable real-to-sim scene modeling — recovering a real indoor scene as separate, editable 3D object assets placed at their observed locations.

Technical level: Advanced. The paper assumes familiarity with multi-view geometry, scene graphs, vision-language models, 3D asset generation, 9-DoF pose estimation, and reinforcement learning from human/AI feedback (GRPO/DAPO-style policy optimization).

Scope: The paper proposes Lucida, a three-stage pipeline (parse, generate, place) that converts posed RGB(-D) indoor video into a scene graph of complete, individually placeable object assets, evaluated on scene-level 3D object detection, object pose estimation, and full scene reconstruction.

What This Paper Is About

Existing real-to-sim pipelines split the problem into parsing observations into instances, generating a 3D asset for each, and placing each asset back into the scene — but each step assumes inputs that a cluttered real capture rarely provides, such as accurate instance geometry, unoccluded views, or assets that already match the observation. Because the steps run in a fixed order, any unmet requirement degrades everything downstream.

Lucida keeps the parse–generate–place order but redistributes the requirements: parsing produces only per-instance multi-view evidence and a coarse 3D estimate, generation completes each object from that evidence, and placement is deferred to a closed-loop vision-language-model agent (GizmoAct) that refines the asset against the scene until it decides alignment is reached.

Key Contributions

1. Reformulating the pipeline's assumptions. The authors recast composable scene modeling so that no stage depends on an input a real cluttered capture cannot provide, deferring precision to a closed-loop final placement step rather than demanding it from the first stage.

2. A scene-level 3D object detection method. A parsing module builds a scene graph with per-instance multi-view evidence bundles, improving mAP over Boxer from 0.351 to 0.592 on R2S-Scene under the all-annotation protocol.

3. GizmoAct, a VLM policy for asset placement. Placement is cast as multi-turn GUI interaction: the policy manipulates an object's gizmo in a closed loop, emits incremental pose edits in the object's local frame, and predicts its own stop action. It is trained by supervised finetuning on synthetic expert trajectories (with injected errors) followed by reinforcement learning, and raises strict-alignment success (ADD-SB@0.05) by up to 25.6 percentage points over the strongest baseline while tolerating mismatched assets and coarse initial poses.

4. An integrated system and benchmark evaluation. Lucida is evaluated on R2S, the authors' real-world real-to-sim benchmark, at three levels — 3D object detection, object pose estimation, and scene reconstruction — with composed scenes reaching a scene F-Score of 0.924 versus 0.794 for the strongest baseline (SAM 3D).

Main Findings

• Scene-level 3D object detection (Table 1). Lucida achieves the highest mAP in all four evaluation protocols. On CA-1M it improves the strongest baseline from 0.171 to 0.180 under the _all protocol and from 0.373 to 0.390 under the _filter protocol. On R2S-Scene, mAP increases from 0.351 to 0.592 under _all and from 0.355 to 0.597 under _filter. On R2S-Scene this is reported as a 69% improvement over Boxer.

• Prompt-frame efficiency. Boxer improves when given prompts on all annotated frames, yet Lucida, initialized only from geometry-aware keyframe prompts, still performs better, using the full sequence for instance propagation, multi-view validation, and 3D box refinement.

• Object pose estimation (Table 2). On CA-1M, GizmoAct (max 4 views) raises ADD-SB@0.05 from 57.8% (RecGen, 1 view) to 83.4%, and 3D IoU from 0.434 to 0.607, compared with SAM 3D. On R2S-Object it raises ADD-SB@0.05 from 79.2% (RecGen, 1 view) to 92.0%, and 3D IoU from 0.500 to 0.719. On CA-1M, GizmoAct (1 view) already reaches 81.9% ADD-SB@0.05 and 3D IoU 0.600.

• Lower average distance error. On CA-1M, GizmoAct (max 4 views) achieves ADD-SB of 0.021 versus 0.062 for SAM 3D and 0.061 for RecGen (1 view). On R2S-Object it achieves 0.017 versus 0.050 for SAM 3D and 0.040 for RecGen (1 view).

• One policy, multiple initializers. A single GizmoAct policy handles initializers with different error profiles without retraining.

• Scene reconstruction. Scenes composed by Lucida achieve a scene F-Score of 0.924, compared with 0.794 for SAM 3D and 0.351 for SceneGen, while improving the object-level F-Score from 0.704 for SAM 3D to 0.736.

• Evaluation also covers ADT. ADT is included among the pose-estimation datasets alongside R2S-Object and CA-1M; the table entries for ADT are cut off in the available text, so the full ADT numbers are not reported here.

Methodology in Plain English

Parse. From posed RGB-D observations, Lucida selects a sparse set of geometry-aware keyframes using a similarity score that combines covisibility with temporal separation, so keyframes stay spread across a long sequence. A VLM and a 3D detector identify instances on the keyframes, and observations are grouped across keyframes by semantic and geometric consistency. Because keyframes are chosen for scene-level covisibility, they may not cover every object well, so the system consolidates evidence over the full sequence: it picks a representative 3D box, projects it into frames as a prompt for per-frame 3D box estimation, and keeps only validated observations. A relation-aware refinement step corrects wrong merges and splits, infers spatial relations such as support and containment, searches extra views for missing supporting objects, and fixes spatial inconsistencies. Each object node carries an evidence bundle of selected views, masks/boxes, partial point clouds, a representative 3D box, and a category or referring description.

Generate. Occlusion and noisy depth make per-object point clouds incomplete, so the system uses RGB instead. It selects a few reliable, complementary views, marks the target with Set-of-Mark prompting, has a VLM designate an anchor plus references and write an editing instruction, and uses an image-editing model to synthesize a complete isolated object image, which an image-to-3D model lifts into a 3D asset.

Place. GizmoAct reframes 3D grounding as multi-turn GUI interaction. The state (object center, rotation, anisotropic scale — 9 DoF) is rendered together with the scene point cloud, the asset overlay, its 3D box, and a gizmo showing the object's local frame. Each turn the VLM reads the rendering and emits one XML-wrapped JSON action: update_pose, switch_obs, permute_axis, or stop. Rotation is applied first, and translation and scale deltas are expressed in units of the current object size, so the policy never predicts an absolute pose or estimates metric scale. For large initial rotation errors, switch_obs re-renders along six signed axes and permute_axis selects one of 24 axis-aligned reorientations in a single action, leaving only the residual for update_pose.

Training. SFT uses synthetic expert trajectories from three data sources (Populated 3D-FRONT, FoundationPose-style scattered assets on textured planes, and CA-1M Objects with SAM 3D assets), with DART-style error injection so the policy learns to recover from its own mistakes. RL then uses GRPO with a quantized reward on generalized 3D IoU (boundaries at 0.75, 0.85, 0.925) and geodesic rotation error (30°, 10°, 5°), dynamic sampling with K = 8 rollouts at temperature 0.7, DAPO-style token-level clipping, no KL penalty, and batches of at least 40 prompts and 320 trajectories optimized for two epochs. Symmetric objects are excluded from RL training because the rotation reward would penalize symmetry-equivalent poses.

Compose. Individual grounded objects are assembled into a scene graph with assets, poses, scales, categories, and spatial relations. Optional postprocessing separately enforces support relations, collision, contact consistency, and placement plausibility as rules, physics checks, or global verification rather than through the agent.

Why This Matters

The work targets a gap between two families of prior methods: differentiable-rendering reconstructions (NeRF, 3D Gaussian Splatting and their object-compositional extensions) that reproduce a room faithfully but as a monolith with occluded surfaces unrecovered, and CAD retrieval-and-alignment real-to-sim systems that produce simulation-ready scenes but cap fidelity at database coverage. Lucida aims for objects that are simultaneously complete, editable assets and faithful to the observed room.

Real-world applications:

• Robot simulation and robot learning, where a captured real room can be turned into a simulation-ready replica whose objects can be manipulated individually.
• Embodied AI training and evaluation in environments that mirror real captured spaces.
• AR/VR and content creation, where a captured indoor scene becomes an editable set of object assets rather than a fixed reconstruction.
• Scene editing and authoring pipelines that need object-level control over captured environments.

Industry relevance: The authors are affiliated with ByteDance Seed, Peking University, and Zhejiang University, and the pipeline explicitly targets the form in which robot simulation, embodied AI, AR/VR, and content creation consume scenes. The approach depends on components that industry labs already deploy at scale (VLMs, image editing models, image-to-3D models, 3D detectors), positioning it as a system-level integration play rather than a new single model.

Future Directions

• Postprocessing and scene-level constraints. The paper separates GizmoAct from optional postprocessing that enforces support relations, collision, contact consistency, and placement plausibility, leaving open how these rules should interact with the learned placement policy.

• Symmetry-aware reward design. RL training currently excludes symmetric objects because geodesic rotation error penalizes symmetry-equivalent poses; scoring them properly is stated to require symmetry annotations in the reward, which is an unresolved problem.

• Ablation of parsing components and training strategies. Section 3.4 is described as ablating scene-parsing components and GizmoAct training strategies — specifically object-centric full-sequence evidence consolidation and relation-aware scene refinement — but those results are truncated in the available content.

• Broadening what a capture can supply. The design principle is that each stage consumes only what a real capture reliably provides; extending the same redistribution of requirements to other capture types (for example, sparse or single-view inputs, which the orthographic views partly address) is a natural next question.

Target Audience

Researchers and engineers working on real-to-sim systems, robot simulation, embodied AI, 3D scene reconstruction, object pose estimation, and vision-language-model agents that operate graphical interfaces. It is most useful to readers already comfortable with multi-view geometry, scene graphs, and 3D asset generation, and to practitioners who need scene reconstructions that are editable per object rather than monolithic. Readers looking for a beginner-level introduction to 3D reconstruction will find the paper dense; readers specifically interested in agentic, closed-loop placement policies will find the GizmoAct formulation the most transferable part.

Authors’ abstract

Composable scene modeling aims to recover a real indoor scene as complete, editable object assets arranged as observed, giving robot simulation and embodied AI a simulation-ready replica of the real environment whose objects can be manipulated individually. Existing pipelines decompose the task into three steps---parse the observations into instances, generate an asset for each, and place each asset back---but every step presumes an input that a cluttered capture rarely provides: accurate instance geometry, unoccluded views, and assets that accurately match the observations. We propose Lucida, which keeps this order but redistributes the requirements, so each step consumes only what a real capture reliably provides and precision is reached at the end of the pipeline rather than demanded at its start. Lucida parses the video into a scene graph whose nodes carry per-instance multi-view evidence, generates a complete asset for each instance from its evidence, and places assets with GizmoAct, a VLM policy that casts placement as multi-turn GUI interaction, manipulating the object's gizmo in a closed loop and deciding itself when alignment is reached. Across scene-level 3D object detection, object pose estimation, and scene reconstruction, Lucida improves mAP over Boxer by 69% on R2S-Scene, raises ADD-SB@0.05 from 57.8% to 83.4% on CA-1M, and increases scene F-Score from 0.794 for SAM3D to 0.924.