HARMONY: Hierarchical Agentic Reasoning for MONocular Image-to-Scene Synthesis

AI summary

Overview

Research area: Computer vision — single-image compositional 3D scene reconstruction, combining vision-language model (VLM) reasoning with visual geometry foundation models.

Technical level: Advanced. The paper assumes familiarity with 3D scene reconstruction, diffusion-based image-to-3D generation, point-cloud geometry (Manhattan frames), camera calibration, and VLM prompting pipelines.

Scope: The paper presents HARMONY, a hierarchical, VLM-guided pipeline that turns one monocular indoor photograph into a complete compositional 3D scene of individually reconstructed objects placed on a calibrated room layout, and evaluates it on 100 synthetic Front3D images, 30 real-world images, and a 300-scene benchmark, with a 16-participant user study.

What This Paper Is About

Recovering a full 3D scene from a single indoor photograph is under-constrained: one view sees only partial geometry, objects occlude one another, and the image contains no explicit information about object scale or spatial relationships. Existing work splits into two camps that each solve half the problem — agentic VLM reasoning that understands spatial semantics but cannot align placements precisely to the input image, and visual geometry foundation models that predict dense point maps but produce limited reconstruction quality. HARMONY's goal is to get a scene that is both semantically coherent (correct pairings, orientations, support relationships) and perceptually aligned with the reference photograph.

Key Contributions

1. A hierarchical chain-of-thought framework for single-image compositional 3D scene reconstruction. The problem is framed as a structured multi-stage reasoning process in which a VLM first grounds the scene spatially (corners, walls, viewpoint) and then places objects in stages: wall-mounted items, free-standing furniture (and ceiling objects), then decorations resting on furniture.

2. A marriage of VLM spatial/semantic reasoning with geometry-grounded refinement. The VLM reasons about what an object leans against, sits on, faces, or pairs with; estimated point clouds and image silhouettes then correct scale and position. The paper states this avoids the weaknesses of prior methods that rely only on point clouds and therefore fail on occlusions or small objects.

3. A reflective feedback loop after each placement stage. The VLM compares a render of the partial scene against the input image to catch missing items, incorrect pairings, wrong orderings, and scale mismatches, repairing errors locally before they propagate downstream. Depth-first traversal orders furniture placement so each new candidate appears in a clean, unoccluded view.

4. State-of-the-art results on synthetic and real-world inputs, plus a qualitative comparison against GPT-6 Astra reported to show more faithful object arrangements and better preservation of scene details.

Main Findings

• Best score on every metric on the 100 Front3D cases (Table 1). HARMONY scores N-CLIP 0.092, PL 0.041, CD 0.049, F@0.1 89.82, F@0.01 19.47, F@0.001 0.130. For comparison: VIGA 0.179 N-CLIP, CAST 0.149, Gen3DSR 0.168, 3D-ReGen 0.163, SAM3D* 0.127, GPT-6 Astra 0.095 N-CLIP and PL 0.052. Lower is better for N-CLIP, PL, CD; higher is better for F-scores.

• Best perceptual scores on real-world images (Table 2). On the real-world inputs HARMONY reports N-CLIP 0.112 and PL 0.045, versus Gen3DSR 0.213, SAM3D* 0.194, VIGA 0.184, CAST 0.172, 3D-ReGen 0.154, and GPT-6 Astra 0.127 N-CLIP / 0.047 PL.

• User study results. With 16 participants across 20 scenes ranking against five baselines, HARMONY is ranked first in 67.0% of choices, in the top two in 86.0%, with a mean rank of 1.54 (± 0.93). The next-best mean rank is SAM3D* at 3.26 (± 1.60); VIGA is last at 4.37 (± 1.23).

• GPT-6 Astra produces plausible but geometrically weaker scenes. Under the evaluated configuration, the paper reports GPT-6 Astra exhibits larger geometric errors, e.g. beds might not align with walls.

• Ablations isolate the interaction of components (Table 3). Full HARMONY: N-CLIP 0.1034, PL 0.0462. Removing camera calibration hurts most (0.1381 / 0.0596), followed by removing the feedback loop (0.1184 / 0.0509), removing VLM placement reasoning (0.1079 / 0.0522), removing placement order (0.1065 / 0.0477), removing placement refinement (0.1051 / 0.0516), removing depth-first traversal (0.1045 / 0.0523), and removing VGGT refinement (0.1069 / 0.0509). The paper concludes the gains come from the interaction of global geometric grounding and structured, image-supervised reasoning, not any single component.

• Failure modes quantified. 25.9% of detections carry low grounding confidence (< 0.35), and genuine placement failures (after routing and de-duplication are excluded) remain rare at 0.3% of objects. A second failure case is heavy occlusion from foreground clipping, where the VLM recovers the correct semantic relation but the mesh is not well constrained from the render viewpoint.

• Evaluation metrics. N-CLIP is defined as 1 − CLIP cos(I_pred, I_ref) using CLIP-ViT-B/32 image embeddings; PL is pixel-wise MSE over normalized RGB in [0,1]; LPIPS is listed as a metric but no LPIPS values appear in the results tables shown. Geometry metrics are Chamfer Distance and F-score at thresholds 0.1, 0.01, and 0.001.

• Baseline setup caveats. SAM3D requires per-object masks, so HARMONY's segmentation mask is used and the baseline is denoted SAM3D*. CAST has no official release, so an unofficial implementation is used. CAST and SAM3D do not reconstruct backgrounds, so they are augmented with HARMONY's background for perceptual metrics.

Methodology in Plain English

HARMONY is a pipeline of stages, each verified before the next begins, and all of its components are off-the-shelf — nothing is trained or fine-tuned.

1. Empty room and camera. A VLM infers approximate room dimensions from semantic cues in the image (the paper's example is (3, 4, 3) m for a bedroom) and identifies the deepest visible room corner as a spatial anchor. In parallel, VGGT produces a point cloud, from which a Manhattan frame and six bounding room planes are fitted. A closed-form similarity transform (rotation, uniform scale, translation) aligns the two, using the room height as the sole external metric reference, giving a calibrated camera pose in the canonical room frame.

2. Objects. Objects are detected hierarchically by category, detected with open-vocabulary detection and segmented (SAM2), then occluded regions are inpainted by an image-editing model guided by a VLM description. Each completed object image is reconstructed into a mesh with an image-to-3D model, then canonicalized by PCA to align its dominant axis with world-up, and the VLM labels its facing direction from multi-view renders.

3. Hierarchical placement. The VLM traverses the scene depth-first from the deepest corner outward, placing wall-mounted items, then free-standing furniture and ceiling objects, then decorations. For each object it assigns an initial size, writes a spatial-relation prompt ("sofa back against the left wall", "vase on the desk", "chair facing the small coffee table"), sets orientation from the object's canonical front, and nudges rotation by comparing renders with the reference. Decorations are attached to the furniture they rest on, with a bounding-box overlap check to prevent attaching a decoration to the wrong host.

4. Geometry-grounded refinement. The VLM placements are semantically right but metrically approximate. Silhouette alignment fixes lateral position and the scale-to-depth ratio s/Z from apparent silhouette width; point-cloud front-surface comparison (robust median with MAD outlier rejection) corrects depth Z′; combining the corrected depth with the angular size then finalizes metric scale. Collisions between a pair are resolved by applying the smaller of the two object-specific escape displacements, escalating to neighbors, and ultimately to a VLM revision of the placement order.

5. Lighting. The VLM assigns per-object PBR materials (e.g. a glass table as transmissive), identifies emissive sources such as lamps and windows, estimates their activation state, color, and intensity, and iterates against a Blender Cycles render until appearance matches.

The paper reports six sequential phases in total: floorplan and camera recovery, wall-mounted object placement, furniture placement, ceiling object placement, decoration placement, and lighting estimation. Camera calibration uses fixed parameters across all scenes: row-band fraction ρ = 0.2, wall-normal alignment threshold τ = 0.7, and extension percentiles p_fc = 98 and p_wall = 97.

Why This Matters

Research impact. The paper argues that the two dominant lines of single-image scene reconstruction — geometry-grounded pipelines and VLM-based agentic reasoning — are complementary rather than competing, and that the right architecture is a hierarchical reasoning process with per-stage verification and geometric supervision at each step. It also releases two benchmarks: HARMONY30 (30 real-world, copyright-free in-the-wild images) and HARMONY300 (300 single-image indoor scenes, including the 30 evaluation scenes).

Real-world applications:

• AR/VR content creation, where a single photo of a room becomes an editable 3D scene.
• Embodied AI and robotic navigation, which need object-level 3D layouts with correct spatial relations.
• Interactive scene editing and relighting for interior design and architectural visualization.
• Asset generation for simulation environments that require many varied indoor scenes.

Industry relevance. The pipeline deliberately composes hosted or locally served foundation models (gpt-5.5 for scene/layout/lighting reasoning, Gemini-Flash-2.5 for amodal completion, Hunyuan3D-2 for mesh and texture, VGGT for Manhattan/metric alignment, LocateAnything for box proposals, SAM2 for segmentation, Blender Cycles for relighting) with no training, which the paper notes allows the prompted models to be swapped between local and hosted serving without changing the pipeline. The work was supported by Apple Inc.

Future Directions

• Extending to multi-view images, which the conclusion explicitly names as future work.
• Inferring object articulations, also named in the conclusion, so that reconstructed objects such as drawers or doors become functional rather than static.
• Mitigating the identified failures, namely rare-type synonym mismatches that depress detection confidence (25.9% of detections below 0.35 confidence) and foreground-clipped objects that are heavily occluded from the render viewpoint. The paper suggests stronger foundation models within the same framework could address the first.
• Reducing reliance on the VLM's inability to reason precise numerical quantities, which the related-work discussion identifies as the core weakness of image-conditioned VLM methods and which HARMONY addresses with post-hoc geometric refinement rather than solving directly.

Target Audience

Researchers and graduate students working on 3D scene reconstruction, image-to-3D generation, and visual geometry foundation models; practitioners building scene-assembly and layout-planning pipelines for AR/VR, interior design, robotics simulation, and embodied AI; and applied scientists interested in how to combine VLM agentic reasoning with geometric grounding in a multi-stage system. Readers should already be comfortable with camera calibration, point clouds, and diffusion-based 3D generation.

Authors’ abstract

Compositional 3D scene reconstruction has recently been explored from two directions: agentic reasoning that provides semantic understanding of spatial relationships but lacks precise alignment with input images; and visual geometry foundation models that predict dense point maps from input images but the reconstruction quality is limited. Therefore, recovering a complete 3D scene from a single monocular image with accurate inter-object relationships and high-fidelity reconstruction quality remains challenging. In this paper, we present HARMONY, a hierarchical chain-of-thought framework that leverages both agentic reasoning and visual geometry foundation. Given an image of an indoor scene, starting from an empty 3D floorplan, HARMONY first calibrates the camera against the reference image to establish a semantically-grounded spatial frame, then uses agentic VLM reasoning to recover the 3D room layout and an initial placement order. It then places the objects in a hierarchical order, from wall-mounted elements, free-standing furniture, to dependent decorations on top of furniture. We also use depth-first traversal for furniture so each placement conditions on previously resolved structure and a reflective feedback loop to avoid error accumulation. After each object placement by VLM, we use the point cloud estimations to perform geometry-based refinement so that the rendered image aligns better with the input. HARMONY can produce 3D scenes that are semantically consistent and perceptually aligned with the reference image, extending single-image compositional reconstruction to complex indoor scene images. Experiments on synthetic and real-world images demonstrate that HARMONY outperforms the evaluated reconstruction baselines, while qualitative comparisons with GPT-6 Astra suggest more faithful object arrangements and better preservation of scene details.