Articulated Object Reconstruction from Rest-State Observation

AI summary

Overview

Research area: Computer vision and 3D reconstruction, specifically articulated object reconstruction, mesh part segmentation, and the use of pretrained foundation models (vision-language, segmentation, and video diffusion) as priors for 3D understanding.

Technical level: Advanced. The paper assumes familiarity with articulated object modeling, SE(3) transformations, novel-view synthesis and mesh reconstruction, multi-view mask lifting, and joint parameter fitting.

Scope: The paper introduces a "rest-state" formulation that reconstructs multi-part articulated objects — their part geometry and their joints — from a single closed configuration, using an explicit mesh to verify and fuse noisy outputs from pretrained models.

What This Paper Is About

Existing articulated object reconstruction methods generally require images or point clouds captured in two or more articulation states, so that the motion of doors and drawers is directly observable. This paper asks whether the same result can be obtained from a single closed configuration, the state in which objects are most commonly photographed and scanned. The authors treat this as an inherently ill-posed problem and solve it by combining semantic and motion priors from pretrained models with geometric verification against an explicit mesh.

Key Contributions

1. A rest-state problem formulation. The paper frames articulated object reconstruction from a single closed configuration as an ill-posed recovery problem, and presents a complete framework (Rest2Art) that operates without any observed motion. The number of parts K is determined automatically, without prior annotation, and joints are modeled as prismatic or revolute.

2. Iterative co-refinement between a VLM and a segmentation model. Rather than chaining pretrained models sequentially, the method uses disagreements between a vision-language model's part hierarchy and a segmentation model's masks as a mutual-correction signal, then lifts validated evidence onto the mesh surface through confidence-weighted per-part aggregation.

3. Video diffusion as a source of motion hypotheses. Since no motion is observed, a video diffusion model synthesizes plausible articulation sequences, from which 2D trajectories are extracted and rigid-body joint models are fitted. The mesh constrains the result: axis configurations that drive a child part into its parent are penalized, and the fitted axis is blended with the principal direction of the adjacent part boundary.

4. Validation across diverse inputs. The framework is demonstrated on multi-view captures, single-image 3D generation, online product photos, scene scans (Replica, ScanNet++), and existing 3D assets, with joint estimation compared against reconstruction-based, generation-based, and modular pretrained-model baselines, including on the real-world MultiScan dataset.

Main Findings

• Rest-state joint estimation is competitive with motion-observing methods. On the ACD-HSSD and ACD-ABO splits, the method reports axis angular errors of 11.35 and 4.78 degrees, position errors of 0.85 and 0.20 dm, and joint type accuracies of 74.49% and 73.38%. The paper states this is the highest type accuracy among methods directly applicable to rest-state inputs on both datasets.

• Two-state reconstruction baselines degrade on synthesized motion. ArtGS, REArtGS (which additionally requires one ground-truth joint type per object as input), and REArtGS++ are two-state methods supplied with paired multi-view images of distinct articulation states synthesized from generated video. The paper reports that these methods struggle, in some cases detecting even the static base part as moving due to slight misalignment between synthesized states, causing the reconstruction to collapse.

• The shared rest-state baseline is sensitive to noisy per-module predictions. Articulate AnyMesh, the only prior reconstruction baseline in the same rest-state setting, reports 11.35 degrees angular error and 67.12% type accuracy on ACD-HSSD, and 25.65 degrees and 38.94% on ACD-ABO. The paper attributes its difficulties to applying pretrained models sequentially, which leaves it vulnerable to noisy individual predictions at subtle articulation boundaries.

• Generative baselines trade metric fidelity for axis alignment. URDFormer and Singapo produce structures in canonical space with axis-aligned joint parameters, yielding low angular error (Singapo: 1.97 degrees on ACD-HSSD, 0.13 degrees on ACD-ABO) but requiring a post-hoc scale factor per predicted axis for meaningful evaluation. Applying the authors' co-refined hierarchy as input to Singapo changes its ACD-ABO type accuracy from 41.47% to 67.94%.

• Real-world performance on MultiScan. With coverage of 48.94% of evaluable scenes, the method reports 17.23 degrees angular error, 0.26 dm position error, and 65.78% type accuracy, compared with 31.91% coverage / 51.93 degrees / 29.60 dm / 64.40% for REArtGS, 36.17% / 62.72 degrees / 6.85 dm / 48.74% for REArtGS++, and 61.70% / 30.51 degrees / 3.45 dm / 12.63% for Articulate AnyMesh.

• Co-refinement improves hierarchy prediction for both VLMs tested. Tree accuracy is 62.3% for GPT-5.2 alone and 52.2% for Qwen3-VL alone. Combined with SAM3 through iterative co-refinement, both reach 72.5% tree accuracy (+10.2 and +20.3 respectively), and per-part count accuracy rises from 47.8% to 63.8% (+16.0) for Qwen3-VL + SAM3. Used alone for per-part counting, SAM3 with ground-truth part labels reaches 82.6%, versus 43.5% for Grounded-SAM2 with ground-truth labels.

• Mask overlay conditioning substantially reduces hallucination. Ablating video synthesis on ACD-HSSD, overlaying part masks reduces hallucination from 23.1% to 11.5% and raises the accurate articulation rate from 69.2% to 80.8%; the VBVR LoRA improves plausibility but the authors state mask overlay is necessary for objects with subtle part boundaries.

• Mesh backend choice trades segmentation quality against joint accuracy. Across backends, IoU/mAP versus axis direction/axis position are: 3DGS 0.30 / 0.83 / 1.69 / 0.154; SVRaster 0.83 / 0.96 / 1.86 / 0.146; 2DGS 0.97 / 0.99 / 3.67 / 0.154. IoU is computed by sampling 100k surface points per part and measuring nearest-neighbor overlap. The main experiments use 2DGS.

Methodology in Plain English

Starting point. The pipeline accepts any source that yields a mesh: multi-view reconstruction, single-image 3D generation, or an existing 3D asset. The mesh is aligned to a canonical orientation and re-rendered from 12 viewpoints around the frontal direction (800x800 resolution, 35.49 degree field of view, camera radius 4.5, azimuth within plus or minus 30 degrees of frontal, elevation between 15 and 45 degrees).

Finding the movable parts. A vision-language model proposes a hierarchy tree of part types (from the vocabulary base, door, knob, handle, drawer, tray) along with expected instance counts. A segmentation model is queried with each part name on every view. The two are then compared: too many detections means the tree is incomplete, so the detected masks are drawn in vivid colors over the image and fed back to the VLM for revision, restarting verification. Too few detections are deferred until evidence across all views is aggregated. The maximum number of refinement rounds is fixed at two. High-confidence masks are also back-projected to 3D surface points and re-projected into other views as prompts, enforcing cross-view consistency.

Turning 2D masks into 3D part labels. Each part selects its own top-scoring views, ranked by detection confidence and mask coverage. Masks are rasterized onto the mesh using z-buffered rendering with back-face culling, and per-face, per-part scores accumulate confidence-weighted evidence across views. Each face takes the label with maximum evidence, and unsupported faces default to the base part. Because this coarse labeling contains boundary inconsistencies, spurious fragments, and under-segmented regions, the method then keeps only the dominant connected component per part, propagates labels to remaining vertices by shortest path along the mesh adjacency graph, and finally verifies each propagated label by projecting it back into that part's best-scoring view.

Imagining the motion. The method overlays the part masks with transparency on the highest-quality view and uses a video diffusion model (Wan2.2 with a LoRA weight from VBVR) conditioned on that image and on a text prompt describing the expected articulation, generating two videos with different random seeds. Dense 2D tracks are extracted with CoTracker3, initialized on each part's mask, and truncated at the first frame where visible tracks fall below a threshold of 0.5, to handle both generated-video degradation and tracker drift. Metric depth from the mesh lets the method unproject frame-0 points into 3D anchor points.

Classifying and fitting joints. Each part is classified as revolute, prismatic, or static from two cues: the coefficient of variation of per-point displacement magnitudes (high for revolute, low for prismatic) and a curvature score over time (curving for revolute, straight for prismatic). Ambiguous cases are resolved by fitting both models and keeping the lower reprojection error. Joint parameters are then optimized by minimizing 2D reprojection error between the motion model's predictions and the tracked points, using a robust loss with a trimmed refit. Because a monocular depth axis is ambiguous, the method penalizes axes whose rotation drives the child part into the parent and blends the axis with the principal direction of the adjacent part boundary. The two independent seed-based fits are combined and refit when they agree on motion type.

Making parts solid. Segmented parts are shell-like patches, so each is converted into a closed mesh by offsetting vertices inward along their normals, with boundary-aware normal smoothing to avoid artifacts at cut edges. Exterior-dominant parts like doors and knobs need only this solidification; parts such as drawers, whose interiors are largely invisible in a closed configuration, additionally receive a generated parametric inner tray blended with the solidified exterior.

Why This Matters

Impact on research. The paper reframes articulated object reconstruction away from the well-constrained multi-state or interaction-video regime toward the ill-posed single-closed-state regime, and demonstrates a general pattern: models with inconsistent, noisy outputs can still produce robust results when their disagreements are arbitrated by explicit geometry. It also provides an unusual use of video diffusion models — not as a generator of the final output, but as a sampler of hypotheses whose plausibility is settled by mesh-based geometric fitting.

Real-world applications:

• Interactive digital twins: producing physically meaningful part meshes and joint parameters from closed-configuration scans or photographs, ready for simulation.
• Robotics and embodied AI: supplying articulated assets with metric-scale joints for training and testing manipulation policies.
• Interior design and spatial planning: reasoning about whether a cabinet door or drawer can actually open given its surroundings, from a single capture.
• XR and immersive visualization: converting closed-state captures of real objects into manipulable virtual replicas (one of the paper's funding programs concerns multi-space fusion and 6DoF free-viewpoint immersive visualization for extended reality).

Industry relevance. Data at scale is far more available for closed objects — online product photos, internet imagery, and scene-level datasets such as ScanNet and Replica — than for carefully staged multi-state captures. A method that turns that existing imagery into simulatable assets lowers the data cost for content pipelines in e-commerce, gaming, simulation, and AR/VR.

Future Directions

• More complex kinematic structures. The conclusion explicitly names extending beyond prismatic and revolute joints as a future direction; the current formulation covers "most furniture articulation mechanisms" but not all.
• Improving the physical plausibility of video generation. The paper identifies this as an open problem, and its own ablation shows hallucination and implausible motion in generated articulation videos, mitigated only partly by LoRA and mask overlay.
• Removing the remaining dependence on pretrained models. The framework still depends on a VLM for hierarchy, a segmentation model for masks, and a video model for motion hypotheses; the paper reports specific failure modes for each in the rest-state setting (incomplete part hierarchy, masks missing in certain views, hallucinated nonexistent drawers).
• Wider real-world coverage. On MultiScan the method is evaluable on 48.94% of scenes, below Articulate AnyMesh's 61.70% coverage, and real-world inputs are noted as noisy — suggesting room to make the mesh-grounded pipeline applicable to a larger fraction of real scans.

Target Audience

Researchers and graduate students working on articulated object reconstruction, 3D part segmentation, digital twins, and robotic simulation, particularly those interested in extracting structure from pretrained foundation models without task-specific training. It is also relevant to practitioners building assets for physics simulation or XR from ordinary photographs and scans, though the paper is written for readers comfortable with multi-view geometry, mesh representations, and joint parameter estimation. The paper reports no training and no object-specific tuning, which makes it accessible to engineers who want to apply, rather than reproduce, the pipeline.

Note: the provided paper content is truncated within the appendix, so the quantitative part-segmentation table referenced as being in Section C.3 is not included here, and the reported segmentation comparisons are therefore described only qualitatively from Figure 5.

Authors’ abstract

Building interactive digital twins requires recovering both 3D geometry and the kinematic structures that govern how objects articulate. Yet existing methods for articulated object reconstruction require explicitly observable motion from multiple articulation states. We introduce a rest-state formulation that reconstructs articulated objects from a single closed configuration, an inherently ill-posed setting where geometry, semantics, and motion priors compensate for the absence of motion cues. Our framework adopts an explicit mesh as an intermediate representation for cross-model verification and fusion, reconciling noisy outputs from vision-language and segmentation models into spatially consistent part structures. To estimate joint parameters without observed motion, we use a video diffusion model to synthesize articulation hypotheses and validate them through geometric consistency. Our approach achieves accurate part decomposition and physically plausible articulation, performing competitively with motion-observing reconstruction-based, generation-based, and modular pretrained-model baselines.