HouseTour: A Virtual Real Estate A(I)gent

AI summary

Overview

Research area: Computer vision and 3D vision-language modeling — specifically generating human-like camera trajectories and real-estate-style descriptions from posed images of indoor spaces.

Technical level: Intermediate. The paper combines diffusion models, vision-language models, spline interpolation, and 3D reconstruction, but the core ideas are explained at a level accessible to readers familiar with deep learning basics.

Scope: The paper introduces a new task (joint 3D camera trajectory and textual summary generation), a method called HouseTour (Residual Diffuser + Qwen2-VL-3D), and a dataset of real-world house-tour videos with 3D reconstructions and professional descriptions.

What This Paper Is About

Professional house-tour videos are labor-intensive to produce: a real estate agent must visit the property with videography equipment and manually write descriptions that emphasize layout, functionality, architectural features, appliances, materials, and ambiance rather than just listing objects. The paper asks whether a model can take only a set of images with known camera poses from an existing 3D space and automatically produce both (i) a smooth, human-like 3D camera trajectory through the property and (ii) a written summary that reads like a real-estate agent's narration.

The authors argue that current vision-language models lack the geometric reasoning needed for this, and that existing 3D-language datasets contain trajectories tailored to reconstruction (close to surfaces, jerky) and descriptions that merely enumerate furniture and object relationships.

Key Contributions

1. A new task: spatially-aware 3D camera trajectory generation and textual summarization from a collection of images, aimed at replicating house-tour videos.
2. A new method, HouseTour: a Residual Diffuser that generates smooth camera trajectories as residuals over spline interpolation, conditioned on sparse known poses, plus Qwen2-VL-3D, which injects 3D spatial features into a vision-language model so the text aligns with the spatial path.
3. The HouseTour dataset: real estate tour videos with 3D reconstructions and real-estate-style textual descriptions, released along with code, trained models, and dataset at house-tour.github.io.
4. A new evaluation metric: the Spatio-Linguistic score (SLS), a harmonic mean of translation recall at 75cm, a rotation score, and a Bradley-Terry preference score, ranging from 0 to 100.

Main Findings

• End-to-end performance: On the joint task (Table 1), HouseTour scores R@75cm 60.2, Rotation Score 97.1, Bradley-Terry 79.5, and SLS 76.0, versus a composed baseline (Catmull-Rom Spline + Qwen2-VL-7B (SFT)) at 57.1, 96.8, 71.4, and 71.7. The baseline is the best-performing method per individual task, since no prior method solves the joint task.

• Trajectory generation: The Residual Diffuser beats Linear Interpolation and Catmull-Rom baselines on all translation and rotation metrics (Table 2). R@50cm is 46.2% and R@1m is 69.4%, versus 45.9% / 64.7% for Catmull-Rom and 41.2% / 59.8% for Linear Interpolation. Euclidean distance drops to 73.9 cm (from 106.2 for Catmull-Rom and 145.8 for Linear Interpolation), Chamfer distance to 75.5, quaternion distance to 0.0073, and geodesic distance to 0.09 radians. The paper attributes the R@1m advantage to fewer large errors, and notes PSNR and SSIM stay comparable across methods because rendering metrics are insensitive to small pose differences (14.20 / 14.22 / 14.24 PSNR and 0.557 / 0.557 / 0.556 SSIM).

• Sparse observations: In the ablation on pose frequency (Table 4), all methods degrade as observations thin out, but the Residual Diffuser stays best. At every 10th frame it cuts Euclidean distance error by 32% versus the closest baseline (151.1 cm vs 219.0 cm), with roughly 28% reductions at every 5th and 15th frames.

• Scene summarization: Qwen2-VL-3D leads on most n-gram metrics (Table 3): BLEU-1 0.433, BLEU-2 0.264, BLEU-3 0.154, BLEU-4 0.090, ROUGE-L 0.24, METEOR 0.193, and a Bradley-Terry score of 0.79. Fine-tuning matters substantially — Qwen2-VL-7B (SFT) reaches BT 0.71, while zero-shot Qwen2-VL-7B and LLaVa-OneVision-7b both score BT 0.04.

• Value of 3D information: When 3D positional information is available, Qwen2-VL-3D wins 67.5% of Bradley-Terry comparisons versus 32.5% for Qwen2-VL-7B (SFT). It also wins 56% of the time without 3D position, which the authors suggest indicates the model partly learns to "localize" images within a scene during training (Table 5).

• Why the 75cm threshold: The authors note HouseTour outperforms baselines at larger recall thresholds but loses its edge at tighter ones, attributing this to rising uncertainty as distance from the nearest known pose grows. They choose 75cm as a practical balance.

• Dataset composition: The dataset contains 1,639 videos of properties ranging from condos to multi-storey apartments. 1,298 videos are transcribed, half with timestamped descriptions, and 3D reconstructions are provided for 878 scenes. Reconstruction outcomes break down as 60% SUCCESS, 15% BAD SCALING, 9% MISALIGNED, 9% FAILED, and 7% INCOMPLETE (Table 6). Summary types split 63% transcripts and 37% descriptions (Table 7); 813 of the descriptions were transcribed with Whisper.

Methodology in Plain English

The input is a set of RGB images plus the camera pose for each image — sparse snapshots rather than a smooth video, which matches what a non-expert with a smartphone could realistically capture. The output is a dense, continuous trajectory and a written summary.

Trajectory generation. Rather than learning an absolute path, the model first draws a spline through the known poses (spherical linear interpolation, or SLERP, for rotations) and then learns a residual to add on top. This framing treats human-like motion as a correction to plain geometric interpolation, which the authors argue is a better learning problem because every property has a different layout, unlike the fixed mazes used in prior diffusion-based planning work. Known poses are pinned in place — their residual is set to zero — and the rest of the trajectory is denoised around them, similar to image inpainting. Training uses a U-Net of 1D convolutions so trajectories of varying length can be handled, and pose sparsity is varied during training.

Trajectory loss. Standard diffusion training predicts noise, but the authors instead define a loss over densely sampled spline points: an L2 norm on translations and a geodesic loss on rotations, since the ground-truth poses come from 3D reconstruction and carry biases (denser sampling in low-texture regions, sparser in high-texture ones). Spline points are evaluated efficiently with Horner's Method, and quaternion residuals are renormalized to unit length.

Summary generation. The authors start from Qwen2-VL and fine-tune it in two stages. First, LoRA fine-tuning teaches the real-estate language style using 96 uniformly sampled frames per tour video, chosen based on training memory limits while still covering the scene. Second, 3D awareness is added by introducing <|traj_start|>, <|traj_pad|>, and <|traj_end|> tokens, and by an adapter that concatenates the denoised pose with bottleneck features from the last trajectory-denoising step, then maps them through a linear layer into the language embedding space. One token encodes each frame's spatial information, and frames can be included with or without spatial features.

Visualization. Final videos are synthesized by training a 3D Gaussian Splat on all ground-truth poses and rendering the denoised poses using only the sparse views. The authors stress these renders are for visualization only; the problem of synthesizing scenes from sparse views is explicitly out of scope.

Evaluation. Trajectories are judged with recall (R@50cm, R@75cm, R@1m), Euclidean, DTW, Hausdorff, Fréchet, and L2 Chamfer distances, plus quaternion and geodesic distances for rotation, and PSNR/SSIM for rendering. Summaries are judged with BLEU, ROUGE-L, METEOR, and CIDEr for style, plus Bradley-Terry preference comparisons using GPT-4o as a judge, which sees the ground-truth summary alongside two generated summaries in randomized order and picks the closer one.

Why This Matters

The paper targets a labor-intensive real-world workflow: providing remote video tours of properties without requiring agents to visit with high-end videography equipment, or requiring end users to shoot smooth video themselves. Using images instead of video also improves privacy, since users can selectively capture content.

Real-world applications:

• Remote real estate listings and virtual property viewings for renters and prospective buyers who cannot visit in person.
• Tourism and hospitality — automated walkthrough videos and descriptions of hotels, rentals, or venues.
• Real estate marketing and listing services that need consistent professional narration at scale.
• Privacy-conscious interior documentation, where a person photographs selected areas rather than recording continuous video of a whole home.

Industry relevance: The paper cites that over 624 million videos on YouTube feature real estate agents and occupants showcasing homes, and that the U.S. real estate market is valued at 3.43 trillion dollars. It also positions the work against RealEstate10K, which contains 7000 video snippets, but with clips of 1 to 10 seconds versus HouseTour's several-minute videos and no textual summaries.

Research impact: The work connects diffusion-based planning (drawing on Diffuser) with 3D vision-language modeling, and contributes a dataset that fills a gap in existing 3D visio-linguistic datasets by pairing professionally written descriptions with scene-observing trajectories rather than reconstruction-oriented ones.

Future Directions

• Incorporating information from vision-language models to jointly guide the trajectory diffusion process, so that what the agent "sees" influences where the camera moves.
• Developing Gaussian splatting methods that can fill gaps between sparse images without generating content that was never actually observed.
• Addressing the residual uncertainty as distance from the nearest known pose grows — the paper identifies this as the reason its relative advantage shrinks at tight recall thresholds and picks 75cm as a compromise.
• Extending evaluation and scene handling to out-of-distribution scenes (the paper defers these results to the supplementary material), and improving reconstruction robustness, given that 15% of scenes showed bad scaling, 9% were misaligned, 9% failed, and 7% were incomplete.

Target Audience

Researchers and practitioners in computer vision, 3D scene understanding, vision-language modeling, and generative trajectory planning; engineers building automated real estate, virtual tour, or spatial-narration products; and dataset builders interested in 3D visio-linguistic benchmarks with in-the-wild video, dense point clouds, and professional text annotations. Readers need working familiarity with diffusion models, vision-language transformers, and camera pose representations to follow the technical sections, though the task framing and results are accessible to a broader audience.

Authors’ abstract

We introduce HouseTour, a method for spatially-aware 3D camera trajectory and natural language summary generation from a collection of images depicting an existing 3D space. Unlike existing vision-language models (VLMs), which struggle with geometric reasoning, our approach generates smooth video trajectories via a diffusion process constrained by known camera poses and integrates this information into the VLM for 3D-grounded descriptions. We synthesize the final video using 3D Gaussian splatting to render novel views along the trajectory. To support this task, we present the HouseTour dataset, which includes over 1,200 house-tour videos with camera poses, 3D reconstructions, and real estate descriptions. Experiments demonstrate that incorporating 3D camera trajectories into the text generation process improves performance over methods handling each task independently. We evaluate both individual and end-to-end performance, introducing a new joint metric. Our work enables automated, professional-quality video creation for real estate and touristic applications without requiring specialized expertise or equipment.