LLM-Enhanced Multimodal House Price Prediction using Structured, Textual, and Visual Features

AI summary

Overview

Research area: Real estate valuation / multimodal machine learning (structured tabular data, natural language processing, and computer vision).

Technical level: Advanced. The paper assumes familiarity with large language model (LLM) prompt engineering, CLIP-style contrastive vision–language pretraining, manifold-based dimensionality reduction, and gradient-boosted / ensemble regression.

Scope: A modular, interpretable pipeline that fuses structured property attributes, LLM-extracted spatio-temporal and contextual text features, and unsupervised category-aware image features to predict residential house prices.

What This Paper Is About

House price prediction models typically rely on structured fields such as bedrooms, bathrooms, and parking counts, which miss the contextual signals buyers actually respond to — proximity to amenities, renovation status, interior quality, curb appeal. Property listings already contain this information in free-text descriptions and photographs, but prior deep learning systems mash text and image embeddings together end-to-end, producing opaque representations that do not explicitly capture domain-relevant factors. The paper's goal is a modular framework that extracts human-interpretable, domain-specific features from each modality and then fuses them to predict prices more accurately than structured-only and existing multimodal baselines.

Key Contributions

1. A modular multimodal framework integrating structured attributes, contextual signals from textual descriptions, and semantic information from unlabeled property images, explicitly avoiding opaque end-to-end fusion architectures.
2. An LLM-based structured extraction method that converts free-form real estate descriptions into spatio-temporal and contextual features in a flat JSON format using zero-shot prompt engineering, with canonical feature naming so extractions are comparable across listings.
3. An unsupervised visual representation pipeline that uses a vision-language model (CLIP) plus "pseudo-prompt" engineering to group unlabeled property photos into four semantically meaningful classes — indoor, outdoor, mixed, and floorplan — enabling category-aware visual features without any manual annotation.
4. Extensive experiments and ablation studies showing, per the abstract, that both LLM-derived contextual features and category-aware visual semantics give consistent and complementary improvements over structured and spatial-feature-based approaches.

Main Findings

• Outperforms baselines: The abstract reports that experiments on a large real-world property dataset show the method beating traditional methods and existing multimodal baselines. The specific benchmark metric values (RMSE, MAE, R², or percentage gains) are not present in the supplied paper content, so no exact numbers are reported here.
• Textual features help: Ablation studies confirm that the LLM-based spatio-temporal and contextual feature extraction substantially contributes to more accurate and reliable price predictions.
• Categorized vision features help: Ablations likewise confirm that the categorized visual embeddings contribute, relative to using uncategorized image representations.
• Complementarity of modalities: The paper states that the LLM-derived contextual features and category-aware visual semantics provide consistent and complementary improvements over traditional structured and spatial feature-based approaches.
• Positioning against prior work (from the paper's comparison table): The proposed method is the only row in Table 1 marked as covering structured, spatio-temporal, contextual, non-categorized visual, and categorized visual modalities; prior families (hedonic, traditional ML, geo-spatial, NLP-based, graph-based, multimodal deep learning) each miss at least one.
• Related-work evidence cited as motivation: The authors note that Geerts et al. (2025a), benchmarking across six housing datasets, found graph neural networks generally failed to outperform well-tuned tree models like LightGBM and CatBoost; and that Geerts et al. (2025b) found LLMs lag dedicated ML models on pure accuracy and tend to be overconfident, making them better suited to feature extraction than to being the valuation engine.

Methodology in Plain English

The pipeline has four stages.

1. Structured features. Missing numeric fields are imputed with medians and categorical fields with the mode. Sale prices are adjusted to a common reference year using a compound growth formula, with a conservative constant annual rate of 6%, described as consistent with long-term average housing inflation in Australia at the time of writing.

Feature engineering includes:

• Distances (e.g., to the nearest train station) converted from meters to kilometers and then squared, reflecting non-linear decay in the value of proximity.
• Pairwise multiplicative interaction features between numeric attributes, with only those retained that reduced validation RMSE — e.g., a rent_bedroom_interaction (bedrooms × local rental prices) and a school-ranking × school-proximity term.
• Neighborhood context: mean and standard deviation of sold prices per locality, merged back into the data.

2. Textual features. A single structured zero-shot prompt casts the LLM as an expert real-estate analyst and asks it to return only a flat JSON object of the most salient features drawn from two families: spatio-temporal (location relative to shops, transport, schools, parks, water, city, employment hubs; proximity and accessibility; orientation; time in market) and contextual (condition, age, build quality, prestige, ambiance, layout and space, indoor/outdoor features, suitability, seller intent). Rules enforce canonical key names (always near_station, never synonyms), numeric values (1 for present/true qualitative attributes, real numbers for quantities like minutes or kilometers), and a cap of roughly max_feats features.

3. Visual features. Images are first quality-filtered by minimum resolution and by sharpness, measured as the variance of the Laplacian operator on grayscale images (higher variance = sharper). Retained images are encoded with CLIP, whose symmetric contrastive loss pulls matching image–text pairs together and pushes non-matching pairs apart in a shared embedding space. High-dimensional CLIP embeddings are then projected to a lower-dimensional space with a manifold learning technique (the section describing this is truncated in the supplied content). Clusters are aligned with carefully designed pseudo text prompts so that photos are automatically assigned to indoor, outdoor, mixed, or floorplan categories — no manual labels — and category-aware visual features are constructed from this grouping.

4. Fusion and prediction. All modality-specific representations are combined into a unified representation and passed to an optimized regressor to predict price. In the pipeline figure, darker blocks mark components introduced in this work and lighter blocks mark standard or reused processing steps.

Why This Matters

Impact on research. The paper targets a specific gap named in prior surveys (Huang et al., 2025): the difficulty of quantifying each modality's individual contribution inside fused predictions. By extracting explicit, named features per modality before fusion, the framework makes modality-level ablation and explanation possible, in contrast to end-to-end architectures where contributions blur together.

Real-world applications.

• Automated valuation for buyers, sellers, and agents, where listing text and photos are already available and structured fields alone are insufficient.
• Mortgage underwriting and financial risk assessment, where an explainable valuation rationale is required alongside the number.
• Property taxation and municipal assessment, where defensible, factor-level justifications matter.
• Listing optimization for sellers and portals, surfacing which qualitative attributes (renovation status, park views, indoor–outdoor flow) actually move price.

Industry relevance. The pipeline works with unlabeled photos and requires no manual image annotation, which maps directly onto how real estate portals already store data: many photos per listing, free-text descriptions, and a structured attribute table. The modular design also allows a portal to swap the LLM, the vision encoder, or the regressor independently rather than retraining a monolithic network.

Future Directions

• Quantifying per-modality contribution systematically. The paper motivates transparent fusion but leaves open a full accounting of how much each modality adds across market segments and price tiers.
• Extending and validating the image taxonomy. The four classes (indoor, outdoor, mixed, floorplan) are hand-designed; whether finer or data-driven categories improve accuracy, and how the approach scales beyond these four, is untested here.
• Generalizing across markets. The price adjustment uses a 6% annual rate described as consistent with Australian housing inflation; the prompt is deliberately written to be location-agnostic, but cross-market transfer (and sensitivity to the assumed inflation rate) is an open question.
• Replacing or upgrading components. Since the framework is modular, substituting stronger LLMs or vision-language models, or richer pseudo-prompt sets, is an obvious next experiment, as is testing the extracted feature schema against learned embeddings directly.

Target Audience

Researchers and graduate students in multimodal machine learning and urban/real-estate analytics; applied scientists at property portals, proptech firms, and valuation or lending institutions who need interpretable automated valuations; and practitioners interested in LLM-based structured feature extraction and unsupervised vision-language categorization as reusable design patterns beyond housing. Readers wanting only headline accuracy numbers should note that the supplied content reports qualitative superiority and ablation conclusions but does not include the numeric result tables.

Authors’ abstract

Accurate house price prediction is essential for real estate buyers, sellers, agents, and financial institutions. Traditional methods rely primarily on structured data, e.g., number of bedrooms, bathrooms, parkings, but they overlook rich contextual information embedded in textual descriptions and property photos. Previous deep learning approaches for house price prediction have fused multimodal data but they fall short to extract structured spatio-temporal and contextual features from texts and semantic visual information from images. In this paper, we introduce a novel multimodal framework that addresses these limitations by systematically integrating structured attributes, textual descriptions, and visual images of properties to achieve higher prediction accuracy while maintaining interpretability. We use a large language model to derive domain-specific spatio-temporal and contextual features from text, and an unsupervised vision-language approach to automatically group photos into meaningful classes like indoor, outdoor, mixed, or floorplan, that are commonly present in real-world house listings. These categories are created without manual labeling by aligning image clusters with carefully designed pseudo text prompts. We fuse all features into a unified representation and use an optimized regressor to predict prices. Experiments on a large real-world property dataset demonstrate that our method outperforms traditional methods and existing multimodal baselines. Ablation studies confirm that the novel textual feature extraction and categorized visual embeddings substantially contribute to more accurate and reliable price predictions.