Learning Multimodal Embeddings for Traffic Accident Prediction and Causal Estimation

AI summary

Overview

Research area: Machine learning for transportation safety — multimodal representation learning (graph neural networks plus satellite imagery) combined with causal inference for traffic accident prediction.

Technical level: Advanced. The paper assumes familiarity with graph neural networks, vision transformers, embedding fusion strategies, and causal estimators such as matching, propensity score matching, and doubly robust estimation.

Scope: The paper builds a six-state multimodal road-network dataset with aligned satellite imagery, evaluates fusion strategies that combine visual and graph embeddings for accident prediction, and uses the learned embeddings to estimate causal effects of weather, road type, and season on accident risk.

What This Paper Is About

Existing accident-prediction models rely mainly on road network structure, traffic counts, and historical crash records, but they do not see the physical road itself — lane width, curvature, surface quality, and surrounding land use — even though those factors shape driver behavior and crash likelihood. This paper asks whether pairing road-network graphs with satellite images centered on each road node can improve accident prediction, and whether the resulting embeddings can support credible causal estimates of what actually raises accident risk.

Key Contributions

1. A large-scale multimodal dataset. The authors construct a dataset spanning six U.S. states (Delaware, Massachusetts, Maryland, Nevada, Montana, and Iowa) containing over nine million accident records from official state Departments of Transportation and one million high-resolution satellite images, with one image aligned to every road network node. Images are 1024 × 1024 pixels and cover roughly 200 m × 200 m of physical space. Accident records extend through 2025 with a maximum temporal span of 24 years.

2. A systematic evaluation of multimodal fusion. The paper compares basic fusion (a multilayer perceptron over concatenated embeddings), gated fusion (a learned scalar gate weighting the two modalities), and a mixture-of-experts (MoE) design, benchmarked against embedding methods (MLP, DeepWalk, CLIP, Vision Transformer), GNNs (GraphSAGE, Graph Transformer, DCRNN, GCN, SupConGCN, GIN), and a supervised contrastive variant.

3. A causal analysis built on the learned embeddings. Using nearest-neighbor matching in the multimodal embedding space, plus propensity score matching and a doubly robust estimator, the authors estimate the average treatment effect on the treated (ATT) for precipitation, road type, and season.

4. Ablation and transfer studies. Leave-one-out feature ablations quantify the contribution of imagery, weather, traffic volume, and road network features, and a cross-state experiment tests how well a model trained in one state transfers to another.

Main Findings

• Multimodal fusion beats graph-only models. Integrating both modalities achieves an average AUROC of 90.1%, a 3.7% gain over GNNs that use only graph structure. The best per-state AUROC for GIN + MoE was 96.4% in Delaware, 94.5% in Nevada, 88.7% in Maryland, 88.1% in Massachusetts, 88.0% in Iowa, and 87.5% in Montana.

• Gains are larger for weaker base architectures. GCN plus Gated Fusion improved average AUROC by 3.8% over standard GCN, with a maximum gain of 7.2% across states; GCN plus MoE improved by an average of 3.9%, up to 7.7%. For the stronger GIN backbone, Gated Fusion improved AUROC by an average of 2.3% with a maximum of 6.7%, and MoE improved it by an average of 3.6% with a maximum of 7.9%.

• Road network features matter most in ablation. In the leave-one-out study on GCN plus Gated Fusion, removing road network features caused the largest drop (3.7%), followed by image features (3.5%), traffic volume (2.4%), and weather features (1.8%). Adding speed limit as an extra feature gave a small but consistent improvement.

• Causal estimates vary by factor and by state. The reported ATT scores are 28.6% for seasonal variation, 21.9% for road type, and 24.2% for precipitation (summarized in the abstract as roughly 29%, 22%, and 24% respectively). Seasonal effects were higher in colder states such as Montana and Iowa and much lower in Nevada, where winter conditions are mild.

• Motorways carry the highest average accident frequency. In the causal framing, treating motorways as the treatment group and other road types as control gives an ATT of 21.9%. Secondary, residential, and primary roads show the highest total accident counts, which the authors attribute to their broad coverage in the network.

• Residential roads account for roughly 74.5% of all roads across the six-state networks, and there are 14 road types in total.

• Hyperparameter sensitivity is limited. Varying GCN depth from 2 to 10 layers changed AUROC by no more than 3%, and training performance plateaued around epoch 30 when epochs ranged from 15 to 50.

• Fusion adds little runtime. For example, one month of prediction took GIN 6.2 seconds in Delaware versus 6.7 seconds for GIN + MoE, and 53.1 seconds in Massachusetts versus 54.5 seconds for GIN + MoE.

• Per-road-type accuracy varies. On Delaware, GIN + MoE reached 91.8% AUROC on Residential, 96.3% on Living street, 94.4% on Road, but only 79.0% on Primary, 78.7% on Trunk, and 80.0% on Motorway.

• Cross-state transfer is strongest for Maryland and Nevada, whose structural and visual features align well with those of other regions in the shared representation.

Methodology in Plain English

The authors model a state's road system as a directed graph where nodes are intersections or key points along road segments and edges are road segments. Each node gets an image: a 1024 × 1024 satellite picture centered at its coordinates, obtained either directly from the Mapbox Static API or by stitching 25 smaller tiles together. Nodes also carry weather statistics (temperature, precipitation, wind speed, atmospheric pressure), and edges carry attributes such as length, road type, and Average Annual Daily Traffic where available.

Accident records come from each state's Department of Transportation and are normalized into a common schema, since states differ in file format, field names, and spatial encoding. The prediction task is edge-level: given a sequence of past monthly graph snapshots, predict whether an accident occurs on each edge in the next month. Train/test splits are temporal, with data up to a cutoff year used for training to avoid leaking future information.

Two encoders produce embeddings. A GNN (tested in several variants including GIN and GCN) learns network embeddings by iteratively aggregating neighbor information. A vision model (CLIP or Vision Transformer) encodes each satellite image. Three strategies then combine the two: a plain multilayer perceptron on the concatenated vectors, a gated fusion that learns how much weight each modality deserves, and a mixture-of-experts where several expert networks each combine features from a different perspective and a gating network computes a softmax weighting over experts per node.

For causal analysis, the authors treat factors such as heavy precipitation, motorway road type, and winter season as binary "treatments." Because only one potential outcome is observed per road segment, they approximate the missing counterfactual by finding each treated segment's nearest neighbor in the multimodal embedding space among untreated segments and using that neighbor's accident count. This yields the average treatment effect on the treated. They also compute propensity score matching and a doubly robust estimator to check stability.

Why This Matters

For research, the paper supplies a publicly released dataset and code (https://github.com/VirtuosoResearch/MMTraCE) that makes multimodal transportation research reproducible at a scale earlier accident datasets did not support. The finding that simple concatenation of modalities underperforms gated and mixture-of-experts fusion is a useful signal for anyone building multimodal models where one modality is high-dimensional imagery and the others are low-dimensional tabular features. Using learned embeddings as the matching space for causal inference also offers a template for confounding control beyond traffic safety.

Real-world applications:

• Identifying high-risk road segments for proactive intervention, such as signage, lighting, or speed limit changes, before crashes accumulate.
• Prioritizing infrastructure spending by combining predicted risk with causal estimates of which road types and weather conditions drive accidents.
• Weather-responsive safety planning, since precipitation is estimated to raise accident rates substantially and seasonal effects differ sharply between cold and mild states.
• Transferring models to states with sparse data, using the cross-state generalization results to gauge whether a model trained elsewhere can be applied locally.

Industry relevance spans transportation agencies and departments of transportation that maintain crash databases, mapping and location-intelligence providers whose imagery feeds such pipelines, insurers assessing geographic risk, and logistics and fleet operators routing around high-risk corridors.

Future Directions

• Understanding cross-state transfer patterns. The authors explicitly flag the reasons why Maryland and Nevada transfer well to other states as an open question for future work.
• Improving accuracy on high-speed roads. Performance was consistently lower on Primary, Trunk, and Motorway segments than on Residential and Living streets, suggesting models still miss something about fast-moving traffic environments.
• Extending and refining causal estimation. The reported ATT for precipitation is truncated in the provided content, and the doubly robust and propensity-score results are described only briefly, leaving room for deeper sensitivity analyses of confounders.
• Broadening geographic and temporal coverage. The dataset covers six states with varying record start years (from 2002 in Massachusetts to 2016 in Nevada and Montana), so generalizing to other states and aligning heterogeneous record windows remain natural next steps. Notably, the provided statistics report no traffic volume availability figure for Iowa.

Target Audience

This paper is best suited to machine learning researchers working on multimodal representation learning and graph neural networks, transportation engineers and road safety analysts, and causal inference practitioners interested in embedding-based matching. It will also be useful to data scientists in mapping, insurance, and mobility companies who need to understand what satellite imagery adds to structured road data, and to researchers seeking a large benchmark dataset for accident prediction.

Authors’ abstract

We consider analyzing traffic accident patterns using both road network data and satellite images aligned to road graph nodes. Previous work for predicting accident occurrences relies primarily on road network structural features while overlooking physical and environmental information from the road surface and its surroundings. In this work, we construct a large multimodal dataset spanning six U.S. states, containing nine million traffic accident records from official sources, and one million high-resolution satellite images for each node of the road network. Additionally, every node is annotated with features such as the region's weather statistics and road type (e.g., residential vs. motorway), and each edge is annotated with traffic volume information (i.e., Average Annual Daily Traffic). Utilizing this dataset, we conduct a comprehensive evaluation of multimodal learning methods that integrate both visual and network embeddings. Our findings show that integrating both data modalities improves prediction accuracy, achieving an average AUROC of $90.1\%$, a $3.7\%$ gain over graph neural network models that use only graph structures. With the improved embeddings, we conduct a causal analysis using a matching estimator to identify the key factors influencing traffic accidents. We find that accident rates rise by $24\%$ under higher precipitation, by $22\%$ on higher-speed roads such as motorways, and by $29\%$ due to seasonal patterns, after adjusting for other confounding factors. Ablation studies confirm that satellite imagery features are essential for achieving accurate prediction.