teLLMe Why (Ain't Nothing but a Jam): Exploratory Causal Analysis of Urban Driving Data

AI summary

Overview

Research area: Causal inference and causal discovery applied to video-derived urban driving data, combined with LLM-based natural-language query interfaces.

Technical level: Intermediate. Readers will get the most from this paper with basic familiarity with causal graphs (DAGs), treatment/outcome framing, and regression-based effect estimation, though the paper explains each component conceptually.

Scope (one sentence): The paper presents teLLMe, a query-driven pipeline that turns dashcam-derived event tables into structured causal questions, estimates treatment effects with backdoor adjustment using OLS and DoWhy, and reports results in an interpretable "Causal Card."

What This Paper Is About

Traffic agencies now hold large volumes of observational video-derived data, but because that data is collected without interventions, causal questions such as "how would rain change traffic density?" are hard to answer cleanly. The authors build teLLMe to connect the entire analysis chain: aggregating dashcam annotations into event-level features, learning a plausible causal graph, selecting an adjustment set per question, estimating average treatment effects, and explaining the result to non-specialists. The system is explicitly framed as a tool for hypothesis generation and expert reasoning, not as a source of definitive causal claims.

Key Contributions

1. An integrated, query-driven causal workflow. teLLMe couples an offline discovery phase (event table construction, causal graph learning, stability scoring) with an online query phase (natural-language parsing, adjustment set selection, effect estimation, explanation) into a single reproducible pipeline.

2. A schema-aware LLM interface that grounds free text in a fixed data schema. The LLM receives the dataset schema, including variable names, types, and allowed categorical values, and must output a structured Causal Query Q = (T, t_treated, t_control, Y, C) in constrained JSON. The output is validated with strict type checking; undefined variables and invalid categorical values are rejected, with an optional fallback to a rule-based parser for some templates.

3. Bootstrap-based stability scoring of the learned graph. The PC algorithm is run with domain constraints (including forbidden edges, such as disallowing edges from traffic_density to weather) and then re-run on B = 20 bootstrap resamples, producing a stability score s(e) ∈ [0,1] for each directed edge that is exposed to the user.

4. The "Causal Card" as a communication artifact. Each query returns a compact summary containing the parsed variables, effect estimate and confidence interval, treated/control sample sizes, any direct T → Y edge and its stability, relevant caveats, and a short natural-language explanation generated from these fields.

Main Findings

• Rain at urban intersections during peak hours showed a small negative association with traffic density. For the query comparing rainy versus clear weather, restricted to urban intersections during peak hours, the adjustment module selected time of day and total objects. The OLS effect was −0.036 with a 95% confidence interval of [−0.047, −0.024], based on 1,840 rainy windows and 7,681 clear windows.

• Peak hours on highways under clear weather showed a small positive associated effect. For the peak-hour query restricted to scene_type = highway and weather = clear, the adjustment module selected time_of_day, weather, and total_objects. The OLS effect was 0.024 with a 95% confidence interval of [0.014, 0.034], based on 4,268 peak-hour windows and 6,154 off-peak windows.

• The outcome distribution differed between the two groups in that highway case. Reported summary statistics for traffic density were: off-peak windows (n = 6,154) with mean 0.3827, std 0.2466, p25 0.20, median 0.35, p75 0.50; peak windows (n = 4,268) with mean 0.5291, std 0.2835, p25 0.30, median 0.50, p75 0.75. Both groups ranged from a minimum of 0 to a maximum of 1.

• Neither case study's learned DAG contained a direct treatment-to-outcome edge. In the weather case, the learned DAG did not contain a direct weather → traffic density edge; in the peak-hour case, it did not contain a direct is_peak_hour → traffic_density edge. Effects were still estimated using the selected adjustment sets, which is a notable modeling choice the paper surfaces.

• Adjustment and sampling choices materially changed the estimates. Comparing DAG-based adjustment against a fixed adjustment set, and repeating analyses on the imbalanced dataset, the authors report that ignoring the DAG sometimes produced larger effects and narrower intervals, consistent with under-adjustment, and that analyses on the imbalanced data occasionally produced more extreme estimates and greater uncertainty. Specific numeric values from the sensitivity analyses are not reported.

• The system is positioned as hypothesis generation, not causal proof. Because the data is observational, all effects depend on assumptions encoded in the learned DAG and the selected adjustment sets.

Methodology in Plain English

The pipeline begins offline. Raw dashcam detections and metadata are aggregated into fixed-length windows, each row carrying weather labels, scene type, traffic density, peak-hour status, and basic temporal indicators such as weekday/weekend and time-of-day bins. Traffic density is computed as the count of detected vehicles per minute. The authors drop windows with missing labels and clip extreme values to avoid outliers. Because some weather and scene-type combinations are rare, they build a balanced subset using stratified sampling for the main analyses and retain the full dataset for sensitivity checks.

On the balanced event table, the authors learn a causal graph over a curated variable set — weather, is_peak_hour, traffic_density, scene_type, weekday, and time_bin — using the PC algorithm as implemented in pgmpy. PC starts from a complete undirected graph and removes edges based on conditional independence tests, then orients the remaining edges with logical rules. Domain knowledge is injected as forbidden edges (for example, traffic_density → weather is disallowed) and optionally as required edges. Variables are discretized or encoded as needed for the independence tests. To check robustness, the authors draw 20 bootstrap resamples with replacement, run PC on each, and record how often each directed edge appears, yielding a stability score between 0 and 1 per edge.

Online, a user asks a question in natural language. A schema-aware LLM converts it into a structured query specifying a treatment, two treatment values (treated and control), an outcome, and conditioning constraints. The result is validated against the schema, and invalid queries are rejected or handled by a simple rule-based fallback.

Given the query and the learned DAG, the system picks a backdoor adjustment set: it starts from a candidate covariate list such as scene_type, weekday, and time_bin, and includes those that are parents of the treatment or the outcome in the graph while avoiding descendants of the treatment. This heuristic approximates a backdoor adjustment set and encodes the modeling choice about what to condition on. The event table is then filtered to the relevant subpopulation.

Effects are estimated two ways. First with ordinary least squares, fitting Y = α + βT + γᵀZ + ε with categorical variables one-hot encoded, where β is the estimate of the average treatment effect and its point estimate, standard error, and confidence interval are reported. Second, DoWhy builds a CausalModel from the filtered data, the treatment, the outcome, and a DOT version of the learned DAG, identifies a backdoor estimand, and estimates it via linear regression. The authors note DoWhy supports inverse-propensity and doubly robust estimators but they use linear regression for consistency with the OLS baseline. Optionally, the filtered data is bootstrapped to examine the spread of the estimated effect.

Finally, each query produces a Causal Card summarizing the parsed variables, the effect estimate and confidence interval, treated/control sample sizes, any direct treatment-to-outcome edge and its stability, and relevant caveats, followed by a short plain-language explanation generated from those fields.

Why This Matters

Impact on research. The paper contributes a concrete architecture for combining causal discovery, LLM-based schema-grounded query parsing, and effect estimation into an inspectable workflow for observational video data. Its emphasis on exposing adjustment sets, edge stability, and assumptions in a single artifact is a design stance about how causal results from messy, imbalanced sensor data should be reported.

Real-world applications.

• Traffic planning and congestion analysis: testing questions about how weather or peak-hour periods relate to traffic density on specific corridor types and at intersections.
• Urban safety analytics: the system is designed as part of Redddot, a broader platform for participatory urban safety and mobility analytics aimed at planners, researchers, and community stakeholders.
• Public-facing "what-if" exploration: Causal Cards can be surfaced alongside other Redddot views to show how candidate effects, uncertainty, and assumptions relate to specific locations and populations.
• Expert hypothesis generation: giving domain experts a structured way to interrogate video-derived evidence before committing to deeper analysis.

Industry relevance. Agencies and vendors sitting on large dashcam, CCTV, or mobile-derived video streams face exactly the problem the paper targets: abundant observational data but no experimental control. A tool that makes treatment, outcome, adjustment set, and stability explicit — while refusing to overclaim — maps onto how transportation analytics, insurance, and mobility platforms need to present quasi-experimental results to non-technical decision makers.

Future Directions

• Evaluate the Causal Cards with actual practitioners. The authors state plainly that they have not yet evaluated the cards with practitioners and do not know how analysts or planners interpret them or what guidance they find most useful.

• Model temporal and spatial structure. The system currently treats event windows as independent and models neither temporal nor spatial dependence, even though traffic patterns depend on both. Capturing these would require different causal discovery methods and richer data.

• Recover currently unobservable factors. Driver intent, road surface conditions, and weather severity are not observable in dashcam footage, so reported effects remain plausible explanations rather than definitive causal claims. Better proxies or additional data sources are needed.

• Develop more safety-oriented outcome measures. The paper notes traffic density is straightforward to compute, but safety surrogates such as near-miss measures are harder to derive reliably from video alone.

• Extend the broader Redddot integration. teLLMe is being developed as the causal reasoning and explanation module within Redddot, which raises open questions about how its queryable "what-if" analyses should mediate human–AI collaboration around urban decisions.

Target Audience

This paper is most useful for causal inference researchers and applied data scientists working with observational sensor or video data, and for transportation and urban analytics practitioners who need quasi-experimental answers from dashcam corpora. It is also relevant to human–AI interaction researchers interested in how causal assumptions and uncertainty are communicated through artifacts like the Causal Card, and to engineers building LLM-to-structured-query interfaces where hallucinated columns must be prevented. Readers without a causal inference background can follow the high-level workflow, but the adjustment-set and DAG-stability details assume some familiarity with graphical causal models.

Authors’ abstract

Traffic agencies now have access to large volumes of video-derived data for studying safety and congestion. Most of these data are observational and collected without interventions, which makes causal questions such as "How would rain change traffic density?" difficult to answer. We present teLLMe, a system for exploratory causal analysis of urban driving datasets. The system starts from a structured event table built from dashcam annotations and combines causal structure learning with the PC algorithm, bootstrap-based stability checks, and query-specific effect estimation using linear regression and DoWhy. Natural-language questions are mapped to structured causal queries through a schema-aware LLM, enabling users to specify treatments, outcomes, and subpopulations. teLLMe returns a "Causal Card" that summarizes effect estimates, adjustment sets, DAG support, and assumptions, followed by a short natural-language explanation. Case studies on BDD-derived traffic events show that the system can surface plausible relationships involving weather, peak hours, and traffic density, while making uncertainty and modeling choices explicit. The system is designed as a tool for hypothesis generation and expert reasoning rather than a source of definitive causal claims.