STRIDE: Automated Evaluation of Text-to-Trajectory Alignment across Diverse Contexts

AI summary

Overview

Research area: Language-conditioned pedestrian trajectory generation and its evaluation; pedestrian sociology, crowd dynamics, and LLM-based benchmarking.

Technical level: Intermediate. The core ideas are conceptual (behavioral decomposition, deterministic measurement functions), but the paper uses trajectory metrics, per-axis aggregation, and statistics such as Cohen's κ and Krippendorff's α.

Scope: The paper proposes STRIDE, a framework that scores how well generated pedestrian trajectories match the scenario described in text, and instantiates it as the crowd-domain benchmark STRIDE-Bench.

What This Paper Is About

Text-conditioned models can now generate pedestrian trajectories from natural-language descriptions of a scene, but evaluating whether those trajectories actually match the described context still depends on collecting real human trajectories for every scenario, which is expensive and impossible for rare events such as evacuations or violent incidents. Existing metrics (e.g., ADE/FDE, collision rate) compare trajectories against reference data rather than testing whether a trajectory reflects the semantic content of a text prompt, and LLM-as-judge and human evaluation are unstable or slow. The paper's goal is a complete, verifiable, automated, and scalable evaluation framework for text-to-trajectory alignment that requires no human trajectory data for the target scenario.

Key Contributions

1. A sociological evaluation protocol (V R D S T). A five-axis protocol — Velocity, Realism, Direction, Spatial, Temporal — derived from pedestrian sociology (notably McPhail and Wohlstein's dimensions of gathering behavior and the EFCA framework) and pedestrian dynamics theory (e.g., Hall's personal space, Helbing's work on normal and evacuation dynamics). The protocol is described as a complete, layered evaluation space covering individual, group, and environment layers.

2. Scenario-adaptive behavioral decomposition. Rather than weighting all axes equally, STRIDE prompts an LLM with the scenario description and the V R D S T protocol to generate scenario-specific evaluation questions (at least five per scenario), each with a justification for traceability.

3. A Deterministic Measurement Tool (DMT) library and verifiable answer space. Twenty deterministic measurement functions operate directly on trajectory coordinates and each is assigned to a protocol axis. For each question, a frontier LLM selects relevant DMT functions and converts the semantic expectation into numerical answer ranges, using a human-trajectory knowledge base called TrajFacts (calibrated with references from events including the Love Parade and the Itaewon crowd crush). Verdicts are therefore reproducible, numerically traceable, and scale-invariant across horizons and agent counts.

4. STRIDE-Bench, a crowd-domain benchmark. The abstract reports 1K scenarios, 6K behavioral questions, and 11K measurements with calibrated expected answers across 30 real-world maps; the body reports totals of 936 scenarios, 6,633 questions, and 11,696 measurements, built over 113 curated Google Maps venues (fixed size 301.7 m × 282.8 m) using Berlonghi's 11 crowd categories, with scenario descriptions from GPT-5.1 and behavioral questions, measurements, and expected answers from GPT-5.2. STRIDE-Bench supports a benchmark mode (deterministic scoring with no LLM at scoring time) and an open mode (questions generated on the fly for user-supplied scenarios).

Main Findings

• Real human trajectories score highly. On 12 recordings from the Fête des Lumières in Lyon (86 questions, 154 measurements), real human trajectories achieve an average STRIDE score of 0.94, with per-recording scores from 0.87 to 1.00. The authors state a perfect score is not expected because expected ranges come from textual descriptions and calibrated references.

• STRIDE-Bench agrees with human judgment. In a Prolific annotation study with n = 50 annotators (paid $14/hour, at least three scenarios each), the analysis covered 15,750 annotations across 56 scenarios and 350 questions. Annotators agreed with STRIDE-Bench on 80% of question–answer pairs (Cohen's κ = 0.730), and pairwise inter-annotator agreement reached 66% (Krippendorff's α = 0.698).

• Judgments are stable across LLM families. Four state-of-the-art LLMs (deepseek-v3, claude-sonnet-4.6, qwen3.6-plus, gpt-5.5) were evaluated on n = 588 items against the consensus human label; all four fell within the leave-one-out human range on both raw agreement and Cohen's κ, with tightly clustered scores.

• Current text-to-trajectory models have limited context alignment. Overall STRIDE-Bench scores: Text-Crowd 0.645 ± 0.233 (best, and highest in every crowd category), LLM-SFM 0.434 ± 0.231, SingularTrajectory 0.421 ± 0.177, Random Walk 0.311 ± 0.117, and Stop 0.221 ± 0.105. The abstract describes the best baseline as achieving 0.64.

• Text input is not the only signal. SingularTrajectory, a diffusion-based predictor that uses no text input, performs competitively with LLM-SFM, suggesting trajectory history alone captures part of the scenario context.

• Model ranking depends on rollout horizon. LLM-SFM performs best in short-to-medium horizons (mean 0.574 ± 0.190 in 0–30s; 0.632 ± 0.228 in 30–120s), while Text-Crowd is stronger at long horizons (mean 0.463 ± 0.217 for 120s+), suggesting better preservation of high-level intent over extended rollouts.

• Text-Crowd leads across crowd sizes. It scores best in all size regimes, with mean 0.638 ± 0.185 (small, 1–25), 0.640 ± 0.191 (medium, 26–100), 0.638 ± 0.172 (large, 101–300), and 0.645 ± 0.161 (mass, 301+). LLM-SFM improves most in mass-gathering settings (mean 0.635 ± 0.324), and SingularTrajectory also benefits from larger crowds.

• On the Lyon scenes, the human–model gap persists. Average STRIDE scores are Human 0.943, LLM-SFM 0.584, Text-Crowd 0.411, and Random Walk 0.317; Text-Crowd underperforms LLM-SFM here, which the authors attribute to sensitivity to complex map layouts.

• Failures concentrate in specific behavioral dimensions. Fine-grained scores reveal persistent difficulty in complex behavioral requirements including spatial reasoning and fine-grained context conditioning.

Methodology in Plain English

STRIDE separates semantic judgment from numerical verification. First, a social-science-grounded protocol fixes what "context alignment" means along five axes: whether people move at speeds appropriate to the scenario (Velocity), whether they behave physically feasibly without overlaps, wall penetrations, or impossible accelerations (Realism), whether their headings match what the scenario implies (Direction), whether they occupy plausible regions and form expected spatial patterns (Spatial), and whether these properties evolve coherently over sliding time windows (Temporal).

Second, for each scenario description, an LLM writes a short set of scenario-specific behavioral questions in natural language, guided by the protocol, with justifications. Third, each question is answered not by an LLM judge but by deterministic functions from the DMT library that compute exact statistics from trajectory coordinates; the LLM only converts the semantic expectation into numeric answer ranges, calibrated against the TrajFacts reference knowledge base. A trajectory's computed values are compared against those ranges to produce binary verdicts, which are averaged hierarchically — measurements within a question, questions within a scenario, scenarios across the benchmark.

For benchmark construction, the authors curated 113 high-traffic public venue maps through Google Maps, converted to obstacle representations via semantic segmentation of the Google Maps color scheme, polygon approximation, and rasterization into obstacle masks; they generated scenario descriptions across Berlonghi's 11 crowd categories, filtered out low-density, obstacle-conflicting, and near-duplicate scenarios (using cosine similarity over sentence embeddings), manually reviewed the remainder, and sampled more heavily from safety-critical categories such as Escaping, Violent, and Dense. Evaluation compared LLM-SFM, Text-Crowd, SingularTrajectory, Random Walk, and Stop.

Why This Matters

Impact on research. STRIDE reframes trajectory evaluation from comparison against reference human data to verification of scenario-conditioned behavioral claims. Because scoring uses fixed deterministic functions, results are claimed to be independent of prompts, runs, and judge-model versions — addressing the instability of LLM-as-judge and the cost of human evaluation. The authors present it as a first step toward principled evaluation of context-aligned pedestrian trajectory generation and release code and a dataset.

Real-world applications:

• Urban planning and public-space accessibility assessment, where pedestrian trajectories are used to study how people move through designed spaces.
• Venue safety auditing and emergency preparedness, including evacuation and mass-gathering scenarios that are heavily sampled in STRIDE-Bench.
• Embodied-agent development, where human-aware behavior is learned and validated before real-world deployment.
• Simulation of rare, ethically unstageable events (e.g., panic egress, violent confrontation) for which no real trajectory datasets exist.

Industry relevance. Any team building or deploying language-conditioned trajectory generators — for robotics, traffic simulation, autonomous driving, or crowd simulation — needs a scoring signal that does not require collecting new human data per scenario. The deterministic scoring path makes benchmark results comparable across model versions, and the open mode allows evaluation of scenarios that are not part of the released benchmark.

Future Directions

• Improving calibration with more real event data. The authors state that STRIDE-Bench's calibration is limited by the availability of real-world event data, and that the benchmark can be refined as more event-based trajectory statistics are collected and added to the TrajFacts knowledge base.
• Expanding the measurement library. The current DMT library covers 20 functions and, per the limitations section, does not cover all behavioral factors; future extensions mentioned include richer measurements for social relations (the limitations text is truncated at this point).
• Closing the behavioral gaps identified by fine-grained scoring. STRIDE's traceable per-axis and per-question breakdowns are presented as a way to identify underperforming behavioral dimensions and inform model development, especially for spatial reasoning and fine-grained context conditioning.
• Extending beyond the crowd domain. STRIDE is a general framework instantiated only in the crowd setting, leaving open whether the V R D S T protocol and DMT approach transfer to other trajectory domains.

Target Audience

Researchers and practitioners working on language-conditioned trajectory generation, crowd simulation, and embodied agents; benchmark and evaluation-methodology researchers interested in replacing LLM-as-judge with deterministic, traceable scoring; and safety, urban-planning, and venue-auditing teams who need to assess scenario-conditioned pedestrian behavior without collecting matched human trajectory data.

Authors’ abstract

Language-conditioned trajectory generation is here, but its evaluation has not kept pace. Existing pedestrian trajectory metrics compare trajectories with real-world human data. This does not scale to text-to-trajectory generation across diverse contexts, as collecting human trajectories for every scenario is costly and infeasible. Moreover, pedestrian behavior is heterogeneous and context-dependent, with no single metric as the correct answer, and current evaluation frameworks are not transferable to this domain. These challenges make scalable, reliable evaluation difficult. We introduce STRIDE, the first framework for evaluating context alignment between scenario descriptions and pedestrian trajectories. STRIDE addresses these challenges through three design choices. First, we derive our VRDST evaluation protocol from sociological theories to define a complete evaluation space. Second, it decomposes high-level context into scenario-adaptive behavioral questions. Third, every question is resolved against a deterministic measurement tool library that yields reproducible answers. Together, STRIDE enables complete, verifiable, automated, and scalable evaluation across diverse contexts without requiring human trajectory data. We instantiate STRIDE in the crowd domain as STRIDE-Bench, comprising 1K scenarios, 6K behavioral questions, and 11K measurements with calibrated expected answers across 30 real-world maps. Comprehensive human validations show that STRIDE-Bench is consistent with human behavior and judgment, achieving 80% human agreement. We further evaluate several text-to-trajectory models, finding limited context-alignment capability and persistent challenges in fine-grained context conditioning. We believe that the STRIDE framework provides a first step toward principled evaluation of context-aligned pedestrian trajectory generation.