MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes

AI summary

Overview

Research area: Machine learning evaluation — specifically agentic tool-use benchmarking, structured-output evaluation, and parameter-efficient fine-tuning (PEFT) for small language models.

Technical level: Advanced.

Scope: This paper introduces a 955-case benchmark that tests whether language models can serve as the policy layer of a passenger-facing transit kiosk, and reports a twenty-three-model leaderboard plus a four-size PEFT distillation sweep.

What This Paper Is About

Transit kiosks normally encode fare rules, route topology, and disruption responses as programmed state machines, so any policy change (a station closure, a new fare bracket, a holiday schedule) must pass through a code deployment cycle. This paper asks whether a language model can replace that policy logic by reading a natural-language system description, calling structured tools, and emitting a machine-renderable terminal state the kiosk hardware can act on. To answer it, the author builds MetroLLM-Bench, a benchmark spanning six real metro systems, and evaluates twenty-six models — including four fine-tuned "student" models — on how reliably they make correct, renderable, and auditable kiosk decisions.

Key Contributions

1. The benchmark itself: 955 cases across six metro systems (MARTA, Doha Metro, BART, Taipei MRT, CTA, Beijing Subway), eleven task categories, and a two-tier scorer in which fourteen deterministic components (Tier 1) are kept separate from eight semantic-quality components (Tier 2), six of which use a language-model judge.

2. Measurement discipline: a system-stratified 75/25 train/held-out split (717 cases for training-data generation, 238 held-out, seed=42) fixed before any training, plus a calibration of the deployed scoring stack against two independent human annotators on 100 case-rubric pairs.

3. A twenty-three-model leaderboard covering local open-weight models served through llama.cpp and proprietary API models, with a documented serving-configuration sensitivity analysis showing that a single global configuration is not neutral across model generations.

4. A four-size PEFT sweep at two to three independent training seeds (2B, 4B, 9B, 27B), which identifies a monotonic decline in adapter utility as base capability rises, ending in a negative delta at 27B.

Main Findings

• A 4B student matches frontier Tier 1 performance at a fraction of the footprint. On the 238-case held-out partition, a 4B Qwen 3.5 student trained with PEFT scores 91.32 on Tier 1, above GPT-5.6 luna at medium effort (90.63) and GPT-5.6 sol at xhigh effort (90.00), and within 0.05 points of GPT-5.4 full at xhigh effort (91.37). Its Q4_K_M build is 2.6 GB.

• Bigger students do not help. On held-out Tier 1, the 4B, 9B, and 27B students score 91.32, 91.03, and 91.41 — a range of 0.38 points. The PEFT gain over the corresponding base model declines from +7.03 points at 2B to −0.91 at 27B, and every training seed agrees on the direction at every size.

• Composite ranking is compressed and led by an open-weight model. Muse Glimmer 30B ranks first on the composite at 92.03. The top eleven rows span 3.18 composite points and no adjacent gap exceeds 0.68 points. On the deterministic tier the order differs, with Qwen3.6-27B leading at 93.63.

• Reasoning effort matters more than model size within the GPT-5.4 family. Moving GPT-5.4 full from medium to xhigh raises its composite by 2.73 points; high to xhigh raises it by 2.25. At medium effort, full, mini, and nano span only 1.35 points. GPT-5.6 does not repeat the pattern: sol at xhigh trails luna at medium by 0.75 composite points.

• The language-model advantage is concentrated in decision-making categories. A deterministic scripted agent that chains route_planner, fare_calculator, and submit_assistant_state scores 77.1 composite and 84.6 on Tier 1. It performs well on Routing (93.5) and Fare (91.7) but drops sharply on Temporal (52.1), Compound Stress (68.9), Accessibility (69.7), and Policy (73.3). Against GPT-5.4 full at xhigh, Qwen 27B base leads Accessibility by 13.5 points (91.1 versus 77.6, n=20), while GPT-5.4 xhigh leads Temporal by 13.7 (87.2 versus 73.5, n=22).

• No model dominates the category breakdown. No model wins more than three of the eleven categories. The 4B PEFT student posts the deployment cluster's best Fare score (97.7), shares the best Policy score (96.9) with GPT-5.6 sol, and trails only sol on Routing (97.2 against 97.7).

• Serving configuration can masquerade as capability. Under a uniform greedy-decoding, 4096-token configuration, Qwen3.8-27B appears to regress 3.60 Tier 1 points against Qwen3.5-27B (89.48 versus 93.08). Roughly 2.7 of those 3.6 points are configuration: 1.72 from raising the output budget to 16384, and a further 1.00 from vendor-recommended sampling. A larger budget is not uniformly better — given 16384 tokens, Muse Glimmer drops from 92.75 to 92.38 and GLM-4.7-Flash from 89.67 to 89.62, falling further to 88.58 at 32768 tokens and forty tool rounds.

• Judge–human agreement exceeds human–human agreement. On 100 case-rubric pairs, judge-versus-author agreement is κw = 0.53 [0.27, 0.75], while agreement between the two human annotators is κw = 0.25 [0.02, 0.46]. Gwet's AC2 places all three pairings between 0.81 and 0.89. The second annotator's near-zero κw = 0.02 is attributed to a prevalence artefact.

• Held-out intervals are wide. On the 238-case held-out partition, the 4B PEFT gain is 1.97 Tier 1 points [−0.17, +4.16], the 27B regression is −0.94 [−2.26, +0.39], and the 4B student versus GPT-5.4 full xhigh is −0.05 [−1.80, +1.70] — all including zero. On the full 955-case matrix the 4B gain is +1.72 [+0.72, +2.74] and the 27B regression is −1.09 [−1.82, −0.38].

• Three models were excluded for budget exhaustion. Gemma 4 E2B and E4B variants exhaust the twenty-round budget without a valid submit_assistant_state on 28 to 33 percent of cases; Llama 3.1-8B is excluded on the same basis. The excluded Gemma variants achieve 99.9 percent scope adherence and 100 percent no-tool-hallucination, but roughly half their submission attempts receive HTTP 422 validation errors — a structured-state synthesis failure, not a tool-selection failure.

• Seed sensitivity also shrinks with size. On the full matrix, the seed spread is 3.96 points at 2B, 0.33 at 4B, 0.17 at 9B, and 0.09 at 27B.

Methodology in Plain English

The unit of evaluation. Each case pairs a "framebook" — a natural-language description of a metro system's terminology, currency, operating hours, and cultural conventions — with scenario events such as origin, destination, passenger count, an optional disruption, and optional free text. These are assembled into a system prompt. The model then enters a ReAct-style loop with native function calling and a budget of twenty tool rounds, during which it may call six tools: route_planner, fare_calculator, station_info, line_info, disruption_feed, and knowledge_base.

The terminal state. A case ends when the model calls submit_assistant_state, which must emit one of five outcomes (route_and_fare_ready, advisory_only, service_unavailable, request_declined, or policy_answer_only), a kiosk action with a reason code, a passenger-facing message, and, when applicable, a per-ticket fare quote with line items. A mock server validates this structure with Pydantic; inconsistent submissions get an HTTP 422 with field-level errors that the model may correct within its remaining round budget.

Scoring. Fourteen Tier 1 components are deterministic — route and fare correctness, tool-call accuracy, renderable-state validity, outcome and reason-code correctness, and others — and double as the PEFT reward signal. Eight Tier 2 components cover semantic quality; six use Anthropic's Claude Haiku 4.5, and every language-model judgment is cached to disk. The composite score is the percentage of available points earned across both tiers.

Case generation. Category-specific templates are combined with per-system station-pair and disruption metadata, and a generator script derives route and fare ground truth from the benchmark graph and fare engines. Generation-time tests check required fields, unique identifiers, graph-valid paths, and exact fare consistency for the Fare category. An independent annotator validated a stratified 50-case sample of the committed answer key.

Distillation. The four students were trained from 600 deduplicated traces drawn only from the 717-case training partition, keeping traces where a Qwen 3.5 27B-dense or 35B-A3B teacher scored at least 90 percent on Tier 1 (540 examples from the 27B teacher, 60 from the 35B, mean Tier 1 score 99.0 percent). All students used QLoRA at rank 16 for three epochs, batch size 2 with gradient accumulation 4, on one NVIDIA RTX 5090. Maximum sequence length was 4096 tokens for 2B, 4B, and 9B, and 2048 for 27B, limited by 32 GB of VRAM. Total training required 9.4 GPU-hours; individual runs took 27 minutes at 2B and 103 minutes at 27B.

Serving. Local models ran through llama.cpp at Q4 to Q8 GGUF quantisation; OpenAI rows ran through Azure OpenAI via the Responses API. Significant effort went into documenting that per-model serving settings (output budget, sampling parameters) materially affect scores, so rows marked in the tables were served at per-model configurations, with the caveat that those configurations were selected on the same held-out cases being ranked.

Why This Matters

Impact on research. The paper argues that transit kiosk policy is a useful, bounded testbed for a broader question about prompt-driven runtimes: can a language model replace code-defined logic when the output must be correct, renderable, adaptable, and auditable? It also contributes a methodological caution — that a single global serving configuration converts configuration mismatch into apparent capability differences, worth up to 2.7 Tier 1 points in the Qwen 3.5-to-3.8 comparison here — and it documents a capacity ceiling at which PEFT gains change sign, which previous distillation work (FireAct, Jhandi et al.) did not identify.

Real-world applications:

• Deploying fare and routing policy updates at a kiosk without a code release cycle, by editing a framebook instead.
• Running passenger-facing kiosk inference locally on operator-controlled hardware: the 4B student ships in 2.6 GB of Q4_K_M, versus 16 GB for the 27B student.
• Serving lower-income or low-connectivity systems where a proprietary frontier API is not affordable — the paper notes GPT-5.6 luna, OpenAI's budget tier, ranks fifth and only two of the six highest-ranked rows are proprietary.
• Validating that a model follows supplied rules rather than recalling a network from pretraining, which the paper states is the benchmark's design intent (with one documented exception in Appendix F).

Industry relevance. The result that a 2.6 GB open-weight model adapted with PEFT delivers Tier 1 parity with frontier APIs is directly relevant to operators who need an auditable, self-hosted deployment stack. The paper is explicit that this is a parity claim on deterministic task performance, not a claim of superiority on the composite score or across every category.

Future Directions

• Cross-judge calibration with a non-Anthropic model remains, in the paper's own words, future work, given that six Tier 2 components rely on Claude Haiku 4.5 and the calibration set was stratified across those rubrics.

• Resolving the safety_response_quality disagreement. Three adversarial scenic-route cases (H-014 for TRTC, MARTA, and CTA) produced a systematic split: the judge penalises agents offering a scenic route despite a system-prompt prohibition, while both human annotators credit the agent for returning a valid route. The underlying question — strict constraint adherence versus outcome utility — is left open.

• Testing whether the capacity ceiling moves. The paper limits its conclusion to the exact recipe tested (QLoRA rank 16, three epochs, 600 high-quality teacher traces) and notes that larger or differently composed datasets, other ranks, and other learning rates may shift the point at which PEFT stops helping. It also does not test whether using a datacenter accelerator to raise the 27B student's sequence length from 2048 to 4096 would alter the 27B delta.

• Establishing an end-to-end latency requirement. Appendix B reports exploratory Apple Silicon inference measurements, but the benchmark does not establish a latency requirement for kiosk deployment. A temperature-zero cross-check of the API rows is also noted as requiring a different model configuration.

Target Audience

Researchers and engineers working on agentic tool-use evaluation, structured-output reliability, and parameter-efficient fine-tuning of small language models will get the most from this paper. It is also directly useful to practitioners building self-hosted, auditable AI runtimes for regulated or public-facing settings — transit operators, kiosk and ticketing vendors — and to anyone who needs a worked example of calibrating a language-model judge against human annotators and reporting confidence intervals honestly, including where those intervals do not resolve the differences being claimed.

Authors’ abstract

We introduce MetroLLM-Bench, a 955-case benchmark for testing language models as the policy layer of a transit kiosk. It covers six real metro systems, ranging from 37 to 414 stations, and eleven categories that include routing, fare calculation, disruptions, accessibility, and adversarial input. In each case, the model must call structured tools and submit a machine-renderable terminal state containing an outcome, a per-ticket fare quote when applicable, and a kiosk action. Fourteen deterministic scoring components form Tier 1; eight semantic-quality components form Tier 2, six of which use a language-model judge. We report Tier 1 and the combined score of both tiers. A stratified 75/25 split reserves 717 cases for training-data generation and 238 for held-out evaluation. We evaluate twenty-six models from six vendors, of which twenty-three are ranked. On the held-out partition, a 4B Qwen 3.5 student trained through parameter-efficient fine-tuning (PEFT) exceeds both GPT-5.6 tiers on Tier 1 (91.3 against 90.6 and 90.0) and matches GPT-5.4 full at maximum reasoning effort (91.4), with a 2.6 GB Q4_K_M footprint. Larger 9B and 27B students provide no further Tier 1 improvement over the 4B student at this training scale. Across the four Qwen sizes, the PEFT gain over the corresponding base model decreases from +7.03 points at 2B (three training seeds) to -0.91 at 27B; every seed shows the same direction at every size. A deterministic rule-based baseline reaches 84.6 on Tier 1, with the remaining language-model advantage concentrated in policy adaptation, compound scenarios, accessibility, and temporal reasoning. Muse Glimmer 30B leads the composite ranking, and serving configuration alone moves the Qwen 3.5-to-3.8 comparison by 2.7 Tier 1 points. The benchmark, harness, reproduction guide, and fine-tuned students are released at https://github.com/continker/metrollm-bench.