Reason or Recite: Complementary Specialists with an LLM Judge for Traffic Anomaly Reasoning

AI summary

Overview

Research area: Computer vision / video-language understanding, specifically traffic anomaly reasoning with vision-language models (VLMs), evaluated on Track 3 of the 2026 AI City Challenge.

Technical level: Advanced. The paper assumes familiarity with LoRA/QLoRA fine-tuning, BERTScore, chain-of-thought supervision, and LLM-as-judge selection.

Scope: A two-specialist system ("Reason-or-Recite", RoR) that separates evidence-grounded reasoning from annotation-style matching for ten heterogeneous traffic-video question types, arbitrated per question by a frozen LLM judge, which ranked first on the in-domain TAR leaderboard with a mean score of 0.6788.

What This Paper Is About

The Traffic Anomaly Reasoning (TAR) benchmark asks a single video-language model to answer ten question types over CCTV traffic footage, nine of which are scored. Five are closed-form (scored by exact match) and seven are open-ended (scored by BERTScore against a single reference answer), and the authors find these two regimes reward opposite behaviors: exact-match tasks reward careful, evidence-grounded reasoning, while BERTScore rewards reproducing the wording and content-selection policy of the machine-generated training annotations. The paper's goal is to build a system that stops conflating these two competences, by training separate specialists for each and then choosing between their answers per question.

Key Contributions

1. Structured reasoning enhancement (TAR-RE): An LLM-driven re-annotation of the official TAR chain-of-thought traces into a five-part, temporally grounded schema (<scene>, <query>, timestamped <evidence>, <reason>, <conclusion>), after auditing and documenting four defect classes in the original annotations. This substantially improves closed-form QA after fine-tuning.

2. Answer-distribution specialization: An answer-only QLoRA fine-tune of a larger 27B model that absorbs the auto-labeled annotation distribution directly and beats reasoning-tuned models on the BERTScore-scored open-ended tasks, showing that under single-reference scoring, matching the annotation distribution can outweigh gains in reasoning quality.

3. Evidence-gated per-question arbitration: A training-free LLM judge that selects between the two specialists per question. Unconstrained, the judge scores below the stronger specialist due to position and family bias; the conservative, winner-last design gains over both.

4. First place: Ranked 1st on the Track 3 in-domain (TAR) leaderboard with mean score 0.6788, including perfect binary event verification (1.000).

Main Findings

• Leaderboard win comes from breadth, not one task: RoR ranks 1st with mean 0.6788 (2nd: FPT AI Vision at 0.6696; 3rd: Smart Vision at 0.6669). It attains the best score on four of nine scored task types (BCQ 1.000, BCQ-OE 0.669, causal linkage 0.531, temporal event description 0.514) and stays within 0.025 of the per-column best on every other task.

• The official annotations are systematically flawed: Conservative phrase matching flags 10.1% of scene-description reasoning (about 1% overall) as citing sources the model cannot see; 66.1% of annotations contain at least one timestamp across four coexisting conventions, with 911 timestamps using a colon as a decimal separator and 113 gold answers mixing two conventions in one JSON object; 33.1% of traces (excluding scene description) anchor no observation to a timestamp, rising to 46.1% for binary questions whose gold answer is "No"; and mean trace length ranges only from 109 to 244 words, with token-level ROUGE-L between trace and answer reaching 0.33.

• TAR-RE produces structurally better supervision: After rewriting, over 99% of traces use the unified timestamp format, the median number of time-anchored observations rises from two to 16–30, the share citing no moment falls from 33.1% to 0.2%, and ROUGE-L overlap with the answer roughly halves on every open-ended task (e.g. causal linkage 0.276 to 0.195; scene description 0.326 to 0.212). Mean trace length rises to 429–505 words.

• Trace supervision helps exact-match tasks and hurts BERTScore tasks: On the 9B backbone, TAR-RE reasoning supervision lifts MCQ from 0.9429 to 1.0000 (+0.0571) and BCQ from 0.9571 to 0.9714 (+0.0143), but loses BCQ-OE by 5.1 BERTScore points (0.6972 to 0.6460); open QA and scene description sit within two points. This is exactly the division of labor the two specialists instantiate.

• Static routing leaves score on the table: Because the 27B answer specialist wins every free-text task on average, a per-task routing table collapses to the 0.4958 answer-only score. Flips (items where the locally weaker specialist is better) are 41% of BCQ-OE (worth +0.060 per flipped item under oracle), 49% of causal linkage (+0.035), 37% of temporal description (+0.027), 24% of open-ended QA (+0.024), 13% of scene description (+0.008), and 0% of video summarization.

• Constrained judging beats both specialists: On the internal test split (70 videos, 490 items), the 9B reasoning specialist alone scores 0.4460, the 27B answer specialist 0.4958, a neutral-rubric judge 0.4887, the judge-guided design 0.5043, and an oracle 0.5347. The 0.0156 gap between neutral and constrained rubrics is purely the value of constraining the judge.

• The judge is strongly biased if left unconstrained: Under a neutral rubric it favors the last-presented candidate on 65–79% of items regardless of author, and only 51.6% of verdicts survive swapping the order (chance level). It selects the same-family 27B candidate on 42.8% of items against the oracle's 37.8% (self-preference), and a 9B sibling returns the last-presented candidate on every item.

• Post-hoc rewording of answers never helps: Every modification that reworded a generated answer regressed: LLM rewriting on open QA (−0.071), LLM cleanup on BCQ-OE (−0.012), LLM fusion of four candidates (−0.016), trimming the last sentence (−0.028) or keeping the first three (−0.074) on scene description, appended gold open-QA content (−0.083), and sibling-description hints (−0.016). The noise floor from paired bootstrap is roughly 0.012–0.016.

• Extra capacity does not help; only fine-tuning did: Vision-tower LoRA on the last 4 or 8 blocks gave +0.002/+0.003, rsLoRA at rank 128 gave −0.012, and 64→32 frames or a larger pixel budget gave approximately 0. Few frames at very high resolution scored −0.077 on the leaderboard.

• The only surviving post-hoc edits injected verifiable external content: Transcribing the location overlay burned into each test frame into scene descriptions (+0.004 on the leaderboard) and anchoring open-ended answers to Gemini-localized event windows (+0.008).

• Local validation is not a leaderboard proxy: The best-ever validation configuration for scene description scored 0.077 below the shipped model on the leaderboard, because the organizer test set is dominated by short, collision-heavy clips (median length about 10 s) while the local split follows the released-data distribution.

Methodology in Plain English

The authors began by inspecting the official training annotations rather than training immediately. Using programmatic checks across all 44,040 annotations plus manual inspection of a small sample of videos, they identified four recurring faults: reasoning that cites captions and annotator notes the model will never see; timestamps in four inconsistent formats; annotations that contradict the video; and traces that assert verdicts without checking the timeline. They deliberately did not relabel the gold answers, but instead repaired the supervision.

First, they canonicalized every timestamp into one textual form (<X.Y seconds>). Second, they used Qwen 3.5 9B in thinking mode to rewrite each training trace, feeding it the video, task name, question, gold answer, and original reasoning, and requiring a five-part trace with minimum length floors. Two guardrails kept the rewrite honest: the gold outcome is frozen so regeneration cannot silently relabel data, and a validator rejects any trace that repeats the gold answer verbatim or misses a structural tag or depth floor.

Then they trained two specialists on an 80/10/10 video-disjoint split. The "reason" specialist is Qwen 3.5 9B with LoRA, trained to emit the full five-part trace inside the model's think block followed by the answer, so it verifies before it answers. The "recite" specialist is Qwen 3.6 27B with 4-bit QLoRA, trained on the gold answers alone with no trace at all, so it absorbs the annotation register into its weights; for the two Scene-group tasks it emits a merged coarse-to-fine target (<scene> then <summary>) that is sliced apart at inference.

At test time, both specialists answer each question independently and a frozen, out-of-the-box Qwen 3.6 27B judge chooses which answer to submit. Because the judge showed strong position and family bias, it is cast as a conservative verifier: the candidate expected to win (from the specialist with the higher validation score on that task type) is presented last and kept unless clearly contradicted by the video, a deterministic pre-filter settles uncontested items, and a short per-task instruction states what separates a correct from an incorrect answer. MCQ-with-explanation items are handled by construction: the option text of the final MCQ prediction is matched to the corresponding option of its MCQ-OE twin and submitted verbatim.

Why This Matters

Impact on research. The paper makes a methodological point that generalizes beyond traffic video: when a benchmark's supervision is auto-labeled and its metric scores similarity to a single reference, reasoning quality and reference matching are separable objectives, and forcing one model to serve both leaves accuracy on the table. The annotation audit is also a reusable template — the defect taxonomy (invisible sources, timestamp chaos, contradictions, unverified absence claims) applies to any distillation pipeline where a teacher model saw inputs the student never will. The finding that unconstrained multimodal judging is unreliable, and that position-bias control is what makes arbitration profitable, is directly actionable for the LLM-as-judge literature.

Real-world applications:

• Automated incident review for traffic operations centers, where operators need verification, description, and cause explanation for collisions and violations, not just an anomaly flag.
• Insurance and claims analysis, where timestamped evidence traces support auditable statements about what happened and when.
• Regulatory compliance and enforcement review, where a system must justify a verdict rather than assert one.
• Video analytics product QA, where per-question arbitration between a reasoning-focused and a style-tuned model can be deployed without retraining.

Industry relevance. The result that a 4-bit-quantized 27B fine-tune plus a 9B fine-tune on commodity single-GPU hardware (one RTX 5090 at 32 GB, one RTX PRO 6000 Blackwell at 96 GB) can top a competitive leaderboard is practically meaningful for teams without large compute budgets. So is the negative result that capacity levers (vision-tower LoRA, higher rank, more frames, more pixels) tied or regressed, implying the binding constraint is perception precision rather than model size.

Future Directions

• Improve perception precision rather than capacity. On scene description, oracle grafting of the references' vehicle sentences gains +0.06, so headroom exists; every deployable route the authors tried traded a small recall gain for a larger precision loss. What supervision or architecture raises the precision of what the model asserts it saw remains open.

• Recover more of the oracle gap. The judge recovers roughly a fifth of the 0.5347 oracle headroom (0.5043 achieved), with the remainder attributed to guarding against the judge's own biases. Better debiasing, multi-order aggregation, or a judge trained specifically for arbitration could close part of that gap without sacrificing independence.

• Address the noisy-supervision ceiling. Because the authors deliberately freeze gold outcomes, upstream hallucinations (such as an entire question family premised on a collision that never occurs) persist in training. Methods that detect and repair contradictory labels without losing the annotation style the metric rewards are a natural next step.

• Validate annotation-style specialization beyond TAR. The claim that matching an auto-labeled distribution can outweigh reasoning quality is demonstrated on seven BERTScore-scored tasks here; testing whether it holds for other single-reference metrics and other auto-labeled benchmark suites would establish how general the lesson is.

Target Audience

Researchers and practitioners working on video-language models, video anomaly understanding, and multimodal benchmark competition entries; engineers who need to fine-tune VLMs under tight compute budgets and want concrete recipes plus documented negative results; and anyone studying LLM-as-judge selection, data quality in auto-labeled supervision, or the mismatch between reasoning quality and reference-similarity metrics. Readers new to LoRA, BERTScore, or VLM fine-tuning will find the methodology section demanding.

Authors’ abstract

Road anomalies such as collisions, violations, and sudden hazards demand more from video understanding than detection: a practical system must verify what happened, describe the scene, and explain the cause. We find these competences pull in opposite directions for vision-language models: closed-form questions reward precise, evidence-grounded reasoning, while open-ended answers are scored by similarity to reference text, where matching the annotation style matters as much as reasoning correctly. We present Reason-or-Recite (RoR), a two-specialist framework for the Traffic Anomaly Reasoning track of the 2026 AI City Challenge. RoR (1) rewrites the official chain-of-thought annotations into a structured, temporally grounded schema and LoRA-tunes a 9B vision-language model on the enhanced traces; (2) fine-tunes a 4-bit-quantized 27B model on answers alone, absorbing the annotation style; and (3) selects between the two candidates per question with a training-free LLM judge. RoR ranks first on the in-domain leaderboard with mean score 0.6788.