VL-RouterBench: A Benchmark for Vision-Language Model Routing

AI summary

Overview

Research area: Machine learning infrastructure — multi-model routing for vision-language models (VLMs), sitting at the intersection of multimodal learning, model selection, and inference cost optimization.

Technical level: Intermediate. The framing (benchmark, accuracy–cost trade-off, routing) is accessible, but the training objective involves Lagrangian soft labels and cross-entropy formulations that assume some familiarity with machine learning.

Scope: This paper builds and evaluates VL-RouterBench, the first large-scale, reproducible benchmark dedicated to routing queries across a pool of vision-language models, covering 30,540 samples, 17 models, and 519,180 sample–model pairs.

What This Paper Is About

Organizations increasingly have access to many vision-language models that differ widely in cost and capability, and picking the right one per query can save money without sacrificing quality. Routing research is well-developed for text-only LLMs, but there has been no standardized benchmark for the multimodal setting, where tasks span visual question answering, scientific reasoning, and chart/OCR reading. This paper creates that benchmark: a shared dataset, training protocol, and evaluation metric suite that lets researchers compare VLM routing methods fairly and reproducibly.

Key Contributions

1. VL-RouterBench itself — a large-scale benchmark for VLM routing built from raw inference and scoring logs, comprising quality and cost matrices over 519,180 sample–model pairs generated from 30,540 samples, 14 datasets, and 17 models (15 open-source, 2 API), totaling roughly 34.5 million tokens.

2. A complete end-to-end pipeline covering router data preparation, router training, and router evaluation, so that results are comparable and new models or datasets can be plugged in incrementally.

3. An adjustable soft-label training strategy derived from a Lagrangian formulation of the accuracy–cost objective, where a single hyperparameter λ continuously controls whether the router favors accuracy, cost, or a balance of the two.

4. A systematic evaluation and ablation study of 10 routing methods and baselines — including the effects of text/visual encoder choice, multimodal fusion strategy, and end-to-end backbone architecture — revealing a large remaining gap between the best routers and the ideal Oracle.

Main Findings

• Routing beats any single model. Most learned routers surpass the "Strongest" single-model baseline, achieving higher accuracy at lower cost. The best router, RouterDC, matches near-strongest accuracy (77.52% vs. 78.01%) at roughly one-fifth of the cost ($1.04 vs. $2.72 per 10K samples).

• A large Oracle gap remains. The ideal Oracle achieves 95.60% accuracy at $0.37 per 10K samples (Rank Score 93.68), versus 74.59 for the best real router. This gap indicates substantial unexploited routing signal in multimodal samples.

• Multimodal inputs help, and simple fusion is surprisingly strong. Using only text or only image embeddings underperforms, but normalizing each embedding and concatenating them (Normalize-Concat) delivers the best Rank Score (74.05), outperforming more elaborate learned fusion modules like GMU and MLB.

• Bigger embeddings help feature-level routing. Across five text encoders and five visual encoders, higher embedding dimensions consistently improved Rank Score; the best pairing was BGE-M3 (text) with SigLIP-L-16 (visual).

• Multimodal backbones beat single-modality backbones end-to-end. BERT (text-only) and MobileNetV4 (image-only) scored markedly lower than VisualBERT, UNITER, LXMERT, and ViLBERT. Among the four multimodal encoders, LXMERT scored highest (71.36), though results were close.

• A speed/accuracy trade-off exists between router paradigms. End-to-end routers achieve slightly better accuracy–cost trade-offs but run at roughly 6–8K tokens/s, while feature-level routers with a small classifier run over 140K tokens/s.

• Routability is real and capturable. Differences in difficulty and structure across multimodal samples are learnable by lightweight routers, confirming that sample-level dynamic selection is a viable strategy rather than a theoretical nicety.

Methodology in Plain English

The authors start by running every one of the 17 candidate vision-language models on every one of the 30,540 test samples. Each sample contains a single image plus a text prompt, and correctness is judged automatically by rule-based answer matching (multiple choice or exact match), avoiding subjective grading.

This produces two big tables: one recording whether each model got each sample right (the quality matrix), and one recording how much each inference cost based on input and output token counts multiplied by each model's published per-token price (the cost matrix).

For training a router, the authors do not simply ask it to predict which model was correct. Instead they compute a "soft label": among the models that answered a given sample correctly, cheaper models receive more probability mass. A hyperparameter λ controls how aggressively cost is weighted — at λ = 0 all correct models count equally, and as λ grows the target approaches "pick the cheapest correct model." The router is then trained to match this target distribution via cross-entropy.

Two router families are tested: feature-level routers, which freeze a text encoder and a visual encoder, combine their embeddings, and feed them to a lightweight classifier (KNN, linear, MLP, and others); and end-to-end routers, which fine-tune a full multimodal encoder (VisualBERT, UNITER, LXMERT, ViLBERT) to output the model choice directly. Evaluation reports average accuracy, average cost in dollars per 10K samples, throughput in thousands of tokens per second, and a single Rank Score — the harmonic mean of accuracy and log-normalized cost, with a default weight favoring accuracy. Accuracy–cost Pareto frontiers are also plotted to compare routers across the whole cost spectrum.

Why This Matters

Routing has quietly become production infrastructure — commercial systems including GPT-5 expose routing as a built-in capability — but multimodal routing research has lacked the shared measurement apparatus that text routing already has. This benchmark supplies that apparatus, making claims about new routers directly comparable and reproducible for the first time in the VLM setting.

Real-world applications:

• API gateways and model-serving platforms that broker requests across many providers can use routing policies benchmarked here to cut inference bills without a quality drop.
• Document and chart processing pipelines (invoice parsing, report extraction, OCR-heavy workflows) where cheap models suffice for easy pages and expensive models are reserved for hard ones.
• Multimodal AI agents and assistants that need to select a vision backbone per user query under strict latency and cost budgets.
• Edge/cloud hybrid deployments where a small on-device model handles routine visual queries and a router escalates only genuinely difficult inputs to a server-side model.

Industry relevance: Cost per token is a direct operational expense for any company running vision-language inference at scale. A router that captures even part of the Oracle gap translates into substantial recurring savings, and the throughput metric reflects the latency concerns that determine whether a routing layer is deployable at all.

Future Directions

• Closing the Oracle gap. The ~19-point accuracy gap between the best router and the ideal Oracle suggests room for architectures that read finer visual cues and model document layout or textual structure explicitly, rather than treating the image as a single pooled embedding.

• Better multimodal fusion for routing. Simple normalized concatenation outperformed learned fusion modules, which the authors read as evidence that fusion designed specifically for routing decisions — not inherited from other multimodal tasks — remains largely unexplored.

• Lower-latency end-to-end routers. End-to-end routers trade throughput for accuracy; making multimodal backbones fast enough for production while preserving their ranking quality is an open engineering problem.

• Keeping the benchmark alive. Because the pipeline is modular, the natural next step is continuous ingestion of newly released models and datasets so the leaderboard tracks a fast-moving model ecosystem. Additional questions include how routing behavior scales with much larger candidate pools and whether routers generalize across task groups they were not trained on.

Target Audience

Researchers working on model routing, model selection, or LLM/VLM serving systems will find this the most directly useful, particularly those who need a standard comparison bed for new router architectures. Practitioners building multimodal inference infrastructure — platform engineers and ML teams managing multi-provider model pools — will benefit from the accuracy–cost methodology and the throughput measurements. Benchmark designers and evaluation researchers will also value the data-construction and metric-design choices, especially the soft-label strategy and the harmonic-mean Rank Score as a way to collapse a two-objective problem into a single comparable number.

Authors’ abstract

Multi-model routing has evolved from an engineering technique into essential infrastructure, yet existing work lacks a systematic, reproducible benchmark for evaluating vision-language models (VLMs). We present VL-RouterBench to assess the overall capability of VLM routing systems systematically. The benchmark is grounded in raw inference and scoring logs from VLMs and constructs quality and cost matrices over sample-model pairs. In scale, VL-RouterBench covers 14 datasets across 3 task groups, totaling 30,540 samples, and includes 15 open-source models and 2 API models, yielding 519,180 sample-model pairs and a total input-output token volume of 34,494,977. The evaluation protocol jointly measures average accuracy, average cost, and throughput, and builds a ranking score from the harmonic mean of normalized cost and accuracy to enable comparison across router configurations and cost budgets. On this benchmark, we evaluate 10 routing methods and baselines and observe a significant routability gain, while the best current routers still show a clear gap to the ideal Oracle, indicating considerable room for improvement in router architecture through finer visual cues and modeling of textual structure. We will open-source the complete data construction and evaluation toolchain to promote comparability, reproducibility, and practical deployment in multimodal routing research.