Model Selection, Routing, and Economics

A small model may extract a known schema faster and more reliably than a general reasoning model. A larger model may be justified for ambiguous planning. Arithmetic and policy invariants should go to deterministic software.

Model selection is therefore a portfolio decision. It matches request classes to the least complex system that meets the release contract. That decision has been measured in public, on named benchmarks, with published prices and published savings. The rest of this lesson is built on those measurements, not on the intuition that bigger is safer.

A leaderboard score covers only part of the decision. It is worth being precise about which part.

Take the best-known leaderboard. Chatbot Arena shows a person two anonymous answers side by side and records which one they prefer. When Chiang and colleagues wrote it up in 2024, the count was already large: “The platform has been operational for several months, amassing over 240K votes.” Over 240,000 crowdsourced pairwise votes, from about 90,000 users, is a real measurement — of which of two answers a person preferred.

That is one axis of the envelope below, and only one. Aggregate pairwise preference carries no latency, no cost and no groundedness. A ranking built from it cannot tell you the time to first token under load. It cannot tell you the price of the traffic it would generate, whether the preferred answer was supported by the retrieved document, or what a version change would cost you to re-certify. Those are separate measurements. Each has to be made against your own traffic.

Task quality is correctness, groundedness, format adherence, tool use and abstention. Operating performance is time to first token, output rate, concurrency and tail latency. Resource cost is input, output, cache, retrieval, tool, accelerator and review costs. Risk and control is safety behavior, data handling, deployment boundary and audit evidence. Lifecycle fit is version stability, migration effort, evaluation burden and provider or platform constraints. That is five dimensions. A leaderboard reports a projection of the first one.

Evaluate complete workflows under matched requirements and budgets. Three designs recur. A single general model takes most traffic. A router with specialists classifies requests and sends them to models or workflows with different strengths. A deterministic hybrid uses rules, search, calculators, templates and models together.

The spread these designs are chosen against is unusually wide. FrugalGPT, published in 2023, found advertised API prices differing by up to two orders of magnitude: a prompt cost for 10M tokens of $30 for OpenAI's GPT-4 against $0.20 for GPT-J on Textsynth. Its answer was an LLM cascade. Its abstract also supplies the magnitude that a phrase like “improves the cost-quality frontier” leaves unstated: “Our experiments show that FrugalGPT can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost.”

Read the two halves of that sentence as two different products. "Up to" marks the best case the paper reports, not the saving to expect on your own traffic. Equal quality at up to 98% lower cost is a margin decision. Four accuracy points above GPT-4 at the same spend is a quality decision. One cascade offers both. Which one you take is set by the release contract, not by the architecture.

The single general model stays simple to integrate and monitor, and it is useful during early discovery. It also costs more, runs slower, and is often unnecessary for easy cases. The router adds routing errors, observability needs, and a fallback and override policy; it earns its place with heterogeneous traffic. The deterministic hybrid gives high reliability for structured steps and reduces the generative burden, at the price of more engineering boundaries. It is the right answer when parts of the task are exact.

A business case should include the resources required to produce and maintain acceptable outcomes. Even the first line has internal structure that a single price-per-million figure hides.

Cached input tokens are a separate price line, not a discount on the same line. Anthropic's prompt-caching documentation says so in its pricing note: “5-minute cache write tokens are 1.25 times the base input tokens price”. 1-hour cache writes are priced at 2x, and cache reads at 0.1x. AWS documents the same shape for OpenAI's GPT-5.6 models — Sol, Terra and Luna — on Amazon Bedrock: writes at 1.25x the uncached input rate, reads at a 90% discount.

The arithmetic follows, and it is worth doing rather than assuming. The 0.25x write surcharge has to be earned back by later reads that each save 0.9x. A cached prefix therefore pays for itself on its first hit. The cache only costs more than no cache once a prefix is written more than about 3.6 times for every time it is read. A long system preamble reused across a session is free money. A prefix rewritten on every request and rarely re-read is a surcharge with no return.

Compare candidates on the complete system contract. Segment traffic into request classes, risk tiers, languages, lengths and tool needs. Establish candidate workflows, including smaller models, deterministic methods, retrieval, tools and larger models. Evaluate quality and safety on representative cases, critical slices and trace-level metrics. Measure performance and cost. Design the routing policy. Then canary and monitor route distribution, drift, regressions and total outcome cost.

The fourth step has a published form to copy rather than invent. MLPerf Inference, run by MLCommons, scores LLM systems only under explicit tail-latency constraints. Its rules document names them: “For LLM benchmarks, 2 latency metrics are collected - time to first token (TTFT) which measures the latency of the first token, and time per output token (TPOT) which measures the average interval between all the tokens generated”. In the Server scenario, Llama2-70B must hold time-to-first-token at 2,000 ms and time-per-output-token at 200 ms in the conversational category. The interactive category tightens those to 450 ms and 40 ms. Both are measured at the 99th percentile. That is what a latency requirement looks like written down: a named model, a named scenario, two numbers, a percentile, and a second, stricter tier for interactive use. The v5.0 round, published on 2 April 2025, carried 17,457 performance results from 23 submitting organisations. That is the scale at which such contracts become comparable across vendors.

The fifth step has a worked form too. Hybrid LLM, published in 2024, routed each query to a small or a large model by predicting how hard it was and comparing that prediction against a quality threshold. The threshold can be tuned at test time. “In experiments our approach allows us to make up to 40% fewer calls to the large model, with no drop in response quality”, says the abstract. The tunable threshold is the design lesson. The policy exposes a quality level as a dial instead of freezing one operating point at training time. The same router then serves a strict tier and a permissive tier without being rebuilt.

A support product's daily traffic is the case for the six steps. The cache price lines (writes at 1.25x, reads at 0.1x) and the tunable routing threshold come from the lesson above. The traffic, the scores and the per-million-token prices below are invented, cache writes and the FAQs that fall back to the large model are left out for simplicity, and every cost is computed.

A cheaper model can become expensive when it triggers retries, escalations or rework. A powerful model can waste resources too, when deterministic components would solve the task more reliably.

There is a second reason not to anchor the decision to today's token price. That price is the fastest-moving number in the stack. Stanford's AI Index Report 2025, working with Epoch AI, tracked the price of a fixed capability rather than the price of a named model: “For instance, the inference cost for an AI model scoring the equivalent of GPT-3.5 (64.8) on MMLU, a popular benchmark for assessing language model performance, dropped from $20 per million tokens in November 2022 to just $0.07 per million tokens by October 2024 (Gemini-1.5-Flash-8B)—a more than 280-fold reduction in approximately 1.5 years.” Same score on the same benchmark. $20.00 down to $0.07, between November 2022 and October 2024: just under two years, although the report gives the span as approximately 1.5.

The report also notes that the rate of decline varies from 9x to 900x per year depending on the task, and that is the part that matters for planning. A selection made on price alone is made on a quantity that may fall a hundredfold before your next migration. It may fall a hundredfold for one of your request classes and barely move for another. The lines that do not collapse on that schedule are the ones in the rest of the cost stack: retrieval, tools, retries, review, monitoring, migration, and the consequences of failure.

The next lesson examines serving mechanics. Prefill, decode, KV cache, batching and memory allocation explain why identical models can produce very different latency and throughput in production.