Model Routing: A Technical Overview of the State of the Art

Model routing dynamically decides which model and how much compute handles a request, aiming to stay on the Pareto frontier of cost, quality, and latency. This post is an overview of the field: the approaches, the state of research, how much can realistically be saved, common router architectures, and known limitations and failure modes. This is not a construction manual, but rather a map of the landscape.

Key Takeaways

• Routing selects a model per request to balance cost, quality, and latency; the primary forms are pre-generation and cascade.

• Paper figures ranging from 85% to 98% represent upper bounds; AWS’s 16% to 56% per model family are more realistic.

• According to RouterArena, real routers perform significantly below the Oracle; typical failure modes include routing collapse and confounder-gadget attacks.

• The primary benefit remains cost savings, complemented by reliability, observability, and, for EU products, compliance.

The conceptual "why"—namely, using the cheapest model that still gets the job done—is covered in the foundational article on price-performance routing. This text evaluates how far the underlying technology has come.

What Routing Is, and What It Is Not

A router is a function that assigns a request from a pool of models to exactly one, in order to maximize expected quality under a cost budget or minimize costs under a quality floor. The threshold at which the choice shifts to the stronger model encodes the trade-off between cost and quality. In literature, this is referred to as "willingness-to-pay."

The fundamental distinction relates to the timing of the decision.

Neighboring techniques should be distinguished from this. Mixture of Experts routes tokens within a single model. Ensembling and Fusion query multiple models and combine the responses, trading higher costs for higher quality. Speculative Decoding accelerates a single model using a draft model. Model routing, on the other hand, selects exactly one model to save costs.

The State of Research

The field is surprisingly mature. FrugalGPT (Stanford, 2023) established cascades, Hybrid LLM (Microsoft, ICLR 2024) introduced the encoder-based difficulty router, and RouteLLM (Berkeley, ICLR 2025) pioneered routers trained on preference data along with metrics like APGR (Average Performance Gap Recovered) and CPT (Call-Performance Threshold). Benchmarks used for evaluation include RouterBench (utilizing the AIQ metric, with zero-router and Oracle-router baselines) and, most candidly, RouterArena (ICLR 2026). Its core finding is sobering: all real-world routers remain significantly below the Oracle, primarily because they struggle to detect when a small model is already sufficient.

How Much Routing Saves

Research cites spectacular figures: FrugalGPT achieves up to 98% cost savings with cascades, and RouteLLM reaches over 85% on MT-Bench while maintaining 95% of GPT-4 quality. These values apply to their respective benchmarks and are upper bounds under favorable difficulty distributions, not expected production values. Closer to reality are the averages published by AWS for Bedrock Intelligent Prompt Routing: depending on the model family, 16% to 56% savings while maintaining top quality.

Whether routing pays off depends on three conditions: heterogeneous request difficulty, a wide price gap between viable models, and high volume. If any of these are missing—for instance, because almost all queries are complex—nearly everything ends up in the top tier anyway. If the price gap is below roughly a factor of 3 to 5, or if over 70% to 80% of requests go to the most expensive tier, the impact remains negligible.

The Three Basic Strategies

Practically all routers can be traced back to three basic patterns, which are often combined in production.

Rule-based Tiering deterministically assigns requests to a tier based on signals like task type, query length, or required capabilities. This approach is transparent and requires no ML, but it is rigid and assumes the task type is known beforehand. According to RouterArena, good tiering alone captures a large portion of the achievable savings.

Classifier Routing estimates complexity upfront using a small, fast model. In research, the encoder-classifier (BERT, DeBERTa, ModernBERT) dominates, with a forward pass in the millisecond range; RouteLLM found tiny matrix factorization to be the most cost-effective. The well-known weakness: routers overfit to their training distribution and drop to random-guess levels out-of-domain unless the training data is augmented.

Cascade Routing operates post-generation. First, the cheap model responds, then its confidence is evaluated, and only in cases of uncertainty is the request escalated to a stronger model. This is more robust because a real response is available, but it incurs additional latency and multiple calls. Confidence-based deferral is often, but not always, optimal according to research (Jitkrittum et al., NeurIPS 2023).

Router Architectures and Latency

The choice of router architecture is itself a cost-latency trade-off, as the overhead of making the decision must not consume the savings. In practice, viable routers operate in the 10 to 50 ms range.

Routing Is First a Constraint Problem

Before cost even becomes a factor, hard capabilities restrict the pool: context window, tool calling, JSON schema support, multimodality, streaming, and, for an EU product, the processing region. A model lacking a required capability is not a valid choice, regardless of price. For large model pools, routing is therefore first a constrained optimization over the permissible sub-pool, and only then a matter of cost. Data residency is one of these hard constraints: a failover that silently sends a request to a non-EU region turns an outage into a compliance violation.

The Prompt Cache Economy

The most frequently underestimated effect is not an ML question, but one of bookkeeping. If a subsequent request switches providers, that provider's prompt cache prefix is lost, and the nominally cheaper model can become more expensive net-net. Additionally, tokenizers differ: the same prompt will have a different token count per model and thus a different price. The real cost of a route is therefore made up of input and output tokens, cache read and write adjustments, as well as expected retry or cascade costs. This is where the common rule of thumb "if you route, pin" comes from: session pinning keeps the cache warm as soon as more than one model is in play.

Semantic Caching

Semantic caching stores answers based on embedding similarity of the query instead of using an exact match. A hit bypasses routing entirely because no model call is required. The trade-off is the potential for false-positive hits if the similarity threshold is set too loosely. For FAQ-like workloads with many repetitions, it is the single largest savings lever.

Failure Modes

Routing Collapse. As the cost budget increases, many routers systematically default to the most expensive model, even when a cheap one would suffice. The paper "When Routing Collapses" (2026) describes this and attributes it to an objective-decision mismatch: routers predict scalar scores but make discrete comparative decisions, and small errors flip the ranking. As a countermeasure, it proposes EquiRouter, a decision-aware, ranking-based approach.

Confounder Gadgets. "Rerouting LLM Routers" (COLM 2025) demonstrates query-independent token sequences that, when prepended to any request, reliably force the router into the expensive model. Such a denial-of-wallet attack does not degrade response quality and bypasses perplexity filters. Input screening and per-user cost anomaly detection serve as protection.

The GPT-5 Lesson. The most public lesson was provided by OpenAI itself. At the launch of GPT-5, the auto-router failed, causing complex queries to land in the cheap model, which made the system appear "dumber"; in parallel, a silent safety router redirected certain prompts without documentation. Since then, this case has served as proof that non-transparent, silent routing changes represent a trust risk.

How Routing Is Evaluated

Reliable comparisons are run against four baselines: random routing, the Oracle as the upper limit, the best single model, and an equal-cost single model. Typical metrics include cost at iso-quality, quality at iso-cost, as well as APGR, CPT, or AIQ. In production, shadow deployments (the router runs in parallel without serving its choice), canary releases on a small portion of traffic, and offline replays from logs are also used. RouterArena shows that real routers consistently perform below the Oracle, primarily because they too rarely detect when a small model is sufficient.

Two Trends Reshaping Routing

Two developments are changing the landscape without eliminating the core benefit. Falling frontier prices are slightly narrowing the price gap, and providers are increasingly building routing in themselves: GPT-5’s internal router, Gemini’s thinking budgets, and the reasoning_effort parameter already decide how much compute a request receives before it ever reaches an external router.

Nevertheless, the most tangible reason for routing remains cost savings. As long as tasks vary in difficulty and the price gap is wide, the cheapest suitable model saves actual money, in practice often in the range of 60% to 80%. A gateway makes this lever practical and adds reliability through failover, observability, and, for EU products, compliance on top. Cost savings and control come from the same layer; they are not mutually exclusive.

Where an EU Gateway Like kontinent.ai Fits In

For an EU product, both aspects come together: the cost savings from routing and the control over it. A gateway that bundles OpenAI, Claude, Gemini, Mistral, and other models behind an OpenAI-compatible endpoint sends every request to the cheapest suitable model and turns switching models into a simple string swap rather than an integration project. If processing is EU-hosted, data residency across the entire pool becomes a hard constraint rather than a task per provider, including EU-only failover. This is where kontinent.ai fits in: as an EU-hosted gateway to 167 models in 240 provider variants, combining real cost savings with compliance, reliability, and observability, instead of playing them off against each other.

Frequently Asked Questions

Pre-generation or Cascade: what is the difference?

Pre-generation (predictive) decides prior to generation based solely on the query—cheap and with low latency, but blind to actual difficulty. Cascade decides after a real response and is more robust, but costs additional latency and multiple calls. In practice, both are combined.

How much does routing realistically save?

Not the 85% to 98% cited in papers, which are benchmark upper limits. Closer to reality are AWS’s own averages of 16% to 56% per model family, and even then only under heterogeneous difficulty, a wide price gap, and high volume.

What is routing collapse?

The phenomenon where routers send everything to the most expensive model as the budget grows, even though cheaper ones would suffice. This is caused by predicting scalar scores for what are actually discrete decisions.

Why can a cheaper model be more expensive net-net?

Because the prompt cache prefix is lost when switching providers. A warm cache is often worth more than the lower list price, which is why the rule of thumb "if you route, pin" has become established.

Is external routing still relevant given falling prices?

Yes. The greatest lever remains cost savings: as long as requests vary in difficulty, the cheapest suitable model saves real money. Falling prices and provider-internal routing only diminish the effect at the margins. On top of that, reliability, observability, and, for EU products, compliance are provided from the same layer.

As of August 2026 · kontinent.ai. Research figures (FrugalGPT, RouteLLM, RouterArena, AWS Bedrock, etc.) are benchmark or manufacturer specifications and do not represent guaranteed production values.