"Hello" went to a small AI, maths went to a big one. Who decided? Model routing in an LLM gateway

Model routing chooses which language model will answer a request. A gateway can use a keyword rule or a learned classifier to make that choice before calling the answering model, so a greeting can go to a small model and a maths problem to a big one. In our lab a keyword rule sent 43 of 60 prompts to the small model, agreed with our hand labels on 55 of them, and decided in about 0.31 microseconds.

How does model routing work?

This is the same job a layer 7 load balancer does with HTTP: read the request, then pick an upstream. Envoy, for example, matches routes on the path, headers and other request attributes. A model router does that with the prompt text, and its backends are models with very different prices and speeds. Unlike picking a server for your request, the two backends here are not equal: one is cheap and weaker, the other costs 20 to 25 times more per token in our assumed price table.

How does a rule-based router score a message?

Our gateway uses only cheap signals. Each kind of signal adds points once, however many matching words a message has:

long message (more than 280 characters)                      +1
maths: an expression like "7 = 2" or "3 + 7", or "solve"     +2
code: a ``` fence, or "debug", "refactor", "SQL", "regex"    +2
reasoning: "prove", "step by step", "explain why", "design"  +2
score >= 2  -> big model, otherwise small model

"Solve for x: 3x + 7 = 2x - 5" matches the maths kind twice ("Solve" and "7 = 2") and still scores 2, so it goes big. "Hello!" scores 0 and goes small. The rule never understands the question. It matches letters, which is why it is fast and why it can be fooled.

Length alone cannot trigger the big model: it adds 1 against a threshold of 2. Any matching maths, code or reasoning category triggers it independently, so the length test never changes the selected route.

// route.js · the rule from labs/ai-19/run.mjs, simplified
const MATH = /\d+(?:\.\d+)?\s*[-+*/^=×÷]\s*\d+|\b(solve|calculate|integral|derivative|equation|probability)\b/i;
const CODE = /```|\b(debug|refactor|sql|regex|stack trace|function|compile)\b/i;
const REASON = /\b(prove|step by step|explain why|compare|trade-offs?|design|plan)\b/i;

function route(text) {
  let score = text.length > 280 ? 1 : 0;
  if (MATH.test(text)) score += 2;
  if (CODE.test(text)) score += 2;
  if (REASON.test(text)) score += 2;
  return score >= 2 ? "big" : "small";
}

console.log(route("Hello!"));                        // small
console.log(route("Solve for x: 3x + 7 = 2x - 5"));  // big
console.log(route("What is 2 + 2?"));                // big: the maths pattern fires

Run it with node route.js. The third call goes to the expensive model, though a child can answer it, because "2 + 2" is a digit, an operator and a digit.

Research routers replace the hand-written rule with a learned one. RouteLLM trains a router on human preference data to predict when a cheaper model's answer would be good enough. FrugalGPT goes further and cascades: ask a cheap model first, score the answer, and only escalate when the score is low.

What our lab measured

We wrote 60 everyday prompts and labelled each easy or hard ourselves: 42 easy (greetings, short facts, rewording) and 18 hard (multi-step maths, code, debugging, logic, design).

Result	Value	What it means
Sent to the small model	43 of 60	17 went big
Agreed with our labels	55 of 60 (91.7%)	label agreement, not answer quality
Easy-labelled sent big	2	"What is 2 + 2?" and a long thank-you note that says "step by step"
Hard-labelled sent small	3	the bat-and-ball riddle, a short logic puzzle, counting the letter r
Decision time	0.31 µs	measured over 120,000 decisions on an Apple M2 Pro

The three misses share a shape: short, plain words, no symbols, and a trap inside. Keyword rules are weakest exactly where questions look simple and are not. Agreement is also optimistic here, because the same people wrote the prompts, the labels and the rule.

To reproduce the numbers, run node labs/ai-19/run.mjs from the repository root. It prints the routes, the label agreement, the timing and the cost model; our run is saved in labs/ai-19/results.json.

Where does the saving come from?

No model was called, so the cost is a model with assumed numbers: small 0.15/0.60 USD and big 3/15 USD per million input/output tokens, input tokens estimated as characters / 4, and replies of 60 tokens for easy prompts and 500 for hard ones.

always big:  $2.923 per 1,000 requests
routed:      $1.966 per 1,000 requests   (32.8% less)

A hard reply on the big model costs about 500 x 15 / 1,000,000 = $0.0075; an easy one about 60 x 15 / 1,000,000 = $0.0009. So the 42 easy prompts are only 22.4% of the always-big bill. When we rerun the same cost model with the 3 hard-labelled misses sent big, the saving drops from 32.8% to 20.3%. About 12 of the 32.8 points come from the prompts the rule got wrong. Savings follow the expensive requests, and moving those is where the risk sits.

What happens when the route is wrong, or the big model is down?

Picking a model does not check the answer. FrugalGPT-style cascades add a scorer after the small model and re-ask the big one when the score is low. Our lab only models that idea: if the small model failed 60% of hard prompts and a checker caught 80% of those failures, the expected wrong answers per 1,000 requests would fall from 30 to 6 and the saving would shrink to 26.5%, before paying for the checker.

wrong without a checker:  1,000 × 3/60 × 0.60 = 30
wrong with a checker:     30 × (1 − 0.80)     = 6

This calculation assumes easy prompts never fail, the big model always succeeds, and only hard-labelled prompts routed small can fail. Checker cost and latency are excluded. Those numbers follow from the assumed rates; we built no checker.

The other failure is the backend itself. A gateway treats models like any upstream: per-request timeouts, a small retry budget, and ejecting a backend that keeps failing, which is what Envoy's outlier detection does for ordinary services. When the big model is out, the safe choices are to fail over to another big model or provider, or to answer from the small model and say so. Downgrading can reduce answer quality; disclose the fallback and enforce the caller's quality requirements.

Trade-offs

Router	Good for	Costs you	Use when
Keyword rule	microsecond decisions, easy to read and audit	misses short trick questions; fires on digit-operator-digit expressions	traffic is mostly simple and you can log misses
Small classifier	learns fuzzy "hard" from examples	its own latency, training data, drift	you have labelled traffic and quality feedback
Cascade with a checker	catches some bad small-model answers	extra calls and latency on escalations	wrong answers cost more than tokens
Always big	simplest routing policy; quality depends on the task	the full bill	volume is low or every request is hard

What we saw when we ran it

We ran node labs/ai-19/run.mjs from the repository root on our Mac on 2026-11-06. It printed a single block: 60 prompts, 43 sent to the small model, and 55 of 60 agreeing with our own labels. Under that it listed the 5 prompts where the rule and the label disagreed, with the longer ones cut short.

Our terminal after the ai-19 run, showing the routes, the cost model and the disagreements.

One line to look at is "What is 2 + 2?" going to the big model. The rule saw a digit, an operator and a digit and called it maths, while the bat-and-ball riddle, which has numbers (1.10 and 1.00) but no operator sign, went small. The cost lines read $2.923 per 1000 requests for always big and $1.966 routed, 32.8% less, using assumed prices and approximate tokens from characters divided by 4. Those figures come from our cost model with assumed prices.

The RouteLLM paper page on arXiv, by Ong et al., 2024.

The page confirms that RouteLLM trains a router on human preference data to predict when a cheaper model's answer would be good enough.

If you try this on your own prompts, read the disagreement list before the savings line. That list shows you which questions your rule cannot see, and those are the ones worth logging.

Our take

A rule that costs 0.31 microseconds is cheap enough to run on every request, and on our prompts it was right most of the time. More than a third of the modelled saving came from three hard questions that looked easy. If you add routing, log every request that went small, sample those answers the way you would check answers that vary, and measure misses by kind before trusting the bill. Evaluate answers separately to measure whether routing preserved quality.

The lab is a routing-only simulation on one machine with prompts and labels we wrote; costs and latencies are modelled from assumed numbers. It describes general patterns from public papers and docs, not any company's private system.