Which AI model should you use?

There is no best model

"Best" is not a property of a model. It is a property of a model against a job and a budget — and the answer changes when either does.

The leaderboard question — which model is smartest — is the wrong one for almost everyone. Frontier models genuinely lead on hard reasoning and long-horizon coding, and if that is your work, pay for one. But a great deal of production AI is extraction, classification, routing, reformatting and summarising, and on those a small model frequently matches a large one at a fortieth of the cost.

Choosing badly is expensive in both directions. Over-specify and you pay forty times over for a job a cheap model does perfectly. Under-specify and you ship something that fails on the cases that matter, which costs more than the API ever would.

Five things we measured ourselves

From routing real production traffic across open models. These are our own numbers — narrow, specific, and more useful than a leaderboard.

• A smaller model beat a much larger one on our own A/B.

  gpt-oss-120b outscored Qwen3-235B and DeepSeek on our quality metrics while costing roughly a third as much per generation. Parameter count is not quality.

• Reasoning effort is a free latency win.

  Setting reasoning_effort to low roughly HALVED latency at equal measured quality on our workload. If your provider exposes a thinking-budget control, test the low setting before paying for a faster model.

• The host matters as much as the model.

  The same model, same weights, on two providers: 3,411ms total at p50 on one and 11,710ms on the other. If you buy through an aggregator, pin the host.

• Average latency hides the problem.

  One model we dropped had a respectable 19.8s median and a 63.5s tail. Users experience the tail. Measure p95 and p99, never the mean.

• Free tiers are capped on tokens, not requests.

  A provider advertising a generous daily request limit gave us roughly 337 real generations a day, because the binding constraint was tokens. Read which unit the quota is denominated in.

The through-line: the model name is not the whole decision. Which host serves it, what thinking budget you set, and which unit the quota is denominated in all moved our numbers more than swapping one frontier model for another would have.

The four axes that actually decide it

The model families, briefly

Generalising about model families ages badly, so this is deliberately shallow and about temperament rather than benchmarks.

OpenAI (GPT) has the widest range from frontier down to genuinely tiny, and the deepest tooling ecosystem. The nano and mini tiers are the strongest argument for the family: very capable models at prices that make high-volume features viable.

Anthropic (Claude) is consistently well regarded for writing quality, long-document work and instruction-following, and tends to be the pick when the output is read by a person rather than parsed by code.

Google (Gemini) leads on context length and on price at the Flash tier, and handles multilingual work well. If you are processing very large inputs, start here.

DeepSeek is the value pick for reasoning and code. Its first-party API also runs an off-peak window at roughly half price, which is unusual and worth building around if your work is batch rather than interactive.

xAI (Grok) is competitive on reasoning and conversation and sells a coding-specialised build model separately. Watch the context tiers — the published rate roughly doubles above 200K tokens, so a long-context job costs more than the headline number suggests.

Mistral runs an unusually wide ladder. Large 3 is a capable general model, and Nemo at $0.02 per million input tokens is one of the cheapest things you can call at all — genuinely good at extraction and classification, and not a writer. It is the model most likely to save you real money.

Amazon (Nova) matters mainly if you are already on Bedrock. The Lite and Micro tiers are priced to compete with the open models while staying inside AWS billing and networking.

Open weights — gpt-oss, Qwen, Llama — have no single price, because nobody sells them: you pay whoever hosts them, and the same weights vary by up to 6× between hosts. Where the tool above shows one of these, the price is the cheapest host we hold a real rate for. That is also the family we have our own numbers on, below.

The step-down method

The most reliable way to choose, and it takes an afternoon.

1. Prototype on the strongest model you can afford. Not to ship on it — to learn what a good answer looks like for your task. Without that reference you cannot tell whether a cheaper model is failing or whether the task is hard.
2. Build a set of twenty real inputs. Real ones, including the awkward cases. This is the step people skip, and it is the one that makes the rest mean anything.
3. Step down one tier and run the same twenty. Compare against your reference outputs, not against your memory of them.
4. Keep stepping down until quality visibly drops. The tier above where it broke is your answer.
5. Re-check in three months. Prices fall, tiers shift, and the model that was right in March is often not the cheapest adequate one by June.

Teams that do this routinely land two or three tiers below where they started. The instinct to reach for the strongest model is expensive and almost never tested.

The escalation pattern

The single most effective cost architecture, and it is barely used: do not pick one model — pick two, and route between them.

A cheap model handles every request. It also decides, as part of the same call, whether the request is one it should be handling. Anything it flags as hard, ambiguous or high-stakes goes to an expensive model. The pattern works because difficulty is not uniformly distributed: in most workloads the large majority of requests are routine and a small minority are genuinely hard.

The arithmetic is worth doing once. Take 10,000 calls a day at roughly 1,000 input and 700 output tokens each:

Roughly an eighth of the all-frontier bill, with the hard cases still getting the frontier model. The cost of the pattern is one extra decision in the prompt and a branch in your code.

What benchmarks do not tell you

Public leaderboards are useful for one thing: ruling out models that are clearly not in contention. Beyond that they are a weak guide, for four reasons worth knowing.

• They measure aggregate ability, not your task. A model two points ahead on a general benchmark can be behind on classifying your support tickets.
• Contamination is real and hard to rule out. Benchmark questions leak into training data, and a model that has seen the test scores better than it performs.
• They ignore cost and latency entirely. The top of a leaderboard is where the expensive slow models live, which is exactly the trade-off you are trying to make.
• They are measured at settings you will not use. Benchmark runs use tuned prompts, high thinking budgets and sometimes multiple attempts. Your production call is one shot at default settings.

Twenty of your own real inputs beat any leaderboard. That is not a slogan — it is the reason the step-down method above starts by collecting them.

Task by task

Switching models without breaking everything

Prompts are less portable than people assume. What transfers cleanly: structure, scope, constraints, audience — everything about what you are asking for. What does not: format adherence, length sensitivity, refusal thresholds, and how literally an instruction is taken.

Practical guidance. Re-test your output format first, because that is where breakage is both most likely and most obvious — a model that quietly starts wrapping JSON in markdown fences will break a parser downstream. Re-check length, since the same word count instruction lands differently across families. And keep the twenty real inputs from the step-down method; a model switch is exactly when they earn their keep a second time.

If you can, keep a fallback configured to a second provider. Outages happen, and a feature that fails entirely when one API is down is a worse product than one that degrades to a cheaper model for an hour.

Where this data comes from

Worth being explicit, because most model-comparison pages are not.

Prices and context windows are facts, taken from published provider rate cards and checked on a stated date, shown in the tool above. They change; the date tells you how stale the number might be.

Task fit is a judgement, based on published benchmarks and general capability reporting. We have not independently benchmarked GPT-5.6 against Claude Opus 5, and we are not going to imply otherwise. Treat the fit ratings as a sensible starting shortlist, not as a measurement.

The five findings above are ours, measured on real traffic through our own routing layer. That is why they are about latency, hosting and quota units rather than about which frontier model is cleverest — those are the things we actually observed, and they are more actionable than another leaderboard.

Common questions

Is a bigger model always better?

No, and we have our own counter-example: a 120B model outscored a 235B one on our quality metrics at roughly a third of the cost. Training data, tuning and serving configuration matter more than parameter count.

Should I use one model for everything?

Operationally simpler, financially wasteful at any scale. A common pattern is a cheap model for classification and routing, with escalation to an expensive one only for the cases that need it. That usually costs a fraction of running everything on the expensive model and performs identically.

What about open-source models?

Competitive, often much cheaper, and worth testing — with two caveats. Quality varies more between hosts than people expect, so pin your provider. And the free tiers are usually capped on tokens rather than requests, which is far more restrictive than the advertised numbers suggest.

How often should I revisit this?

Every three to six months, or whenever a provider announces a price change. Both prices and capability move fast enough that an eighteen-month-old decision is usually leaving money on the table.

Does the ranking account for latency?

Not directly, because published latency figures are unreliable and vary by host and region. Where it matters, measure it yourself on your own traffic — and measure p95, not the average.

Glossary

• Context window — the maximum tokens a model holds at once, covering prompt, history and reply together.
• Frontier model — the most capable current tier. Expensive, and necessary less often than it is used.
• Parameter count — model size. A weak proxy for quality.
• p95 latency — the response time 95% of requests come in under. What users actually experience, unlike the mean.
• Batch API — asynchronous processing at roughly half price.
• Prompt caching — a reduced rate on a repeated prompt prefix.
• Aggregator — a service reselling many providers behind one API. Convenient, but pin the host or your model may be served by anyone.
• Escalation — routing only the hard cases to an expensive model. The single most effective cost pattern there is.