Choosing a language model for a product, not a benchmark

Benchmarks answer someone else's question

A leaderboard tells you how a model does on a fixed academic set. Your product has one job, a latency budget, a cost ceiling and a set of languages. None of those appear in the table.

Five questions that decide it

What is the hardest thing it must do, exactly? Write that task down as twenty real examples from your own domain before you read any comparison. That list is your benchmark.

What latency can the interface tolerate? A chat reply can take three seconds. An autocomplete cannot. A batch job overnight can take a minute. This alone eliminates half the candidates.

What does a wrong answer cost? Summarising an internal note and approving a refund are not the same risk. Higher stakes justify a larger model, or a smaller model plus a verification step.

Which languages, really? Model quality drops unevenly across languages. If you serve Vietnamese, Japanese and English, test all three on your own examples. Aggregate multilingual scores hide exactly the gap that will hurt you.

What happens at ten times the volume? Price per call multiplied by realistic traffic, not today's traffic. Many designs that work in a pilot are unaffordable at scale.

A practical pattern

Start with the most capable model you can afford and get the feature correct. Only then try to move the same task to something cheaper, with your twenty examples as the gate. Optimising cost before correctness usually means paying twice.