AnyLearn
All lessons
AIintermediate

Choosing the Right LLM for Your Use Case

A practical decision framework for picking an LLM in 2026 — define the task, build an offline eval, measure quality + latency + cost on real candidates, and avoid the classic trap of optimizing for benchmarks instead of your task.

llmevaluationmodel-selectionevalsbenchmarkscostlatencypractical
Part of:🧭LLM Models in Practice
Not signed in — your progress and quiz score won't be saved.
Lesson progress1 / 10

Stop reading leaderboards

Most LLM-selection mistakes happen before any code is written. A team reads a leaderboard, picks the top model, ships it, and discovers six weeks later that the model is wrong for the actual task — too slow, too expensive, hallucinates on their schema, fails on French customer messages.

The fix is to start from your task, not from a leaderboard. This lesson walks through a sequence of five questions that, answered honestly, gets you to the right model in under a week. None of the questions are "what's the SOTA on MMLU."

Full lesson text

All 10 steps on one page — for reading, reference, and search.

Show

1. Stop reading leaderboards

Most LLM-selection mistakes happen before any code is written. A team reads a leaderboard, picks the top model, ships it, and discovers six weeks later that the model is wrong for the actual task — too slow, too expensive, hallucinates on their schema, fails on French customer messages.

The fix is to start from your task, not from a leaderboard. This lesson walks through a sequence of five questions that, answered honestly, gets you to the right model in under a week. None of the questions are "what's the SOTA on MMLU."

2. Step 1 — Define the task and the failure modes

Write one paragraph describing the task as if explaining it to a new colleague. Include: input format, output format, who consumes the output, what a perfect answer looks like, and — most importantly — what specific failure modes you cannot tolerate.

Examples of useful failure-mode statements:

  • "Hallucinated drug names are unacceptable; missing a real drug is recoverable."
  • "Latency over 800 ms breaks the voice interface; quality variance is acceptable."
  • "Wrong SQL is fine; wrong SQL that runs and silently returns bad numbers is catastrophic."

These statements drive every later decision. Models that pass benchmarks may still fail on your failure mode.

3. Step 2 — Build an offline eval set

Before testing any model, collect 50-200 real examples of the task with known good outputs. Sources: production logs (with PII redacted), customer-support transcripts, annotated edge cases, adversarial examples your engineers can think up.

Guidelines:

  • At least 50 examples. Below that, you can't distinguish two real candidates statistically.
  • Stratify by user segment, language, length, difficulty. Don't let the easy cases dominate.
  • Include known failure modes explicitly — at least 10-20% of your set should be cases you expect to fail.
  • Define grading. Exact match, LLM-as-judge with a rubric, human review, or a hybrid. Decide before you grade.

This is the most under-invested asset in LLM projects and the highest-leverage one.

4. Step 3 — Shortlist candidates

Pick 3-5 candidates that span the realistic space. A balanced shortlist in 2026 might look like:

  • One closed frontier (Claude Opus 4 or GPT-5 or Gemini 2.x Pro) — sets the quality ceiling.
  • One closed mid-tier (Claude Sonnet, GPT-4.1, Gemini Flash) — usually the production sweet spot.
  • One reasoning model (o3 or Claude with extended thinking) — only if the task is hard reasoning.
  • One open-weights candidate (Llama 4, Qwen 2.5-72B, DeepSeek V3) — your fallback if cost or residency bites.
  • Optionally one small/local model (Phi, Gemma, Llama 3 8B) — if latency or edge deployment matters.

Resist the urge to evaluate 15 models. Three serious candidates evaluated well beats a dozen evaluated badly.

5. Step 4 — Measure three numbers, not one

Run each candidate against your eval set and record three dimensions:

  1. Quality — pass rate or score against your grading rubric.
  2. Latency — p50 and p95 time-to-first-token and time-to-completion, measured from your region.
  3. Cost per task — total tokens (input + output, including any reasoning tokens) times per-token price.

Report these as a table. The "best" model is rarely the highest quality — it is usually the one with the best quality at acceptable latency under your cost budget. Make the constraint binding before you optimize quality.

6. Selection decision flow

From task definition to deployed model.

flowchart TD
A["Define task and failure modes"]
A --> B["Build offline eval set (50-200 ex.)"]
B --> C["Shortlist 3-5 candidates"]
C --> D["Measure quality, latency, cost"]
D --> E["Filter: meets latency budget?"]
E --> F["Filter: meets cost budget?"]
F --> G["Pick highest quality remaining"]
G --> H["Pilot in production with shadow traffic"]
H --> I["Promote if eval and pilot agree"]
I --> J["Re-run eval at each model release"]

7. Step 5 — Watch for context and tool-use needs

Quality on the eval set isn't enough if the production shape is different. Two pitfalls show up late:

  • Context length. Your eval may use 4K-token examples but production sees 80K-token transcripts. Verify your top candidate's effective recall at production length — many models advertise 128K+ context but lose accuracy past ~32K.
  • Tool use / function calling. If your app calls tools (search, code-exec, SQL, internal APIs), test the candidates on your tool schema. Claude and GPT-5 lead on long tool-use trajectories; some open models trail badly.

Add a slice of your eval set that mirrors the long-context and tool-use shape of production, even if it's tedious to construct.

8. Don't optimize for benchmarks — a worked example

Hypothetical: a team building a customer-support summarizer. Public leaderboards say o3 is the strongest model. They ship o3, get high quality, and discover three things:

  • p95 latency is 18 seconds (reasoning tokens) — agents complain.
  • Cost per ticket is 12 cents — finance complains.
  • The summaries are too detailed — managers wanted three bullets, got six paragraphs.

They re-eval with Claude Sonnet 4 and Gemini Flash 2.x. Sonnet hits the same quality on their eval, with p95 of 1.8 seconds and 0.8 cents per ticket. The team ships Sonnet and saves $10K/month. The benchmark winner lost the task it wasn't designed for.

9. The selection checklist

Pin this somewhere visible before picking a model:

  • Task is described in one paragraph including failure modes.
  • Offline eval set has 50-200 real examples, stratified, with at least 10-20% adversarial.
  • Grading method is defined (exact-match, judge LLM, human, hybrid).
  • Latency budget (p95) is written down.
  • Cost-per-task budget is written down.
  • Data-residency / privacy constraints documented.
  • Context length and tool-use needs mirrored in the eval.
  • At least 3 candidate models spanning closed frontier / mid-tier / open.
  • Quality, latency, and cost reported as a table — winner is the best on all three constraints, not just quality.
  • A plan to re-run the eval on every major model release (monthly is a reasonable cadence).

10. Re-evaluate on every release

Model selection is not a one-time decision. New checkpoints land every few weeks; pricing drops faster than that. Treat your eval set as a standing test you re-run automatically when any candidate releases a new version. A model that loses 2% quality but drops 70% in price often deserves to swap in.

Make the eval one command. Pin the prompts, the seed where possible, and the grading rubric in version control. Six months in, your eval set itself becomes the most valuable engineering artifact in the project — more than any specific model choice.

Check your understanding

The lesson ends with a 5-question quiz. Take it in the player above to see your score.

  1. What is the single most important thing to do *before* picking a model?
    • Read the latest leaderboard rankings
    • Define the task and its failure modes, then build an offline eval set
    • Sign up for the most expensive API and test it
    • Pick the model with the largest context window
  2. Roughly how large should an initial offline eval set be to discriminate between two real candidate models?
    • 5-10 examples
    • 50-200 examples, stratified by user segment and difficulty
    • 10,000 examples minimum
    • Whatever fits in one prompt
  3. When measuring candidate models, the right thing to record is:
    • Quality only
    • Cost only
    • Quality, latency (p50 + p95), and cost per task
    • MMLU score from the provider's marketing page
  4. You picked the SOTA reasoning model and shipped, then discovered p95 latency is 18 seconds. The cleanest fix is:
    • Tell users to be patient
    • Re-run your eval set on cheaper, faster mid-tier chat models — promote whichever hits the same quality bar at acceptable latency
    • Raise the cost-per-task budget
    • Disable streaming
  5. Your offline eval has 4K-token examples but production sees 80K-token transcripts. What should you do?
    • Trust the eval — context length doesn't affect quality
    • Only use models with 1M context windows
    • Add a slice of long-context examples that mirror production shape and re-evaluate effective recall on each candidate
    • Always truncate production inputs to 4K

Related lessons

AI
advanced

Loop engineering: verification, orchestration, and anti-patterns

Make loops trustworthy. Adversarial verification panels, sub-agent orchestration, loop-until-dry for unbounded discovery, loop-until-budget for paid depth, multi-modal sweeps with diverse prompts, eval-driven cap selection, and the five anti-patterns — silent caps, infinite plans, drift, premature termination, thrashing — that ship to prod more than they should.

9 steps·~14 min
AI
intermediate

RAG Evaluation in Production: Metrics, Tools, and Cadence

Learn how to systematically evaluate Retrieval-Augmented Generation systems in production using RAGAS, TruLens, and Phoenix — covering golden sets, retrieval drift, embedding drift, and cost-aware eval scheduling.

12 steps·~18 min·audio
AI
intermediate

Evaluating AI Agents: From Final Answers to Full Trajectories

A rigorous look at how to measure agent performance — trajectory-level vs final-answer evals, canonical multi-step benchmarks (SWE-bench, WebArena, OSWorld, GAIA), LLM-as-judge pitfalls, and why your eval is your spec.

12 steps·~18 min·audio
AI
intermediate

Loop engineering in production: state, errors, and budgets

Once an agent loop runs past a few iterations the failure surface shifts: state bloats, tool calls fail in three distinct ways, retries multiply if you stack them wrong, and unbounded budgets rack up four-figure bills overnight. The state, errors, and budgets you need to make the loop survive contact with reality.

8 steps·~12 min