LLM Pricing and Latency: What Actually Drives Cost

LLM cost has two unintuitive properties that drive almost every production decision. First, output tokens cost 3-5x more than input tokens on every major API — Anthropic, OpenAI, Google all use this asymmetry. Second, latency and cost are coupled: the same things that slow a model down (more output tokens, reasoning chains, large context) also raise the bill.

If you internalize only one thing from this lesson: cost is dominated by output tokens, latency is dominated by time-to-first-token (TTFT) plus generation speed. Optimizing for one usually helps the other; they trade off only at the margins.

Rough 2026 numbers for context (always check current pricing):

• Claude Sonnet 4.x: input ~$3/M tokens, output ~$15/M — 5x ratio.
• GPT-5 / GPT-4.1: input ~$2-3/M, output ~$10-15/M — 4-5x ratio.
• Gemini 2.x Pro: input ~$1-2/M, output ~$5-10/M — similar ratio.
• Open models (DeepSeek, Llama via Fireworks/Together): $0.1-1/M, much tighter ratio.

The practical implication: a 5000-input / 500-output task and a 500-input / 5000-output task can cost the same number of tokens but ~5x different dollars. Push work into the prompt; constrain the output. Use stop sequences, max_tokens, JSON mode, and explicit "answer in one sentence" instructions ruthlessly.

Anthropic, OpenAI, and Google all offer prompt caching — cached input tokens cost 80-90% less than fresh ones, often with sub-100ms TTFT improvements. The user marks a long stable prefix (system prompt, documentation, retrieved context) and subsequent calls reuse the cached prefix.

Anthropic's pattern (illustrative):

messages = [{
    "role": "user",
    "content": [
        {"type": "text", "text": LONG_DOC,
         "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": user_question}
    ]
}]

If your system prompt is 5K tokens and you serve 100K calls/day, caching it cuts ~$300/day at Sonnet pricing. This is usually the single highest-ROI optimization in an LLM app.

If a task is not user-facing — overnight backfills, embedding regeneration, eval runs, bulk classification — every major provider offers a batch API with roughly 50% off in exchange for hours of turnaround (typically up to 24h). OpenAI's /batches, Anthropic's Message Batches, Google's Vertex Batch.

Real pattern: route real-time traffic to the regular API, accumulate anything async into a daily batch. A team running a million-document monthly summarization saves serious money for one line of code change. Check whether your batch supports the same models, tools, and caching as your real-time path — sometimes there are subtle gaps (e.g. no streaming, fewer tool schemas).

Where the seconds go.

Two latency numbers matter, not one. Time-to-first-token (TTFT) is the wall clock until the first visible character — dominated by queue, prefill of the prompt, and reasoning tokens. Tokens-per-second (TPS) is how fast generation streams after that — typically 30-150 tps for big models, 200-500 tps for small ones.

Total latency ≈ TTFT + (output_tokens / TPS).

For a chat UI, TTFT under 1 second is the make-or-break threshold; users tolerate slow continuation if the first words appear fast. Always stream if your UI can display partial output. For voice, you need TTFT under ~400 ms — only Gemini Flash, Haiku, and a handful of small open models hit that consistently.

Reasoning models (o1, o3, Claude Opus with extended thinking, Gemini 2.x Thinking, DeepSeek R1) emit invisible reasoning tokens before the visible answer. You're billed for those tokens at the higher output rate.

Real numbers vary, but a single hard problem can burn 5,000-50,000 reasoning tokens before answering — adding seconds-to-minutes of latency and multiplying cost by 5-50x vs the same model in non-reasoning mode. On most providers you can cap or disable reasoning effort (reasoning_effort=low/medium/high).

Rule of thumb: use reasoning models only for tasks where a wrong answer is genuinely expensive. For most chat, classification, summarization, and RAG, a non-reasoning chat model is the right default.

Pricing scales linearly with input length — feeding a 200K-token document costs 200x what a 1K prompt costs at the same per-token rate. Some providers also charge a premium tier for ultra-long context (Gemini's 1M-token tier, Claude's 1M variant).

More subtly, latency scales worse than linearly with context. Prefill of a 200K prompt can take seconds even on the fastest hardware. Common mitigations:

• Cache stable prefixes (system prompt, documentation).
• RAG — retrieve and inject only the 2-10K tokens you actually need.
• Summarize earlier turns when conversations get long.
• Choose the model wisely — long-context models aren't always cheaper per token even when you need the length.

When does self-hosting an open model beat paying a closed API per token? Rough back-of-envelope, assuming an H100 at ~$2-4/hr renting capacity:

• One H100 serves a 70B model at ~100-300 tokens/sec aggregate, comfortably for several concurrent requests with batching.
• Steady throughput per H100 per month: roughly 200B-700B tokens, depending on input/output mix.
• API equivalent cost at Sonnet rates: tens of thousands of dollars.

The break-even arrives around $50K-100K/month of sustained closed-API spend on a single model — below that, ops overhead eats the savings; above it, self-hosting wins quickly. The math improves further when you fully utilize the GPU and worsens when traffic is bursty (idle GPUs are the killer).

Walk this list before paying for capacity expansion:

• Enable prompt caching on every stable prefix over ~1K tokens.
• Move all non-real-time work to the batch API.
• Set tight max_tokens and clear "answer in N words" instructions.
• Cap or disable reasoning effort except where it pays off.
• Always stream to the UI; measure TTFT, not just total latency.
• Use RAG to keep prompts short instead of stuffing huge context.
• Tier models: cheap default (Haiku/Flash/4.1-mini), escalate to a strong model only when a cheap-model confidence check fails.
• Re-evaluate against open-weights candidates if your sustained spend on one model exceeds ~$50K/month.

These together routinely cut LLM bills by 3-10x with no visible quality regression.