You're Using GPT-4 for Everything. That's a 30x Mistake for Half Your Features.

The Default Trap

When a team ships its first LLM feature, the model choice is usually made once, for the hardest feature in the roadmap, and then reused everywhere else by default. Classification, extraction, simple chat replies, and translation all get routed through the same top-tier model that was picked for complex reasoning. It works — every task gets a correct answer — but most of those tasks never needed that much model.

A Real Case

A mid-size support tool ran every ticket through a GPT-4-class model for three jobs: tagging the category, summarizing the thread, and drafting a reply. Tagging and summarizing were simple, high-volume, low-ambiguity tasks. After measuring real token counts and routing just those two jobs to a smaller model, the team cut spend on that pipeline by 78% with no measurable drop in tagging accuracy.

Interactive: Model Match Analyzer

Quality-Cost Confusion

The case against switching to cheaper models is usually "quality will drop." Sometimes it does. But the real cost of a cheaper model isn't just its token price — it's also its retry rate. A model that's 10x cheaper but fails validation twice as often can end up costing more once you count the retried calls. The fix is to measure retry rate per model per task, not just sticker price per token, before deciding a switch was a mistake.

The Salih Yildirim Case

One of the most cited real-world examples of this pattern comes from engineer Salih Yildirim, who documented switching a production pipeline from a top-tier model to a smaller tier for tasks that didn't need full reasoning capability, reporting a 97% cost reduction on that workload after validating output quality on a real sample before the full rollout. The case is notable not because 97% is typical — it usually isn't — but because the validation step (testing on real production samples before switching all traffic) is exactly the part most teams skip.

Interactive: Overpaying Calculator

The Benchmarking Fallacy

Public benchmarks compare models on tasks that rarely resemble your actual traffic shape — your prompt lengths, your output format constraints, your retry behavior under your validation rules. A model that wins a benchmark can still be the wrong (or right) choice for your feature. The only benchmark that matters is your own request data run through both models side by side.

Copy// Route by feature, not globally — measure before switching traffic
const MODEL_BY_FEATURE = {
  'ticket-tagging': 'gpt-4o-mini',
  'ticket-summary': 'gpt-4o-mini',
  'reply-draft': 'gpt-4o'
}

fetch('https://llm-track.com/api/ingest', {
  method: 'POST',
  headers: { 'x-api-key': process.env.LLMTRACK_KEY },
  body: JSON.stringify({
    provider: 'openai',
    model: MODEL_BY_FEATURE[featureName],
    feature_name: featureName,
    total_tokens: response.usage.total_tokens,
    latency_ms: Date.now() - startedAt,
    status: 'success'
  })
}).catch(() => {})

FAQ