Your System Prompt Is Your Biggest LLM Cost Driver. Most Developers Don't Know This.

Why This Is Counterintuitive

Most cost conversations focus on which model you call or how long the output is. Nobody budgets for the system prompt because it "isn't the real work" — it's just instructions, persona, tool schemas, and formatting rules. But pricing doesn't care what tokens are for. A 4,000-token system prompt costs the same as 4,000 tokens of user content, and it gets billed on every request — including the 99% of requests where nothing in it changed since the last call.

Prompts grow organically: a sentence here, a tool schema there, a paragraph of edge-case handling. Most teams don't realize they've built a multi-thousand-token prefix that gets retransmitted and rebilled on every single interaction.

How Caching Works: OpenAI vs Anthropic

Both major providers offer prompt caching, but the mechanics differ:

• Anthropic (Claude): you mark an explicit cache breakpoint. Tokens before it are cached for roughly 5 minutes and billed at about a 90% discount on cache hits, with a smaller write cost on cache misses.
• OpenAI: caching is automatic for prompts over 1,024 tokens with an identical prefix — no explicit breakpoint needed. Cache hits depend on the prefix staying byte-identical, so dynamic content placed before static instructions breaks the cache.

Either way: put static content (system prompt, tool definitions, few-shot examples) first, and dynamic content (user input, retrieved context, timestamps) last. Order determines whether you get the discount at all.

Token Breakdown of a Typical Request

For a typical chat feature with a static system prompt, history, and a short user message, the token budget usually looks like this:

The largest lever on input cost — by far — is the part of the request that never changes between calls.

Copy// Anthropic-style cache breakpoint: static system prompt first
const response = await anthropic.messages.create({
  model: 'claude-sonnet-4-6',
  system: [
    {
      type: 'text',
      text: STATIC_SYSTEM_PROMPT,
      cache_control: { type: 'ephemeral' } // marks the cache breakpoint
    }
  ],
  messages: [{ role: 'user', content: userMessage }]
})

FAQ