The Silent Cost Multiplier in Every Chat Feature: Your Conversation History

The Math No One Shows You

Most cost estimates are built from a single request: one prompt, one response, one token count. But chat features don't send one request — they re-send the entire conversation history with every single turn. A small amount of waste in turn one doesn't stay small. It gets re-sent on turn two, turn three, and every turn after that, compounding until 100 wasted tokens in turn 1 becomes roughly 3,000 wasted tokens by turn 30.

Context Accumulation Visualizer

Click "Send message" to simulate a conversation growing turn by turn and watch tokens, cost, and efficiency move in real time.

The Quality Cliff

This isn't only a cost problem. The RULER benchmark found that GPT-4o accuracy drops into the high-60% range once context length reaches roughly 32K tokens — long context windows don't just cost more, they measurably degrade answer quality. Sending unnecessary history is both more expensive and worse for the user.

Trim vs Full Calculator

Compare the cost of sending full conversation history every turn against trimming to only the last 6 turns.

Three Strategies Ranked

1. RAG / retrieval-based context (best, ~80-90% savings) — only pull in the specific prior content relevant to the current turn instead of the full transcript.
2. Summarize older turns (~60-80% savings) — replace older history with a compact running summary, keeping recent turns verbatim.
3. Trim to last N turns (~40-60% savings) — simplest to implement, drop everything beyond a fixed recent window.

All three beat sending full, unbounded history by a wide margin, and all three can be implemented incrementally without rearchitecting a chat feature.

Copy// Track per-turn token growth across a conversation
fetch('https://llm-track.com/api/ingest', {
  method: 'POST',
  headers: { 'x-api-key': process.env.LLMTRACK_KEY },
  body: JSON.stringify({
    provider: 'anthropic',
    model: response.model,
    feature_name: 'chat-completion',
    conversation_id: sessionId,
    turn_number: turnCount,
    total_tokens: response.usage.input_tokens + response.usage.output_tokens,
    status: 'success'
  })
}).catch(() => {})

FAQ