We broke our own cache hit rate. The bill explained everything.
Prompt caching is the largest line-item discount in LLM billing: DeepSeek bills cache hits at roughly one-tenth of the miss price, Anthropic reads cost 10% of base input, OpenAI halves repeated prefixes automatically. Notes from a 60-request field test, including the timestamp bug that dropped our hit rate from 91% to zero.
A user report started this: “I ran two continuations back to back and the second one billed almost everything at full price.” That should be impossible. Continuation is the most cache-friendly workload there is. So we took a 500,000-word novel, ran 60 continuation requests against DeepSeek, and copied the billing fields off every single response. What follows is the lab notebook, including the part where the bug turned out to be ours.
The numbers, before we broke anything
Request 3: 21,400 input tokens, 18,900 of them cache hits. Request 20: 33,700 in, 31,200 hits. Request 41: 47,800 in, 44,600 hits. That last one is a 93% hit rate, and DeepSeek charges roughly one-tenth of the normal input price for the matched portion (they were the first to turn this on by default for everyone, back in August 2024). Effective input cost on request 41: under 15% of list price.
The mechanism is simple enough to state in one sentence. While reading your prompt, the model computes an intermediate state per token (the KV cache), that state depends only on what came before, so the provider stores it and reloads it whenever a new request starts with an identical prefix. Continuation requests are almost entirely identical prefix: same system prompt, same story bible, same chapters, plus one new instruction at the end. Nearly free by construction.
How we drove 91% to zero
Midway through the test we shipped a build that added one line to the system prompt: the current time, “21:47” style, so the model could reference it. Hit rate that evening: 0%. Thirty-plus requests at full price. The cache matches prefixes verbatim, the timestamp changed every minute, so every request rewrote its own prefix. We switched to date-only injection and the rate was back above 90% the next day.
We have since hit two more versions of the same bug. Sampling parameters prepended to the prompt on rerolls. Lorebook entries re-sorted by relevance so their order changed between turns. One discipline covers all three cases: stable content first, volatile content last. No provider will enforce this for you. It lives entirely in the application.
Provider rules, the short version
| Provider | Activation | Hit price | Detail |
|---|---|---|---|
| DeepSeek | always on | ~1/10 of miss price | 64-token block matching |
| OpenAI | automatic ≥1,024 tokens | 50% of input | no code changes |
| Anthropic | explicit breakpoints | 10% of base input | writes cost +25% |
| Gemini | implicit + explicit | discounted | explicit caches take a TTL |
Sources: OpenAI, Anthropic, Gemini. One caveat we have not fully tested: Anthropic's 25% write premium means a write-heavy, read-light pattern (switching books every request, say) could plausibly cost more with caching than without. Treat that as an open question rather than advice.
The old advice this overturns
“Keep your context short” used to be the standard cost tip, and apps trimmed history and lore aggressively to follow it. Cache pricing flips the trade. Repeated long context is nearly free after the first request. What costs money is whatever changes each turn, plus output. So keeping the full character card and the pinned worldbook in every request stops being a luxury. Your characters keep their memory, and you pay one-tenth price for the privilege.
Do not take our word for any of this. Every provider returns cache fields in the response: DeepSeek calls it prompt_cache_hit_tokens, OpenAI calls it cached_tokens. Foreverse surfaces these fields per request in the API record screen. Run two continuations back to back. If the second shows zero hits, your app has a timestamp somewhere, and now you know exactly where to look.
FAQ
What exactly does prompt caching cache?
The model's intermediate attention state (the KV cache) for a prefix it has already processed. When a request starts with the exact same content as a previous one, the provider loads that state instead of recomputing it, and bills the matched portion at a steep discount. Matching is strictly prefix-based: identical from the first character up to the divergence point.
How big are the discounts across providers?
Order-of-magnitude, per official pricing pages: DeepSeek bills cache-hit input at roughly one-tenth of the miss price; Anthropic cache reads cost 10% of base input (writes carry a 25% premium); OpenAI automatically halves input for repeated prefixes over 1,024 tokens; Gemini offers implicit and explicit context caching with discounted hit pricing.
Why do long conversations get cheaper per turn?
Turn N's input is the full history of turns 1..N-1 plus one new message. That history is precisely the prefix the provider just processed, so it hits cache naturally. The deeper the conversation, the higher the cacheable share. By turn 50, over 95% of input tokens are discount-priced, and marginal cost comes almost entirely from new text and output.
What breaks caching?
Anything that mutates the prefix: injecting a per-request timestamp into the system prompt, reordering lorebook entries each turn, or editing an old message mid-history. Matching is exact. Change one character early in the prompt and everything after it bills at full price. Design rule: stable content first, volatile content last.
Questions or ideas? Join our Discord →