How prompt cache hits cut LLM input cost, when cache writes matter, which workloads benefit, and how to estimate savings in TokenCALC before you ship.
Try it in TokenCALC
Open the cache savings tool with a model that publishes cached-input rates, set your hit %, and compare monthly spend.
What prompt caching is
Prompt caching lets many providers bill a repeated prompt prefix (system instructions, large context, or tool schemas) at a discounted cached-input rate. New or changing tokens still bill at the normal input rate. Output tokens are unchanged by cache hits.
Caching is an input-side optimization. It does not make a chatty model cheap if output dominates the bill.
Hits, writes, and break-even
A cache hit means that portion of input is billed cheaper. Some providers also charge a cache write when the prefix is first stored or refreshed. If writes are published in the TokenCALC catalog, the cache savings tool shows the one-time write cost and how many requests it takes to repay it from per-request savings.
If your prefix changes every request, you may pay writes without enjoying hits. Stability is the product requirement, not only the pricing feature.
When caching wins
Caching pays off when the same large prefix appears across many requests: agents, support bots, and RAG systems with a stable system prompt. It helps less when every request is unique short text.
Always measure with your real prefix size, hit rate, and monthly volume. A marketing “up to 90% off input” claim means nothing without your traffic shape.
How to estimate in TokenCALC
Use the cache savings tool for hit-rate math, then open the main cost calculator with the same model to combine Exact or Approx token counts, cache %, and monthly users × messages. Only models with published cache rates expose the controls.
Compare against a no-cache baseline on the same prompt. If savings are small, fix output length or model tier before you overfit cache engineering.
Caching vs batch vs model downgrades
Cache helps repeated prefixes. Batch helps latency-tolerant jobs. A smaller model helps when quality allows. These levers stack only when the provider and workload support them.
Use batch pricing and cache savings as satellites, then confirm on the cost calculator. Read the LLM API pricing guide for the wider billing picture.
Common mistakes
Avoid these cache mistakes.
- Expecting cache to fix output-heavy bills
- Changing the prefix every request and wondering why hits are low
- Ignoring published cache write costs
- Enabling cache UI mentally for models with no published cache rates
- Skipping invoice checks after launch
Best practices
Put stable instructions and schemas at the front of the cached prefix. Keep volatile user content outside it. Log hit rates in production. Re-estimate monthly savings when traffic mix changes.
Start in TokenCALC with a realistic prefix token count, not a slogan.
Real-world scenarios
An agent platform ships the same tool schema and safety policy on every call. The user message changes, but two thousand tokens of prefix repeat. Cache hits turn that prefix from full input price to cached input price on supported models, which materially changes unit economics at scale.
A content generator rebuilds the system prompt per user session with dynamic timestamps and random examples. The prefix never stabilizes, cache writes churn, and savings never materialize despite engineering effort.
A RAG app caches the system instructions plus a large static knowledge summary, but injects fresh retrieval chunks after the cached block. Only the stable portion qualifies for hits; retrieved text still bills at standard input rates.
Step-by-step in TokenCALC
Open the cache savings tool from this guide or the tools hub. Select a model that shows published cached input rates in the catalog.
Enter prefix token count, expected cache hit percentage, and monthly request volume. Review break even against any listed cache write cost.
Return to the main cost calculator with the same model, paste full prompt tokens, set cache hit percent, and compare monthly totals against a zero cache baseline.
Related concepts
Prompt cost explains base input and output math. Batch pricing addresses a different lever for latency tolerant jobs. LLM API pricing situates caching among other rate modifiers.
Exact vs Approx matters for prefix sizing: an Approx prefix count still works for directional savings if you apply margin before committing.
Expert notes
Cache keys are provider specific. Prefix byte identity, ordering, and TTL rules differ across OpenAI, Anthropic, and Google. Read the provider doc for minimum prefix length and eligibility windows.
Measure hit rate in staging with production shaped traffic. Demo prompts that change every refresh hide real economics.
Caching optimizes input. If output tokens dominate your bill, shorten completions or choose a smaller model before you invest in cache engineering.
Warnings
Marketing claims about maximum cache discounts assume ideal prefix reuse. Your hit rate may be lower. Do not enable cache dependent pricing in customer contracts without measured data.
Cache write fees can dominate low volume workloads. Model break even request counts before you treat caching as free money.
Expert notes on rollout
Roll out caching in stages: measure hit rate on a canary deployment, compare invoices week over week, then expand traffic. TokenCALC gives pre launch directional savings; production logs validate them.
Document which parts of your prompt are stable enough to cache. Product and legal changes to disclaimers or policy text can invalidate a previously hot prefix overnight.
Frequently asked questions
What is prompt caching?
A provider feature that bills repeated prompt prefixes at a discounted cached-input rate when eligibility rules are met.
Do cache hits reduce output cost?
No. Output tokens still bill at output rates. Caching is an input side optimization.
What is a cache write?
Some providers charge to store or refresh a prefix. Writes can offset hit savings if traffic is low or prefixes churn frequently.
When should I use the cache savings tool?
When your model publishes cache rates and you reuse a large stable prefix across many requests. Enter realistic hit rates, not best case marketing numbers.
Is caching available for every model in TokenCALC?
No. Controls appear only when curated cache rates exist for that model.
What belongs in the cached prefix?
Stable system instructions, tool definitions, and static context blocks that repeat unchanged across requests. Keep volatile user text outside when possible.
Next steps
Use the calculator links above for Exact or Approx counts on your own prompts, then browse related guides and model pages to compare pricing assumptions.