Prompt caching
- Context
- Industry term
Prompt caching saves and reuses work a model has already done when requests begin with exactly the same content. It can reduce cost and delay. The model still writes a new answer each time.
Before answering, a model, the trained AI inside an app, does calculations to process what it receives. The provider, the company running the model, can save those results. Reuse starts at the beginning of the next request and continues while the content matches. Once the content differs, the rest needs processing again.
Where you’ll meet it
You've built a staff question-answering app that sends the company handbook with every question. Its usage report labels some of that incoming text “cached input,” even though employees asked different questions and received different answers.
What to do
Put the unchanged handbook and instructions that apply to every request before the employee's question. If a date or employee name changes on every request, place it afterward. The usage report can then show how much input was reused.
Easy to mix up
| Prompt caching | Response caching |
|---|---|
| Work already done to process the request's beginning | An answer stored from an earlier request |
Check yourself
Your app puts a 20-page catalog before each new question. You correct a price on page 2; everything else stays identical. The earlier computation is still saved. Which complete pages could still qualify for reuse?
Go deeperMatching text must also meet the provider's rules for reuse
Gim and colleagues' 2023 paper, “Prompt Cache,” studied reusing attention states: calculation results a model uses to relate parts of its input while generating text. Vendor features have their own eligibility rules. Anthropic and OpenAI document minimum input lengths, retention periods, and separate rates for saving and reusing those values. These rules affect whether repeated input produces a saving.
OpenAI's prompt-caching documentation warns that tool definitions and request settings can affect a match, even when the visible message hasn't changed. Reusing the calculations also leaves the input's size unchanged: the text still counts toward the context window, the limit on how much the model can work with in one request.
Sources
- Prompt caching, Anthropic Claude Docs
- Prompt caching, OpenAI API docs
- Prompt Cache: Modular Attention Reuse for Low-Latency Inference, Gim et al., arXiv