Prompt caching

Intermediate · 9 min

How a cached prefix removes repeated prefill work, how to arrange a prompt so the cache actually hits, and what it saves.

Before thisContext engineering

Efficient NLP · 9 min

Prompt caching

The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.

Taught by Efficient NLPReport a problem with this video
Not started
Set this lesson's status

What you'll learn

  • Caching stores the key-value tensors computed during prefill, so a repeated prefix is loaded rather than recomputed.
  • The cache matches on an exact prefix from the first token, so anything variable must come after everything stable.
  • The saving grows with how much of the prompt is shared, which makes it largest exactly where costs hurt most — long system prompts and agent loops.

The problem: paying for the same work twice

Serving a model call has two phases. Prefill reads the entire input in one parallel pass, building the key and value tensors for every token at every attention layer — the same KV cache described in the context window lesson. Decode then generates output tokens one at a time, each step reading what prefill built. Prefill is the expensive part, and it scales with input length. Now notice something: if your next request opens with the exact same two thousand tokens as the last one — the same system prompt, the same tool definitions, the same long reference document — the provider redoes all of that prefill work from scratch, even though the answer would come out identical every time. That waste is the whole problem caching exists to fix.

What gets reused

Think of it like a kitchen that preps a big pot of stock once at the start of service, instead of making fresh stock for every single order. Caching keeps the computed key and value tensors for a prefix around between calls, keyed to that exact sequence of tokens. When a new request opens with a prefix that matches one already cached, the provider skips redoing that part of prefill and just reuses the stored tensors, picking up fresh computation only from the first token that differs. The cached portion bills at a small fraction of its normal per-token price and adds almost no latency, because the expensive parallel work behind it was already paid for once. Nothing about what the model can attend to or how it reasons changes — this is purely about how much redundant computation the provider has to redo.

Why the match has to be exact, from token one

The cache keys on an identical prefix starting at position zero — not on “close enough,” not on semantic similarity. One differing token anywhere in the prefix, and everything from that position onward falls out of cache, because each token’s key and value tensors depend on every token before it through attention. That single fact decides how you should lay out a prompt: put the most stable material first — system instructions, tool definitions, long reference documents, since these barely change call to call — then retrieved documents, and only at the very end anything request-specific, like a timestamp or a session ID. Put that timestamp near the top out of habit and you’ve invalidated the cache on every single call, since no two requests ever share that value.

Doing the arithmetic

Say a system prompt plus tool definitions run three thousand tokens, and an agent loop reuses them across twenty iterations per task. Without caching, that’s three thousand tokens of prefill work repeated twenty times over — sixty thousand tokens of computation spent on content that never changed. With caching, only the first call pays full price for that chunk; the other nineteen pay the steeply discounted cached rate, with full price landing only on the new content appended at the end. The saving scales with how much of the prompt is shared and how often it repeats, which is exactly why agent loops — where most of the prompt is identical turn after turn — gain far more from this than a single-turn chat ever would.

What differs by provider

Providers disagree on the minimum prefix length that qualifies for caching, on how long a cached entry survives before it’s evicted, and on whether caching happens automatically or needs an explicit marker in the request showing where the cacheable prefix ends. Check the current specifics directly rather than assuming last year’s numbers still hold. What doesn’t change across providers is the underlying shape of the trick — stable content first, variable content last — and it stays one of the cheapest, biggest cost reductions available to most LLM applications, tying straight into the broader economics in cost and latency.

Check yourself

  1. 01Why must a timestamp or user id go at the end of a prompt rather than the start?

    Matching is on an exact prefix from the first token. A value that changes per request invalidates everything after it, so putting it first means nothing is ever cached.

  2. 02Why do agent loops benefit from caching more than single-turn chat?

    Each iteration re-sends the whole accumulated history, so the shared prefix grows with every step. Without caching an agent pays repeatedly to prefill the same tokens.

Go deeper

The maths underneath

Notes

Private to you

Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark