KC 301-02 · Caching: prompt caching, KV cache

01One object, two bills

Start with the smallest useful vocabulary. Everything else in this lecture builds on these five words.

Jargon translation

Token — the unit a language model reads and writes. Roughly 3–4 characters of English, or one character in Chinese/Japanese/Korean. It is the unit of both pricing (dollars per million tokens) and work (one token generated per forward pass of the model).

Jargon translation

Cache — a place to keep the result of expensive work so you don't redo it. On a website, a cache saves a rendered page. In an LLM, the cache saves the attention state the model computed for the tokens it already read.

Jargon translation

KV cache — the key/value cache. Inside a transformer's attention, every token is turned into three vectors — a query (Q), a key (K) and a value (V). The K and V vectors of earlier tokens are what a new token looks back at. Those vectors don't change when you append a new token, so there is no reason to recompute them: you store them. That store is the KV cache. K = key, V = value, KV = “keys and values”.

Jargon translation

Prefix — the beginning of a request. Requests are a sequence: system prompt, tools, memory, history, then the new user message. Everything before the new message is the prefix. Caching only ever works on prefixes, and only on exact matches — this is the single most important fact in section 3.

Jargon translation

Prefill and decode — the two phases of a request. Prefill processes the whole prompt at once (parallel, compute-heavy — this is where a cache hit saves you money and time-to-first-token). Decode generates the answer one token at a time (sequential, memory-bandwidth-heavy — caching does not help here, the new tokens are genuinely new).

With those five words, the whole unit collapses into one table:

The two bills

	Hosted API (DeepSeek, Anthropic, OpenAI)	Your own box / phone (llama.cpp, Ollama)
Who holds the KV cache	The provider, on their disk and GPU memory	You, in VRAM or system RAM
What you pay	Dollars per token: a hit (reused prefix) vs a miss (recomputed prefix)	Memory bandwidth per generated token + RAM capacity for the cache
What limits you	The hit rate you can achieve	GB/s of memory bandwidth; how much context fits
What breaks it	Any change in the prefix, a model switch, expiry (TTL)	Longer context, bigger KV dtype, an OS that reclaims your RAM

Read that table twice. Gap 1 of this unit is the left column: an accounting question with a 50× price difference. Gap 2 is the right column: a physics question about how fast bytes can move. Same cache, two bills.

PrefixPrefix02Why a cache exists at all

The plain explanation

A transformer generates text one token at a time, left to right. To produce token 500, it needs attention over tokens 1–499. If you recomputed everything from scratch each step, producing a 500-token answer would cost on the order of 500 × 500 attention operations — the classic quadratic blow-up. The KV cache is the observation that the previous tokens' key and value vectors are already known and never change, so you compute each token's K and V once and keep them. New token, one new row appended; attention reads the whole table. That turns “recompute the past” into “read the past”, which is why it exists.

Jargon translation

Attention — the mechanism by which each token decides how much to “look at” every other token. Practically: a weighted average of past value vectors, where the weights come from comparing the new token's query against each past token's key.

This is why the HF Transformers docs describe the KV cache the way they do: KV scores “are calculated every time because the model predicts one token at a time... A KV cache stores these calculations so they can be reused without recomputing them.”

It is not free — it is RAM that grows with your conversation

The cache is a second copy of part of your state, and it grows linearly with context length. Meta's 2026 on-device survey[4] puts it plainly: the KV cache “grows linearly with sequence length and can dominate memory usage during long-context inference, often exceeding the model weights themselves.”

Here is the arithmetic, with real numbers from a model you may well be running locally — Qwen3-4B, whose published config.json says 36 layers, 8 key/value heads (grouped-query attention), head dimension 128:

Calc — KV bytes per token

KV cache bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_element
                         = 2 × 36 × 8 × 128 × 2 bytes  (f16)
                         ≈ 144 KiB per token

Cache-only size, Qwen3-4B

Context length	f16 KV cache	q8_0 KV cache
4,096 tokens	~0.60 GB	~0.30 GB
32,768 tokens	~4.8 GB	~2.4 GB

Those are cache-only numbers. On top of them sit the weights (2.5 GB for Qwen3-4B at Q4_K_M) and the runtime's own overhead. This table is the reason Apple ships a 4,096-token window on-device (section 6) and the reason your phone's “15.5 GB of RAM” behaves like 5 GB (section 6).

Where you meet it in practice

llama.cpp: -c, --ctx-size N is literally “how big is the KV cache you're willing to allocate”. Default 0 means “take it from the model”. -ctk / --cache-type-k and -ctv / --cache-type-v choose the cache's data type — f16 is the default, and q8_0, q4_0 and friends are quantization formats for the cache itself.
Ollama: same idea with different names. Ollama's default context window is 4096 tokens (OLLAMA_CONTEXT_LENGTH overrides it, /set parameter num_ctx per run), and OLLAMA_KV_CACHE_TYPE selects f16 (default), q8_0 (~1/2 the memory, “very small loss in precision”) or q4_0 (~1/4 the memory, more noticeable quality loss at high context).
Hugging Face Transformers: the cache is an explicit object you can choose — DynamicCache (default, grows), StaticCache (pre-allocated, for torch.compile), QuantizedCache (smaller), and offloaded variants that keep only the current layer's cache on the GPU. The doc's framing of the trade-off is the one to remember: the KV cache “can occupy a significant portion of memory and become a bottleneck for long-context generation.”

Why it matters: the KV cache is the resource that ties together three things you normally think about separately — your API bill, your time-to-first-token, and your maximum usable context. You cannot tune one without touching the others.

BillBill03Prompt caching on the API: hits, misses and “cache prefix units”

The plain explanation

When you call a hosted model, you send the whole conversation every turn — the API is stateless. But the provider is not: they can keep the KV cache from your last request on their disk, and if your next request starts with the identical tokens, they read that saved state instead of recomputing it. Identical prefix → cache hit (cheap, fast). Any difference → cache miss (full price, full prefill work).

Three properties are load-bearing, and DeepSeek's Context Caching guide[1] states all three explicitly:

Prefix only. Caching covers the beginning of the request, not the middle or the end.
Exact match. A hit requires that the request fully matches a persisted “cache prefix unit”. Not “mostly matches”. Not “semantically similar”. The same tokens, in the same order, byte for byte.
Best effort. DeepSeek's docs: “The cache system works on a ‘best-effort’ basis and does not guarantee a 100% cache hit rate.” And it is not instant: “Cache construction takes seconds”, and an unused cache is cleared “usually within a few hours to a few days.”

What counts as a cacheable unit

DeepSeek persists cache prefix units at:

Request boundaries — the end of your input, and the end of the model's output. So in a multi-turn conversation A + B (round 1) followed by A + B + C (round 2), round 2 fully matches the unit A + B and hits.
Detected common prefixes — if round 1 is A + B and round 2 is A + C, round 2 does not hit (A + C never matches the unit A + B). But the system notices the shared A, persists it as its own unit, and a third request A + D then hits on A. Hits can therefore take a request or two to “warm up”.
Fixed token intervals — for long inputs and outputs, units are carved out at intervals so a very long prefix isn't uncacheable forever just because it never reaches an end position.

Reading the hit rate off the response

DeepSeek exposes the accounting in the usage object, so you can measure instead of guess:

Usage — DeepSeek response

{
  "id": "chatcmpl-...",
  "model": "deepseek-flash",
  "usage": {
    "prompt_tokens": 61248,
    "completion_tokens": 1842,
    "total_tokens": 63090,
    "prompt_cache_hit_tokens": 58752,
    "prompt_cache_miss_tokens": 2496,
    "prompt_tokens_details": { "cached_tokens": 58752 }
  }
}

Read it like a bill: prompt_cache_hit_tokens is the part of your input that was served from the provider's disk cache (cheap tier); prompt_cache_miss_tokens is the part it had to actually compute (full price). In this example, 96% of a 61K-token prompt was a hit. The other vendors use different names for the same two numbers: Anthropic reports cache_creation_input_tokens and cache_read_input_tokens; OpenAI reports usage.input_tokens_details.cached_tokens. Whatever the field is called, that field is your cost lever's dashboard.

The 50× lever, verified

On the current DeepSeek price page[2] (fetched 2026-09-13, for the model now called deepseek-flash, i.e. DeepSeek-V4.1-Flash):

Price — per 1M input tokens

Input tier (per 1M tokens)	Off-peak	Peak
Cache hit	$0.003	$0.006
Cache miss	$0.15	$0.30
Output	$0.60	$1.20

Key idea

That is a 50× spread between a hit and a miss on input, in both pricing windows (off-peak is half of peak; peak hours are 01:00–04:00 and 06:00–10:00 UTC, Mon–Fri). The audit's anchor figure — “a cache hit costs roughly 1/50th of a miss ($0.003 vs $0.15 per 1M tokens)” — reproduces exactly on the live page, and I re-checked it directly rather than trusting the note.

Two honest caveats about that table, since prices are the fastest-moving fact in this course:

DeepSeek retired and repriced on 2026-09-10 with the V4.1 launch; some mirrors of the price page still serve the previous tier (V4-Flash at $0.007 hit / $0.22 miss, a ~31× spread). The lesson is identical either way; the numbers must be read from the live page.
The spread is big enough that the exact multiplier barely matters. A 50× or a 31× on input is the difference between “budget for it” and “don't think about it”.

Why can a provider sell a hit that cheap? Because a hit is a disk read, not a GPU forward pass. DeepSeek's own V4.1 announcement[3] makes the link explicit: “Compared with the previous generation, V4.1-Flash's KV cache needs just 1/4 the HBM and 1/8 the SSD storage. Cache-hit charges often account for a large share of agent costs. Compressing the cache cuts those costs significantly.” The vendor is telling you, in a product announcement, that pushing the KV cache down the memory hierarchy (GPU memory → host RAM → SSD) is how they make agent work affordable. That is section 5's physics, monetized.

The same mechanism, three vendors

The same lever, three vendors

	DeepSeek	Anthropic (Claude)	OpenAI
How you opt in	Automatic, on by default	`cache_control` (automatic or explicit breakpoints)	Automatic (`prompt_cache_options` to tune)
Unit of caching	Prefix; persisted “cache prefix units”	Prefix; cache breakpoints, hierarchy `tools → system → messages`	Prefix; implicit or explicit breakpoints (`prompt_cache_key`)
Write cost	(bundled — no separate write tier)	1.25× base input for 5m TTL; 2× for 1h	1.25× base input
Read (hit) cost	1/50 of miss on input	0.1× base input (0.025× on the newest flagships)	0.1× base input
Lifetime	Hours to days, best effort	5 min default, refreshed on use; 1h optional	30 min default (`ttl: "30m"`), refreshed on use
Minimum cacheable prefix	Not documented as a token floor; short prefixes just may not persist	Model-specific minimum; check your model's row	1,024 tokens (GPT-5.6+), 2,048 for older
Where the numbers show up	`prompt_cache_hit_tokens` / `prompt_cache_miss_tokens`	`cache_creation_input_tokens` / `cache_read_input_tokens`	`input_tokens_details.cached_tokens`

Why it matters, in one worked example. A 60K-token agent turn (system prompt + memory + skills + tools + history) with a 2K-token answer, on deepseek-flash off-peak:

Miss on the whole input: 0.06M × $0.15 = $0.00900 input + 0.002M × $0.60 = $0.00120 output → $0.0102
Hit on the whole input: 0.06M × $0.003 = $0.00018 input + $0.00120 output → $0.00138

Per turn that is 7.4× cheaper; over 50 such turns, $0.51 vs $0.069. The reason it isn't 50× is that output tokens are never cached — they are genuinely new work. Which is exactly the point of the next two sections: caching is a discount on reading the past; it is never a discount on producing the future.

Bill04Your stack's prefix: what is stable, what is volatile

Now the part that changes your behaviour. Hermes assembles its system prompt in three ordered tiers — stable → context → volatile — and the docs state the intent directly: “This ordering matters... This separation keeps the stable prefix stable for caching.”

Walking the layers in the order the API sees them:

The prompt layers, front to back

Prompt layer	Source	Stability	Cache consequence
Agent identity	`SOUL.md`	Stable across turns	Part of the cacheable prefix — edit it between sessions, not mid-run
Tool-aware behaviour guidance, skills index	code + `skills/`	Stable if the skill set is stable	Skills index is inside the cached prompt; loading a different set of skills mid-run changes it
Project context	`.hermes.md`, `AGENTS.md`, `CLAUDE.md`	Stable per project	Rebuilding it (new file, changed file) invalidates from that point
Memory snapshot	`MEMORY.md`	Frozen for the session	Mid-session writes update disk but “do not mutate the already-built cached system prompt until a rebuild path runs”
User profile	`USER.md`	Frozen for the session	Same as above
API-call-time layers	`ephemeral_system_prompt`, `pre_llm_call` plugin context	Changes every call	Deliberately kept out of the cached prompt — appended to the current user message instead

That last row is the design lesson in miniature: Hermes has a mechanism for “volatile context that must not break the prefix”, and it works by putting volatile content after the stable content. You should do the same thing by hand when you write your own prompts, cron job bodies, or skill text.

In practice

Platform side, Hermes also manages the provider cache for you: prompt caching is enabled automatically for Anthropic-family models (native API or via OpenRouter), the TTL is configurable in config.yaml (prompt_caching.cache_ttl: "5m" or "1h"), and the CLI announces its state at startup — Prompt caching: ENABLED (Claude via OpenRouter, 5m TTL). If your Claude-backed sessions feel expensive per turn, that TTL line is the first thing to check: a 5-minute cache and a slow, stop-start working rhythm is the most common silent miss.

What the tool definitions cost you

The cache hierarchy is ordered tools → system → messages, and Anthropic's invalidation table is brutal about the top of it: changing tool definitions (names, descriptions, parameters) invalidates the entire cache — tools, system, and messages. Changing tool_choice invalidates only messages. Changing the system prompt invalidates system and messages.

That maps straight onto something you run: a stack with a large tool surface behind progressive disclosure (tool_search / tool_describe / tool_call, 298 tools not all resident). The reason progressive disclosure is cheaper is not only that fewer schemas ride in the prompt — it is that a smaller, more stable tool array keeps the top of the cache hierarchy constant. Reshuffling which tools are loaded mid-run — enabling a skill, attaching an MCP server, switching tool sets between turns — is a cache event, not just a context-length event.

The hit/miss table for your daily actions

Action level — hit or miss

Action	Hit or miss?	Why
Another turn in the same session, same model, nothing edited	Hit on everything before the new message	The prefix is byte-identical; the reused part is served from cache
Editing `MEMORY.md` mid-session	Hit for this session, miss next session/rebuild	The snapshot is frozen until the prompt is rebuilt — you pay at the rebuild, not immediately
`/model` switch, provider fallback, or credential-pool rotation	Miss — the whole prompt	Hermes' own docs warn this: “means the next request gets zero cache hits and re-reads the full conversation at undiscounted input price. This is inherent to how provider caches work”
Loading different skills mid-run	Miss from the skills-index layer down	Skills index sits in the cached system prompt
Attaching an MCP server / changing the tool set	Miss — entire cache	Tool definitions are first in the hierarchy; a change there invalidates everything below
Context compaction fires (default at 50% of the window)	Miss once, then re-warm	Compaction rewrites the live message list; OpenAI says it plainly: “the first request after compaction may reuse less of the previous cache even when the conversation is logically the same”
A cron job with a fixed prompt	Hit across runs while the cache survives	Identical prefix → identical unit; DeepSeek's cache lives hours-to-days
A cron job that interpolates a timestamp at the top of its prompt	Miss, every run	You changed the prefix before anything stable
Retrieved chunks (RAG / Mnemosyne lookup) injected in the middle	Miss from the insertion point	Caching is prefix-based; material inserted mid-prompt truncates the reusable region
A 5-minute gap and a short turn, then the next call	Hit (Anthropic TTL refreshed on use)	But a 5-minute TTL with a long gap goes cold — a 4-minute streaming response already eats most of it

Two habits fall out of that table, and they are the whole practical content of this section:

Stable content first, volatile content last. Identity, tools, memory, skills, project context → front. Today's date, the current query, retrieved snippets, per-run IDs → back. If you must inject volatile context, inject it at the end of the user message (which is what Hermes' API-call-time layer does).
Don't change the middle of a run. Model switches, skill churn, tool-set changes and memory rewrites are all fine — but do them between runs, not during one, because the provider's cache is prefix-exact and everything after a change is a miss.

Jargon translation

TTL — “time to live”: how long a cached entry survives without being used. DeepSeek: hours to days, best effort. Anthropic: 5 minutes by default, refreshed for free every time it is read; 1 hour if you pay 2× for the write. OpenAI: 30 minutes. A TTL is why “the same prompt, run tomorrow” is not automatically a hit.

WallWall05Memory bandwidth: why decode is not a compute problem

Switch columns in the table from section 1. On your own hardware, nobody charges you per token — so why is a 4B model on a phone stuck at single-digit tokens per second?

The plain explanation

Generating one token requires running the whole model once. For every layer, the hardware must read the layer's weights out of memory, and also read the KV cache for the current context. Then it does a comparatively tiny amount of arithmetic. The ratio of arithmetic to bytes moved — arithmetic intensity — is low, so the chip spends its time waiting for memory, not computing.

This is why TOPS (trillions of operations per second) is the wrong number to judge a device by. Meta's on-device survey[4] says it outright: mobile NPUs “now deliver serious TOPS, getting close to the capability of data-center GPUs in 2017 (for example, V100 is 125 TOPS)” — Apple's A19 Pro Neural Engine ~35 TOPS, Snapdragon 8 Elite Gen 5 ~60 TOPS, Dimensity 9400+ ~50 TOPS — “But TOPS alone doesn't tell you much.” Then the decisive sentence:

Quoted source

“The deeper constraint is memory bandwidth. Mobile devices have 50–90 GB/s; data center GPUs have 2–3 TB/s. That's a 30–50x gap. For LLM inference, this gap is decisive because decode is memory-bound: you load the entire model weights for each token generated, so the compute units sit idle waiting for memory.”

Chandra & Krishnamoorthi — On-Device LLMs: State of the Union, 2026 [4]

Jargon translation

Memory bandwidth — how many bytes per second the processor can pull from RAM. Measured in GB/s (gigabytes) or TB/s (terabytes). Not to be confused with storage speed (SSD) or capacity (how many GB you have).

The napkin formula

The lever

tokens per second  ≈  memory bandwidth ÷ bytes read per token

“Bytes read per token” is dominated by the model weights, plus the KV cache you are carrying. Run it on two real, verifiable datapoints.

Datacenter (from the official llama-bench README[5]). Qwen2.5-7B-Instruct-Q4_K_M, 4.677 GB of weights, on an RTX 4080:

4080 memory bandwidth: 716.8 GB/s (Wikipedia's RTX 40-series desktop table)
Ceiling: 716.8 ÷ 4.677 ≈ 153 tokens/s
Measured: tg128 = 120.6 tokens/s → 79% of the theoretical ceiling

Phone (from the Sep 13 audit's own bench). Qwen3-4B-Q4_K_M, ~2.5 GB of weights, Termux + llama.cpp on a Pixel-class device:

Measured: llama-bench pp512 / tg128 = 9.28 tokens/s for generation
Implied achieved bandwidth: 2.5 GB × 9.28 ≈ 23 GB/s
Against the survey's 50–90 GB/s range, that is 26–46% of the ceiling — while the 4080 sits at 79%

Both numbers come from the same formula; the phone is simply further from its ceiling, for reasons that are all visible in section 6: KV-cache reads on top of weights, CPU-side overhead in Termux, thermal throttling, and an OS that reclaims memory behind your back.

Bandwidth by device class

Class of device	Typical memory bandwidth	What that means for a 2.5 GB Q4 model
Phone / tablet SoC	~50–90 GB/s	20–36 tok/s ceiling; realistically single digits to low teens
Datacenter GPU (current)	~2–3 TB/s	800–1,200 tok/s ceiling on the same bytes
Consumer GPU (RTX 4080, 2022)	716.8 GB/s	~153 tok/s ceiling

Jargon translation

Quantization — storing model weights (and, optionally, KV cache entries) in fewer bits. Q4_K_M ≈ 4 bits per weight; f16 = 16 bits. The reason quantization speeds up generation is not that the math is cheaper to compute — it is that fewer bytes cross the memory bus per token, which is the actual bottleneck.

That last translation is worth its own sentence: this is why going from 16-bit to 4-bit weights is roughly a 4× throughput win and not a 4× size-only win, and why speculative decoding (a small draft model proposing several tokens that the big model verifies in one pass) is described as nearly “free” — it amortizes one memory-bound pass over multiple accepted tokens.

Where it shows up in practice

llama-bench is the tool that makes this measurable on your own hardware. From the official README:

Bench — llama-bench on your own hardware

# prompt processing (-p) vs text generation (-n); -ngl = layers offloaded to GPU
./llama-bench -m qwen3-4b-q4_k_m.gguf -ngl 99 -p 512 -n 128

# how much does context depth cost you? prefill the KV cache to 0, 512, 4096 tokens
./llama-bench -m qwen3-4b-q4_k_m.gguf -d 0,512,4096 -p 512 -n 128

# how much does the KV cache data type cost? f16 vs q8_0
./llama-bench -m qwen3-4b-q4_k_m.gguf -ctk q8_0 -ctv q8_0 -n 128

Reading the output (the README's own example, retitled for clarity):

Bench — reading the output

| model  | size     | params | backend | ngl | test  | t/s            |
| qwen2 7B Q4_K - Medium | 4.36 GiB | 7.62 B | CUDA | -1 | pp512 | 7340.20 ± 23.45 |
| qwen2 7B Q4_K - Medium | 4.36 GiB | 7.62 B | CUDA | -1 | tg128 |  120.60 ± 0.59  |

pp512 = prefill (prompt processing) of 512 tokens — thousands of tokens/s, because it is parallel and compute-bound.
tg128 = text generation (token generation) of 128 tokens — tens of tokens/s, because it is sequential and memory-bandwidth-bound. This is the number a phone lives and dies by.
-d N runs the test with the KV cache pre-filled to N tokens, which is how you see context length costing you speed instead of just believing it.
The README notes the measurements exclude tokenization and sampling — real-world numbers are a little lower.

If this fails

One more practical trap, from llama.cpp's own completion docs: when the context window fills up, “some of the earlier tokens (half of the tokens after --keep) will be discarded. The context must then be re-evaluated before generation can resume. On large models and/or large context windows, this will result in a significant pause in output.” That pause is a cache miss, felt through your chair rather than on an invoice. The local equivalent of “freeze your prefix” is --keep N (or --keep -1 to retain the initial prompt when the window rolls).

Wall06The phone case, concretely

The Sep 13 audit ran a real bench on a Pixel-class Android device — Termux, llama.cpp, Qwen3-4B-Q4_K_M (~2.5 GB). The numbers, and what each one is telling you:

Bench observations — what each number says

Observation	Reading
`llama-bench pp512 / tg128` = 9.28 tok/s generation	~23 GB/s achieved from a 50–90 GB/s memory system. Slow decode is a bandwidth/overhead story, not a “small chip” story
`-ngl 99` (offload all layers to the Adreno GPU via OpenCL) crashed on SET_ROWS with a q8_0 KV cache	Quantized KV cache is not universally supported by accelerator backends. Falling back to f16 KV fixed the crash — and doubled the cache size. Speed and correctness trade against each other at the backend's mercy
MemAvailable 5.2 GB against 15.5 GB headline RAM	Weights (2.5 GB) + KV cache (grows with context) + Android's own services + the GPU driver's allocations. “16 GB of RAM” never means 16 GB for your model
Context length was the knob that mattered	Per section 2's arithmetic: f16 KV at 32K context is ~4.8 GB for a 4B model. Long context is not free locally — it is a RAM bill paid up front

Jargon translation

SET_ROWS — a low-level array-writing operation in the compute backend. When a backend (here, the Adreno OpenCL path) hasn't implemented an operation the quantized KV-cache path needs, the process doesn't slow down — it crashes. Backend coverage, not raw FLOPS, is what decides which flags are usable on a phone.

If this fails

-ngl 99 with -ctk q8_0 -ctv q8_0 crashing on SET_ROWS is a backend-coverage failure, not a memory failure. Fall back to f16 KV and accept the doubled cache size, or drop the offload. Read the backend's op coverage before you read its TOPS.

Apple's 4,096 tokens is a memory decision

Apple's on-device Foundation Models framework[6] documents a 4096-token context window per session, and states what happens when you exceed it: the framework “throws a LanguageModelError.contextSizeExceeded error and the session stops responding.” Apple's prescribed responses are not “be less ambitious” — they are memory engineering:

Split large tasks across multiple sessions, chaining summaries forward (their own worked example is summarizing a long article in chunks).
Start a new session with a summary plus the first and last transcript entries: “The first transcript entry often contains important instructions and the last entry contains the most recent context.”
Limit tool use: three to five tools per request, short descriptions — because “tool definitions and their input and output” consume the same context budget as your prompt.
Keep prompts and instructions concise, and ask for shorter answers.
Use contextSize (the model's actual maximum) and tokenCount(for:) instead of hardcoding 4096.

Read that list as a KV-cache budget statement written in English. A 4,096-token window is small because each token costs KV memory, on a device with 50–90 GB/s of bandwidth shared with the whole operating system. It is the same 50–90 GB/s number from section 5, and the same 144 KiB/token from section 2. Not stinginess: arithmetic.

The local knobs, gathered

In practice — the local knobs

Goal	llama.cpp	Ollama
Cap context (cap the KV cache)	`-c, --ctx-size N`	`num_ctx` option / `OLLAMA_CONTEXT_LENGTH` (default 4096)
Shrink the cache per token	`-ctk q8_0 -ctv q8_0` (f16 default)	`OLLAMA_KV_CACHE_TYPE=q8_0` or `q4_0` (needs Flash Attention)
Enable Flash Attention (needed for quantized KV)	`-fa on`	`OLLAMA_FLASH_ATTENTION=1`
Keep the cache on the CPU, not the GPU	`-kvo / --no-kv-offload`	— (managed automatically)
Keep the initial prefix when the window rolls	`--keep N` (`-1` = keep all)	—
Cache prompt state to a file across runs	`--prompt-cache FNAME` (plus `--prompt-cache-all`, `--prompt-cache-ro`)	— (server keeps state while loaded)
Keep the model (and its KV cache) resident	`--sleep-idle-seconds` controls when it stops being resident	`keep_alive` (default ~5 min; `ollama ps` shows the countdown)
Restore saved cache state	`--slot-save-path`	—

Gotcha

Two notes that will save you a debugging session. First, in Ollama the KV-cache quantization type is global — “all models will run with the specified quantization type” — and quantizing the cache “can harm latency if the context length is short and there is enough GPU memory”, so the win is context-length-dependent. Second, llama-server's --sleep-idle-seconds unloads “the model and its associated memory (including the KV cache)” after idleness: a warm local cache is not eternal either, it just expires on your hardware's clock instead of the vendor's.

CrossoverCrossover07The two halves are the same object

Put the two gaps side by side, and the unit's spine appears:

The join

	Prompt caching (API)	KV cache on your hardware
What is stored	The KV cache for a prefix of your request	The KV cache for your whole context
Who pays to store it	The provider (their HBM, their SSD)	You (your VRAM, your RAM)
What a “hit” costs	~1/50 of a miss on input tokens	A read of N bytes per token at 50–90 GB/s
What a “miss” costs	Full input price + slower time-to-first-token	Re-prefill: the “significant pause” llama.cpp warns about
What invalidates it	Prefix edit, model change, tool-set change, TTL expiry	Context growth, cache dtype, OS memory pressure, backend limits
What you can control	Prompt layout, model stability, TTL strategy, watching hit-token counters	`ctx-size`, KV dtype, Flash Attention, quantization, context discipline
What stays constant	Output tokens are never cached	Decode is always memory-bound

The join is the sentence to take away:

Key idea

Prompt caching is renting somebody else's KV cache. Running a model locally is owning yours — and paying the bill in bandwidth and RAM instead of dollars.

Both are governed by the same two rules, which is why one lecture covers both:

The cache is prefix-exact. On the API, a changed byte at position 4,000 makes position 400,001 uncacheable. On your device, a cache dtype your backend can't execute makes the whole thing nonexistent. Cacheability is a property of the beginning of your input and of your hardware's compatibility, not of your intentions.
You only ever discount the past. Prefill (reading the prompt) is what caching helps. Decode (producing the answer) is what memory bandwidth limits, and it is never cached — not by DeepSeek's disk tier, not by your phone's LPDDR. Every serious cost or latency optimization eventually has to answer: how much of this request is re-reading, and how much is genuinely new work?

08Playbook

Freeze your prefix. Put identity (SOUL/system prompt), tool definitions, memory, skills and project context at the front, in a stable order. Nothing volatile above them.
Push volatility to the end. Current time, session IDs, retrieved chunks, per-run variables → end of the user message. If your platform offers an ephemeral-context hook, use it rather than editing the system prompt.
Stabilize the tool array. Keep the resident tool set (including progressive-disclosure/tool_search names and descriptions) constant within a run. Tool definition changes invalidate the entire cache.
Don't swap models, providers, or credentials mid-session. Hermes' docs are explicit that this means zero cache hits on the next request. If a fallback is needed, expect a full-price turn and accept it knowingly.
Edit memory between runs, not during them. A frozen MEMORY.md snapshot is a feature: you keep the session's cache while your writes land on disk for the next one.
Measure the hit rate, don't assume it. Log prompt_cache_hit_tokens / prompt_cache_miss_tokens (DeepSeek), cache_read_input_tokens (Anthropic), or cached_tokens (OpenAI). A hit rate is a metric; a feeling is not.
Design cron prompts with a stable body. Fixed instructions first; interpolated values (dates, run ids) last. Identical prefix across runs is free money while the cache lives.
Right-size context on local hardware. ctx-size × KV dtype is a RAM allocation. Start at the smallest window your task works in, and only then reach for q8_0 KV (≈ half) or q4_0 KV (≈ a quarter) if your backend supports it.
Benchmark before optimizing. llama-bench -p 512 -n 128 tells you prefill vs decode; add -d 0,512,4096 to price context depth; add -ctk q8_0 -ctv q8_0 to price cache quantization. Change one thing at a time.
Expect a prefill pause when the window rolls. In llama.cpp that is the --keep/context-shift path; on a phone it's the difference between a responsive assistant and a visible stall. Budget context accordingly.

09Key takeaway

There is one KV cache and two ways to meet it. On a hosted API it is the provider's — they keep it on disk, and they charge you by whether your request's prefix matches something they already have: on DeepSeek today a cache hit costs $0.003 per 1M input tokens against $0.15 for a miss, a 50× lever that is real, measurable (via prompt_cache_hit_tokens / prompt_cache_miss_tokens) and entirely under your control through prompt layout — stable content first, volatile content last, no model or tool-set changes mid-run, and memory edits between sessions rather than during them. On your own hardware the cache is yours, and the bill arrives as memory bandwidth and RAM capacity, because decode is memory-bound: roughly, tokens/s ≈ bandwidth ÷ bytes-read-per-token. That single formula explains why a phone (50–90 GB/s) does ~9 tok/s on a 2.5 GB model while a datacenter GPU (2–3 TB/s) does hundreds — and why Apple's 4,096-token on-device window, Ollama's 4,096-token default, and llama.cpp's -c/-ctk/-ctv flags are all the same decision wearing different clothes. And in both columns, caching only ever discounts the past: the tokens you generate next are new work no cache can absorb. Freeze the prefix, keep the volatile parts at the end, and count your context — that is the whole discipline.

10Go deeper

Gap 1 — prompt caching (the 1–3 best free references)

[1]

DeepSeek — Context Caching
api-docs.deepseek.com/guides/kv_cache
The primary source for this lecture: persistence at request boundaries, common-prefix detection and fixed token intervals; the “fully matches a cache prefix unit” rule; best-effort semantics; the prompt_cache_hit_tokens / prompt_cache_miss_tokens fields; the A+B / A+C / A+D worked examples. Read this one first.
Cited in §03
[2]

DeepSeek — Models & Pricing
api-docs.deepseek.com/quick_start/pricing
The live hit/miss price table, the off-peak/peak split, and the model-rename footnote. Check it again before you quote any number from this lecture.
Cited in §03
[3]

Introducing DeepSeek-V4.1-Flash
www.deepseek.com/en/news/deepseek-v4-1-flash
The vendor's own statement that “cache-hit charges often account for a large share of agent costs”, with the 1/4 HBM and 1/8 SSD KV-cache compression figures. Pairs with the technical report PDF and the model card.
huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf · huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash Cited in §03

Gap 1 — the other vendors' versions of the same lever (worth comparing line by line)

·

Anthropic — Prompt caching
docs.anthropic.com/en/docs/build-with-claude/prompt-caching
cache_control, the tools → system → messages hierarchy, 1.25×/2× write multipliers, 0.1× reads, the 20-block lookback window, and the single most useful table in the discipline: what invalidates the cache. Tool definition changes kill everything; tool_choice changes kill only messages.
Cited in §03, §04
·

OpenAI — Prompt caching
platform.openai.com/docs/guides/prompt-caching
Minimum cacheable prefix (1,024 tokens on GPT-5.6+, 2,048 before), implicit vs explicit breakpoints, prompt_cache_key, and a genuinely instructive gotcha about implicit caching writing through dynamic content.
Cited in §03, §04

Gap 2 — memory bandwidth and KV-cache sizing

[4]

Vikas Chandra & Raghuraman Krishnamoorthi (Meta) — “On-Device LLMs: State of the Union, 2026”
v-chandra.github.io/on-device-llms
The best free source for gap 2: the 50–90 GB/s vs 2–3 TB/s comparison, “decode is memory-bound: you load the entire model weights for each token generated”, the mobile-NPU TOPS table, and a whole “KV Cache Management” section (KV quantization to ~3 bits, StreamingLLM, DuoAttention, ChunkKV, EvolKV).
Cited in §02, §05
[5]

llama.cpp — llama-bench (tools/llama-bench README)
github.com/ggml-org/llama.cpp/tree/master/tools/llama-bench
The measurement tool, with real tables for pp/tg, threads, GPU layers and prefilled context depth. Paired with the server README (-ctk, -ctv, -kv-offload, --sleep-idle-seconds) and the completion README (--prompt-cache, --keep, the context-management section).
github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md · github.com/ggml-org/llama.cpp/blob/master/tools/completion/README.md Cited in §05
[6]

Apple — Managing the context window
developer.apple.com/documentation/foundationmodels/managing-the-context-window
The 4096-token window, contextSizeExceeded, tokenCount(for:), and the prescribed strategies (split tasks, fresh sessions, ≤3–5 tools) that are really a memory budget written as developer advice. Coverage: InfoQ.
www.infoq.com/news/2026/03/apple-foundation-models-context Cited in §06

Supporting references (free, verified 2026-09-13)

·

Hugging Face — Cache strategies and Caching
huggingface.co/docs/transformers/en/kv_cache · huggingface.co/docs/transformers/en/cache_explanation
The cache as an explicit object: dynamic vs static vs quantized caches, cache offloading, and “prefill a cache (prefix caching)” as a first-class operation.
Cited in §02
·

Hugging Face — Llama 3.1 inference memory requirements (blog)
huggingface.co/blog/llama31
The KV-cache sizing working shown end-to-end for a real model family.
·

vLLM — Automatic Prefix Caching and the deeper prefix caching design doc
docs.vllm.ai/en/latest/features/automatic_prefix_caching.html · docs.vllm.ai/en/latest/design/prefix_caching.html
What a self-hosted prefix cache looks like: enable_prefix_caching=True, block hashing, and the honesty that APC “only reduces the time of processing the queries (the prefilling phase) and does not reduce the time of generating new tokens”. Watch for “Prefix cache hit rate” in vLLM's logs — a free dashboard for this lecture's first gap.
·

Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (arXiv:2309.06180, SOSP 2023)
arxiv.org/abs/2309.06180
The paper behind vLLM; the origin of treating the KV cache as paged memory with explicit waste accounting, and the reason prefix sharing became practical at scale.
·

Dao, “FlashAttention-2” (arXiv:2307.08691) and Wikipedia — FlashAttention
arxiv.org/abs/2307.08691 · en.wikipedia.org/wiki/FlashAttention
Why attention is written to minimize HBM↔SRAM traffic; the same “bandwidth is the bottleneck” argument one level down. (Flash Attention is also what makes quantized KV caches usable in Ollama.)
·

DeepSeek-V2 (arXiv:2405.04434)
arxiv.org/abs/2405.04434
Multi-head Latent Attention, the architecture lineage behind DeepSeek's compressed KV cache; read it as “how a vendor makes cache hits cheap by making the cache small”.
·

NVIDIA — Mastering LLM Techniques: Inference Optimization and Databricks — LLM Inference Performance Engineering
developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization · www.databricks.com/blog/llm-inference-performance-engineering-best-practices
Industry treatments of prefill vs decode, batching, and the memory-bandwidth ceiling.
·

Ollama docs — FAQ, GPU, API
docs.ollama.com/faq · docs.ollama.com/gpu · docs.ollama.com/api
Context length default 4096, OLLAMA_KV_CACHE_TYPE, OLLAMA_FLASH_ATTENTION, ollama ps.
Cited in §02, §06
·

Wikipedia — Memory bandwidth, Cache (computing), GeForce RTX 40 series
en.wikipedia.org/wiki/Memory_bandwidth · en.wikipedia.org/wiki/Cache_(computing) · en.wikipedia.org/wiki/GeForce_RTX_40_series
The 716.8 GB/s figure used in section 5.
Cited in §05
·

MIT HAN Lab — TinyML and Efficient Deep Learning Computing and StreamingLLM
hanlab.mit.edu/courses/2024-fall-65940 · github.com/mit-han-lab/streaming-llm
The research line on KV-cache compression and attention sinks; the “you don't need to cache everything, you need to cache the right things” argument.
·

Stanford CS336 — Language Modeling from Scratch and CS224N
cs336.stanford.edu · web.stanford.edu/class/cs224n
The course-level treatment of attention and inference, if you want the derivation rather than the recipe.

Watch (free)

·

Andrej Karpathy — “Let's build GPT: from scratch, in code, spelled out”
www.youtube.com/watch?v=kCc8FmEb1nY
Attention implemented line by line; the clearest way to see what K and V are. Follow with “Let's reproduce GPT-2 (124M)” for the training-scale view, and “[1hr Talk] Intro to Large Language Models” for the whole-system framing.
www.youtube.com/watch?v=l8pRSuU81PU · www.youtube.com/watch?v=zjkBMFhNj_g
·

3Blue1Brown — “Transformers, the tech behind LLMs” (Ch. 5) and “Attention in transformers, step-by-step” (Ch. 6)
www.youtube.com/watch?v=wjZofJX0v4M · www.youtube.com/watch?v=eMlx5fFNoYc
The visual intuition for queries, keys and values that section 2 assumes.
·

MIT 6.S191 — Introduction to Deep Learning
introtodeeplearning.com
Free lecture series; useful scaffolding if you want a course rather than a lecture.

Next lecture: KC 301-03 — Quantization & VRAM math (the other half of the memory equation: how many bytes your weights cost, and why Q4 is the default), with KC 301-04 — Serving: prefill vs decode (where this lecture's pp/tg split becomes a deployment decision).

Caching:prompt caching, KV cache

01One object, two bills

PrefixPrefix02Why a cache exists at all

The plain explanation

It is not free — it is RAM that grows with your conversation

Where you meet it in practice

BillBill03Prompt caching on the API: hits, misses and “cache prefix units”

The plain explanation

What counts as a cacheable unit

Reading the hit rate off the response

The 50× lever, verified

The same mechanism, three vendors

Bill04Your stack's prefix: what is stable, what is volatile

What the tool definitions cost you

The hit/miss table for your daily actions

WallWall05Memory bandwidth: why decode is not a compute problem

The plain explanation

The napkin formula

Where it shows up in practice

Wall06The phone case, concretely

Apple's 4,096 tokens is a memory decision

The local knobs, gathered

CrossoverCrossover07The two halves are the same object

08Playbook

09Key takeaway

10Go deeper

Caching:
prompt caching, KV cache