Put the Cache Breakpoint Before the Timestamp

Prefix caching pays from the second request that shares a prefix, so the block you mark decides whether your agent ever reads one

*Image: BalticServers data center by BalticServers.com, licensed under CC BY-SA 3.0.*

Take a prompt with this shape. It opens with operating rules, then the tool definitions, then a line carrying the current timestamp, then the user's message, and the cache marker sits on the last block. Every response reports a cache write and no cache read: cache_creation_input_tokens above zero, cache_read_input_tokens at 0. The prefix is long enough to cache, the provider caches it, and the bill carries the write premium on every request.

Anthropic's prompt caching documentation describes this exact structure and calls it a common mistake. The request writes an entry whose hash includes the timestamp. The next request has a different timestamp, so the hash differs and the read walks backward looking for entries that earlier requests wrote. "The lookback does not find stable content behind your breakpoint and cache it," the page says. The fix is one line: move the breakpoint to the last block that stays the same across requests, which here is the tool definitions above the timestamp (Claude prompt caching documentation, captured October 26, 2026).

A template that changes one line per request pays the write premium on every call, while the same prefix sent unchanged twice costs 1.35 units against 2.00 for two uncached passes.

What a prefix cache holds

A prefix cache stores the model state computed for a run of tokens at the front of a request. A later request that begins with the same tokens reuses that state instead of recomputing it. vLLM's serving documentation describes the mechanism plainly: automatic prefix caching "caches the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix with one of the existing queries" (vLLM automatic prefix caching, captured October 26, 2026). SGLang's paper attributes part of its throughput gain on agent and multi-turn workloads to the same idea, reuse of the KV cache across requests that share a prefix (SGLang paper, revised June 2024).

Three properties of the hosted implementations decide whether you get the reuse, and all three appear in the vendors' documentation.

A cache write happens only where you place a breakpoint. Anthropic's page states that marking a block writes exactly one entry, a hash of the prefix ending at that block, and that no entries exist for earlier positions. The hash is cumulative, so a change anywhere at or before the breakpoint produces a different hash on the next request.

A read searches backward through a bounded window. On the Claude API, the system checks at most 20 block positions per breakpoint, counting the breakpoint as the first, and a request can carry up to 4 breakpoints. A conversation that grows by 20 blocks or more between requests falls out of the window, and the documented remedy is a second breakpoint at an earlier position.

The prefix has to match exactly, including request settings. OpenAI's diagnostics page states that reuse "requires an exact prefix match and compatible request settings, including the model, service tier, and tools" (prompt cache diagnostics, captured October 26, 2026).

The arithmetic that makes placement worth the trouble

Anthropic bills a 5-minute cache write at 25 percent above base input and a cache read at 10 percent of base input, with lower read rates on some models: 5 percent on Opus 5.5 and Sonnet 5.5, 2.5 percent on Fable 5.1 and Mythos 5.1. A 1-hour lifetime writes at 2x base input.

Take those multipliers at face value and price a prefix used N times, in units of one uncached pass over the same tokens. Uncached, N passes cost N. Cached, they cost 1.25 for the write plus 0.1 for each of the N-1 reads.

A prefix used once costs 1.25 against 1.00, so caching a single-use prompt loses money. Used twice it costs 1.35 against 2.00. Used ten times it costs 2.15 against 10.00. The break-even sits at about 1.3 uses, which means the second request that shares a prefix already pays for the write. Switch to the 1-hour lifetime and the write costs 2.00, which moves break-even against no caching to about 2.1 uses. The longer lifetime pays only when a gap longer than five minutes would otherwise drop the entry: a warm 5-minute cache at three uses costs 1.45 while the 1-hour write costs 2.20.

OpenAI publishes the same shape with its own numbers. For GPT-5.6 and later, a write costs 1.25x ordinary input at the default 30-minute lifetime and 2.5x at a 12-hour minimum lifetime, and a read costs 0.1x on most of those models, 0.05x on GPT-6.1 Sol. Its documentation does the two-request version of the sum directly: "writing a prefix once and fully reusing it once costs 1.35× its ordinary input cost, compared with 2× for processing it twice without caching," and one write with nine reads costs 2.15x against 10x (OpenAI prompt caching, captured October 26, 2026).

For an agent loop the conclusion is narrow: a prefix reused even twice per cache lifetime is worth caching, and a prefix that varies per request is not worth the premium.

Three conditions the pricing assumes

Minimum length. Every provider refuses to cache short prompts, and the threshold differs by model and by provider. Anthropic's list includes 512 tokens for Opus 5.5, Sonnet 5.5 and Haiku 5.5, 1,024 for Sonnet 5, Opus 4.8, Sonnet 4.6 and Opus 4.1, 2,048 for Opus 4.7 and Haiku 3.5, and 4,096 for Opus 4.6, Opus 4.5 and Haiku 4.5. A request below the minimum "is processed without caching, and no error is returned," so the failure stays invisible without usage data. OpenAI sets the minimum at 1,024 visible input tokens for GPT-5.6 and later, excluding the hidden system content it adds, and rounds reported cached_tokens down to a multiple of 128 on earlier models. Gemini's implicit cache needs 4,096 input tokens on the Gemini 4, 3.8, 3.6 and 3.1 models and 2,048 on the 2.5 models (Gemini context caching, captured October 26, 2026).

Lifetime. Anthropic's default entry lives 5 minutes and is refreshed at no cost each time it is read, so a steady stream of requests keeps it warm and a gap of more than five minutes starts over. OpenAI's default minimum lifetime is 30 minutes.

Concurrency. Anthropic notes that an entry becomes available only after the first response starts, so parallel requests that all need the cache have to wait for that first response. OpenAI notes that cached states live on individual machines, that traffic above roughly 15 requests per minute per cache key can overflow routing to another machine, and that a request reuses a prefix only if it lands on the machine holding that entry.

What moves the prefix

Anthropic describes the cache as a hierarchy, tools, then system, then messages, where a change at one level invalidates that level and everything after it. Tool definitions sit at the top. The page states that modifying tool definitions, including names, descriptions and parameters, invalidates the entire cache. Toggling web search or citations modifies the system prompt and drops the system and message caches. Changing tool_choice affects only message blocks.

That hierarchy explains a failure that reads as a caching bug on the bill, with the cause in the tool list. An agent framework that rebuilds the tool list per request, filtering tools by user permissions or by the current task, changes the first block of the prompt on every call, and nothing downstream can match.

OpenAI's documentation offers the same fix from the application side: keep tool definitions, ordering and schemas stable, and control availability with tool_choice: "none" or allowed_tools instead of removing definitions. Its diagnostics page classifies the misses it can see, and the reason codes read like a list of ordinary framework habits: tools_changed, input_changed for a timestamp, request ID or edited earlier message, model_changed from a fallback or A/B test, service_tier_changed, reasoning_effort_changed, verbosity_changed, text_format_changed, prompt_cache_key_changed, and context_compacted.

History rewriting belongs on that list. OpenAI advises appending messages rather than rewriting earlier turns, because summarization or context truncation changes the prefix. Compaction that replaces earlier turns can still be the right call on total cost, since fewer input tokens can beat a cache hit, and the diagnostics page says so. Both effects show up in the same usage fields, so the comparison is a subtraction.

The audit

Six steps, in this order: step one needs a day of traffic and the rest fit in an afternoon.

Instrument usage before changing anything. Log the three fields Anthropic returns in usage: cache_creation_input_tokens, cache_read_input_tokens and input_tokens. On OpenAI, log usage.input_tokens_details.cached_tokens. Compute the read share as reads divided by reads plus writes plus uncached input, per endpoint and per prompt template.

total = creation + read + uncached
hit_share = read / total if total else 0.0

A template whose write tokens climb while its read tokens stay at zero is the prompt at the top of this article.

Render two consecutive requests and diff them. Do not diff the template. Render the final prompt strings, including tool definitions and any injected metadata, and find the byte offset where they first differ. Everything before that offset is the only region that can ever be cached.

Move volatile content after the breakpoint. Timestamps, request IDs, user names, retrieved chunks, per-tenant configuration and the incoming message all belong after the cache marker. Put the stable material first: system rules, tool definitions, the long reference documents, few-shot examples.

Keep the tool array fixed. Ship one tool list per endpoint with stable order and schemas, and restrict availability through the request parameters instead of rewriting the array.

Place the breakpoint at the end of the stable prefix. On the Claude API, mark the last block that is identical across the requests you want to share, and add a second breakpoint at an earlier position when a conversation grows by 20 blocks or more between turns. Choose the lifetime by the gap between requests: the 5-minute default when requests arrive within minutes of each other, written at 1.25x, and the 1-hour write at 2x only when the same prefix is reused across gaps longer than five minutes.

Verify with a repeat, then with diagnostics. Send the same prefix twice in a row and confirm the second response reports reads instead of writes. On OpenAI's Responses API, compare the second request against the first and read prompt_cache_diagnostics.reason when reuse is lower than expected. Anthropic's page points at the same two fields, cache_creation_input_tokens and cache_read_input_tokens, both zero meaning the prompt was never cached at all.

Where this stops paying

A prompt used once and thrown away should not be cached, because the write premium is a loss. That covers most evaluation runs and most one-shot extraction jobs.

A prompt below the model's minimum cannot be cached at any price, and padding it to reach the threshold helps only when the padded prefix is reused. Anthropic suggests exactly that trade when a prompt falls just short of the minimum and the content repeats.

A prefix that changes per tenant, per customer or per request is closer to uncacheable than to optimizable, and moving the tenant block behind the breakpoint only helps if there is enough stable material in front of it to clear the minimum on its own.

The mechanism also promises less than the pricing implies. OpenAI states that a minimum lifetime "governs eligibility for cache-read pricing" and "does not guarantee that every request reuses stored KV tensors or has lower latency." Anthropic's concurrency note makes the same point operationally: two requests that arrive together cannot both read an entry before it is written.

One habit covers most of this. Before adding a template to the agent, read its first block and its last block and ask which one survives to the next request. Put the marker on the last block that does. When the first block changes on every call, no breakpoint anywhere can pay, and the usage fields will show it as writes climbing while reads stay flat.