#Reuse stable prefixes

Prompt caching reuses the stable part of a request (system prompt, tool schemas, conversation prefix) so multi-step tool loops do not re-pay full input cost every turn.

Most providers either cache automatically or need only a sticky session key. Anthropic is the exception: it needs explicit cache breakpoints (or a gateway that inserts them).

#Provider matrix

What crabcode does today for each routing path:

Anthropic is the unfriendly one. Without cache_control (or Gateway caching: auto), Claude traffic does not cache-read. That was the main gap crabcode fixed.

#What gets marked (direct Anthropic)

Hybrid of OpenCode auto + Grok Build placement:

1. Last tool — tool schemas are large and stable across a tool loop
2. Last system block — instructions / project context
3. Transcript tip — last markable content block (skips thinking / redacted_thinking)
4. Previous user tip — where the prior request ended, so turns past the 20-block lookback still hit cache

At most 4 breakpoints; the 4th slot stays free when tools or previous-user are missing so gateways can auto-mark.

Breakpoints are applied after message regrouping so adjacent tool_use / tool_result blocks stay valid.

#xAI Grok Build (cli-chat-proxy)

In addition to sticky prompt_cache_key = session_id, crabcode stamps Grok Build–style affinity headers:

Grok Build’s sampler sends that header so cli-chat-proxy emits response.doom_loop_check with tail_repetition:{n}@thinking. Crabcode treats those as confident the same way (DoomLoopRecoveryPolicy::is_confident: thinking channel, threshold 2..=64).

• Policy / window: .devrefs/references/xai-org/grok-build/crates/codegen/xai-grok-sampling-types/src/doom_loop.rs
• SSE collector: .devrefs/references/xai-org/grok-build/crates/codegen/xai-grok-sampler/src/doom_loop.rs

Doom-loop recovery matches Grok Build’s sampler, not a client tool-name babysitter. Only confident tail_repetition:{n}@thinking from response.doom_loop_check resamples: skip that generation’s tools, inject RECOVERY_REMINDER into the next request only (same <system_reminder> user-role item as ConversationItem::system_reminder — not a visible user turn). After DEFAULT_MAX_RETRIES (2) the abort is disarmed and the turn continues. Repeating reads or shell commands are not a loop — they usually mean earlier tool results were cleared.

Tool-result pruning matches Grok Build prune_conversation (.devrefs/references/xai-org/grok-build/crates/codegen/xai-chat-state/src/actor/request_builder.rs): only when estimated tokens exceed 50% of a 500k window (should_prune), last 3 user turns stay intact, older large results soft-trim at 4000/1500/1500, age ≥ 10 hard-clear. Those numbers have not changed in Grok Build; the important gate is not pruning a short session at all.

Parent-cached aux (e.g. max-steps text-only summary): keeps parent prompt_cache_key + session/conv so the conversation prefix can reuse the main turn's KV cache; assigns a fresh aux-… req id.

Subagents use the child session id on purpose. They have a different system prompt and tool set, so parent-prefix reuse would miss and can pollute sticky routing. Isolation beats a false shared key.

#Prefix stability (anti-bust)

• Empty system/user rows are dropped before the wire (they pad the sticky prefix).
• Images use hysteresis: compact only above a trigger (~6 MiB total data-urls), then reclaim to a lower target (~3 MiB) so later turns stay cache-warm instead of re-evicting every step.

#How to verify it works

Run with logs enabled and watch app.log for [prompt-cache] lines:

crabcode --emit-logs

Logging writes to app.log in the working directory when --emit-logs is set.

#Direct Anthropic

[prompt-cache] anthropic input=… output=… cache_read=… cache_creation=… total_input=… hit_pct=…

#AI Gateway / OpenAI-compatible

[prompt-cache] openai-compatible prompt=… completion=… cached_tokens=… cache_read=… cache_creation=… hit_pct=…

#OpenAI / xAI Responses

[prompt-cache] openai-responses input=… output=… cached_tokens=… hit_pct=…

#Healthy multi-step session

Notes:

• Anthropic's input_tokens is non-cached only. Total input ≈ input + cache_read + cache_creation.
• Gateway may surface cache as prompt_tokens_details.cached_tokens and/or forwarded Anthropic fields — crabcode logs both when present.
• No dashboard required: stream finals carry usage; crabcode logs them when logging is enabled.

#What we intentionally do not do

#Related

• AI Gateway automatic caching: Vercel docs
• Anthropic prompt caching: provider docs for cache_control / ephemeral