Prompt Caching

Keep sansa-auto conversations on a warm prompt cache with a prompt cache key

Prompt caching reuses the part of a request that hasn't changed since the last turn. Cached input tokens cost less and the first token arrives sooner, which adds up quickly in agents and long chats that resend the whole conversation every turn.

A cache only helps if the next turn goes to the same model. On sansa-auto, a prompt cache key is how you tell Sansa which turns belong together.

Send a prompt cache key

Send the same prompt_cache_key on every turn of a conversation, and a new one for each new conversation:

{
  "model": "sansa-auto",
  "prompt_cache_key": "conversation-42",
  "messages": [
    {"role": "system", "content": "You are a coding agent."},
    {"role": "user", "content": "Summarize the repo layout."}
  ]
}

Without a key, every turn is routed on its own and can land on a different model, starting with a cold cache each time. This is the single biggest factor in what a long conversation costs on sansa-auto.

Use one key per conversation or agent run. A key shared across unrelated conversations pins them all to one model.
session_id (or the x-session-id header) works the same way and wins if both are sent.
On Messages, Claude Code's session header does this for you.

See the code panel for TypeScript, Python, and curl examples.

What to expect

First turn: routed normally. Sansa remembers the model it chose for your key.
Later turns: go to the same model while its cache is warm, between about 5 minutes and an hour depending on the model. Every request restarts the clock.
Sansa won't move a warm conversation to another model to save cost or spread load. If the model is briefly rate-limited, Sansa waits for it (up to a minute on long conversations) instead of switching.
Length never forces a move. Every model sansa-auto routes to supports a 1M-token context.

When a turn goes to another model

Situation	What happens
The turn adds an image or audio the model can't read	Sansa routes again, and the new model becomes the conversation's model. Its cache starts cold.
A setting on one request the model can't serve, such as reasoning off or forcing a specific tool	That request goes to another model. The next turn returns to the warm one.
The model is unavailable	Another model answers that turn. The next turn tries the warm model again.

Keep the start of the request stable

A cache hit needs the start of the request to match the previous turn exactly. Keep the system prompt, tool definitions, and earlier messages unchanged, and add new turns at the end. Anything that changes early, such as a timestamp in the system prompt or reordered tools, makes everything after it cold.

Some models only cache content marked with a cache marker. Add a top-level cache_control so they cache too:

{ "cache_control": { "type": "ephemeral" } }

Check that it's working

Cached input tokens are reported in usage:

Endpoint	Field
Chat Completions	`usage.prompt_tokens_details.cached_tokens`
Responses	`usage.input_tokens_details.cached_tokens`
Messages	`usage.cache_read_input_tokens`

The first turn shows 0. From the second turn on, most of the prompt should be cached:

"usage": {
  "prompt_tokens": 10339,
  "completion_tokens": 60,
  "prompt_tokens_details": {
    "cached_tokens": 10318,
    "cache_write_tokens": 0
  }
}

If later turns keep showing 0, check that the key is the same on every turn and that the start of the request isn't changing.

Billing

Cached input tokens are billed at the model's cache-read rate, which is lower than its input rate. Some models charge extra to write the cache, reported as cache_write_tokens. Rates are on the Models page.