What Is Prompt Caching? How It Cuts LLM Cost and Latency

Language models process every token of every request, and many requests begin with the same text: a long system prompt, a set of tool definitions, a shared reference document. Prompt caching lets providers store the processed state of that repeated prefix and reuse it, billing cached tokens at a discount and returning responses faster. It is one of the largest cost and latency levers available without changing models, and it depends on how prompts are structured. This explainer covers how it works and how to use it, drawing on FISTA Solutions' AI enablement practice. The wider cost toolkit is in llm api cost optimization and token economics in llm token cost explained.

What is prompt caching?

Prompt caching is a provider-side mechanism that stores the internal state produced when a model processes a prompt prefix, so that later requests sharing the exact same prefix skip that processing. The model's attention computation over the prefix, held in what is called the key-value cache, is preserved across requests rather than rebuilt each time. Providers expose it with explicit cache markers or automatically, bill cached tokens at reduced rates, and expire cached entries after a period of inactivity. Underlying mechanics are in what is a kv cache.

How does it work in practice?

Step	What happens	Effect
First request	Full prompt processed; prefix state stored at cache point	Normal cost; cache created
Subsequent request with same prefix	Prefix state loaded; only suffix processed	Discounted prefix tokens; faster first token
Prefix changes	Cache miss; full processing; new cache entry	Normal cost
Inactivity beyond lifetime	Entry expires	Next request pays full cost and recreates
Multiple cache points	Nested prefixes cached at several boundaries	Partial hits when later sections change

Provider implementations differ in minimum cacheable sizes, lifetimes, and marker requirements; verify current documentation.

What content should be cached?

System prompts, tool and function schemas, few-shot examples, shared reference documents, long instructions, and any large content sent identically across many requests. In agents, tool definitions and instructions often dominate prompt size and repeat on every step, making agents prime beneficiaries. System prompt design is in what is a system prompt and tool schemas in how to build tool use for llm agents.

How should prompts be structured for hits?

Place stable content first and variable content last: system prompt, then tool schemas, then shared documents and examples, then retrieved context, conversation history, and the user message. Never insert timestamps, user identifiers, or per-request data into the stable section. Keep system prompt versions stable across a deployment. Any byte change before the cache point breaks the match.

What savings should be expected?

Cost savings equal the discount on cached tokens times the share of tokens that hit the cache; latency savings come from skipping prefix processing, which shortens time to first token, especially for long prefixes. Applications with large stable prefixes and steady traffic see the largest gains; short or unique prompts see little. Model your own token mix. Cost dashboards that track cache hit rates are in how to build an ai cost dashboard.

How does caching interact with conversation history?

Multi-turn conversations grow by appending, so earlier turns form a stable prefix that can be cached turn over turn, with only the newest messages processed fresh. Summarizing or trimming history changes the prefix and resets the cache, so history management strategies should account for cache behavior. Context limits are in what is a context window.

How does caching relate to other caching layers?

Prompt caching happens inside the provider on the model's computation; response caching happens in your application, returning stored answers for repeated questions; semantic caching matches similar questions. They stack: a response cache avoids the model call entirely, and prompt caching makes unavoidable calls cheaper. Gateway-level implementation is in how to build an llm gateway.

What are the limits?

Caches expire, so low-traffic applications may rarely hit; minimum cacheable sizes exclude short prompts; exact matching means any drift defeats it; cached content still counts toward context limits; and pricing and behavior vary by provider and change. Design for high hit rates and measure them rather than assuming savings. Latency budgeting is in what is latency in ai systems.

What does effective use look like?

A support agent with a long system prompt, a dozen tool schemas, and a policy document restructures its prompt to place all of that first, adds cache markers, keeps versions stable per deployment, and moves customer data and retrieved articles after the cache point. Cache hit rates climb, input cost per request falls substantially, and first-token latency drops, with no change in behavior. Agent cost patterns are in ai agent maintenance cost.

How FISTA Solutions applies prompt caching

FISTA Solutions structures prompts for caching from the first design, keeps stable content versioned and ordered first, places cache markers where providers require them, and tracks hit rates and savings on cost dashboards alongside quality. The AI enablement practice delivers the cost architecture, AI agents are built with cached prefixes by default, and forward deployed engineers embed with client engineering teams. The record behind the approach is 150+ projects with 47% efficiency gains for clients.

To cut model cost and latency without changing behavior, message FISTA on WhatsApp, or read what is an llm router for the complementary lever of sending each request to the right model.