Caching Prompts and LLM Responses: How to Reduce Cost and Latency in AI Applications

Caching for LLM applications means reusing work you have already paid for: the fixed part of prompts (the provider's prompt caching) and responses to identical or very similar requests (exact and semantic cache). Well designed, it cuts call costs and latency by 30-70% in use cases with many repetitive requests, such as internal assistants, FAQs and document classification.

Where LLM Calls Waste Money

In a business application, much of every prompt is always the same: system instructions, examples, documentation, tool definitions. That part is sent and billed on every request. On top of that, different users often ask the same question in different words. Caching addresses both.

Provider Prompt Caching

Several providers, including OpenAI and Anthropic, recognize the repeated prefix of a prompt and process it at lower cost and latency. The practical requirement is to structure the prompt with the stable part first (instructions, documents, examples) and the variable part last (the user's question). Moving a timestamp or an identifier from the first paragraph to the end is often enough to make the cache work. Minimum token thresholds, cache lifetime and discounts vary by provider and should be checked in the official documentation.

Exact Response Cache

The simplest form is a key built from a hash of model, prompt version, parameters and request text, with the response stored in Redis or a database. It works where input is repeatable: classification, extraction from identical texts, translation of interface strings. Use it only with low or zero temperature, since reuse makes no sense with highly variable outputs.

Semantic Cache with Embeddings

A semantic cache computes the embedding of the question and looks in a vector index for already-answered requests with similarity above a threshold. It suits FAQs and support assistants, where "how do I change my password?" and "forgot my password, what now?" share the same answer. The risk is the false positive: two similar questions with different answers. Mitigate it with a conservative threshold, metadata filters (customer, language, product) and spot checks.

Invalidation and Safety

• Short time to live (TTL) for changing content, long for stable answers.
• Prompt and model version inside the key: an update invalidates the cache automatically.
• Never share responses across users or tenants when they contain personal or confidential data.
• Exclude requests that depend on real-time data from the cache.

Metrics to Monitor

Track cache hit rate, savings in tokens and euros, average latency with and without cache, and the quality of cached answers compared with fresh ones. A very high hit rate with falling quality points to a threshold that is too permissive.

Key takeaway: combining provider prompt caching with response caching, using versioned keys, TTLs and per-customer isolation, cuts cost and response time without sacrificing answer reliability.