Context Management in LLMs: Context Window and Conversational Memory

An LLM does not remember: it only sees what you send with each request, within the limit of the context window. Managing what goes into that window, between instructions, history, documents and summaries, determines the quality of answers, latency and the cost of a business chatbot.

What the Context Window Is

It is the maximum number of tokens, input and output, the model can consider in a single call. It includes the system prompt, the conversation, retrieved documents and the answer. Even with very large windows, filling them is not free: more tokens mean higher costs and longer times, and the model may give less weight to information in the middle of a long text.

Short-Term Memory: the History

The simplest solution is to resend the latest messages with every request. It works for short conversations. As the history grows you can truncate it, keeping the last turns and system instructions, or compress older messages into a summary.

Progressive Summaries

Periodically you ask the model to summarise the conversation so far, keeping decisions, preferences and key data, and replace the old part with the summary. The risk is losing detail: state explicitly what to preserve, such as names, numbers, constraints and open requests.

Long-Term Memory and Retrieval

• Session memory: state and summary of the current conversation.
• User memory: preferences and stable facts saved in a database, with the person's consent.
• Company knowledge: documents retrieved with semantic search (RAG) only when needed.

Instead of putting everything in the prompt, store information externally and recall it by relevance. It is cheaper and more controllable, including for privacy: you decide what to keep, for how long and how to delete it.

Cost, Quality and Security

Every token sent is paid for on every turn, so a long history costs more and more. Stable content such as instructions and reference documents belongs at the start of the prompt, to benefit from prompt caching where the provider offers it. Personal and confidential data must not enter the context without a legal basis and minimisation.

How to Test Context Management

Prepare long test conversations with questions that recall information given many turns earlier and check whether the system remembers. Measure average cost per conversation, latency and correct-answer rate, comparing different truncation and summarisation strategies.

Key takeaway: a chatbot's memory belongs not to the model but to the system around it: trimmed history, targeted summaries and selective knowledge retrieval keep answers consistent, costs under control and personal data governed.