With every request, a language model only takes into account what fits into its context window. Everything outside it does not exist for the model. This explains many everyday observations: why an assistant seems to forget earlier instructions in a very long chat, why you cannot evaluate any number of documents at the same time, and why long requests cost more.
The size of the context window is an important criterion when selecting a model. It has grown significantly in recent years but still differs greatly between models.
How does the context window work?
A large language model does not store anything between two requests. To make a chat appear coherent nonetheless, the application resends the previous history with every new message. The context window typically includes:
- the system prompt with the assistant's role and rules
- the previous conversation history
- inserted or automatically retrieved documents
- the current question
- for most models, also the answer to be generated
All of this is measured in tokens. When the window is full, the application has to shorten: older messages are cut off or summarized, and documents are shortened or rejected.
A large context window does not automatically mean that all content is taken into account equally well. With very long inputs, individual pieces of information can be overlooked.
Typical examples
- An extensive contract is inserted in full and checked for deadlines.
- In a long chat history, the assistant increasingly deviates from the initial instructions.
- A knowledge assistant selects only the relevant sections from thousands of documents, because not all of them fit into the window at the same time. This principle is used by retrieval-augmented generation.
What companies should keep in mind
- Relevance over volume: Send only the content that is needed for the task. This saves costs and often improves the results.
- Restart long chats: When changing topics or with very long histories, start a new chat and restate important instructions.
- Minimize data: Everything in the context window is transmitted to the respective service. Personal data should only be included to the extent necessary.
- Test the limits: For applications with large documents, check whether important details are reliably recognized even in the middle or at the end of long texts.
How it differs
The context window is not long-term memory. Some assistants offer a memory feature that remembers preferences or facts across multiple conversations. Technically, this stored information is loaded back into the context window when needed and also takes up space there.
View AI software development