Context windows keep growing, and each increase is announced as if it solved memory: paste in all your documents, your whole codebase, every past conversation, and the model "knows" it all. In practice, teams that build this way run into the same problems. Costs climb with every request, answers get slower, the model misses details that are plainly in the input, and nothing persists once the conversation ends.
The root cause is a category error. A context window is the amount of text a model can read in a single request. Memory is a system that decides what to keep, how to find it again and when to forget it. This article explains the difference, why a larger window does not give you memory, and how to design systems that combine both.
What a context window actually is
A language model is stateless. Each request is independent: the model receives a sequence of tokens (the system prompt, the conversation so far, any documents or tool results) and produces a response. The context window is the maximum length of that sequence.
Everything that looks like memory in a chat interface is the application re-sending earlier messages with every new request. When the conversation exceeds the window, something must be dropped, truncated or summarised. Nothing is "remembered" between requests except what the application chooses to send again.
Three properties follow:
- You pay for the whole window every time. Input tokens are billed per request. A 100,000-token context sent with every message is 100,000 tokens of cost each time, though prompt caching can reduce this for stable prefixes.
- Latency grows with input length. The model must process the whole input before producing the first token.
- Attention is not uniform. Models do not use every part of a long input equally well; see lost in the middle.
Why a bigger window is not memory
Memory requires selection
Human memory is valuable because it is selective. Of everything you experienced last year, you retain what mattered. A context window has no notion of importance: a throwaway remark and a critical constraint occupy the same space. As you add more, the important facts become a smaller share of the input and are easier to miss.
Memory requires persistence
A context window lasts one request. If a user tells your assistant their preferences today, nothing carries over tomorrow unless your application stores it somewhere and puts it back in the context.
Memory requires retrieval
To use the right memory at the right time, you need to find it. With a big window, the implicit strategy is "include everything and let the model find it." That works for small amounts of material and degrades as the material grows, both in quality and in cost.
Memory requires forgetting and updating
Facts change. A user moves house, a policy is replaced, a decision is reversed. A context full of old and new versions of the same fact invites the model to use the wrong one. Real memory systems overwrite, expire and resolve conflicts.
Context window, retrieval and memory compared
| Approach | What it is | Strengths | Weaknesses |
|---|---|---|---|
| Long context | Put everything in the prompt | Simple; model sees full documents | Costly, slow, uneven attention, no persistence |
| Retrieval (RAG) | Search a store and insert the best matches | Scales to huge corpora; cheaper per request | Depends on search quality and chunking |
| Stored memory | Write and update facts about users or tasks | Persists across sessions; can forget | Needs rules for what to store, privacy controls |
Most good systems use all three: a stable system prompt, retrieval for knowledge, stored memory for user- or task-specific facts, and the context window as the place where the selected pieces are assembled for one request.
Designing the context: what goes in each request
Think of each request as assembling a briefing for a capable colleague who has never met you. A useful order:
- Instructions and rules (system prompt), kept stable so it can be cached.
- Durable facts about the user or task from stored memory: preferences, constraints, decisions already made.
- Retrieved knowledge relevant to the current question, ordered by relevance.
- Recent conversation, possibly with older turns summarised.
- The current request at the end, with any key constraint restated.
Placing the question last and keeping the most relevant material near it counteracts the position effects of long inputs. Retrieval quality depends heavily on how documents are split; see RAG chunking strategies compared.
Summarisation: useful, but lossy
A common technique for long conversations is to summarise older turns and keep the summary in context. It works, with caveats:
- summaries lose details, and the model cannot recover what the summary dropped;
- repeated summaries of summaries drift from the original;
- important specifics (numbers, names, decisions) should be extracted into structured memory rather than left to prose summaries.
A good pattern is to keep a short running summary plus a structured list of facts and decisions, each with a source and a timestamp.
Building real memory
For assistants and agents that work across sessions, design memory explicitly:
- Decide what to store. Preferences, stable facts, decisions and their reasons. Not every message.
- Store with metadata: source, date, confidence, and who can see it.
- Retrieve selectively based on the current task.
- Update and expire. When a new fact contradicts an old one, replace it and keep the history if needed.
- Give users control. Let people see, correct and delete what is stored about them.
We cover these choices in detail, including privacy, in memory for agents: what to store, what to forget.
When a long context really is the right tool
Long context windows are genuinely valuable for:
- reading one long document thoroughly, such as a contract or report, where retrieval might miss connections;
- code tasks that need several related files at once;
- analysing a complete transcript or dataset that fits comfortably;
- as the final stage of a retrieval system, letting the model read generous retrieved material rather than tiny snippets.
The architecture behind large windows is covered in sparse attention explained. The key is to treat the window as working space for the current task, not as a database.
A quick diagnostic
If your system shows these symptoms, you are probably using context as memory:
- input token counts grow steadily over a conversation or over weeks of use;
- the model contradicts facts the user stated earlier, or uses outdated ones;
- costs per request are dominated by input tokens;
- users repeat information every session.
The fix is rarely a bigger window. It is selection, storage and retrieval.
Frequently asked questions
What is the difference between a context window and memory in LLMs?
The context window is the maximum amount of text a model can process in one request. Memory is an application-level system that stores information between requests and decides what to retrieve, update or forget. Models are stateless; memory must be built around them.
Does a larger context window make a model remember more?
It lets the model read more in a single request, but nothing persists afterwards, and models do not use every part of a long input equally well. Persistence and selection require a memory or retrieval system.
Should I use RAG or a long context window?
Use retrieval for large or changing knowledge bases and long context for reading a single long document or a set of related files thoroughly. Many systems combine both: retrieve relevant material, then let a long-context model read it.
How do chatbots remember previous messages?
The application re-sends earlier messages, or a summary of them, with every new request. Some products also store selected facts in a database and add them to the context in future sessions.
