Context windows have grown dramatically. Models that once read a few pages can now accept entire codebases, books or months of chat history in a single prompt. Vendors often demonstrate this with near-perfect "needle in a haystack" charts: hide one sentence in a huge document, ask for it, and the model finds it almost every time.
Yet anyone who has pasted a long contract or report into a model has seen it miss things that were clearly there. Both observations are true, and the gap between them is one of the most important things to understand about long-context models. This article explains the "lost in the middle" effect, why simple needle tests overstate real-world recall, and how to build a long-context evaluation that reflects your actual tasks.
The "lost in the middle" finding
In 2023, Liu and colleagues published "Lost in the Middle: How Language Models Use Long Contexts." They gave models a question plus a set of documents, only one of which contained the answer, and varied where that document appeared. Performance followed a U-shaped curve: models did best when the relevant document was at the very beginning or the very end of the context, and noticeably worse when it sat in the middle. In some settings, accuracy with the answer in the middle fell below the models' accuracy with no documents at all.
Models have improved since then, and many current models show a much flatter curve on simple retrieval. But position effects have not vanished. They reappear on harder tasks, at longer lengths and when the relevant information is phrased differently from the question.
Why needle tests are too easy
The classic needle-in-a-haystack test hides a distinctive sentence ("The best thing to do in San Francisco is eat a sandwich in Dolores Park") inside unrelated filler text, then asks a question that closely matches the needle's wording. That design is easy for three reasons:
- The needle stands out. It is semantically unrelated to the filler, so it is easy to spot.
- The question matches the wording. The model can match words rather than understand.
- Only one fact is needed. No combining, comparing or reasoning across locations.
Real documents are the opposite. The relevant clause in a contract looks like every other clause. The question uses different words from the answer. And the answer often requires combining two or three facts from different sections, for example a definition on page 2, an exception on page 40 and a date in an appendix.
What actually degrades in long contexts
| Task type | Example | Typical difficulty |
|---|---|---|
| Single-fact retrieval, distinctive needle | Find the planted sentence | Easy for most current models |
| Single-fact retrieval, similar distractors | Find the right clause among similar clauses | Moderate |
| Multi-fact aggregation | Count or list all items matching a condition | Hard |
| Multi-hop reasoning | Combine a definition and an exception from different sections | Hard |
| Global understanding | Summarise how a policy changed across a whole document | Hard, and hard to grade |
Benchmarks such as RULER (Hsieh et al., 2024) were designed to go beyond simple needles with multi-needle retrieval, aggregation and tracing tasks, and they found that many models' *effective* context length is considerably shorter than their advertised maximum.
How to build a better long-context eval
1. Use your own documents
Take real documents of the length you plan to use: contracts, reports, transcripts, code. Synthetic filler hides the main difficulty, which is similar-looking distractors.
2. Write questions that need paraphrase
Phrase questions the way a user would, not the way the document does. If the contract says "termination for convenience," ask "Can we cancel without a reason?"
3. Include multi-hop and aggregation questions
At least a third of questions should require combining two or more facts from different locations, or listing every instance of something.
4. Vary the position systematically
For a subset of questions, move the relevant passage to the beginning, quarter, middle, three-quarter and end positions while keeping everything else fixed. Plot accuracy by position to see your model's curve.
5. Vary the length
Test the same questions at several total lengths, for example 8K, 32K and 128K tokens, padding with *related* documents rather than random text. The length at which accuracy drops meaningfully is your model's effective context for that task.
6. Include "not in the document" questions
Ask some questions whose answers are absent. A model that confidently answers them is filling gaps from training knowledge, which is dangerous in legal or compliance work.
For the general mechanics of building and scoring a set, see build an eval set in an afternoon.
Scoring long-context answers
Long-context questions are harder to grade than short ones, because answers are longer and partially correct answers are common. A few rules keep scores meaningful:
- Grade facts, not prose. For each question, list the specific facts a correct answer must contain, then check each one. An answer that names two of three required exceptions scores two thirds on the fact checklist and fails on the strict pass/fail metric. Report both.
- Penalise unsupported claims. Ask the model to cite section numbers or quotes, and mark any claim that cannot be found in the document. Hallucinated detail is more harmful in long documents because readers are less likely to check.
- Keep position and length as columns in your results. A single average hides exactly the effect you are trying to measure. A small table of accuracy by position and by length is far more informative.
- Re-run after every model or prompt change. Long-context behaviour is sensitive to small changes in instructions and to model updates you may not be told about.
Practical fixes when your model forgets the middle
- Put the question last, and repeat key instructions at the end of very long prompts.
- Retrieve, then read. Instead of sending a whole corpus, retrieve the most relevant passages and place them near the question. Chunking choices matter here; see RAG chunking strategies compared.
- Order retrieved passages by relevance, with the most relevant at the start or end rather than buried in the middle.
- Ask for quotes first. Instruct the model to quote the relevant passages, then answer from those quotes. Quoting forces it to locate evidence and makes answers checkable.
- Split the task. Map over sections (extract relevant facts from each), then reduce (answer from the extracted facts).
Long context and memory are also different things; context windows are not memory explains why a bigger window does not replace good retrieval. For the architecture behind long contexts, see sparse attention explained.
Frequently asked questions
What does "lost in the middle" mean for LLMs?
It describes the tendency of language models to use information at the beginning and end of a long input more reliably than information in the middle, producing a U-shaped accuracy curve by position. The effect was documented by Liu et al. in 2023.
Is the needle-in-a-haystack test a good long-context benchmark?
It is a useful sanity check, but it overstates real-world recall, because the needle is distinctive, the question matches its wording and only one fact is needed. Multi-fact, paraphrased questions on real documents are a better test.
What is effective context length?
It is the input length up to which a model maintains acceptable accuracy on a given task. It is often well below the maximum context window advertised for the model, and it differs by task type.
Should I use RAG or long context?
For large or frequently changing corpora, retrieval is usually cheaper and more accurate. Long context works well when the relevant material is a single document of moderate size. Many systems combine both: retrieve generously, then let a long-context model read the results.
