When the windows got long, a particular temptation arrived with them: stop choosing. If everything fits, put everything in, and let attention sort it out.
We tried it, on the retrieval paths where it was cheapest to try. It does not hold. Capacity and usable capacity are different quantities, and the gap between them widens as you fill the window.
What the traces show
The shape is consistent across the paths we have measured, and it is not a cliff at the limit. Quality starts to move well before the window is full — first as a small increase in variance between runs on the same input, then as a specific failure: the model answers from material that is present and relevant-looking but not the material that actually settles the question.
Three effects are separable, and they matter for different reasons.
Position. Evidence in the middle of a long context is used less reliably than the same evidence near either end. This is well documented elsewhere and it survives every prompt-level workaround we have tried. It is a property to design around, not a bug to phrase around.
Competition. Adding a passage that is topically similar to the correct one but does not answer the question makes the answer worse than not adding it. Near-misses are more expensive than irrelevant material, because irrelevant material is easy to ignore and a near-miss is not.
Accumulation. In multi-turn and agentic paths, the context grows by appending — prior turns, tool output, intermediate summaries — and most of what accumulates is stale by the time it matters. A superseded intermediate result that stays in context does not sit there inertly. It competes.
None of this says long context is not useful. It says that a long window changes what is affordable, not what is correct.
What we removed
The rules that survived contact with production, in the order we apply them:
- Retrieve for the question, not for the topic. The unit of retrieval is a claim that could settle the question, not a document that mentions its subject. This single change removes more near-misses than any re-ranking we layered on afterwards.
- Drop superseded state instead of appending to it. When an intermediate result is replaced, the old one leaves the context. Agentic paths that append their entire history are the ones whose behaviour drifts over long runs.
- Keep the decisive material near the ends. Given a position effect we cannot prompt our way out of, ordering is a design parameter: the material that determines the answer goes where the model reliably reads it.
- Compress narration, never evidence. Prior turns become short factual summaries; the passages an answer must be checkable against stay verbatim. Compressed evidence produces answers that are impossible to audit, because the thing being cited no longer exists in the form it was cited from.
- Measure with the window half full. An eval suite that only exercises short contexts certifies a system nobody runs. We run the same cases at realistic context lengths, and the divergence between the two is itself a metric.
The economics changed, the discipline did not
It is now cheap to send more tokens, so the cost of over-including has largely stopped being a budget question. That is exactly why the discipline has to come from somewhere else. When including everything was expensive, the bill enforced curation. Now nothing does, except a decision to keep curating.
The useful reframing: a context window is not storage, it is a working set. The question was never what fits. It is what does this specific answer need to be correct, and what would compete with it. That question had the same answer when the window was small.