What is a context window?
A context window is the quantity of text a model can consider in a single request. Everything the answer rests on competes for that space, from the instruction through to the retrieved passages and the conversation so far. Its size is a hard limit rather than a target, so retrieval exists partly to decide what deserves the room.
Also called: Context length
Why a larger window moves the problem rather than removing it
Vendors now quote windows large enough that teams assume selection no longer matters, and the assumption is expensive. Accuracy degrades well before the stated limit, most noticeably for evidence placed in the middle of a long input, so a window filled indiscriminately answers worse than a smaller one filled carefully. Cost and latency also scale with what is supplied, which an organisation discovers at production volume rather than in testing. The window is a budget to be allocated, and the allocation is the engineering.
- The stated limit is some distance above the point where accuracy begins to fall.
- Evidence positioned mid-input is weighted less than evidence at either end.
- Every token supplied carries a cost that multiplies by query volume.
- A window large enough to hold everything removes the incentive to select well.
How a context window works
- The budget is divided between evidence and conversational history before any query arrives.
- Retrieved candidates are reranked so the strongest passages occupy the available room.
- Passages are placed with the most important evidence at the edges of the input.
- History is summarised or truncated once it exceeds its share of the budget.
- Token counts are measured per request against the allocation agreed.
- Accuracy is retested whenever the allocation or the model version changes.
How a context window is evaluated
A context window is assessed on where accuracy actually degrades rather than on the figure a vendor publishes. Measure the following on the question set the system will face:
- accuracy by position of the supporting evidence within the input
- accuracy as total supplied tokens increase towards the stated limit
- latency and cost per query at each context size
- frequency of requests truncated before they reach the model
- share of the window occupied by evidence rather than history
Limitations, risks and common failure modes
- A long window encourages teams to supply everything retrieved without reranking.
- Truncation silently removes the passage the answer depended upon.
- Accuracy measured on short inputs is reported as the figure for long ones.
- Cost per query is modelled at pilot volume and never at production volume.