What Is a Context Window? – Definition and How It Works
Learn what a context window is, how tokens are counted, what happens when a conversation overflows it, and why a large window is not the same as good recall.
The context window is the total amount of text a language model can consider in a single request. It is a hard architectural limit, and almost every practical constraint in building on top of a model traces back to it.
1. What Is a Context Window?
The context window is a fixed token budget covering everything the model sees and everything it produces in one call.
- Key idea: it is working memory for a single request, not storage. Nothing persists between calls.
- Mechanism: the model's attention operates over a sequence of at most N tokens; the limit is fixed when the model is trained.
- Goal: bound the computation, which grows steeply with sequence length.
What Counts Toward It
system prompt + conversation history
+ retrieved documents + tool definitions and tool results
+ the answer being generated
= must fit within the window
Tool schemas are easy to forget and can be substantial when many tools are attached.
2. How Tokens Are Counted
A token is a word fragment, and the ratio to plain words is not fixed.
| Content | Approximate tokens |
|---|---|
| Ordinary English prose | ~0.75 tokens per word |
| Code and markup | Higher — punctuation and indentation tokenise separately |
| JSON with long keys | Higher again — every key repeats in every record |
| Languages with non-Latin scripts | Often several tokens per character |
That last two rows matter for retrieval: pasting raw HTML or verbose JSON into a prompt consumes budget that carries almost no information.
3. What Happens at the Limit
Three behaviours are common, and they differ in how visible the failure is.
- Rejection. The request errors before generation. Loud and easy to handle.
- Truncation. The client silently drops the oldest turns to fit. The model then answers without information it appeared to have been given.
- Degradation. The content fits, but attention is spread so thin that details in the middle are effectively ignored.
The second and third are the dangerous ones, because output still looks confident and well-formed.
4. Large Windows Are Not Free Recall
A larger window admits more text; it does not guarantee the model uses all of it evenly. Retrieval accuracy commonly drops for material in the middle of a very long context compared with material at the start or end.
The engineering response is the same as it was with small windows: select fewer, better passages rather than pasting everything and hoping. A retrieval step that returns five relevant chunks usually beats one that returns fifty.
5. Managing the Budget
- Retrieve, do not paste. Fetch the passages that answer the question instead of the whole document.
- Strip before sending. Convert pages to clean text; navigation, scripts, and styling are pure overhead.
- Summarise history. Replace old turns with a compact running summary rather than dropping them silently.
- Prune tool definitions. Attach the tools a task needs, not the whole catalogue.
- Measure, do not estimate. Count tokens with the model's own tokeniser before shipping a prompt template.
6. Real-World Examples
- A long support chat starts giving inconsistent answers because early turns were silently truncated away.
- A document assistant improves markedly after switching from "paste the whole PDF" to "retrieve three sections", despite using a smaller window.
- A scraping agent overflows on a single page because raw HTML was passed instead of extracted text, and the same page fits easily once converted to Markdown.
- A multi-tool agent slows and degrades once dozens of tool schemas are attached, and recovers when the tool set is scoped per task.
7. Common Misconceptions
- "A larger window removes the need for retrieval." It raises the ceiling and leaves selection quality untouched; detail in the middle of a long context is still used unevenly.
- "Tokens are words." Ratios vary by content type, and code, JSON, and non-Latin scripts all tokenise far less efficiently than English prose.
- "The conversation is remembered." Nothing persists between calls. History is re-sent every time, which is why it consumes budget.
- "Only the prompt counts." Generated output, tool schemas, and tool results all share the same window.
8. Summary
The context window is the model's single-request working memory, measured in tokens and shared between input and output. Exceeding it produces errors, silent truncation, or quiet degradation — and only the first is obvious.
Treat context as a scarce resource to be curated. Clean text, targeted retrieval, and summarised history consistently outperform simply moving to a bigger window.