What Is Token Usage? – Definition and How to Control It
Learn how token usage is counted, why input and output are priced differently, where waste accumulates, and which reductions actually work.
Token usage is the count of tokens consumed by a request — everything sent plus everything generated. It determines both cost and whether a request fits the context window.
1. What Gets Counted
| Component | Counted as | Notes |
|---|---|---|
| System prompt | Input | Re-sent on every request |
| Conversation history | Input | Grows with every turn |
| Retrieved passages | Input | Often the largest share |
| Tool definitions | Input | All attached tools, every call |
| Tool results | Input | Appended to the conversation |
| Generated response | Output | Usually priced higher |
Output tokens typically cost several times more than input tokens, which changes where optimisation pays.
2. Where Waste Accumulates
- Raw HTML pasted instead of extracted text.
- Verbose JSON whose keys repeat in every record.
- Whole documents supplied where a passage would do.
- Unbounded history, re-sent in full each turn.
- Every tool attached to every request regardless of the task.
- Rambling outputs where a constrained format was possible.
3. Reductions That Work
- Extract before sending. Clean text instead of markup removes the largest single source of waste.
- Retrieve passages, not documents. Fewer, better chunks beat more, weaker ones.
- Summarise history rather than re-sending every turn.
- Scope tools per task. Definitions consume the window even when unused.
- Constrain output. A schema and a length limit reduce the expensive half.
- Cache repeated prefixes where the provider supports it — an unchanged system prompt need not be reprocessed at full cost.
4. Measure Before Optimising
Log input and output tokens per request, attributed by feature or endpoint. Almost every optimisation effort is initially aimed at the wrong place.
The common surprise is that retrieved context, not the prompt or the answer, dominates the bill.
5. Token Counts Are Not Portable
Model families tokenize differently, so the same text yields different counts. Cost comparisons must use each model's own tokenizer, and a context window advertised as the same size can hold different amounts of the same text.
6. Usage and the Context Limit
Cost and capacity are the same measurement. A request approaching the window limit is both expensive and fragile: any growth in history or retrieved context pushes it over, producing an error or silent truncation.
Leaving deliberate headroom is a reliability practice as much as a cost one.
7. Practical Targets
- Keep the system prompt short and stable, so prefix caching can apply.
- Convert every fetched document to clean text before it enters a prompt.
- Cap retrieved passages by count and by size.
- Set an output length limit appropriate to the task.
- Track tokens per successful outcome, not per request — that ratio is what actually matters.
8. Summary
Token usage counts everything sent and generated, with output priced highest and retrieved context usually largest.
Measure per feature before optimising, extract and retrieve rather than pasting, scope tools per task, constrain outputs, and leave headroom below the limit so cost control and reliability improve together.