What is a token budget?
A token budget is the allowance of model input and output a task is permitted to consume. It is stated per request and in aggregate per workflow, since the cost of a design accrues in the input far more often than teams expect. Retrieved context and accumulated conversation history usually dominate the total, while the visible answer accounts for a small fraction.
Where the cost of an agent design actually accrues
Unit economics are rarely examined until a pilot is approved for a second business unit, at which point a cost per conversation that looked immaterial meets a volume nobody modelled. The accrual is almost always in the input, since each retrieved passage is resent on every call along with each prior turn. A design retrieving ten documents where two would answer the question costs several times as much for no gain in quality. Budgeting is therefore a design activity performed before deployment rather than a reporting activity afterwards.
- Input tokens dominate the bill in retrieval and agent architectures.
- A budget per task makes an expensive design visible during engineering.
- Caching and context pruning are measurable only against a stated allowance.
- Approval for scale depends on a defensible cost per completed task.
How a token budget works
- The token cost of each component of the prompt is measured in isolation.
- An allowance is set per task type from the cost the business case supports.
- Retrieval is limited to the passages that materially change the answer.
- Conversation history is summarised or pruned once it passes the allowance.
- Prompt prefixes are arranged so the cacheable portion remains stable.
- Consumption is monitored per task type and compared against the allowance.
How a token budget is evaluated
A budget is assessed against measured consumption rather than against a projection. Examine the following per task type in production:
- input and output tokens per completed task, reported separately
- share of input accounted for by retrieved context
- cache reuse rate on the stable prompt prefix
- cost per completed task against the figure in the business case
- tokens consumed by retries and by abandoned attempts
Limitations, risks and common failure modes
- Retrieval depth is set generously during development and never revisited.
- Conversation history accumulates until a long session costs many times a short one.
- Retries double the token cost of exactly the hardest requests.
- A prompt prefix is edited often, which destroys the caching the design assumed.
- Evaluation traffic is charged to the same budget and distorts the unit figure.