What is inference cost per task?
Inference cost per task is what one completed unit of work costs in model usage. It is counted against a task a person would recognise, such as an answered query or a processed form, rather than against a thousand tokens. The figure includes retries and the steps that produced nothing useful.
Also called: Cost per task, Unit cost per task
Why the finance function eventually asks a question engineering cannot answer
Nine months into a deployment, somebody in finance asks what a single processed application costs, and the engineering answer arrives in tokens. Token figures are meaningless to the person holding the budget, who is comparing this work against the staff cost it displaced. The translation is harder than it looks, because an honest figure has to carry failed attempts and all the orchestration a task required before it succeeded. Teams that defer this calculation usually find the number is dominated by retries nobody was counting.
- A per task figure is the only form a budget holder can compare against staff cost.
- Retries and abandoned steps dominate the cost in many deployments.
- Cost per task makes model choice an economic decision rather than a preference.
- A rising figure is an early indicator that something upstream has degraded.
How inference cost per task works
- The unit of work is defined as something a business owner would recognise.
- Every model call belonging to one task is tagged with the task identifier.
- Token usage is priced per model and summed across the whole task.
- Failed attempts and retries are attributed to the task they belonged to.
- Retrieval and orchestration costs are included rather than reported separately.
- The figure is reported per task type, since averages conceal the expensive cases.
How inference cost per task is evaluated
The figure is only useful with its distribution and its failures included. Report the following per task type rather than across the deployment:
- median and ninetieth percentile cost per completed task
- share of total cost attributable to retries
- cost of tasks abandoned before completion
- cost per task by model where routing is in use
- change in the figure after each prompt or model revision
Limitations, risks and common failure modes
- Averages conceal a small class of tasks consuming most of the budget.
- Retries are excluded, which understates the figure by a wide margin.
- Context grows quietly with every added instruction until the cost doubles.
- A cheaper model reduces the figure while increasing human review time elsewhere.