What is a latency budget?
A latency budget is the total time available for a response, apportioned across each component that has to act within it. On a telephone call the whole figure is usually under a second, which is then divided between recognition and retrieval before generation and speech synthesis receive what remains. Each component is given a ceiling rather than an average, because the slowest turn is the one a caller notices.
Why a latency target without an allocation cannot be enforced
Everyone agrees a voice agent should respond quickly, and almost nobody writes down what each component may consume. The arithmetic is unforgiving: endpointing takes its silence window; recognition finalises; a retrieval call reaches an enterprise system over a corporate network; generation then produces a sentence and synthesis streams the first audio out. Any one of those overrunning by a few hundred milliseconds is audible, and without an allocation nobody can attribute it. Budgets are also where the honest conversation about retrieval happens, since an enterprise lookup rarely returns within the time a conversation allows.
- A target for the whole response cannot be enforced on any single component.
- Ceilings matter more than averages, since callers hear the worst turn.
- Retrieval against an enterprise system is usually the component that overruns.
- An allocation makes an architectural trade-off explicit before deployment rather than after.
How a latency budget works
- The acceptable response time for the conversation is agreed as a ceiling.
- That ceiling is apportioned across each component in the response path.
- Each component is instrumented to report its own duration per turn.
- Components exceeding their allocation are given a timeout and a fallback response.
- Slow retrieval is moved off the response path through caching or a holding phrase.
- Percentile timings per component are monitored after release rather than averages.
How a latency budget is evaluated
A budget is evaluated at the tail rather than the mean, because the slow turns are the ones callers report. Measure the following per component over a week of calls:
- ninety-fifth percentile duration against each component allocation
- end of speech to first audio out, measured on the line
- proportion of turns exceeding the total ceiling
- timeout rate and the fallback used when one occurs
- latency added by the telephony network itself
Limitations, risks and common failure modes
- Component timings are measured in a data centre and exclude the telephony path.
- Averages conceal the tail where every exceeded turn actually lives.
- A retrieval dependency is added after release with no allocation available for it.
- Streaming synthesis masks a slow component until the sentence stalls mid-word.