What is a retry policy?
A retry policy is the set of rules governing when a failed step is attempted again. It states how long to wait before each attempt, how many attempts are permitted and where the work goes once they are exhausted. The policy distinguishes a failure worth repeating from one that will fail identically every time.
Why most repeated attempts make the failure worse
A retry policy is where a well engineered automation becomes an outage. An agent meeting a slow dependency and responding with an attempt every second adds load to a system already in difficulty; a fleet doing so in unison removes it. The distinction that matters lies between a transient failure and a deterministic one, because repeating a request the system has already rejected on its merits achieves nothing. Operations teams tend to discover which kind they have during the incident rather than before it.
- Repeated attempts against a struggling dependency convert slowness into failure.
- A deterministic error attempted ten times is ten identical errors.
- Without a limit, a failing step consumes capacity that healthy work needs.
- Repeated attempts without idempotency duplicate the effect of the step that failed.
- An exhausted item has to go somewhere a person will actually look.
How a retry policy works
- Failures are classified as transient or deterministic before any policy applies.
- Transient failures are attempted again after a delay that grows with each attempt.
- A random element is added to the delay so that agents do not synchronise.
- A maximum attempt count and a total time budget both apply.
- Exhausted items are moved to a dead letter queue with the failure recorded.
- A circuit breaker stops attempts entirely once a dependency is clearly unavailable.
How a retry policy is evaluated
A retry policy is assessed on what it does during a dependency failure rather than in steady state. Measure the following during a deliberate fault injection:
- attempt count distribution per failed step
- added load on the dependency during its slowest period
- proportion of failures resolved by a second attempt
- items reaching the dead letter queue per day
- duplicate effects attributable to a repeated attempt
Limitations, risks and common failure modes
- Every failure is treated as transient, so permanent errors are attempted forever.
- Fixed delays cause agents to synchronise and arrive together.
- The dead letter queue has no owner, so items accumulate unexamined.
- Attempt limits are configured per step while the total time budget is absent.
- A repeated attempt on a step lacking idempotency issues a second payment.