What is a long-lived task?
A long-lived task is work whose completion spans hours or days rather than a single session. It has to survive a process restart or the loss of a worker, resuming from the point it reached rather than from the beginning. Its state therefore lives outside the executing process.
Why hours of elapsed time change the engineering
Nothing about a long-lived task is difficult until the first deployment lands halfway through one. At that moment the organisation learns whether the work resumes or begins again from nothing; it also learns whether beginning again means a supplier is paid twice. A demonstration completes inside a minute and never meets a restart, which is why this is the gap operations teams find first. Elapsed time is what turns a sequence of steps into a system with state worth protecting.
- A restart during the task is a normal event rather than an exception.
- Progress held only in memory disappears with the worker that held it.
- Resuming without idempotency repeats an action already taken in another system.
- The business needs a status for work that will not complete today.
How a long-lived task works
- The work is decomposed into steps that can each be recorded as complete.
- A durable store holds the current step alongside the data gathered so far.
- Each step is made idempotent, so a repeat produces no second effect.
- A worker claims the task under a lease that expires if the worker disappears.
- On resumption the task continues from the last recorded step.
- Status and elapsed time are exposed to the operations team throughout.
How a long-lived task is evaluated
A long-lived task is judged on what happens when it is interrupted. Measure the following against deliberate interruption:
- proportion of tasks resuming correctly after a forced restart
- duplicate effects observed per thousand resumptions
- median and worst case elapsed time to completion
- age of the oldest task still in progress
- share of tasks abandoned without a recorded outcome
Limitations, risks and common failure modes
- Progress is recorded at a granularity coarser than the step that fails.
- A lease expires while the worker is alive, so two workers proceed at once.
- Resumption is tested on an empty task rather than a half-completed one.
- A deployment mid-task is treated as an incident instead of a normal occurrence.
- Nobody owns a task that has been in progress for three days.