Which restart policy suits a long-running telemetry ingester, and which suits a run-to-completion import job?

## The policy is a reaction rule, not a repair A **restart policy** is a standing instruction attached to a workload, read by whatever supervises it at one moment only: when an instance ends. It cannot prevent anything. It decides whether another attempt begins, and that is the whole of its power. Every platform that runs containers has some form of it, because the alternative — a workload that quietly vanishes the first time a dependency blips — is not operable. The three policies in general use are the same three everywhere, whatever they are spelled: | policy | after an orderly ending | after a bad ending | the workload it fits | |---|---|---|---| | restart on any ending | yes | yes | a long-running service | | restart only on a bad ending | no | yes | a run-to-completion job | | never restart | no | no | work that must not repeat unattended, or an instance you want frozen | ## Why a long-running service takes the first row A telemetry ingester is supposed to be running now, and at every moment after now. Its useful states are "running" and "broken"; it has no successful ending. So when the instance ends, the reason barely matters — whether the process decided to stop, or was ended from outside, the service is no longer ingesting, and something should try again. Restarting on any ending encodes that: the desired state is *present*, and the policy keeps pushing toward it. This is also why a clean, orderly ending from a long-running service should make you suspicious rather than relieved. Something asked it to stop, or it decided its own work was over, and neither is normal for a workload whose job never finishes. ## Why a job takes the second row A nightly import has the opposite shape. It has a defined unit of work and a real successful ending, and "finished" is the outcome you wanted. Put it under restart-on-any-ending and you have built a loop out of a success: the job completes, the supervisor observes an ending, starts another attempt, that one completes too, and the import runs continuously until someone notices. This is one of the most common self-inflicted restart loops in production, and the giveaway is that nothing is failing — each attempt succeeds. The policy that fits reacts to the *outcome*, not merely to the ending: restart only when the attempt ended badly, so a failed import is retried and a successful one is left finished. ## When never-restart is the right choice Never-restart has two honest uses, and one dishonest one. - Honest: repeating the work unattended is not safe, so a failed attempt should wait for a person to decide whether re-running it is correct. - Honest: you want the failed instance preserved as it is, because the next attempt would displace the evidence you are about to read. - Dishonest: using it to silence a noisy workload. That converts an intermittently-available workload into an unavailable one and changes nothing about why it was failing. Never-restart also removes the automatic recovery that the other two policies provide for **transient** causes — a dependency that was briefly unreachable, a node that was briefly starved. Those are real, and they are the reason the default on most platforms leans toward restarting rather than not. ## What the policy interacts with Two interactions are worth naming, because they are what makes the choice non-obvious in practice: 1. **The delay between attempts.** A policy that restarts is almost always paired with a growing delay, so that a workload which cannot start does not attempt it thousands of times a minute. The policy decides *whether*; the delay decides *how often*. 2. **What forces the ending in the first place.** An instance can end because its process exited, or because a failing check asked for it to be replaced, or because the host reclaimed it. The policy reacts to the ending the same way regardless — which is precisely why the policy choice never tells you anything about the cause. ## A working rule Ask what the workload's *successful* state is. If success means still running, restart on any ending. If success means finished, restart only on a bad ending. If success means a human looked at it first, do not restart — and make sure something alerts, because a workload that is deliberately not restarting is also a workload that is deliberately down.