Why is idempotency the property that makes declarative infrastructure code usable, and what typically makes a hand-written provisioning shell script non-idempotent?

## The definition, and why it matters here An operation is **idempotent** when applying it more than once has the same effect as applying it once. Formally, `f(f(x)) == f(x)`. In infrastructure terms: apply the configuration, apply it again, and the estate is in the same place — the second run reports that there is nothing to do. That sounds like a small technical nicety. It is actually the property that makes automated infrastructure change *operable at all*, because of what it removes: the need for anyone to know the history. Without idempotency, before every run someone must answer "what state is this account in, and how far did the last attempt get?" With it, that question stops mattering. Three concrete consequences: - **Retries are safe.** A run that dies half-way — network blip, expired credential, someone's laptop closing — can simply be run again. This is the big one, because partial application is the normal failure mode of infrastructure change, not an exotic one. - **Anyone can run it.** The result does not depend on who applied last or what they did manually beforehand. - **It can be run continuously.** Scheduled or triggered application only makes sense if a no-op run is genuinely a no-op. ## Why declarative code is idempotent by construction Because each statement is a **condition to assert**, not an action to perform. "A bucket with this name and this configuration exists" is either already true — in which case the correct behaviour is to do nothing — or not, in which case the engine acts. The declaration itself carries no verb, so there is nothing to repeat. Configuration-management tools reach the same property one step at a time: a well-behaved task checks the condition, acts only if needed, and reports back whether it changed anything. That per-task `changed` versus `unchanged` reporting is the visible signal of idempotency, and a run that reports changes on every single execution against an untouched host is the classic symptom that something in the code is not idempotent. ## How scripts lose it Scripts are built from verbs, and verbs stack. The usual offenders: **Create calls with no existence check.** A script that calls a cloud CLI to launch an instance launches another instance on the next run. Nothing in the call says "one of these"; it says "make one". **Append instead of write.** `>>` is the single most common cause. Three runs, three copies of the same export line in a profile script, and now a config file has drifted in a way nobody notices until a value is defined twice. **Operations that fail on the second run.** `mkdir /opt/app` succeeds once and errors afterwards. Under `set -e` that turns a harmless re-run into a failed deploy, which is why people reach for `mkdir -p` — a genuinely idempotent form. **Additive resource operations.** Adding a firewall rule, appending to a list, attaching a policy. Some of these silently duplicate; some hit a limit weeks later. The fix is always the same shape: replace the action with an assertion. Write the whole file rather than appending a line. Guard the create with a lookup. Use the `-p` or `--if-not-exists` variants where the tool offers them. Notice that this is exactly what a declarative engine does for you, resource by resource — which is the point the interviewer is usually driving at. ```bash # not idempotent: each run appends again and the second useradd fails echo 'export APP_ENV=prod' >> /etc/profile.d/app.sh useradd appuser # idempotent: same end state however many times it runs printf 'export APP_ENV=prod\n' > /etc/profile.d/app.sh id -u appuser >/dev/null 2>&1 || useradd appuser ``` ## The nuances worth raising **Idempotent does not mean "makes no changes".** The first run absolutely changes things — that is its job. The claim is about the *end state* after N runs, not about the number of actions taken. Candidates who say "idempotent means it does nothing" have half the definition. **Idempotent is not the same as safe.** A run that destroys and recreates a database every time it executes is not idempotent, but a run that idempotently converges on a wrong declaration is perfectly repeatable and still an outage. Repeatability is about the mechanism, not about whether the intent is correct. **A declarative tool is not automatically idempotent end to end.** The moment your configuration shells out to an arbitrary command as part of an apply, the engine cannot know whether that command has already had its effect, so the guarantee only holds as far as your escape hatches allow. If you must shell out, make the script itself idempotent, or gate it on a condition, and be honest that the tool can no longer tell you in advance what will happen. **Not everything can be idempotent.** Genuinely one-shot actions — send a notification, rotate a credential, trigger a restore — have no natural "already true" condition. Those belong in operational tooling rather than in the desired-state description, which is the honest answer when an interviewer pushes on the limits.