A group writes a job record and adds its key to a pending set, and the second step errors while applying — how do you repair the state it left?

## What the store actually left behind The group promised one thing and delivered it: no other caller's operation landed between the steps. It never promised the two effects would stand or fall together. After the membership step errors, the keyspace holds a **job record nothing points at** — and, if the group had a third step, that step's effect as well. Nothing in the store remembers that these operations belonged to one unit, so nothing will ever come back and tidy up. The per-step results in the caller's hand are the only record that exists, and they live only as long as that process does. That last point is the senior one. The repair is not hard; the repair *surviving the thing that owed it* is hard. ## Repair one: the caller compensates 1. **Read the per-step results** and determine exactly which effects landed. Do not infer it from the fact that an error came back. 2. **Choose a direction**: complete the work (add the missing member) or withdraw it (delete the record). Completing is usually right when the failed step was incidental; withdrawing is right when the record is meaningless unless it is listed. 3. **Write the compensation idempotently** — add a member to a set, write a whole value, delete by key. Each of those is safe to run twice; an operation that adjusts a value by a delta is not. 4. **Bound the attempt.** If the compensation itself fails, the state must still be findable by someone else, which is the next section. ## Repair two: make the partial state harmless The cheapest repair is the one you never run. Order the steps so that any prefix is a state the system can live with, and put the step that makes the work **visible to readers** last. If the pending set is what consumers read, writing the record first and the membership second is already the right order: a failure leaves an unreferenced record that no consumer will pick up, rather than a queued key pointing at a record that does not exist. Readers should also be written to tolerate the middle state — a member whose record is missing is skipped, not treated as corruption. | Repair strategy | What it needs | What it costs | |---|---|---| | Caller compensates | The per-step results and an idempotent write | Useless if the caller dies first | | Order steps so a prefix is benign | Control over step order and reader behaviour | Only works when one step is the visible one | | Reconciling sweep | A way to enumerate or timestamp candidates | Runs continuously; adds load and lag | | Self-clearing marker | An entry lifetime on the unfinished record | Only safe when losing the record is acceptable | | Collapse into one entry | The data fits under one key, written whole | Larger value, coarser contention | ## Repair three: something that does not depend on the caller Assume the process crashes between the error and the compensation, because eventually one will. Two shapes cover it. A **reconciling sweep** walks records and re-derives the membership set — the invariant becomes something the system continuously restores rather than something one caller must get right. Or the record is written with an **entry lifetime** and only made permanent once the group's remaining steps confirmed, so an unfinished record disappears by itself; that is only acceptable where losing an unconfirmed job is better than leaking one. ## Changing the shape so the repair is not needed - **Put what must change together under one key.** If the record and its queued state are one value written whole, the change is a single operation and there is no partial state to repair. The price is a larger value and coarser contention on it. - **Move the decision to the store as one unit.** Where the store accepts a submitted program, the multi-step decision is applied as one unit — but note that this still gives you no rollback if a step inside it fails, only fewer moments where the caller can vanish. - **Put the invariant where rollback exists.** If "a record is listed if and only if it exists" must hold strictly, the authority for it belongs in an engine with transactions, and this tier holds the fast copy that the reconciler repairs. ## What varies Some stores in this class offer no grouping construct at all, in which case this failure mode arrives even earlier — the two writes were always separate operations. Where the keyspace is split across nodes, a group touching both entries requires them to be co-located in the first place. And whether the steps following the failing one were applied is worth confirming for your store rather than assuming; the repair differs if step three never ran. ## One group, three steps, one error while applying. Two effects landed and one did not; nothing is undone, and the per-step result list is the only record of which is which ``` group submitted with three steps step operation result keyspace afterwards ---- ------------------------------------------------- -------- ------------------------------- 1 write job record under key jobs:8123 applied record present 2 add jobs:8123 as a member of the pending set errored pending set unchanged 3 server-side in-place update of queued-count applied count includes the invisible job returned to caller: one result per step -> [ok, error, ok] stored state: a record no consumer can see, and a count that disagrees with the set ```