How do you stop queued concurrent.futures work when a 6-hour nightly billing run hits a fatal error?

A nightly subscription-billing run that fans 200,000 accounts across a pool and takes six hours has to answer a hard question halfway through: the payment gateway starts rejecting everything, and you want the run to stop **now** rather than push through three more hours of guaranteed failures. `concurrent.futures` gives you exactly two levers, and it is important to know which one does what. ### Lever one: cancelling what has not started A `Future` moves through pending, running, then finished or cancelled. `Future.cancel()` returns `True` only from the pending state — it flips the future to cancelled and the worker skips it. Once a worker has picked the item up, `cancel()` returns `False` and does nothing. There is no `Future.kill()`, and there cannot be: interrupting arbitrary Python code at an arbitrary bytecode boundary would leave half-updated state behind. `Executor.shutdown(wait=True, cancel_futures=True)` is the bulk form of the same thing. It drains the pending work queue, cancels each of those futures, refuses new `submit` calls with `RuntimeError`, and then — because `wait=True` — blocks until the currently running items finish. `cancel_futures` was added in Python 3.9; before that you had to loop over your futures calling `cancel()` yourself, which is still a fine way to cancel a subset. The trap is the context manager. `Executor.__exit__` calls `shutdown(wait=True)` and nothing else, so leaving the `with` block on an exception **still drains the entire backlog** — the run you thought you aborted keeps billing for three more hours before the traceback prints. If a `with` block might be exited early, either construct the executor outside it and shut it down explicitly, or wrap the body in `try/except` and call `shutdown(cancel_futures=True)` on the way out. ### Lever two: stopping what is already running Running items can only be stopped cooperatively. The standard shape is a `threading.Event` created in the parent, closed over (thread pool) or passed via an `initializer` (process pool), and checked by the worker between units of work: ```python abort = threading.Event() def charge(account): if abort.is_set(): return "skipped" ... ``` That gives a bounded stop time equal to one item, not one queue. With a `ProcessPoolExecutor` the only forcible alternative is killing the child processes, which breaks the pool: every pending `Future` completes with a `BrokenProcessPool` error and the executor is finished. ### Noticing the failure early enough to act None of this helps if you learn about the failure last. Draining results with `Executor.map` or by iterating your futures in submission order means the first error you see is the first *by input position*, possibly hours late. Two better shapes: * `done, not_done = concurrent.futures.wait(futures, return_when=FIRST_EXCEPTION)` returns the moment any job raises, without raising itself, and hands you both sets. Follow it with `shutdown(wait=True, cancel_futures=True)`. * `for fut in concurrent.futures.as_completed(futures)` with a failure counter, so you can apply a policy — abort after N consecutive failures, or after a failure rate crosses a threshold — instead of aborting on the first blip. On a six-hour billing run a threshold is usually right: one declined card is normal, a 100% decline rate is an outage. ### Cancellation is not a rollback This is the part that separates a senior answer. Cancelling futures stops *future* work; it undoes nothing. The accounts already charged stay charged, and the executor has no notion of a transaction spanning items. A partial-failure rollback has to be designed at the job level: * Make each item idempotent and keyed — an idempotency key per invoice, so a re-run charges nobody twice. * Record each item's outcome durably as it completes, not at the end, so an aborted run leaves a ledger you can resume from rather than an all-or-nothing mystery. * Decide explicitly whether the unit of atomicity is the item or the batch. Per-item is almost always the answer for a long run, because a six-hour batch that must roll back wholesale has a six-hour recovery. * Have the failure path report counts — succeeded, failed, cancelled, never-started — because "the run stopped" is not an operable statement. ### One shutdown detail worth knowing Since Python 3.9, `ThreadPoolExecutor` workers are ordinary non-daemon threads that interpreter shutdown joins. A process holding an executor with a long queue will therefore not exit until that queue drains, even on `SIGINT`. Explicitly calling `shutdown(wait=True, cancel_futures=True)` in your signal handling path is what makes Ctrl-C actually stop a batch job in reasonable time.