Why does an exception inside a ThreadPoolExecutor job stay hidden until Future.result()?

`Executor.submit(fn, *args)` does two separate things: it puts a work item on the executor's internal queue, and it hands you back a `concurrent.futures.Future` — a thread-safe handle to a result that does not exist yet. The call returns in microseconds. `fn` runs later, on one of the pool's worker threads in a `ThreadPoolExecutor`, or in a worker process in a `ProcessPoolExecutor`. ### Why the exception cannot simply propagate A `raise` unwinds the stack of the thread that raised it. Your calling thread is somewhere else entirely by then — it has already returned from `submit` and moved on. There is no stack frame of yours for the worker's exception to travel through. If the pool let the exception escape the worker's run loop, the worker thread would die and the pool would silently shrink until nothing ran at all. So the executor wraps each call. It invokes the callable inside a `try`, and either records the return value on the `Future` or records the exception on it. It catches `BaseException`, not just `Exception`, so even something like a `KeyboardInterrupt` raised inside the callable lands on the `Future` rather than killing the worker. ### Where the exception comes back `Future.result()` blocks until the job reaches a final state, then returns the value or re-raises the stored exception in the *calling* thread. The exception object still carries the worker's traceback, and CPython appends the frames of the `result()` call site, so a printed traceback shows both halves: your loop at the top, the failing worker frames at the bottom. `Future.exception()` is the non-raising counterpart: it blocks the same way and returns the exception object, or `None` when the job succeeded. Both take a `timeout=` argument, and both raise `TimeoutError` if the job has not finished in time. Since Python 3.11, `concurrent.futures.TimeoutError` is an alias of the built-in `TimeoutError`, so a plain `except TimeoutError` catches it; on older versions it was a distinct class. ### `done()` is not `succeeded()` This is the trap juniors fall into. `Future.done()` returns `True` for a job that finished, a job that raised, and a job that was cancelled. `Future.cancelled()` singles out the cancelled case, and `result()` on a cancelled `Future` raises `CancelledError` rather than returning anything. The only way to learn whether a job *succeeded* is to ask for its outcome, with `result()` or `exception()`. ### The bug this design creates in real code ```python with ThreadPoolExecutor(max_workers=8) as pool: for record in records: pool.submit(handle, record) print("done") ``` This prints `done` and exits cleanly whether every job succeeded or every single one raised. `Executor.__exit__` calls `shutdown(wait=True)`, which waits for the queue to drain — it never looks at a result, and there is nowhere for a worker error to go. A batch job written this way reports success while quietly dropping a fraction of its work. The fix is to keep the futures and drain them: ```python futures = [pool.submit(handle, r) for r in records] failures = [] for fut in futures: try: fut.result() except Exception as exc: failures.append(exc) ``` Or, when you only want to log rather than re-raise, use `Future.exception()` and skip the `try` entirely. A third option is `Future.add_done_callback(fn)`: the callback fires when the job finishes, fails or is cancelled, receives the `Future` itself, and must call `result()` or `exception()` to see the outcome. Note that an exception raised *inside* a done-callback is logged by `concurrent.futures` and otherwise swallowed, so callbacks are not a place to put fragile code. ### One extra twist with process pools With a `ProcessPoolExecutor` the exception has to travel back from a child process, which means it is pickled. An exception class that cannot be pickled, or one whose custom `__init__` signature does not survive a round trip, comes back degraded. And if the worker dies outright — a segfault in a C extension, or the OS killing it for memory — there is no exception to store at all: the pool is declared broken and every pending `Future` is completed with a `BrokenProcessPool` error from `concurrent.futures.process`. ### How to say it in an interview One sentence carries it: *the worker cannot raise into my thread, so the pool stores the outcome on the Future and `result()` replays it at the call site — which means an unread Future is an unread failure.* Then name the fire-and-forget bug, because that is what the interviewer is actually probing for.