A nightly ETL export runs under python -O in production but not in CI; what breaks?

This is a configuration-divergence bug, and it belongs to a family: production runs an artefact that no test ever executed. What makes the optimization level a particularly nasty member of that family is that the difference is invisible in the source, invisible in the process listing if the level came from `PYTHONOPTIMIZE`, and produces no error of its own. ## What actually differs At level 1 the compiler emits no bytecode for `assert` statements and drops `if __debug__:` blocks as dead code. Three consequences follow. **Invariants stop being enforced.** The check that made CI fail fast on a malformed batch is absent in production, so a bad batch flows onward and fails much later, somewhere with less context — a constraint violation in the warehouse, or worse, silently wrong data. **Side effects inside assert expressions disappear.** This is the one that produces genuinely different control flow. `assert claim_partition(worker_id)` acquires the claim under test and does nothing in production. Two export workers that were serialised in CI now overlap, and the race on shared state that the assertion was quietly preventing becomes real. The bug is not "an assertion was skipped"; it is "a function was never called". **Expensive debug work vanishes.** That part is intended, and it is the only legitimate reason the flag exists. ## Why it is hard to catch The level can be set far from the code. A base image, a Kubernetes manifest or a job template can export `PYTHONOPTIMIZE`, and nothing in the repository records it. In a fan-out pipeline — say a 17-service dependency graph where the export job pulls a shared transform library that eleven other services also import — a single service whose image sets the variable makes that shared library behave differently in exactly one place. Reviewers reading the library see assertions; one deployment does not run them. The reproduction step fails too. An engineer runs the failing job locally, without the flag, and it behaves correctly, which sends the investigation toward data, timing or the warehouse rather than toward the interpreter's startup configuration. ## How to diagnose it Start by making the level observable. Log `sys.flags.optimize` and `sys.version` once at startup for every service; a one-line addition turns the whole class of question into a query. If you suspect it retroactively, reproduce the job with the same flag rather than the same code — the difference in behaviour under `-O` versus level 0 is the confirmation. Then audit the assertions. The dangerous ones are those whose expression calls something. A short script that walks the abstract syntax tree with the `ast` module and reports every `assert` whose test contains a call is enough to find them across a repository, and a third-party linter will flag several of the same shapes. Fix each one by lifting the call onto its own statement and asserting on the result — or, more often, by deciding it was never an assertion in the first place. ## How to prevent it **One level everywhere.** Whatever level production uses, CI must use it too, including for the integration jobs. If they cannot be aligned, do not use the flag. **Prefer level 0.** `-O` provides no speed; the only real payoff on the ladder is the memory that `-OO` reclaims by dropping docstrings. If nobody can name the benefit being bought, the flag is pure risk. **Reserve `assert` for internal invariants.** An assertion says "this is impossible if the program is correct", and its cost of failure is a crash during development. Anything a correct program must still check at runtime — a partition claim, an argument that came from outside, a precondition that protects downstream data — needs an ordinary `if` and a raised exception. **Keep assert expressions pure.** Read state, compare it, produce a bool. If deleting the line would change what the program does, it is not an assertion. **Make it a platform rule, not a per-service one.** The level is a property of the runtime image and belongs in the same tier as the interpreter version: set once, verified in a smoke check, and identical across the graph. ## What the interviewer is listening for The weak answer is "assertions are disabled in production". The strong answer adds that the statement is *removed at compile time*, that removal takes its side effects with it, that the level can arrive from the environment without touching the code, and that the remedy is alignment plus moving real checks out of `assert` — not adding a try/except around an `AssertionError` that will never be raised.