How do you use fcntl.flock to guarantee only one copy of a Python job runs?

### The guarantee you actually want A job that must not overlap itself — a minute-by-minute cron entry, a queue drainer, a nightly sweep — needs mutual exclusion whose lifetime is tied to a **process**, not to a file's existence. On a single Unix host `fcntl.flock` is that primitive: the lock is kernel state attached to an open file description, and the kernel tears it down when the holding process goes away, however it goes away. ### The idiom ```python import fcntl, os, sys fd = os.open("/tmp/convert-worker.lock", os.O_CREAT | os.O_RDWR, 0o644) try: fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) except BlockingIOError: sys.exit(0) # another copy is running; this tick is a no-op ``` Four decisions are packed into those lines. **`os.O_CREAT | os.O_RDWR`, never a truncating mode.** The file must exist before it can be locked, and `os.O_CREAT` creates it when absent without disturbing it when present. Opening with mode `"w"` (`os.O_TRUNC`) blanks a file the running instance may have written its pid into — and the truncation happens at open time, *before* you have the lock, so it happens even in the copy that is about to lose the race. **Hold a descriptor, not a value you drop.** `fcntl.flock(open(path, "a"), fcntl.LOCK_EX)` looks fine and is useless: no reference survives the call, CPython closes the file object on the spot, and closing releases the lock. Bind the descriptor (or the file object) to a name that lives as long as the work does, at module scope or on a long-lived object. **`fcntl.LOCK_EX`, not `fcntl.LOCK_SH`.** A shared lock is compatible with other shared locks, so two copies asking for `fcntl.LOCK_SH` both succeed and both run. Only the exclusive mode gives single-instance. **`fcntl.LOCK_NB`.** Without it the second copy *blocks* until the first finishes. For a job fired every minute that turns a slow run into a growing queue of waiting processes, all of which eventually run back-to-back. With it, the call fails immediately with `BlockingIOError` (an `OSError` subclass, `EWOULDBLOCK`) and you choose the policy: skip the tick, log, or exit non-zero so a supervisor notices. ### Lifetime, precisely A `flock` lock belongs to the open file description, not to the file and not to the process. It is released when the last descriptor referring to that description is closed, when you call `fcntl.flock(fd, fcntl.LOCK_UN)`, or when the process terminates — cleanly, on an unhandled exception, or under SIGKILL. There is no cleanup handler to write and no stale state to detect on the next start. That single property is why this beats a hand-rolled pidfile, whose contents survive the process that wrote them. ### Advisory, not mandatory "Advisory" means the kernel only holds back processes that *also* lock the same file. Anything that just opens the path and writes it succeeds regardless. For single-instance that is fine, because every participant is your own program; but it means the lockfile is a token, not protection for data. Keep the lockfile dedicated to locking — do not lock the data file the job is writing. ### Things that quietly break it *Deleting the lockfile on exit.* Process A unlinks the file as it finishes while B has already opened that path and is holding the lock on the now-unlinked inode; C then creates a fresh file at the same path and locks that. B and C both believe they are the only instance. Leave the file in place — an empty file per job costs nothing. *A per-run path.* Locking a file whose name contains a timestamp or a pid excludes nothing. The path must be fixed and shared by every copy, and it must be on a filesystem that survives — a directory cleaned between runs reintroduces the same problem. *Assuming it reaches other hosts.* The guarantee is one kernel. Advisory locks on a network filesystem are mediated by the server's lock manager with its own recovery windows; single-instance across machines is a lease problem, not an `fcntl` one. *Portability.* There is no `fcntl` module on Windows; the locking call there is a different, platform-specific API, so a cross-platform tool needs a branch. ### Writing the pid in After locking, `os.ftruncate(fd, 0)` and `os.write(fd, str(os.getpid()).encode())` make the file self-describing for an operator running `ls` and `cat`. That is diagnostics only: correctness never reads the contents back, which is exactly what makes the scheme immune to the stale-content problems of a pidfile-driven check.