A distributed lock service grants a lease to a client believed to hold exclusive access to a shared resource, such as a storage volume. The client experiences a long GC pause, its lease expires, and a second client acquires the lease and starts writing. The first client then resumes and, unaware its lease expired, also writes. How does fencing prevent this from corrupting the shared resource, and why isn't simply checking the lease is still valid on the client side sufficient?

## The gap fencing closes **Fencing** is the mechanism that closes a specific, dangerous gap in lease- and lock-based coordination: the gap between a client believing it holds exclusive access and that belief actually still being true. Distributed locks (via ZooKeeper, etcd, Chubby, or a database-backed lease table) are typically implemented as time-bounded **leases** - a client is granted ownership for, say, 10 seconds, must periodically renew it, and the lock service considers the lease expired and re-grantable if renewal doesn't arrive in time. The problem is that a client can be paused for an unbounded, unpredictable amount of time: - a stop-the-world garbage collection pause - a hypervisor migrating the VM - an overloaded kernel scheduler - a slow disk I/O syscall blocking a thread During such a pause it cannot renew its lease, has no way to observe that time has passed, and has no way to know, once it resumes, whether its lease already expired and was reassigned. From the paused client's own perspective, no time appeared to pass at all; it resumes execution exactly where it left off, fully believing it's still the lock holder, and proceeds to write to the shared resource. Meanwhile the lock service, having seen no renewal, has already reassigned the lease to a second client, which is now also writing. Both clients now believe, correctly by their own local reasoning, that they are the exclusive owner - this is the **zombie** or **split-brain writer** problem. ## Why client-side revalidation falls short Client-side revalidation - having the client re-check whether its lease is still valid right before writing - looks like an obvious fix but doesn't actually close the gap, because the check and the subsequent write are two separate steps with a time window between them, and the pause that caused the original problem can just as easily land in that window. - A **check-then-act** sequence is never atomic against an unbounded external pause; you can always construct a schedule where the pause happens after the check succeeds but before the write lands. - No amount of checking harder or more recently on the client side eliminates this, because the client cannot make guarantees about its own future scheduling. ## The token the resource enforces The fix fencing provides is to move the correctness check out of the untrustworthy, pausable client and into the resource being protected, where it can be made a real invariant rather than a best-effort courtesy check. Concretely: - Every time the lock service grants or renews a lease, it issues a **fencing token** - a number that increases monotonically across every grant, typically a simple counter maintained by the lock service itself. - The client must attach this token to every write it sends to the protected resource. - The resource - the storage node, the database, the file server - is modified to track the highest fencing token it has ever accepted and to reject any incoming write whose token is lower than that **high-water mark**. In the scenario above: 1. Client A gets token 33, pauses, its lease expires. 2. Client B gets token 34 and writes successfully, and the resource's high-water mark becomes 34. 3. Client A resumes and sends its write tagged with token 33, and the resource, which now requires tokens 34 or higher, rejects it outright, regardless of how confident client A is that it still holds the lock. The resource doesn't need to know or care why token 33 is stale; it just enforces **monotonicity**, which is a property it can check locally and correctly without needing to reason about clocks, pauses, or leases at all. ## Safety moves to the last-writer boundary This is a meaningful architectural shift: fencing pushes safety enforcement to the last-writer boundary - the resource - rather than relying on any actor upstream of it (client, lock service, network) behaving correctly or promptly. It's the same principle behind fencing in the older, more literal sense from cluster and SAN environments - **STONITH** (shoot the other node in the head) forcibly powers off or disconnects a suspect node from shared storage at the hardware level, because you can't trust a possibly-still-running zombie node to voluntarily stop. Software fencing tokens achieve the same isolation logically, without needing to physically kill anything, by making the shared resource itself the arbiter. ## The trade-off The trade-off is that fencing only works if the protected resource is willing and able to enforce the check - it requires that resource to be **fencing-aware**: - maintaining per-key or per-resource high-water marks - comparing tokens on every write which not every storage system supports natively. Plain object stores don't have a built-in fencing-token check, so applications must layer one on, for example via conditional writes keyed on a version or `ETag` field. If the resource can't enforce token ordering, fencing degrades back to a best-effort convention that a sufficiently pathological zombie client could still violate. This is why systems that need strong lock-based mutual exclusion design the storage layer itself to be fencing-aware rather than trusting the lock service or client discipline alone.