skip to content

Which schema changes can block reads or writes on a large, busy table, and what techniques keep the lock window short enough to be invisible to users?

level: seniorimportance: must knowfreq 52%

answer

  1. Lock strength vs lock duration — two different hazards
  2. DDL queues behind long queries; everything queues behind DDL
  3. lock_timeout + retry with backoff
  4. Concurrent index build; constraint NOT VALID then VALIDATE
  5. Rewrites flood replication and stall replica replay

basics

~20 s

Anything that rewrites the table or holds an exclusive lock: type changes, most index builds, validating new constraints, some column additions. Use concurrent or online index builds, add-then-validate constraints, a short lock timeout with retries, and batched work.

solid answer

~60 s

Two separate hazards, and candidates often conflate them. **Lock strength.** DDL that takes an exclusive table lock blocks everything, including reads, for as long as it holds it. Even a metadata-only change is dangerous on a busy table because the DDL must first *acquire* the lock: it queues behind running queries and, in engines with fair queuing, every new query then queues behind the DDL. One long-running SELECT plus one instant ALTER equals a full stall. **Work volume.** DDL that rewrites the table or reads every row holds its lock for a duration proportional to table size: type changes, rewriting defaults on older engine versions, validating a CHECK or foreign key, non-concurrent index builds. Mitigations I actually use: - Set a short `lock_timeout` (a second or two) and retry with backoff, so a failed attempt costs nothing instead of stalling traffic. - Build indexes with the engine's concurrent/online option. - Add constraints as NOT VALID, then validate separately with a weaker lock. - Prefer add-new-column-and-migrate over ALTER TYPE. - Kill or wait out long-running transactions first; they are what turns a fast DDL into an outage. - Run it deliberately, off-peak, with a plan to abort.

code

sql · 8 lines
sql
SET lock_timeout = '2s';   -- fail fast instead of queueing behind a long query

-- fast metadata step: enforced for new rows only
ALTER TABLE orders
  ADD CONSTRAINT orders_total_nonneg CHECK (total >= 0) NOT VALID;

-- separate statement, weaker lock, concurrent writes allowed
ALTER TABLE orders VALIDATE CONSTRAINT orders_total_nonneg;

go deeper

for a junior

Know that some schema changes lock the table and that index creation and type changes are the usual suspects.

for a middle

Distinguish lock strength from lock duration, name the rewriting operations, and know that concurrent index builds and NOT VALID constraints exist.

for a senior

Own the operational procedure: lock_timeout with retries, clearing long transactions first, add-then-validate, batching, replication-lag throttling, rehearsal on production-sized data and an abort path.

for a principal

Set the policy and the guard rails — a migration review checklist, an enforced default lock timeout, when an online schema-change tool is warranted versus expand-contract, and how migration risk is scheduled against traffic and failover windows.

## Two different questions "Is this DDL blocking?" decomposes into: **how strong is the lock**, and **how long is it held**. A statement can be dangerous for either reason, and the mitigations differ. ## Lock acquisition is the hidden hazard Even a purely metadata change needs an exclusive lock on the table. To get it, the statement must wait for every conflicting transaction to finish — including an analytics query that has been running for ten minutes, or an idle-in-transaction session that touched the table and never committed. Meanwhile, in engines with a fair lock queue, incoming queries queue *behind* the waiting DDL. The result is the classic incident: an ALTER that takes 3 ms takes down the service for ten minutes, because it sat in the queue and everything piled up behind it. The standard defence is to make waiting cheap: set a low `lock_timeout` (or the engine equivalent) so the DDL either grabs the lock almost immediately or fails, and wrap it in a retry loop with backoff. Failing fifty times costs nothing; waiting once costs an outage. Before running, check for long transactions and either wait them out or terminate them. ## Operations that rewrite or scan the whole table These hold their lock for a size-proportional duration: - **Changing a column type** in a way that changes storage (int → bigint, text → numeric). Usually a full rewrite, doubling the table's space during the operation. - **Adding a column with a non-constant default** — a function evaluated per row forces a rewrite. Adding a column with a *constant* default is metadata-only on current major engine versions, but was a rewrite on older ones. - **Setting NOT NULL** on an existing column: requires proving no NULLs exist, which is a full scan under a strong lock unless the engine can lean on an already-validated check constraint. - **Adding a CHECK or FOREIGN KEY constraint**: validating existing rows means scanning the table (and, for a foreign key, locking the referenced table too). - **Building an index** the ordinary way: reads the whole table and blocks writes for the duration. - **Reordering, clustering or repacking** a table. ## Techniques that keep the window short **Concurrent / online index builds.** All major engines offer a form that allows reads and writes while the index is built, at the cost of more total work, more than one pass, and — in some engines — the possibility of leaving an invalid index behind if it fails, which must be dropped and retried. Always the right default on a large table. **Add-then-validate for constraints.** Add the constraint in a not-yet-validated state (a fast metadata operation that enforces the rule for *new* rows), then validate the existing rows in a separate statement that takes a weaker lock and allows concurrent writes. Two steps, no long exclusive lock. **Avoid ALTER TYPE entirely.** Instead of changing a column's type in place, add a new column of the target type, dual write, backfill in batches, switch reads, drop the old one — the expand-contract sequence. It is more steps but every step is short. **Batch the data work.** Anything touching many rows belongs in bounded batches with commits between them, so no single transaction holds locks or accumulates undo/version data for long. **Online schema-change tools.** Where the engine's own facilities are insufficient, tools that build a shadow copy of the table, keep it in sync with triggers or by tailing the replication stream, and swap it in with a brief rename are standard practice. They trade extra disk and load for a lock window measured in milliseconds. **Take the small stuff seriously.** Wrap DDL in an explicit transaction where the engine supports transactional DDL, so a failure leaves nothing half-applied; run one change per migration so a retry is cheap; and never batch a fast metadata change together with a long-running one in the same transaction, because the transaction holds the strongest lock for the longest duration among them. ## Replication and follower impact A rewrite generates a large volume of write-ahead log or binlog, which floods replication and can push follower lag high enough to break read-your-writes routing or delay failover. On engines that replay DDL serially on the replica, a long DDL blocks *all* replay, so replicas fall behind for the whole duration even though the primary is fine. Throttling on measured lag is part of the plan, not a nicety. ## Rehearsal and observation Measure on a production-sized copy, not on a development database with a thousand rows: rewrite duration is proportional to size, so a change that takes 200 ms locally can take forty minutes live. During the run, watch lock waits, active query counts, error rates and replication lag, and have an abort path — for a batched job that means stopping the loop; for a single DDL it means cancelling and letting the engine roll back, which for a rewrite may itself take time. ## Vendor caveat The specific catalogue of what is metadata-only differs by engine and version, so the professional answer names the class of hazard and then says "and I check my engine's documented behaviour for this version" rather than asserting a universal table of safe operations.

  • An ALTER TABLE that should be instantaneous caused a multi-minute outage. Explain how.
    The statement needed an exclusive lock and could not get it because a long-running query or an idle-in-transaction session still held a conflicting lock on the table. While the ALTER waited in the lock queue, newly arriving queries queued behind it rather than overtaking it, so the table became effectively unavailable even though no work was being done. The fix is a short lock_timeout with retries, plus checking for and clearing long transactions before running DDL.
  • How do you add a NOT NULL constraint to a large existing column without a long exclusive lock?
    Backfill the column so no NULLs remain, add a CHECK (col IS NOT NULL) constraint in a not-yet-validated state so new rows are covered immediately, then validate it in a separate statement that takes a weaker lock. On engines that can use an already-validated check constraint as proof, the subsequent SET NOT NULL becomes a metadata operation instead of a full scan. Otherwise the check constraint itself is a perfectly adequate substitute.
  • Why does a table rewrite matter for replicas even if the primary handles it fine?
    A rewrite writes the entire table again, producing a large burst of write-ahead log or binlog that replicas must transfer and replay, so lag spikes and any read routing that assumes freshness breaks. On engines that replay DDL serially, the replica applies the whole operation as one blocking step, stalling all other replay for its duration. That is why long migrations are throttled on measured lag and scheduled when a lag spike is tolerable.

saying these in an interview costs you the question

  • Judging a migration safe because it ran quickly against a development database with few rows
  • Treating a metadata-only ALTER as free while ignoring the time spent waiting to acquire the lock
  • Running ALTER TABLE with no lock timeout on a busy table
  • Building an index on a large table without the concurrent or online option
  • Adding a foreign key or check constraint in one step and being surprised by the validation scan
  • Ignoring replication lag caused by a table rewrite

context