skip to content

A lakehouse table's files are registered in both a Hive Metastore and AWS Glue — what breaks?

level: seniorimportance: should knowfreq 50%

answer

  1. two entries, two pointers, no shared knowledge
  2. neither writer's commit ever conflicts
  3. one side's history never learns about the other's
  4. cleanup asks the wrong question about ownership
  5. exactly one writable owner per table

basics

~20 s

Each catalog keeps its own pointer to a current metadata version, so their conditional swaps never see each other. Commits made through one become invisible or get overwritten from the other's view, and a cleanup job run through one deletes files the other still references.

solid answer

~50 s

Atomicity depends on **exactly one** compare-and-swap loop over **one** pointer. Register the same table location twice and you get two independent loops. Writer A commits through the metastore and advances its pointer; writer B commits through Glue, whose pointer still names an older metadata version, and the resulting version silently drops A's files. Readers then disagree about the table depending on which catalog they resolved. The failure gets worse with maintenance: expiring snapshots or removing orphan files through one catalog deletes objects that the other catalog's current metadata still lists, so queries on that side fail with missing-file errors rather than wrong answers. The rule is one owning catalog per table. Other engines should read through that catalog, or through a read-only federation of it; registering the same table into a second writable catalog is a one-time migration step, not a steady state.

code

text · 12 lines
text
-- both catalogs initially point at metadata M7
metastore.sales.orders  -> s3://lake/sales/orders/metadata/M7.json
glue.sales.orders       -> s3://lake/sales/orders/metadata/M7.json

-- Spark commits through the metastore
metastore.sales.orders  -> M8   (= M7 + files X)   OK, no conflict
glue.sales.orders       -> M7   (unchanged, unaware of X)

-- a Glue-based job commits, rebasing on M7
glue.sales.orders       -> M9   (= M7 + files Y)   OK, no conflict

-- result: M8 has no Y, M9 has no X, neither commit reported an error

go deeper

for a junior

Recall the rule rather than the mechanism: a table has one owning catalog, and pointing a second catalog at the same files is not how you share a table between engines.

for a middle

Explain why the conditional swap cannot detect a conflict across catalogs, and walk through two commits rebasing on the same old version to show how one loses rows.

for a senior

Rank the failures by damage — silent lost commits, then maintenance deleting live files — and give the recovery and detection story, including auditing overlapping table locations before enabling cleanup jobs.

for a principal

Set the estate-wide policy: designate the owning catalog per domain, permit only read-only mirrors, and treat registration as a cutover primitive with writers stopped and old entries deleted.

## What the duplicate registration actually creates A catalog entry for an open table format table is, at heart, a mutable pointer to the table's current metadata version. Registering the same storage location in two catalogs does not create a shared table with two names; it creates **two tables that happen to share files**. Each has its own pointer, its own conditional-update loop, and no knowledge of the other. The conditional swap that makes commits atomic — "set the pointer to `M8` only if it is still `M7`" — is scoped to one catalog's record. A commit through catalog A cannot fail because of a commit through catalog B, because B's change never touched A's record. ## Failure one: lost commits Both catalogs start pointing at `M7`. 1. Spark, via the metastore, commits `M8` = `M7` + files X. The metastore now points at `M8`. Glue still points at `M7`. 2. A Glue-based job reads its table, sees `M7`, appends files Y, and commits `M9` = `M7` + Y. Glue points at `M9`. `M9` was built on `M7` and never mentions X. Readers on Glue see the table without X. Readers on the metastore see the table without Y. Neither commit failed; nothing logged an error. This is the silent-data-loss shape of the bug, and it is why it shows up in interviews: nothing alerts, and the divergence is discovered weeks later when two dashboards disagree. ## Failure two: deleted files under a live pointer Maintenance is where silent divergence becomes a loud outage. Snapshot expiry, and orphan-file removal in particular, work by asking "which files does any retained version of **this** table reference?" and deleting the rest. Run through the metastore, that question is answered from the metastore's version history. Files referenced only by Glue's current metadata look like garbage and get deleted. The next query on the Glue side plans successfully — its metadata still lists those files — and then fails during the scan with a missing-file error. Recovery means rolling that side back to an older version whose files survive, if one does, or rebuilding the table. ## Failure three: divergent schema and layout Schema changes, partition-spec changes and table properties all live in metadata, so each catalog can evolve its copy independently. One side adds a column and starts writing files with it; the other never learns, so its metadata does not describe the column and readers do not see it. Time travel becomes meaningless across the pair, since the two version histories are different lineages that occasionally share objects. ## How teams end up here - A migration that registers tables into a new catalog and never decommissions writes to the old one. - A warehouse that can only see tables through its own catalog, so someone registers the lake's tables there too — and then a pipeline writes through it. - Copying a table definition between environments by pointing at the same bucket path. - A crawler or auto-discovery process that inventories a bucket and creates entries in a second catalog for the same locations. ## The correct shapes **One owning catalog per table.** Every writer resolves the table through it. This is the only configuration that preserves atomicity. **Read-only elsewhere.** If another catalog must expose the table for discovery, make it a read path that resolves the owning catalog's current pointer, or make it strictly read-only and accept that it can be stale. Federation or catalog-to-catalog sync in one direction is acceptable when nobody writes on the mirror. **Registration as a migration step.** Registering an existing table's current metadata into a new catalog is legitimate exactly once, as part of a cutover: stop writers, register, repoint every writer, and delete the old entry. It is a cutover primitive, not a way to share a table. **Detection.** Audit for two catalog entries whose storage locations overlap; the cheap version is comparing table locations across catalogs. Before enabling any deletion-based maintenance, confirm the table has exactly one writable registration. ## What to say when asked Name the invariant first — atomicity requires a single pointer and a single compare-and-swap loop — then the three failures in order of nastiness: silent lost commits, deleted files under a live pointer, divergent schema. Then give the remedy: one owner, read-only mirrors, registration only during cutover.

  • Why is orphan-file cleanup the most dangerous job to run on a doubly-registered table?
    It deletes objects no retained version of the table it knows about references. Files that only the other catalog's current metadata lists look exactly like debris from a crashed writer. Once deleted, the other side keeps planning queries that name them and fails mid-scan, with no rollback target unless an older surviving version exists.
  • Is it ever safe for a second catalog to expose the same table?
    Yes, read-only. A mirror that resolves the owning catalog's pointer, or a one-way sync that nobody writes through, is fine for discovery and for query engines that can only see their own catalog. What is never safe is a second registration that accepts commits or runs deletion-based maintenance.
  • How would you detect this across an estate before it bites?
    Compare storage locations across every catalog entry and flag overlaps, including prefixes of one another. Pair that with a policy check that maintenance jobs run only against tables with a single writable registration, and make cutovers explicit: stop writers, register, repoint, delete the old entry.
  • Two dashboards on the same table disagree. How do you confirm this is the cause?
    Resolve the table through each catalog and compare the current metadata location and version history. If the two pointers name different lineages that share a common ancestor, you have divergence rather than staleness. Then identify which writers use which catalog before choosing an owner and reconciling.

saying these in an interview costs you the question

  • Assumes the two catalogs coordinate through the shared storage location
  • Thinks the file system resolves the conflict since files are shared
  • Says the newest commit simply wins across both catalogs
  • Believes read-after-write consistency in the bucket prevents this
  • Treats registering the same table twice as normal multi-engine sharing

context

resolving links…