How would you choose one catalog for a platform where Spark, Trino, Flink and a warehouse share tables?

## Why this is a design question, not a shortlist There is no best catalog. There is a set of constraints, and the honest answer names them in priority order and shows what you would give up. An interviewer at this level is checking whether you know which constraints are hard. ## Constraint 1: who writes This dominates everything. Reading a table through a second, read-only path is tractable; writing through two catalogs is not, because atomicity lives in a single pointer and a single conditional swap. So enumerate the writers — batch jobs, streaming jobs, the warehouse, dbt-style transformation tools, ingestion services — and require that every one of them has a maintained client for the candidate catalog at a version you can deploy. A candidate catalog that a single writer cannot speak is either eliminated or forces that writer into a read-only role. ## Constraint 2: governance requirements Ask what "secure" has to mean here: - **Discovery only** — the catalog resolves names and holds the pointer; permissions live in storage policy. Cheapest, weakest. - **Table-level authorization enforced at access time** — the catalog authorizes and vends scoped, short-lived storage credentials, and the direct bucket path is closed. This is what regulated environments usually need, and it eliminates plain metastores. - **Row and column controls, lineage, tagging, retention policy** — a governance catalog, at the cost of a heavier dependency and often a vendor. Do not buy governance you have no requirement for; do not pretend a metastore provides it. ## Constraint 3: operability and blast radius Once chosen, the catalog is on the resolution path of every query and, if it vends credentials, on the data path too. Treat it as a tier-one service: availability target at least as strong as the engines that depend on it, tested restore of its backing store, a failover story, and rate limits that survive a thousand-task Spark job resolving tables at once. Managed offerings trade control for that burden; self-hosting an open implementation trades cost for the pager. ## Constraint 4: protocol and lock-in Prefer a catalog whose protocol is specified rather than one whose interface is a specific product's client library. A specified HTTP protocol means today's clients keep working if you replace the implementation, and new engines are cheap to onboard. It also puts commit logic server-side, so evolving behaviour is a catalog deploy rather than a fleet upgrade. Ask explicitly: if we want to move off this in three years, what has to change — table metadata, or just an endpoint and some credentials? ## Constraint 5: migration cost Count the tables already registered somewhere, and the pipelines whose configuration names that catalog. The cutover per table is: stop writers, register the current metadata in the new catalog, repoint every writer and reader, delete the old writable entry. Anything that leaves two writable entries alive is worse than not migrating, because it silently loses commits. Plan for a dual-run period where the old catalog is read-only, and for a detection sweep that flags any two entries with overlapping storage locations. ## The warehouse problem Warehouses frequently insist on resolving tables through their own catalog. Options, roughly in order of preference: 1. The warehouse can act as a client of the lake's catalog, or the lake's catalog speaks a protocol the warehouse understands natively. One owner, everybody sees the same commits. 2. The warehouse mirrors the lake catalog read-only — federation or one-way sync — and never writes those tables. Acceptable; accept some staleness and be explicit about it. 3. The warehouse owns some tables and the lake catalog owns others, with a clear ownership boundary per table and no overlap. Workable but requires discipline. 4. Both write the same tables. Not an option; this is the doubly-registered failure. ## Decision shape to state out loud "One owning catalog per table, chosen by the union of writers that must speak it; a specified protocol so engines stay replaceable; governance level matched to an actual requirement rather than aspiration; run as a tier-one service; migration table by table with the old entry made read-only, then deleted." Then name what you gave up — usually some engine's native features, or a period of staleness on the mirror — because a principal-level answer that claims no tradeoff is not credible. ## Anti-patterns to call out Choosing by benchmark; choosing the catalog your favourite engine defaults to; letting each team pick their own; deferring the decision until "we see what sticks", which in practice means several catalogs claiming the same tables; and treating a crawler that auto-registers bucket contents as a catalog strategy.