What does the Iceberg REST catalog spec change compared with using a Hive Metastore?

## Two different kinds of thing A useful first move in the interview is to refuse the false symmetry. **Hive Metastore (HMS)** is a *product*: a long-running Thrift service backed by a relational database, storing databases, tables, partitions, columns and storage descriptors. **The Iceberg REST catalog** is a *protocol*: an OpenAPI specification for how a client asks a catalog to list namespaces, load a table, and commit a change. Polaris, Unity Catalog, Nessie, Gravitino, cloud vendors' catalogs and small in-house services can all speak it; even HMS can sit behind a REST-speaking front end. ## Where the logic lives With HMS, the pointer to a table's current metadata is stored as a table property, and the client library — running inside Spark, Trino or Flink — is what builds the new metadata, decides whether the commit is valid, and performs the conditional update. That means: - Every engine ships its own copy of catalog logic, and behaviour depends on which version each shipped. - A change to commit semantics is a fleet-wide client upgrade. - The catalog cannot enforce much, because it does not understand the operation; it sees a property update. Under the REST spec, the client sends a structured commit: a set of **requirements** — assertions such as "the table's current snapshot is still the one I read" — and a set of **updates** to apply. The server evaluates the requirements and applies the updates atomically or rejects the whole request. The commit decision moves server-side. That is the architectural change everything else follows from. ## Consequences that matter in production **Thin, polyglot clients.** Talking Thrift and the Hive object model in Python, Rust or Go is painful; talking HTTP and JSON is not. This is why the REST spec is what made non-JVM readers and writers practical, and it is the answer interviewers are usually fishing for. **Server-side evolution.** Fix a conflict-detection bug, add support for a newer table format version, change how retries behave — deploy the catalog, not fifty engine images. **Configuration handshake.** A client fetches catalog configuration at initialization, so the server can hand out defaults and overrides (warehouse location, storage settings) instead of every engine holding its own copy in a config file. **Authorization boundary.** Because the server understands the operation and identity, it can authorize at table and namespace level, and can return temporary, scoped storage credentials for the specific table being read rather than requiring every engine to hold broad bucket credentials. **A real multi-tenant surface.** HMS was designed for a Hadoop cluster's trusted interior; putting it on the open internet with proper authentication and tenancy is uncomfortable. An HTTP service with standard auth is a normal thing to operate. ## What does not change - **The table format is untouched.** Data files and metadata still live in your object storage in the same layout. A REST catalog is not a storage system and does not hold your rows. - **The atomicity requirement is identical.** Whether the swap happens as a conditional database update inside HMS or as requirement evaluation in a REST server, exactly one writer may win a given transition. - **A table still needs one owning catalog.** Speaking a standard protocol does not make it safe for two catalogs to accept writes to the same files. - **Migration is not free.** Existing tables registered in HMS or a cloud catalog must be moved deliberately, and every engine in the estate needs a client that speaks the new protocol at a compatible version. ## Why HMS persists anyway Because it is everywhere. A decade of Hive, Spark and Trino deployments resolve tables through it; non-Iceberg tables in the estate may depend on its partition model; and tooling, permissions and lineage systems are wired to it. Many platforms therefore run both: HMS for legacy Hive-style tables, a REST-speaking catalog for open table format tables, or a REST façade in front of the existing metastore so clients converge on one protocol while the storage of record stays put. ## How to answer this in an interview Lead with "specification versus implementation", then name the one structural change — the commit decision moves to the server — and let the consequences (polyglot clients, server-side evolution, credential vending, real multi-tenancy) fall out of it. Candidates who instead describe the REST catalog as "a faster metastore" or "a metastore in the cloud" have missed the point entirely.