What does the Hive Metastore provide that query engines outside Hive still need?

## The problem it solves A distributed filesystem stores files. It knows a path, a size, a modification time and a set of blocks. It does not know that the CSV under `/warehouse/sales/` has a header, that the third column is a `DECIMAL(12,2)`, that the directories named `dt=2026-01-01` are a partition column rather than a literal folder, or that the whole thing should be read as ORC with a particular SerDe. Something has to hold that interpretation. In the Hadoop ecosystem that something is the **Hive Metastore (HMS)**. ## What it actually stores The metastore is a catalog of *metadata only* — never table rows. Per object it records: - **Databases** (namespaces) and their default warehouse location. - **Tables**: name, owner, table type (`MANAGED_TABLE` or `EXTERNAL_TABLE`), the list of columns with types and comments, and free-form `TBLPROPERTIES`. - **Storage descriptors**: the base location (an HDFS or S3 URI), input format, output format, SerDe class and its properties, compression, bucketing columns and sort columns. - **Partitions**: one row per partition value combination, each with *its own* location and storage descriptor. This is why a partition can live outside the table's base directory. - **Statistics**: row counts, file sizes, and per-column stats such as NDV, min/max and null counts, used by cost-based optimizers. `DESCRIBE FORMATTED <table>` prints most of this back at you, and is the fastest way to see what the catalog actually believes. ## Architecture HMS is a Java service backed by a relational database, accessed through DataNucleus/JDO. Three deployment shapes exist: - **Embedded** — an in-process Apache Derby database. This is the default when nothing is configured and it is the classic first-day trap: Derby in embedded mode allows a single JVM to hold the database, so the second client fails to connect, and the `metastore_db` directory is created wherever you happened to launch the shell. Never use it beyond a tutorial. - **Local** — the metastore code runs inside the client JVM but talks to a shared MySQL/PostgreSQL. - **Remote** — a standalone metastore process exposing Thrift on port 9083, which every client reaches via `hive.metastore.uris=thrift://host:9083`. This is the production shape, and the only one that gives you a single authoritative catalog plus a place to enforce authorization. Since Hive 3 the metastore ships as a **standalone artifact** that can be run without the rest of Hive at all — an acknowledgement that most of its clients are no longer Hive. ## Why non-Hive engines depend on it The Thrift API became the de-facto catalog interface for the whole ecosystem. Trino/Presto, Flink and Spark can all register HMS as a catalog. The payoff is that a table is defined once and read by many engines with the same schema, the same partition layout and the same location — the concrete mechanism behind "separate storage from compute". Without a shared catalog every engine needs its own DDL and the definitions drift. This also explains why HMS is a **hard dependency** long after nobody runs Hive queries: rip it out and a hundred `SELECT`s across three engines stop resolving table names. ## Partitions, the sharp edge Because each partition is a row in a relational database, partition metadata is both the metastore's greatest value and its main failure mode: - New directories written directly to HDFS by another job are **invisible** until the catalog learns about them — hence `MSCK REPAIR TABLE t;` (scan the location and add what is missing) or explicit `ALTER TABLE t ADD PARTITION (...)`. - Over-fine partitioning (hour, then customer, then region) produces hundreds of thousands of partition rows. Planning a query then means a large metastore query, and `MSCK REPAIR` on such a table can run for a very long time or time out. - Because pruning happens against the catalog, wrong or missing statistics silently degrade plans in engines that use cost-based optimization. ## Where it is going In 2026 the direction of travel is toward catalogs designed for table formats rather than directories: the **AWS Glue Data Catalog** (deliberately HMS-API-compatible so existing clients work), **Iceberg REST catalogs**, Unity Catalog and Polaris. These add atomic metadata commits, snapshot/time-travel and file-level tracking that HMS never had. The role, though, is unchanged — and in an interview the useful point is that HMS defined the role, so knowing what it stores transfers directly to whatever replaces it. ## Interview framing If you say only "it stores Hive table metadata", you have answered a 2014 question. The 2026 answer is: it is the shared catalog that turns files into tables for several engines at once, it lives in an RDBMS behind a Thrift API, its partition table is where scale problems appear, and its successors copy its interface.