Why do lakehouse catalogs vend short-lived storage credentials to query engines?

## The gap being closed In the classic setup, a query engine holds object-storage credentials — an instance role, a service principal, a key in a config file — that grant access to the buckets holding your tables. The catalog, meanwhile, records that team A may read `finance.ledger` and team B may not. Those two facts are unrelated. Team B simply points a storage client at the table's prefix and reads the Parquet files. The catalog grant was never an enforcement point; it was documentation. This is uncomfortable in exactly the environments where open table formats are attractive: many engines, many teams, one shared object store. ## How vending works 1. The engine authenticates **to the catalog** as a principal (a user, a job identity, a workload identity). 2. It asks to load a table for read or for write. 3. The catalog authorizes: does this principal hold that privilege on that table or namespace? 4. If yes, the response includes not only the metadata location but temporary credentials scoped to that table's storage location, with only the required permissions and a short expiry. An alternative shape is **remote signing**: the engine sends each storage request to the catalog to be signed, and never holds credentials at all. 5. The engine reads or writes the files with those credentials and discards them. The Iceberg REST catalog specification includes an access-delegation option along these lines, and governance-oriented catalogs such as Unity Catalog and Apache Polaris (incubating) are built around it. It is one of the concrete reasons a REST-style catalog is more than a faster metastore: because the server understands both the identity and the operation, it can be the place authorization happens. ## What it buys you - **Table-level authorization that actually holds**, because reaching the bytes requires going through the catalog. - **Least privilege in time and scope**: credentials expire in minutes and address one table's prefix rather than a bucket. - **Audit per table and per principal**, since every access begins with a catalog call. - **No long-lived secrets in engine configuration**, which removes a whole class of leak. - **Cross-engine consistency**: one grant model applies whether the reader is Spark, Trino, a Python client or a notebook. ## Where it stops **It is not a substitute for storage policy.** If the principal can already assume a role with bucket access, vending changes nothing — they read the files directly. Vending only bites when the direct path is closed, so the storage policy must be tightened at the same time. Candidates who present vending as a complete answer without saying this are missing the point. **Granularity is path-shaped.** Vended credentials scope to locations. Row filters, column masks and views are enforced by whatever component reads the data — the engine or a catalog-mediated read path — not by the credential. A credential that lets you read a table's files lets you read all of its rows. **Lifetime versus long queries.** A scan that outlives its credential fails partway. Clients must refresh, and the catalog must survive being on the hot path of every table load. That makes the catalog a real availability dependency: if it is down, nothing reads, which is a stronger coupling than a discovery-only metastore. **Client support is uneven.** Every engine and every library in your estate has to understand the delegation mechanism; the ones that do not fall back to their own credentials, quietly reopening the gap. ## Why this is a differentiator, not a screener Most candidates working with these formats have never configured vending, and a strong engineer can be entirely competent without it. But it is where the "which catalog?" decision stops being about interoperability and becomes about governance — and it is the reason organizations with real compliance requirements choose a governance catalog over a plain metastore.