skip to content

How would you migrate a large Hive-style Parquet lake to a table format without rewriting the data?

level: principalimportance: should knowfreq 35%

answer

  1. the table layer points at files, it does not own them
  2. what does the conversion actually have to read
  3. cheap to adopt, but you inherit the layout
  4. the hard part is not the data, it is the writers

basics

~20 s

Write a table layer over the existing files instead of copying them: the conversion reads each file's footer for schema and statistics and commits a file list referencing the current paths. You inherit the old file sizes and layout, so plan compaction as a separate follow-up.

solid answer

~50 s

Because a table format only *references* data files, adoption can be metadata-only. An in-place conversion enumerates the existing Parquet files, reads their footers for schema and statistics, and writes an initial commit whose file list points at the paths already on storage — minutes to hours of metadata work instead of a full rewrite, and no duplicate storage. The costs are what you inherit: existing file sizes, existing partitioning, and any type sloppiness from the Hive era. The alternative is a shadow migration — create-table-as-select into a new table — which buys you target file sizes, a sort order and a fresh partition spec, but costs a full read and write plus double storage during cutover. The decisive constraint is usually neither: it is stopping every legacy writer, because any job still writing directly into the directory produces orphan files the new table cannot see.

code

text · 9 lines
text
Option A -- in place
  existing files stay put ....... s3://lake/orders/dt=2026-03-01/part-*.parquet
  new: table metadata references those exact paths
  cost: footer reads only        inherits: file sizes, partitioning, types

Option B -- shadow rewrite
  new files written .............. s3://lake/orders_v2/data/*.parquet
  cost: full read + write, 2x storage during cutover
  buys: target file size, sort order, new partition spec, corrected types

go deeper

for a junior

Know the headline: a table format references existing data files, so an existing Parquet lake can often be adopted by writing metadata over it rather than copying anything.

for a middle

Explain what conversion actually does — enumerate files, read footers for schema and statistics, translate directory partitions, commit an initial file list — and what it deliberately does not do.

for a senior

Show the operational plan: writer inventory and cutover, permission lockdown on the storage prefix, removal of lifecycle rules, parallel-run validation, and compaction scheduled as follow-up work.

for a principal

Own the strategy call: metadata-only adoption now versus a full rewrite, how the rewrite cost is sequenced against which partitions are actually queried, and how the rollback path shapes the risk you are willing to take on cutover day.

## Why this is even possible The entire migration question turns on the two-layer model. A table format does not own the bytes; it owns a versioned list of file paths plus schema, partitioning and statistics. So adopting an existing Parquet lake can be a matter of *writing the table layer over data that is already there*. Understanding that is the difference between quoting a two-week rewrite and quoting an afternoon. ## Option A — in-place conversion Enumerate the existing files, read each footer to capture schema and per-file statistics, translate Hive directory partitions into partition metadata, and commit an initial version referencing the current paths. Nothing is copied. What you get immediately: atomic commits, snapshot isolation, schema evolution, and planning that prunes from metadata instead of listing directories. What you inherit: - **The existing file layout.** Ten million 4 MB files stay ten million 4 MB files, and now they are also ten million metadata entries, which can make planning slower before maintenance improves it. - **The existing partitioning**, including any over-partitioned scheme that produced those small files. - **Type and nullability drift.** Files written over years by different engines may disagree, and the conversion has to settle on one canonical schema. Timestamps with and without time zone are the classic landmine. - **No physical clustering.** Statistics get recorded, but if the data was never sorted, the bounds will not prune. ## Option B — shadow migration Create the new table with the layout you actually want and load it with a full read-and-write, then cut over the name. You choose target file size, sort or clustering order, a modern partition scheme with coarser grain, and corrected types. Cost: a complete pass over the data, roughly double storage until the old copy is dropped, and a longer window during which two copies must be kept consistent if writes continue. ## Option C — the pragmatic middle, and usually the right answer Convert in place to get transactional guarantees quickly and cheaply, then treat layout as ordinary background maintenance: compact small files to target size, sort or cluster on real filter columns, and evolve partitioning if the format allows it without a rewrite. This spreads the compute cost over weeks, keeps the table queryable throughout, and lets you prioritize the partitions that queries actually touch. Most historical data never gets read; paying to rewrite it up front is often waste. ## The constraint that actually decides it: writers Whichever option you pick, every job that writes into that directory must go through the new table layer on the cutover date. A legacy job that keeps writing Parquet straight into the path produces files the table cannot see, and which the table's own cleanup job will eventually delete. So the migration plan is mostly a writer inventory: 1. Enumerate every producer, including ad-hoc scripts, external partners and backfill jobs that run quarterly and are easy to forget. 2. Convert or repoint each one, then revoke direct write permission on the storage prefix so a missed writer fails loudly instead of silently orphaning data. 3. Remove or repoint any object-lifecycle expiry rules on that prefix — retention now belongs to version expiry, not to object age. Readers are the easier half but still need attention: engines and versions that can read the chosen format, and BI tools that may be pinned to the old catalog entry. ## Validating the cutover Run both tables in parallel for a window and compare: total row counts, per-partition counts, and checksums or aggregates over the columns that matter. Keep the old catalog entry pointing at the old table until the comparison is clean, so rollback is renaming a pointer rather than rebuilding anything. Plan the rollback explicitly — for an in-place conversion, the underlying files are untouched, which makes reverting genuinely cheap and is a strong argument for that path. ## What a strong answer sounds like Name both mechanisms, state that in-place is possible *because the table layer only references files*, then talk about what actually costs time: writer cutover, type reconciliation, validation and rollback. A weak answer assumes migration means rewriting petabytes; a weaker one assumes conversion also fixes the small-files problem, which it explicitly does not.

  • What does an in-place conversion still have to read, if it is not copying data?
    Each data file's footer, to capture the schema and per-column statistics needed for planning, plus the directory structure to translate Hive partition values into partition metadata. That is a small read per file rather than a full scan, but on a lake with millions of tiny files even footer reads take real time — another reason file count, not data volume, drives the estimate.
  • After an in-place conversion, why might queries not get faster right away?
    Because nothing physical changed. The same small files and the same partitioning remain, and now each file is also a metadata entry to evaluate during planning. You gain correctness — atomic commits, isolation, time travel — immediately, and performance only after compaction and sorting run. Promising a speed-up on conversion day is a common way to lose credibility.
  • How do you handle producers you cannot convert on the cutover date?
    Give them a landing zone outside the table and ingest from it with a real commit on a schedule. That keeps them working without letting them write into the table's prefix. Then revoke their write access to that prefix, so any forgotten job fails loudly rather than silently creating orphan files that the cleanup job will later delete.
  • When is a full rewrite the better call despite its cost?
    When the existing layout is the actual problem — pathological small files, a partition scheme that produced thousands of tiny directories, or types you must correct — and when the data volume is small enough that a rewrite is hours, not weeks. It is also better when you want a clean cutover with the old copy retained untouched as the rollback.

saying these in an interview costs you the question

  • Assuming adopting a table format requires rewriting all the data
  • Expecting conversion alone to fix the small-files problem
  • Forgetting to inventory and cut over every legacy writer
  • Leaving object-lifecycle expiry rules on the migrated table's prefix
  • Cutting over with no parallel-run comparison or rollback plan

context

resolving links…