In an Apache Hudi write, what do the record key and precombine field control?

## The problem these two configs solve A directory of Parquet files has no notion of row identity — you can append files to it, but you cannot say "row 42 changed". Hudi's upsert path exists to give a lake table that ability, and it rests on two write configurations you set the first time you write the table: which columns identify a row, and which column decides which version of that row is newer. ## The record key `hoodie.datasource.write.recordkey.field` names one column, or several comma-separated columns, that uniquely identify a row: a source primary key, a CDC key, a UUID. Hudi materialises the value into a hidden metadata column, `_hoodie_record_key`, which it adds to every base file alongside `_hoodie_commit_time`, `_hoodie_commit_seqno`, `_hoodie_partition_path` and `_hoodie_file_name`. A composite key is stored as a list of `field:value` pairs so it round-trips exactly. The record key is what the index is keyed by. When an `upsert` batch arrives, Hudi *tags* each incoming record: it asks the index which file group already contains that key. Tagged records are updates and are routed to that file group — merged into a new version of the base file on a Copy-on-Write table, appended to a log file on a Merge-on-Read table. Untagged records are inserts and are placed into a file group with room. With no record key there is nothing to tag against, and every write can only append. Two practical rules follow. The key value may not be null — Hudi fails the write rather than guessing. And the key definition is effectively immutable for the life of the table: change which columns make up the key and every stored row now has a different identity, so no future update can ever match it. That is a full rewrite, not a config change. ## The partition path `hoodie.datasource.write.partitionpath.field` decides the directory a record lands in. Uniqueness is enforced per (partition path, record key) unless you configure a global index, which is why a record whose partition value changes can end up existing twice. ## The precombine field `hoodie.datasource.write.precombine.field` names the column that orders two versions of the same key. It is consulted in two places: 1. **Within the incoming batch.** A CDC batch commonly carries three changes to the same row. Hudi collapses them before writing, keeping the record with the largest precombine value, so one commit never writes the same key twice. 2. **Against the record already stored.** Here the behaviour depends on merge semantics. With *event-time ordering* Hudi compares the incoming precombine value against the stored one and keeps the larger, so replaying an old file or a lagging stream partition cannot resurrect a stale version. With *commit-time ordering* the newest commit wins outright regardless of the value. Hudi 1.x exposes this choice as an explicit record merge mode; earlier versions expressed it through the record payload class you configured. Choose a monotonically non-decreasing value produced by the source: the database transaction timestamp, a binlog sequence number, a Kafka offset made comparable across partitions. Ingestion wall-clock time is a poor choice, because a backfill that re-reads history will stamp old rows with a *newer* precombine value and overwrite good data. ## Deletes The same key drives deletion. A hard delete is a write with `hoodie.datasource.write.operation=delete`, which removes the row entirely; a soft delete sends the record with the `_hoodie_is_deleted` marker column set so the row stays present with its non-key fields nulled. Both need the record key to find the file group holding the row. ## Common failure modes - **A key that is not actually unique.** Hudi will happily collapse two genuinely different business rows into one because they share a key, and the loss is silent. - **No precombine field, or a constant one.** Duplicates in a batch then resolve arbitrarily, and re-running the same batch can flip the winner, which makes the pipeline non-deterministic. - **Assuming precombine also orders rows on disk.** It does not — it is a comparison used during merge, not a sort order. Clustering and the bulk-insert sort modes control physical ordering. - **Assuming the newest write always wins.** Under event-time ordering it does not: an incoming record with a smaller precombine value is discarded even though its commit is newer, which is exactly the protection you asked for and exactly the surprise when a backfill "does nothing". In interviews these two configs are the entry point to everything else on the Hudi timeline-and-index topic: the key is what the index maps, and the precombine value is what makes replaying an upstream stream safe.