An Apache Hudi upsert job's index tagging slows as the table grows — what would you change?

## What tagging actually costs On an `upsert`, Hudi must answer one question per incoming record: which file group already holds this record key? That step is called tagging, and in a slow Hudi upsert job it is usually where the time goes. Understanding the fix means understanding how each index answers the question. ## The bloom index and why it degrades With `hoodie.index.type=BLOOM`, Hudi stores in each base file's footer a bloom filter over the record keys in that file, plus the minimum and maximum record key it contains. Tagging then proceeds in stages: 1. **Range pruning.** For each incoming key, discard files whose min/max key range cannot contain it. 2. **Bloom check.** For surviving candidates, test the key against the file's bloom filter. A negative is definitive; a positive may be false. 3. **Confirmation.** Actually read the candidate file's keys to confirm, because a bloom filter has false positives. Stage 1 is the load-bearing one, and it only works if keys are correlated with files. If your record key is a UUID or a hash, every file's range spans essentially the whole key space, nothing prunes, and every incoming key fans out to bloom checks and confirming reads across a growing number of files. The job gets slower every week for the same input volume — the classic symptom in the question. A simple index (`SIMPLE` / `GLOBAL_SIMPLE`) makes the opposite tradeoff: it joins the incoming keys against keys read from the relevant existing files, with no bloom structures. It can beat bloom when updates are spread randomly across many files, because it avoids the false-positive confirmations, but it still reads data proportional to the candidate set. ## The record-level index Hudi 0.14 added a record index stored as a partition of the table's metadata table under `.hoodie`. It is an explicit mapping from record key to the partition and file group that holds it, so tagging becomes a lookup in a compact, key-partitioned structure rather than a probe against data files. Cost per key stops scaling with table width, which is precisely the property the failing job lacks. It is enabled with `hoodie.index.type=RECORD_INDEX` alongside the metadata-table record-index setting, and it is global by nature — the mapping spans partitions. The trade: the index is state that must be built and maintained. Enabling it on a large existing table means an initialisation pass, and every write updates the mapping, so writes carry extra work in exchange for a much cheaper lookup. It is the right default for large mutable tables with random keys. ## The bucket index `hoodie.index.type=BUCKET` removes the lookup entirely. The record key is hashed into one of a fixed number of buckets per partition (`hoodie.bucket.index.num.buckets`), and the bucket determines the file group. Tagging becomes arithmetic. This is why bucket indexing is popular for high-throughput streaming ingest, especially with Flink. The trade is rigidity: the bucket count is chosen up front and fixes the parallelism and file sizing of each partition. Choose too few and each bucket's file group grows unboundedly; too many and you manufacture small files. A consistent-hashing variant exists to allow bucket counts to change, and is the answer to "what if my volume grows tenfold?". ## Cheaper things to check first Before changing the index, rule out the avoidable causes: - **Is the index global when it need not be?** A global index searches every partition for each key. If your updates always land in the same partition as the original record, a partition-scoped index cuts the search space enormously. - **Are updates spread across all history?** If the batch touches only recent partitions but the job searches all of them, constrain the write so tagging looks where the data is. - **Was the table bulk-loaded without sorting?** Global-sorted loading gives each file a narrow key range and can revive bloom pruning without any index change. - **Is the key itself hostile?** A key with a leading sortable component — a tenant id, a date, a monotonic id — makes ranges meaningful; a bare UUID never will. ## How to present the answer Diagnose first: report tagging time as a share of job time, and how many candidate files a typical key resolves to. Then state the choice as a tradeoff. Record index buys O(1)-ish lookups for maintenance cost and metadata-table dependency. Bucket index buys zero lookups for a rigid layout. Bloom stays viable only if you can make key ranges mean something. A candidate who answers "switch to the record index" without saying what it costs, or who blames the write path rather than the index, is guessing. Be careful not to import machinery from a neighbouring format here: this problem is Hudi's, and the answer is a Hudi index type, not a clustering command from another table format.