A Hudi Merge-on-Read table's snapshot queries slow down every hour — what would you check?

## The mechanism behind the symptom A Merge-on-Read table stores each file group as a **file slice**: a Parquet base file plus log files appended since it was written. Ingestion is cheap precisely because it never rewrites the base file — it appends log blocks and lands a `deltacommit` instant on the `.hoodie` timeline. That deferred work does not disappear. A **snapshot** query must open the base file and replay every log block on top, merging by record key, before it returns a row. Read cost therefore grows monotonically with the amount of unmerged log data on each slice. Compaction is the only thing that resets it: it reads a slice's base file plus its logs and writes a new base file with no logs. It is scheduled as a `compaction` instant (requested, then inflight) and lands as a completed `commit`. If compaction stops or falls behind ingestion, queries degrade steadily and predictably — hour by hour, exactly as described. ## Diagnose from the timeline first List `.hoodie` and answer three questions before touching any config: 1. **When did compaction last complete?** Find the newest compaction instant that reached completion. If it is hours or days old on a table ingesting every few minutes, you have your answer. 2. **How many `deltacommit` instants have landed since?** That number, times the average per-commit log volume, is the backlog a snapshot query is now merging. 3. **Is there a stuck instant?** A `compaction.requested` with no matching completion, or an instant sitting inflight, means compaction was scheduled and then failed, was killed, or is blocked. Look for `rollback` instants around it — repeated rollbacks mean compaction jobs keep dying, often on memory. A directory listing confirms it: pick a hot partition, group files by file ID, and count log files per base file. A slice with a dozen logs is a slice whose reads are a dozen merges deep. ## Then check how compaction is supposed to run Compaction runs one of two ways, and the failure modes differ. **Inline.** The writer performs compaction itself as part of a write, triggered after a configured number of delta commits (`hoodie.compact.inline` enables it; `hoodie.compact.inline.max.delta.commits` sets the trigger). It is simple and self-healing, but it makes some commits dramatically slower than others, and if ingestion has a tight latency budget the operator often disables it and then forgets to run anything else. That is the single most common root cause. **Asynchronous.** Compaction runs as a separate process — a service inside a long-running streaming writer, or a scheduled batch job pointed at the table. Ingestion latency stays flat, which is why streaming deployments prefer it, but now compaction is an independent job that can be down, under-resourced, or silently failing without anyone noticing, because ingestion keeps succeeding. Check that the job is actually running, that its resources scale with ingest volume, and that its failures alert someone. Either way, when several processes write to the same table you need Hudi's concurrency control configured correctly. A mismatch there can cause compaction to keep aborting on conflicts, which shows up as those repeated rollback instants. ## Other contributors worth ruling out - **Compaction is running but cannot keep up.** Completed compactions exist, yet the backlog still grows. Then it is a throughput problem: raise compaction parallelism and resources, compact more often, or reduce the write volume per commit. - **Too many small file groups.** If ingestion opens new groups constantly instead of packing into existing base files, compaction has far more slices to process. Revisit the file-sizing settings. - **Skewed file groups.** A few hot keys concentrate updates into a handful of groups, so those slices grow enormous logs while the rest are fine. Look at log volume per file ID, not just the total. - **The cleaner.** Cleaning is a separate action from compaction; a stalled cleaner inflates storage and listing cost but is not what makes the merge deeper. Do not confuse the two. ## Fixing it Short term, restore query latency by routing heavy readers to the read-optimized view, which reads base files only — accepting that its results are as of the last compaction and that recently deleted rows will still appear. Say that out loud to whoever consumes the data; silently serving stale results is worse than a slow query. Medium term, drain the backlog with a dedicated compaction run sized for it, then re-establish a steady schedule. Compact often enough that log depth per slice stays bounded, and make that bound your published freshness guarantee for the read-optimized view. Long term, monitor it as a service-level indicator, not as a folklore. The two metrics that matter are the age of the newest completed compaction and the number of delta commits (or bytes of log) outstanding per file group. Alert on both. And ask the design question: if this table's read pattern cannot tolerate any read-time merge, it may simply belong in a Copy-on-Write table, paying the cost on the write side instead.