In Apache Hudi, what is a file group and how does a file slice relate to it?

## Why the grouping exists at all A plain directory of Parquet files has no notion of *where a given row lives*. Apache Hudi needs that notion, because its core operation is an upsert by record key: given an incoming key, the writer must find the existing copy in order to merge or replace it. The structure that provides it is the **file group**. A file group is a set of records inside one partition, identified by a **file ID** — a stable identifier baked into the names of every file belonging to the group. Once a record key is assigned to a file group, it stays there for the life of the table (barring operations that explicitly replace file groups, such as clustering). Hudi's index is essentially a map from record key to file group, and the file ID is the value it returns. ## The slice is the version A file group changes over time, and each version of it is a **file slice**. A slice is identified by the instant time that created its base file. What a slice physically contains depends on the table type: - **Copy-on-Write.** A slice is exactly one Parquet **base file**. Every write that touches the group merges the group's contents and emits a new base file, so a new instant time means a new slice. - **Merge-on-Read.** A slice is a base file *plus* any **log files** appended after it. Ingestion appends log blocks rather than rewriting, so the slice grows in place until compaction reads base plus logs and emits a new slice with a fresh base file and no logs. The naming makes this readable straight off a directory listing: base files carry the file ID and the instant that wrote them, and log files carry the same file ID together with the base instant of the slice they belong to. Group the listing by file ID and you can see every version of every group. ## What the structure buys you Four things in Hudi are defined in terms of file groups and slices. **Upserts.** Because the index resolves a key to a file ID, an update touches exactly one file group, not the whole partition. This is what makes record-level updates on immutable files tractable at all. **Compaction.** In Merge-on-Read, compaction operates per file slice: read this slice's base file and its logs, write a new base file. That makes compaction incremental and parallelizable — slices with the most log data can be prioritized without rewriting the table. **Versioning and cleaning.** Older slices of a group are not deleted at commit time. They remain so that time-travel and incremental readers, and any query already in flight, still have consistent files to read. Hudi's cleaner removes superseded slices later according to the configured retention. The relationship is direct: retention policy is expressed in slices per group and instants of history, and cleaning too aggressively is what breaks time travel. **File sizing.** The small-file problem is managed at group granularity. The writer bin-packs new inserts into existing base files still under `hoodie.parquet.small.file.limit`, opening new file groups only when existing ones approach `hoodie.parquet.max.file.size`. That is why a well-configured Hudi table does not spray a new tiny file per commit the way a naive append-only writer does. ## Common confusions A file group is **not** a partition. A partition typically contains many file groups; the partition path is a directory, the file group is a logical set of records inside it identified by a file ID. A file slice is **not** a snapshot of the table. Slices are per file group. A table-level read assembles the current slice of every relevant file group; different groups can have been last written at different instants, and that is normal. A log file does **not** belong to a partition in general — it belongs to a specific file group and a specific base instant. That association is what lets a reader know which base file to merge it with, and it is why a stray log file with no matching base instant is an orphan rather than data. ## Reading a listing Given a partition directory containing files for two file IDs, one with a single Parquet file and one with a Parquet file plus two logs, you can state immediately: this is a Merge-on-Read table; the first group has been compacted or has received no updates since its base file was written; the second has unmerged updates that a snapshot query is paying to merge and a read-optimized query is ignoring. That inference — from listing to behaviour — is the reason the vocabulary is worth knowing.