How does an AWS Glue crawler decide which parts of an S3 key prefix become partition columns?

## The rule A crawler's S3 target is the root of a table. Everything below it is structure. The crawler descends, and for each folder level it asks two questions: does every branch at this level look the same, and is the level named in Hive style? **Hive-style** means a folder whose name is `key=value`. That convention comes straight from the Hive metastore model the Data Catalog implements, and it is the only naming the tooling can interpret. A path of ``` s3://lake/events/dt=2026-08-20/region=eu/part-0000.parquet ``` yields a table `events` with partition keys `dt` and `region`, and a partition record whose values are `2026-08-20` and `eu`, pointing at that folder. Without the `key=` prefix, as in ``` s3://lake/logs/2026/08/20/part-0000.parquet ``` the crawler still sees three consistent levels but has nothing to name them after, so it produces partition columns `partition_0`, `partition_1` and `partition_2` in depth order. The table works, but every downstream query reads `WHERE partition_0 = '2026' AND partition_1 = '08'`, which is opaque, easy to get wrong, and impossible to change later without rewriting the layout or hand-editing the table. Partition column values arrive as strings; if you want dates or integers you either name the columns yourself in DDL or handle the cast in queries. ## Why one prefix sometimes becomes many tables The crawler's second job is **table grouping**: deciding whether the folders it found are partitions of one table or independent tables. It groups them into one table when their schemas are compatible and the structure is uniform. It splits them when they are not. Common causes of an unwanted split: - **Mixed formats** under one prefix — some folders JSON, some Parquet. Those cannot share a SerDe, so they cannot share a table. - **Incompatible schemas** — one folder's `amount` is an integer and another's is a quoted string, or column sets diverge sharply. - **Ragged depth** — files directly under the root alongside files two levels down. The crawler cannot decide what level the table starts at. - **Writer debris** — `_temporary/`, `_SUCCESS`, `.spark-staging` folders and zero-byte folder-marker objects can be picked up as if they were partitions. The crawler configuration option **"Create a single schema for each S3 path"** (in the API, the grouping policy `CombineCompatibleSchemas`) tells the crawler to prefer one table per path and merge schemas that differ only slightly, which fixes most accidental splits. Exclude patterns fix the debris. Ragged depth is a layout problem and has to be fixed by the writer. ## Two more knobs that bite **Table level.** For nested layouts you can tell the crawler how many folder levels down the *table root* lives, which prevents it from mistaking a top-level grouping folder for a partition key — useful when one bucket holds many datasets. **Partition metadata inheritance.** Each partition record carries its own storage descriptor. If the table's SerDe or column types are updated but partitions keep old ones, engines raise a schema-mismatch error at read time. The crawler option "Update all new and existing partitions with metadata from the table" (partition `AddOrUpdateBehavior` = `InheritFromTable`) makes partitions follow the table definition and heads that off. ## What good looks like Write Hive-style prefixes from the start: `dt=YYYY-MM-DD`, and add a second key only when queries genuinely filter on it. Keep the cardinality sane — a partition per minute produces millions of catalog records and slow query planning, while a partition per day with reasonably sized files is easy on both the catalog and the scan. Keep one dataset per prefix and one format per dataset. Do those three things and partition inference is boring, which is exactly what you want from it. If you inherit a non-Hive layout you cannot rewrite, you are not stuck with `partition_0`: define the table yourself with `CREATE EXTERNAL TABLE` naming the partition columns, register partitions with explicit `ADD PARTITION … LOCATION` statements, and stop crawling that prefix.