Why do landing zones store dropped files under dated prefixes instead of one flat folder?

## Why layout is a correctness concern, not tidiness Object stores present a flat key space with `/` as a display convention; there are no real directories. What a "folder" gives you is a prefix filter on the LIST API. That single fact drives the whole pattern: whatever you put in the key prefix is the only thing a consumer can cheaply filter on without opening files. A layout like `raw/<source>/<dataset>/ingest_date=2026-08-20/part-0000.parquet` therefore buys four distinct things. ## Bounded listing LIST is paginated — typically a thousand keys per call — and the cost of finding today's files in a flat prefix grows with everything ever written there. A pipeline that has run for two years can spend most of its wall clock enumerating history before it reads a byte, and the degradation is gradual enough that nobody notices until it is severe. A dated prefix caps the listing at one period's objects, permanently. ## A replay unit The deeper benefit. When one day lives entirely under one prefix, "reload 2026-08-14" is a well-defined operation: delete or overwrite the target partition, re-read that prefix, done. The load becomes *replaceable* rather than *additive*, which is the cheapest route to idempotency — rerunning yields the same state instead of doubled rows. In a flat folder there is no such boundary; you would have to know which files belonged to which day, which is exactly the knowledge the prefix was carrying for you. ## Lifecycle and cost Retention rules on the raw zone are usually expressed as "keep 90 days hot, then archive, then expire". Object lifecycle policies apply by prefix and age, so a dated layout makes retention a configuration line. It also makes selective deletion possible: a bad partner drop on one day can be removed without touching neighbours. ## Partition pruning If anything queries the landing zone in place — an external table, a lake table, a Spark job — the prefix components become partition columns, and a predicate on the date lets the engine skip whole prefixes without opening a file. Hive-style `key=value` naming (`ingest_date=2026-08-20`) is what most engines auto-discover; bare `2026/08/20` works too but usually requires you to declare the partition scheme yourself. ## The choice that actually gets argued about: arrival date vs event date Two different dates can name a prefix, and the difference matters more than the format. **Arrival (ingestion) date** — the date the file landed. A prefix, once its window closes, never changes again. That immutability is what makes "reload one prefix" a safe, deterministic operation, and it means a straggler that shows up three days late simply lands in today's prefix and gets picked up by today's run. The cost is that the prefix does not answer business questions: rows for 14 August are scattered across several arrival prefixes. **Event (business) date** — the date inside the records. Convenient for consumers, hostile to replay: a late file for 14 August modifies a prefix you already loaded, so reloading it changes history and any "load yesterday" job silently misses the correction. If you go this way you need an explicit lookback window and a reload policy for it. The common resolution is to partition the *landing* zone by arrival date, because the landing zone's job is to be a faithful, immutable archive of what was received, and to partition the *modelled* tables downstream by event date, where correcting history is a supported operation. ## Granularity Granularity should follow volume, not habit. Hourly prefixes for a dataset that produces one file a day create thousands of empty prefixes and one tiny file each. Daily prefixes for a firehose create prefixes too large to list or reprocess. Rough rule: choose the grain at which one prefix is a comfortable unit of reprocessing and holds files worth reading. If you find yourself with a partition per file, the partitioning has become the small-file generator. Also pin the timezone. `ingest_date=2026-08-20` in whose clock? UTC everywhere is the boring correct answer; a pipeline that switches between local and UTC at a DST boundary produces a duplicate hour and a missing one, and the bug surfaces twice a year. ## Keys inside the prefix File names carry the rest of the identity: dataset, window, a run or sequence id, and a part number. A name like `orders_20260820_run7_part-0000.parquet` lets you recognise re-drops, spot gaps in sequence, and attribute a bad file to a run without opening it. Random UUID names are safe against collisions but tell an operator nothing at three in the morning.