A shared dataset's files sit in directories named for a high-cardinality column — how do you judge that granularity for every job that reads it?

## What a reader inherits In a **finite job** — a run over an input that ends, so the stored bytes can be measured before it starts — the division is derived from what is on disk. The **piece count** (how many slices the run is divided into, each worked by one **worker thread** end to end) comes from the surviving files, and which files survive comes from matching the filter against **column-value directories**: files grouped into directories named for one column's value. So a layout decision made once, by a producer, sets the first shape of every job that later reads the dataset. Nobody downstream gets a vote at run time. ## The two directions it fails in | granularity | directories | typical file size | elimination | what a reader inherits | |---|---|---|---|---| | coarse, e.g. one per month | tens | large | weak | few pieces, each large; most jobs read far more than they need | | moderate, e.g. one per day | hundreds to thousands | moderate | good for date filters | a piece count in a sane range for most jobs | | fine, e.g. one per customer or per minute | hundreds of thousands | tiny | excellent for that one filter | one piece per tiny file, expensive listing, planning dominating the run | The fine end is the trap, because it is chosen by someone measuring the query it was chosen for. That query improves enormously. Every job that filters on something else now lists hundreds of thousands of objects, derives a piece per file and spends more on bookkeeping than on reading — and it gets nothing back, because elimination only serves filters on the directory column. ## How to judge it 1. **Measure the reading population, not the flagship query.** Collect which columns appear as filters across the jobs that actually read the dataset, weighted by what they scan. A column filtered by most of the scanned bytes is a candidate; a column filtered by one important job is not, on its own. 2. **Turn each candidate into two numbers**: how many directories it produces, and the median file size inside one. Directories should stay few enough that listing is cheap, and files large enough that the work of handing out a piece is small next to reading it. Those two pull in opposite directions and the granularity is where they balance. 3. **Check the distribution's skew.** A directory per customer where one customer is forty percent of the rows gives you both extremes at once: a huge population of near-empty directories and one directory no elimination will ever make small. Uneven pieces, and what to do about them, is its own subject; here it is a reason to reject the column. 4. **Price the change.** A shared layout is a contract with consumers you cannot see. Changing it rewrites the dataset and breaks anything that addressed paths directly, so the honest comparison is against the bytes scanned and the planning time the current layout costs per day, summed over the population. ## What you cannot assume about the readers - Some readers can rewrite a filter that wraps the directory column in a function back into a directory match, and some cannot. If the dataset serves engines you do not control, assume the weaker behaviour and choose a column people will filter on literally. - Some readers can pack several small files into one piece of the input and some give one piece per file unconditionally, so the same fine-grained layout is survivable on one reader and ruinous on another. - A **continuous job** does not derive a piece count from the files at all: the author sets a **declared operator width** that stands until the job is restarted. Such a reader still pays the per-file open cost of a fragmented layout, but the planning cliff shows up differently. ## Adjacent decisions this is not Declaring and evolving a table's partition specification in a table format, and rewriting a fragmented file population into fewer larger files, are both real and both belong to the table-format and table-maintenance subjects rather than to what a reader inherits. The shape a producing job leaves behind — how many output files its own piece count creates — is the writing side of this same handover. Skipping blocks inside an already-opened file using statistics kept for them is the columnar-storage subject and does not change the piece count at all. ## The organisational part Publish one layout standard per shared dataset, with the directory column, the target file size and the expected directory count written down, and measure compliance by the two numbers above rather than by intention. Review it when the query mix moves, not on a schedule, and keep a record of what the current granularity costs the jobs it does not serve. That number is what justifies the next migration, and it is the number nobody has when the argument starts.