Stored files are grouped into directories named for one column's value — what does a filter on that column change before reading?

## Two different things are called a partition This subject collides two senses of one word, and an answer that does not separate them is unreadable. - **A piece of the input** is one slice of the stored input that a single **worker thread** reads and processes from start to finish. How many exist — the **piece count** — is the ceiling on how many lanes can be busy. - **A column-value directory** is stored files grouped into a directory named for one column's value, for example a directory per calendar day. It is an arrangement on disk, made when the data was written. The two meet in exactly one place, which is this leaf's subject: eliminating directories changes how many pieces of the input exist, because in a **finite job** — a run over an input that ends, so the stored bytes can be measured first — the pieces are derived from the files that survived planning. ## What happens while the plan is built 1. The reader **lists the directory tree** and recovers each directory's column value from its name. 2. It **matches the filter** against those values. A directory that cannot contain a matching row is dropped from the plan entirely. 3. The files under the surviving directories are **measured and turned into pieces** of the input, subject to whether each file is divisible at all. 4. Those pieces are **handed out** to worker threads. Every one of those steps happens before a data file is opened. That is what makes the mechanism worth so much: it removes work from the plan rather than making the work faster. ## What it changes, and what it does not | mechanism | when it applies | what it reduces | |---|---|---| | eliminating whole column-value directories | while the plan is built, before any file is opened | the piece count and the bytes read | | skipping blocks inside an opened file using statistics kept for them | after the file is already in the plan | bytes read only; the piece count is unchanged | | discarding rows as they are decoded | after decode | neither; the reading and the scheduling are already paid for | The distinction matters in an interview because candidates use the word "pruning" for the first two indiscriminately, and only the first one changes how many pieces the job is divided into. The second belongs to the columnar-storage subject and is a real, separate win. ## When it silently does not happen - **The filter is on a different column.** Nothing is eliminated; every file in the dataset enters the plan. A layout only ever serves the queries that filter on its directory column. - **The column is wrapped in a function or a cast.** The filter is then a computed value rather than one that can be compared to a directory name. Some readers rewrite such an expression into a directory match and some do not, so assume the weaker behaviour and filter on the stored value. - **The value is only known at run time**, for instance because it comes from another input. Some engines can eliminate directories once that value materialises, others fix the file list at planning time and cannot. - **Types disagree.** A directory name is text; comparing it against a value of another type may or may not be handled, and where it is not, nothing is dropped. - **The dataset is addressed as a bare path** rather than through something that knows the layout, in which case the directory names may simply be opaque path segments. Because all five failures are silent — the query returns the right answer, only slowly — the habit worth teaching is to **verify by measurement**: read off the number of files or pieces the plan reports, or the bytes scanned, with and without the filter. The difference is the proof; the wording of the query is not. ## The cost side of the same choice Elimination is not free to arrange. Directories named for a column with many distinct values eliminate beautifully for a filter on that column and leave a large population of small files behind, and a job that filters on something else then inherits one piece per file with all the scheduling overhead that implies. Deciding that granularity is a judgment call about the whole reading population rather than about one query. Repairing a file population that is already too fragmented — rewriting many small files into fewer larger ones — is a table-maintenance subject in its own right and not part of what a reader does at plan time. ## What varies between engines - Whether a function over the directory column can be inverted into a directory match varies, as does whether a value discovered at run time can eliminate directories after planning has begun. - A **continuous job** — one over an input with no end, where nothing can be measured in advance — has a **declared operator width** the author states rather than a piece count derived from files, so directory elimination for it is about which files are read, not about how many lanes run. - Where the listing itself happens differs: some engines list from the single **coordinating process** that turns the program into a graph and hands out pieces, others distribute the listing or accept a prepared file list.