How does column-level lineage differ from table-level lineage for impact analysis in a data platform?

## The same graph at two resolutions Lineage is a derivation graph. **Table-level (or dataset-level) lineage** records edges between whole datasets: this job read `raw.orders` and `raw.customers` and wrote `mart.orders_daily`. **Column-level lineage** records edges between fields: `mart.orders_daily.net_total` was derived from `raw.orders.gross_amount` and `raw.orders.discount`, while `mart.orders_daily.region` came from `raw.customers.region`. Everything table-level lineage supports, column-level supports too — you can always collapse field edges up to their datasets. The reverse is not true, and that asymmetry is the whole subject. ## Where table-level is enough Operational questions are usually dataset-shaped: - **Staleness propagation.** The load into `raw.orders` failed. Everything transitively downstream is stale, regardless of columns. - **Blast radius during an incident.** Which dashboards and feeds sit below the failure, and who owns them. - **Deprecation of a whole table.** Nothing reads it, so it can be deleted. - **Cost attribution and orchestration checks.** Which pipelines feed the ten most expensive marts. For all of these, field-level detail adds nothing and costs effort to maintain. ## Where table-level fails and column-level pays The question column-level lineage exists for is **schema change impact**: a source column is being renamed, retyped, dropped, or its semantics are changing. Ask that of a table-level graph and it over-reports catastrophically. A staging table with three hundred downstream models produces three hundred "affected" answers, so the team either ignores the analysis or spends days manually reading SQL to find the real consumers. The signal-to-noise ratio is so bad it changes behaviour: people stop asking, and ship the change blind. Column-level lineage turns the same question into a precise list. It also enables things table-level cannot express at all: - **Sensitive-data tracking.** Where did `customer.email` end up? A PII tag on a source column can be **propagated** down the column graph, so a mart that derives from it inherits the tag automatically instead of relying on someone to remember. - **Explaining a single metric.** A user asks how `net_revenue` is computed; the column path plus the expressions on each hop is the answer. - **Precise regression scoping in review.** A pull request changes one expression; the reviewer sees exactly which downstream fields shift. - **Column-level cleanup.** A wide table where two thirds of columns have no downstream reader is a real and common finding. ## Direct versus indirect dependencies Good column-level lineage distinguishes two edge kinds, and candidates who have used it know the difference: - **Direct**: the input column's value flows into the output column, via projection or an expression. - **Indirect**: the input column influenced *which rows* exist without contributing a value — it appeared in a `WHERE` predicate, a `JOIN` condition, a `GROUP BY`, or a window's `PARTITION BY`. ```sql select o.id, o.gross_amount - o.discount as net_total, -- direct: gross_amount, discount c.region -- direct: c.region from orders o join customers c on c.id = o.customer_id -- indirect: customer_id, c.id where o.status = 'PAID'; -- indirect: o.status ``` Dropping `status` breaks this model even though no output column carries it. Lineage that records only direct edges would report it as safe — a false negative, which is a far worse failure mode than the false positives of table-level lineage. ## Where column-level lineage comes from, and where it breaks Field edges cannot be observed from the outside; a job that reads a table and writes a table looks identical at runtime whichever columns it touched. So column-level lineage is **derived by analysing the query or code**: parse the SQL into a syntax tree, resolve identifiers against real catalog schemas, and walk the projection through subqueries and CTEs. That pipeline has known blind spots: - **`select *`** — resolvable only if the parser knows the upstream schema at that moment, and stale if the upstream gains a column later. - **Dynamic or generated SQL** — the text does not exist until runtime, so static analysis sees nothing. - **Opaque code paths** — a user-defined function, a Python transform, a stored procedure. Parsers typically degrade to "all inputs may affect all outputs" or drop the edge entirely. - **The reporting layer** — a BI tool's own calculated fields form a second graph that the warehouse parser never sees, so lineage frequently stops at the last table. Because of these, treat column coverage as partial by construction. The right posture is to know which parts of the estate have trustworthy field-level edges and which do not, rather than presenting a graph that silently omits a dependency. ## Choosing Start table-level: it is cheap, obtainable from runtime events, and covers the incident-shaped questions that dominate day to day. Add column-level where change is frequent and expensive — core source tables, published marts, anything carrying regulated fields — and accept coarse edges elsewhere. Being explicit about the resolution you have is more useful than pretending to uniform coverage.