How would you build end-to-end lineage across a platform where several different tools transform data?

## The problem: lineage stops at tool boundaries A typical estate moves data through four or five systems: an ingestion tool lands raw data, an orchestrator schedules the work, a SQL transformation layer builds models, a processing engine handles the heavy or unstructured parts, a BI tool serves it, and a reverse-ETL job pushes results back into operational systems. Each of those tools shows lineage — *within itself*. The graph everyone actually needs crosses all of them, and it is exactly the crossings where impact analysis fails: the analyst asks what breaks if a source field changes, and the answer stops at the boundary of whichever tool they happened to open. So the design goal is not "turn on lineage" but "produce one connected graph from heterogeneous producers, with known coverage". ## Choose one store and one format The first decision is to have a single destination — a metadata service or catalog — and a single event or ingestion format that every producer targets. Without it you get N×M adapters and a graph that exists in fragments. An open interchange format is usually the right default because it lets you swap the store later and because integrations for common tools already exist. If the organisation has already standardised on a commercial catalog, make it the destination and normalise everything into it; the principle, not the vendor, is what matters. ## Three capture mechanisms, and where each fits **Runtime emission.** The executing tool reports, per run, the datasets read and written. This is the strongest source: it reflects what actually happened, includes failed runs, and can carry row counts and timing so the same pipeline feeds observability as well as lineage. Cost: an integration per tool, and the need for tools that expose a hook at all. **Parsing.** Read the SQL — from a repository, or better from the warehouse's own query history — and derive edges by resolving identifiers against catalog schemas. This is the only practical route for ad-hoc queries and for tools with no integration, and it is the main source of **column-level** edges, which runtime observation cannot produce. Cost: dialect-specific parsers, and blind spots at wildcard projections, dynamic SQL and opaque functions. **Declaration.** For hops nothing can observe — a vendor file drop, a spreadsheet, a script someone runs quarterly — accept a small declared manifest. The rule is that declared edges must be visibly marked as unverified, so nobody mistakes a claim for an observation, and they should be a shrinking category rather than a permanent one. Most mature platforms run all three concurrently, and the graph is the union. ## Identity is the hard part The failure that sinks these projects is not capture, it is naming. The ingestion tool calls a table `prod_db.public.orders`; the warehouse calls it `analytics.raw.orders`; the processing engine sees `s3://bucket/orders/`; the BI tool sees a semantic model. Unless they resolve to the same node, you get a graph that looks populated and is silently disconnected — the worst outcome, because it produces confidently wrong impact analysis. So settle, up front: - a **namespace convention** identifying the storage system and environment; - a **canonical name** per physical dataset, plus an aliasing mechanism for the same object seen under a different path (an external table over object storage is the recurring case); - environment separation, so development runs never contaminate production lineage. Write it down, and make new integrations conform before they are enabled. ## Scope by value, and publish coverage Full coverage is neither achievable nor worth buying. Work backwards from the outputs that matter — regulated reports, revenue and billing feeds, the executive dashboards, the machine-learning features in production — and cover their upstream paths completely. Elsewhere, coarse table-level edges are fine. Make **coverage a measured number**: the share of production datasets with a known producing job, the share of critical paths with unbroken lineage to source, the share with column-level edges. A lineage graph whose completeness is unknown gets used once, produces a wrong answer, and is never trusted again. One whose gaps are published is useful immediately, because users know where to stop trusting it. ## Rollout that survives contact with teams - **Start with one high-value path** end to end, across every tool it touches, rather than enabling everything shallowly. It proves the naming convention and produces a demo people believe. - **Make emission a platform default**, not a per-team project — wire it into the shared orchestrator image, the shared job template, the warehouse's query-history ingestion — so coverage grows without asking for adoption. - **Attach it to a workflow people already have.** Lineage that appears in the schema-change review, or in the incident channel as "this failure reaches these dashboards and these owners", earns its keep. A standalone graph explorer does not. - **Bind ownership to it.** Impact analysis whose output is a list of datasets nobody owns cannot be acted on; the graph's value multiplies when every node resolves to a team. ## Where to accept gaps Be explicit about what you will not cover: transformation logic inside opaque user code, joins that happen inside the BI tool's own calculated fields, exports to systems outside the platform's control, and anything downstream of a manual spreadsheet step. For those, the honest position is a boundary marker on the graph — lineage ends here — plus, where it matters, an organisational control such as a required change notice. Pretending coverage you do not have is worse than admitting the edge of the map.