Data Governance, Lineage & Catalogs

Technical metadata describes structure (schemas, types, location); business metadata describes meaning (descriptions, glossary terms, owners, classification); operational metadata describes behaviour (freshness, run history, usage, quality results). Systems supply the first and third, people the second.

Domain ownership moves analytical data to the teams that know it; data as a product makes it usable by others; a self-serve platform makes that affordable for every domain; federated computational governance keeps it interoperable through shared rules the platform enforces.

A direct identifier names a person by itself (email, national ID). Quasi-identifiers identify people only in combination (birth date, postcode, sex). Sensitive attributes are what must not leak (diagnosis, salary). Each class needs a different protection.

Lake data sits in immutable files copied through raw, cleaned and curated layers, kept in old snapshots, backups and extracts. Deleting a person means rewriting files everywhere they appear, expiring old versions, and stopping replays from reloading them.

Start at the dashboard and walk the lineage upstream one hop at a time, comparing each dataset's figure with a trusted reference. The first hop where the value diverges is where the error entered; then check what changed there recently.

Use a keyed hash when analysts only need a stable join key and nobody needs the email back. Tokenize when an authorised process must recover it. Encrypt when the value itself must be recoverable by key holders. None of the three makes it anonymous.

Lineage shows which sources and transformation steps feed the number, and in what order. It does not say whether the data is correct, what the metric is supposed to mean, or whether the logic matches the business definition.

The data owner is accountable for a dataset and decides its use and access; the data steward maintains its definitions, quality and metadata day to day; the data custodian operates the systems and technical controls that store and protect it.

An access review asks each data owner to confirm that every holder of access to their data still needs it. Read logs show which grants were never used, so owners revoke on evidence instead of approving everything by default.

A glossary defines business concepts in plain language, independent of any system; a data dictionary describes the columns of specific tables. Linking terms to columns shows where each concept lives and exposes columns that claim a term but compute it differently.

Pull ingestion has the catalog crawl sources on a schedule, so it is stale between crawls and misses short-lived objects. Push ingestion has systems emit metadata when things change, so it is fresh but depends on every producer emitting correctly.

Keep consent as a governed dataset of each person's current choice per purpose, tag datasets and jobs with the purpose they serve, filter by consent wherever a purpose-tagged use reads data, rebuild derived segments, and stop exports to downstream tools.

Give every dataset a retention class tied to a named date column, partition by that date, run automated jobs that drop expired partitions and then physically purge old versions, and log evidence. Legal holds and exceptions are explicit, recorded overrides.

A static graph shows today's dependencies, not what ran then. You need run-level records naming which job version ran, which input dataset versions it read, and when, plus retained copies or snapshots of those inputs.

Domain and platform owners jointly agree a small set of global rules, and the platform enforces them automatically. Global rules cover interoperability, such as how a shared entity is identified, security and discoverability; each domain decides its own data model.

A data product has an accountable owner and is built for consumers - discoverable, addressable, self-describing, trustworthy with stated objectives, interoperable through global standards and secured by global access rules. A published table has none of those guarantees.

Scanners sample column values and match them with patterns plus checksums, reference dictionaries and trained entity recognisers, then propose tags with a confidence score. Names alone miss badly named columns, free text and JSON that hide personal data.

Table rules run in the query engine, but the files sit in object storage with separate permissions, so a direct storage reader skips filters and masks. Deny users direct storage access and give only governed engines scoped, short-lived credentials.

With no tag, no tag-based policy applies, so the emails are readable by anyone who can read the table. Close the gap by scanning new columns, inheriting tags along lineage, and defaulting unclassified columns in sensitive tables to masked.

Reassign ownership, automate every fact a system can supply, focus human effort on the most-used datasets, surface trust signals such as certification and freshness, deprecate dead assets, and embed the catalog in daily tools so it gets used and corrected.

Verify the request, resolve the person to every identifier, locate their data through classification and lineage, delete or anonymise per store while honouring retention duties, notify processors, purge old versions, suppress re-ingestion, and log evidence.

Lineage sees only tools that emit or expose it, so ad hoc queries, notebooks, spreadsheet exports, dynamic SQL and external copies go missing. Warehouse query and access logs, plus owners, reveal readers the graph cannot see.

Start at the classified source column and follow its column-level lineage downstream to every derived column, noting which steps copy it and which mask or aggregate it. Tracing breaks at free-text fields, opaque code, exports and other unobserved copies.

Source-aligned products publish a domain's business facts close to how they happen; consumer-aligned products reshape and combine them for a group of use cases. A cross-domain dataset should be owned by a team whose purpose is serving that use, not by one source domain.

k-anonymity guarantees each record shares its quasi-identifier values with at least k-1 others, achieved by generalising and suppressing values. It still leaks when a group shares one sensitive value or an attacker has background knowledge.

The data owner should be accountable for access to sensitive data, but not click every request. Tier data by sensitivity, auto-approve low tiers by policy, require owner review with a stated purpose for high tiers, make grants expire, and track lead time.

Weigh the real value of full rebuilds against the risk, erasure cost and legal limits of keeping identifiable raw data. Usually keep identifiable raw data for a bounded rebuild window, then keep pseudonymised or aggregated forms, with legal minimums as explicit exceptions.

Adopt it only if the central team is a real bottleneck, the business has several distinct domains, domains can staff data ownership, and a self-serve platform exists or can be funded. Otherwise improve the central model, adding domain ownership step by step.

Crypto-shredding encrypts each person's identifying fields with a per-person key; erasure destroys the key, making every copy unreadable, including backups and old snapshots. It wins where rewriting copies is impractical, but costs key management at scale and query-time decryption.