What are the signs that a medallion lakehouse has degraded into a data swamp?

## What "swamp" means here A data swamp is a store that holds a lot of data and yields few trustworthy answers. Medallion layering is a defence against it, but the layering is only a naming convention — apply the names without the guarantees and you get a swamp with three schemas instead of one. The signs are recognisable long before anyone says the word. ## Sign 1: bronze nobody can interpret Raw landing is supposed to be uninterpreted, but it must still be **catalogued**. The swamp version has hundreds of tables with no record of which source produced them, which pipeline still writes them, what the columns meant in the source system, or whether anything downstream depends on them. Nobody can delete anything because nobody can prove it is unused, and nobody can use anything because nobody can prove what it is. The fix is metadata, not cleaning: source, owner, load schedule, downstream dependents. ## Sign 2: layers with no declared grain A silver table that cannot answer "what is one row?" is a copy, not a layer. Its duplicates surface as inflated sums in reports weeks later, and each consumer patches around them differently — one uses `DISTINCT`, another a window function with a different tie-break, a third does not notice. When you find several downstream queries deduplicating the same table in different ways, the layer boundary has already failed. ## Sign 3: copy sprawl The signature shape: ``` silver_events_clean silver_events_clean_v2 (fix applied here; v1 still queried) silver_events_enriched_fin (finance's copy, one extra filter) gold_kpi_daily (built on which one? both, in places) ``` This grows when a team needs a change, cannot get it made in the shared model quickly, and forks. Each fork is individually reasonable and collectively fatal: the same metric now has three lineages and the meeting spends its first ten minutes reconciling numbers. The organisational cause — no ownership, no responsive change process for shared models — matters more than the technical one. ## Sign 4: business logic in bronze When someone filters out test accounts, excludes cancelled orders, or joins in a lookup during ingestion, the raw layer stops being raw. Now a reprocess cannot recover what was dropped, and the exclusion rule is invisible to everyone reading downstream tables. The rule of thumb holds: bronze is allowed to *add* provenance columns and nothing else. ## Sign 5: consumers reading whatever layer is convenient Dashboards on bronze, an executive report on an intermediate table, a reverse-ETL job on a staging model. Each of these makes an internal table a public interface without anyone deciding to publish it, so a routine refactor breaks a report and the platform team learns about it from an angry message. Layer discipline includes knowing which tables are consumable and enforcing it — a small set of published gold tables, everything else internal. ## Sign 6: layer sprawl in the other direction The opposite failure is more layers than there are guarantees to make: bronze, cleansed, standardised, conformed, enriched, curated, presentation. Every additional copy costs storage, adds pipeline latency, gives logic another place to hide and makes lineage harder to reason about — while adding nothing a consumer can name. If you cannot state the promise each layer adds, collapse it. ## Sign 7: no owner and no tests The common root. Nobody is paged when a table stops loading, nobody reviews what gets published, nobody bears the cost of adding another copy. Tests either do not exist or have been failing so long that failure is background noise. At that point layer names are decoration. ## Remediation, in the order that works Start with **inventory and usage**: what tables exist, who queries them, what still writes to them. Query logs answer this better than opinions do, and they usually reveal that a large fraction of tables have no readers at all. Then **name owners** per dataset — an unowned table cannot be fixed or retired. Then **declare and test grains** on the tables that actually have consumers, which converts the biggest copies back into contracts. Then **consolidate duplicates**: pick one definition, migrate readers, and *delete* the losers rather than leaving them as deprecated shims, because a shim is a table someone will query next quarter. Finally, close the loop that caused the forking: a change process for shared models that is fast enough that forking is not the rational choice. ## The uncomfortable part Most of this is not a modelling problem. The technical remedies — a catalogue, grain tests, fewer layers — are straightforward; what is hard is agreeing whose definition of revenue wins and who is accountable for the shared model. A candidate who names that honestly is describing the actual job.