Your ELT warehouse bill tripled as models multiplied — how do you decide where transformation belongs?

## Why the bill grew ELT converted a fixed, visible transformation budget — a cluster someone procured once — into a variable cost that grows with every model anyone adds. No individual query looks expensive. Each new model is defensible on its own. The total is nobody's specific decision. Tripling is therefore the *expected* outcome of a successful platform with no cost feedback loop, and the interesting question is what changed: more models, more frequent rebuilds, more data per rebuild, or more consumers. Those have different fixes. ## Step one: attribution before architecture You cannot make a placement decision without knowing what is being placed. Tag every transformation run so spend can be attributed to a model, an owner, a schedule and a downstream consumer, and rank the results. Expect a long tail: a handful of jobs typically account for most of the cost, which is good news — it means a small number of decisions can move the number materially. While you are there, capture two facts that are usually missing: **how often each model rebuilds**, and **when its output was last actually read**. Together those find the most embarrassing category of spend — expensive models nobody consumes. ## Step two: classify by cause, not by size For each top consumer, ask *why* it costs what it does. **It rebuilds everything every time.** A model that scans full history nightly to append one day is paying for the whole table daily. Rebuilding only the affected window is usually the single largest saving available, at the price of designing for late-arriving data and periodic full reconciliation. **It rebuilds more often than anyone reads it.** Frequency is a business decision disguised as a technical setting. Hourly refresh of a table feeding a weekly report is pure waste, and dropping to daily is a conversation, not a migration. **It recomputes shared work.** When ten models each derive the same heavy intermediate, materialising that intermediate once and referencing it converts ten expensive scans into one. **Its logic fights the engine.** Row-at-a-time user-defined functions, repeated sorts over huge tables, joins that cannot prune. These are optimisation problems inside the warehouse before they are placement problems. **It scans more than it needs.** Layout choices — partitioning, clustering, pruning-friendly filters — often cut scanned volume dramatically with no change to the logic at all. Only one of these categories argues for leaving the warehouse. The others are shape and schedule. ## Step three: the placement decision, made honestly When a job survives all that and still looks structurally mispriced — heavy per-row work, unstructured parsing, model inference — compare properly: - Run the job both ways on real data and compare **total** cost, including the read-out and write-back, not just the engine's compute charge. - Add the standing cost of the alternative: cluster idle time, a second permission model, a second deployment path, a second on-call rotation, and the engineering time to keep them aligned. This is paid every week regardless of how many jobs use it. - Ask whether the resulting lineage gap is acceptable and how the orchestrator will hold the dependency across it. - Ask who maintains it after its author changes team. A job outside the mainstream path is the first thing to rot. Moving three jobs rarely justifies an engine. Moving a whole class of work — all feature generation, all raw file parsing — sometimes does, and that is the granularity at which the decision should be made. ## Step four: make the fix durable Without a feedback loop the bill regrows, because the mechanism that produced it is untouched. - **Ownership.** Every model has a named owner; unowned models are candidates for deletion, and deletion is the cheapest optimisation there is. - **Showback.** Teams see their own transformation spend on a regular cadence. Visibility alone reliably changes behaviour. - **Guardrails.** Defaults that fail safe: incremental unless justified, a default schedule that is not the fastest one available, alerts on individual jobs whose cost jumps. - **Review at introduction.** New models declare consumer, freshness requirement and expected cost. Refresh frequency should be justified by a stated requirement rather than chosen by habit. - **Retention.** Raw retention policy and intermediate cleanup, decided deliberately rather than accumulating. ## What to resist Resist re-platforming on the strength of a bill alone — the same badly shaped jobs cost badly on any engine, and you will have paid a migration to find out. Resist a blanket refresh-frequency cut that quietly breaks a freshness commitment someone depends on. Resist optimising the long tail; it is where effort goes to die. And resist the framing that this is a technology failure. It is a governance gap, and the durable fix is that transformation cost has an owner, a number and a review — after which "where does transformation belong" becomes answerable per workload instead of as an architecture-wide referendum.