In a warehouse with storage and compute separated, stored data triples yearly while query volume stays flat — how do you plan cost?

## Frame the question correctly The naive version of this scenario says: storage triples, storage is cheap, compute is flat, therefore the bill barely moves. That is roughly right and it is also the answer that gets interrupted, because it misses where growth actually leaks into compute. A strong answer separates the first-order model from the second-order coupling and says what you would measure to catch the coupling early. ## The first-order model Build two forecasts with different drivers. **Storage.** Cost is retained bytes times a per-TB-month rate. The drivers you control are retention policy on raw and intermediate layers, the history/versioning window (retained old table versions keep their files alive, so a heavily rewritten table can hold a multiple of its logical size), duplicate copies made by teams for convenience, and compression and file layout. Note that this curve is smooth and predictable — it is the easiest part of a warehouse bill to forecast and usually the smallest. **Compute.** Cost is duty cycle times cluster size, or bytes scanned times a rate, depending on the model. The driver is workload — number of queries, concurrency, and how much data each touches. At genuinely flat query volume with stable data layout, this curve is flat. The contrast worth naming explicitly: in a shared-nothing cluster with local disks, tripling data forces you to triple node count to hold it, and you buy CPU you did not need. Separation is precisely what breaks that, and it is the strongest single argument for the architecture in a growth scenario. ## Where the curves recouple **Pruning that stops holding.** A query filtering on last 30 days against a table partitioned or clustered by date reads a roughly constant volume as history grows — this is the good case. A query with no usable filter, or a filter on a column the layout does not support, reads a fraction of the whole table and therefore triples with it. At flat query counts your scan volume still triples. Under per-byte billing that is a tripled invoice; under provisioned billing it is tripled runtime, which either lengthens the batch window or forces a bigger cluster. This is the dominant failure mode and it is why "query volume is flat" is not the same as "compute cost is flat". **Working set outgrowing local cache.** Compute clusters cache column chunks on local SSD, and that capacity is fixed by cluster size. While the hot data fits, most reads are local; once it does not, entries are evicted before reuse and reads go back over the network. The transition is not gradual in effect — query times can step up noticeably as the hit ratio falls past the point where the recurring workload no longer fits. Flat query volume, unchanged SQL, slower queries. **Metadata and planning.** Growth expressed as many more small files rather than more large ones raises planning cost, because file-level statistics must be considered per file. A streaming pipeline is the usual source. This shows up as latency on every query, not just big ones. **Concurrency drift.** Longer queries at constant arrival rate means more queries in flight, which can push a cluster into queueing that it previously never saw. Worth naming as a consequence; the design of isolation and queueing itself is a separate subject. ## What you actually do 1. **Put retention on a policy, not on inertia.** Decide per layer how long raw, staged and curated data lives, and set the history/versioning window per table according to how much it churns. This is the single biggest lever on the storage curve and it costs nothing to exercise early. 2. **Audit pruning on the recurring workload.** Take the top queries by scan volume and check that bytes scanned per execution is flat over time. Any query whose scanned bytes track table size is a future cost incident with a known date. 3. **Watch the cache hit ratio, not just cost.** The fraction of bytes served locally versus remotely on your recurring workload is the leading indicator that the working set has outgrown the cluster. Falling hit ratio predicts the latency complaint before users file it. 4. **Manage file count as a distinct dimension.** Ensure ingestion consolidates small files, because metadata cost scales with file count while storage cost scales with bytes. 5. **Forecast in unit terms.** Cost per TB-month stored and cost per recurring report are far more useful to a finance conversation than a single total, because they separate "we have more data" from "we got less efficient", and only the second is an engineering problem. ## The judgment to demonstrate A principal-level answer commits to a position: in this architecture, storage growth is a budget line, not an architecture problem, and it should be handled with retention policy rather than platform change. The engineering attention belongs on the coupling — pruning discipline and cache fit — because that is where a tripling dataset quietly becomes a tripling compute bill. It is also worth saying what would change your mind: if scan volume tracks storage despite layout work, or if the working set can no longer be made to fit any reasonable cluster, then the workload is genuinely growing, and the conversation moves from retention to aggregation, pre-computation and workload redesign.