Retention and Cost Control

A log platform's bill is not one number with one dial on it. It is four largely independent terms, and cost conversations go wrong when a team assumes the dial it can reach is attached to the term that dominates. ## The four terms 1. **Ingest** — the bytes accepted at the front door. A hosted platform usually charges this directly per gigabyte. Self-hosted, you pay it as network transfer, as the CPU the node-level log shipper burns batching and compressing, and as the capacity of the collection tier itself. This term is a function of raw volume and of nothing else. 2. **Processing** — the work done once per record at write time: parsing an unstructured line into fields, enrichment lookups that attach orchestrator or geographic metadata, and building whatever search structures the store maintains. Those structures cost twice: CPU to build, and then they are themselves stored bytes. For verbose records they can rival the size of the original text. 3. **Storage** — retained bytes multiplied by the replication factor, summed over every tier and every copy. The multipliers surprise people: eighteen gigabytes a day of raw text, expanded by write-time structures and held in two replicas, is a far larger billable figure than the number anyone quoted, and that is before snapshots. 4. **Query** — the scan, decompression and compute a search burns. The human part is small and bursty. The part that dominates is automated: alert rules and dashboard panels re-evaluating on a fixed interval, forever, each one a real query over real data whether or not anybody is looking at the answer. ## Why shortening the retention window disappoints Shortening how long data is kept removes the oldest days from term 3, and only from term 3. Those bytes were already ingested and already processed; that charge was paid once, at write time, and nothing refunds it. The saving is therefore bounded by the storage share of the bill multiplied by the fraction of retained days removed — and on an indexed, hot-heavy platform the storage share is frequently the smaller half. It disappoints twice, because it also arrives late: nothing is deleted when the setting changes, so the saving accrues only as existing days age past the new horizon. Teams reach for it first anyway, because it is a single setting needing no code change and no negotiation with a service owner. Every lever that genuinely moves the bill requires somebody to change what a service emits. ## What moves what | Lever | Ingest | Processing | Storage | Query | |---|---|---|---|---| | Shorten the retention window | no | no | removed days only | slightly | | Drop a class of record before it is shipped | yes | yes | yes | yes | | Sample a high-volume class at collection | yes | yes | yes | yes | | Pre-aggregate a class into a counter at the edge | yes | yes | yes | yes | | Index fewer fields of each record | no | yes | yes | queries cost more | | Move older data to a cheaper tier | no | no | rate only, not bytes | slower | | Reduce replicas on older data | no | no | yes | availability risk | | Lengthen an alert rule's evaluation interval | no | no | no | yes | The table states the point plainly: **only levers applied before ingest move all four terms.** Everything applied after the front door is a discount on one of them. ## Measure before you cut - Rank producers by bytes per day. The distribution is nearly always extreme — a handful of streams carry most of the volume. - Rank the same producers by demonstrated readers: which have been touched by a human search, or named by an alert rule, in the last quarter? - Join the two rankings. High volume with no readers is the free money. High volume with many readers is where you sample rather than drop. - Measure the expansion factor your store applies, so that a capacity forecast starts from billable bytes rather than from raw text. - Look at the automated query load separately from the human load. It is a fixed cost you chose, and it is usually invisible on the very dashboard that is causing it. ## A worked example A vinyl-record marketplace ships 18.4 GB of application and access logs a day into a fully indexed hosted platform with a 30-day window. Someone proposes cutting the window to 14 days. If ingest and write-time processing are 62% of the invoice, stored bytes 31%, and query the remainder, then removing 16 of the 30 days can take at most about 16% off the total, and only once the old days have aged out. Meanwhile, ranking by producer shows that 7.5 GB of the daily 18.4 GB is access lines for a readiness endpoint polled twice a second by 109 replicas, roughly 400 bytes each, and that nothing has queried them in ninety days. Discarding that one class before it is shipped removes 41% of the volume from all four terms at once, and it takes effect the day the rule ships. The retention change is not wrong; it is simply the smaller of the two, and it was reached for first because it was easier.