How would you design a data stream and ILM policy to keep 400 GB of logs a day for 13 months on a fixed budget?

## Start with access, not volume 400 GB a day is a distraction until you know who reads it. The design question is: for each age band, what latency is acceptable and how often is it queried? A workable interrogation is to ask for the last thirty days of query telemetry and bucket it by the time range requested. In almost every logging system the answer is heavily skewed — the overwhelming majority of queries cover the last day or two, a thin tail covers the last month, and access beyond that is a handful of investigations and audits per quarter. That shape is what makes tiering pay. If instead the audit team runs heavy aggregations over the full thirteen months every week, frozen is the wrong answer and the budget conversation has to happen up front rather than after the first slow query. The second question is what "keep for 13 months" actually obliges. Retention for regulatory retrieval, where a request may take hours to answer, is a completely different engineering problem from retention for interactive search. It is worth establishing whether the requirement can be met by snapshots in a repository at all, since a snapshot that can be restored on demand satisfies many retention obligations at a fraction of the cost of anything mounted. ## Shape the stream Separate streams per meaningful class of data, not one stream for everything. Different retention obligations, different mappings and different query patterns should not be forced to share a lifecycle — and separation also isolates a mapping explosion in one noisy service from everything else. Conversely, resist a stream per microservice: thousands of tiny streams multiply shards, cluster state and policy evaluation cost for no benefit. Size the rollover so each backing index is a comfortable unit of work: a size trigger on the largest primary shard, with an age ceiling so a quiet stream still rolls on a predictable cadence, and a `min_docs` guard so quiet periods do not manufacture empty generations. Roughly daily generations are a good default because retention granularity can never be finer than the rollover cadence — you delete whole backing indices, so a weekly index means expiring a week at a time. Get the shard count right at the template level: it is fixed for each generation's life, and the only cheap way to change it is to edit the template and let the next rollover apply it. ## The phases **Hot** carries the write index and the last few days. Fast storage, replicas for both redundancy and query throughput, no merging or read-only games — this data is still changing. **Warm** takes over once writes stop. Mark read-only, force merge to few segments, drop the recovery priority so hot indices restore first after a restart, and consider `best_compression` since this data is now written once and read occasionally. Denser, cheaper nodes; still local disk, so latency stays low. **Cold** for the month-plus band: mount as a fully mounted searchable snapshot so the repository copy provides redundancy and local replicas can go, roughly halving local storage for that band while keeping query speed close to warm. **Frozen** for the long tail: a partially mounted searchable snapshot, where only a bounded shared cache lives on local disk and blocks are fetched from object storage on demand. This is what makes thirteen months affordable — the tail becomes an object-storage line item rather than provisioned disk — and the price is first-query latency measured in seconds. Confirm that whoever queries year-old data can live with that before committing. **Delete** at the retention boundary, with `wait_for_snapshot` naming an SLM policy so an index is never deleted before a snapshot taken after it entered the delete phase exists. Also decide deliberately whether deleting a mounted index should delete its underlying snapshot; keeping the snapshot beyond the searchable retention is a common and cheap way to satisfy an archival obligation. ## The snapshot side An SLM policy defines schedule, repository, which indices, a name pattern and a retention block with expiry and minimum and maximum counts. Two things are worth stating explicitly at design time. First, retention in SLM and retention in ILM are separate systems that must not contradict each other: an SLM retention that expires snapshots still underpinning mounted cold or frozen indices is a data-loss event, not a cleanup. Second, snapshot policies deserve the same monitoring as the ingest path, because a `wait_for_snapshot` guard turns a silently failing SLM policy into an ever-growing set of undeletable indices — the failure is safe but not free. ## Verify, then defend the numbers Write down what each transition buys and costs — hardware saved in warm, replicas removed in cold, disk avoided in frozen, object-storage spend added — and validate the boundaries against real query telemetry after a month. Expect to move the frozen boundary at least once. The design is not the policy JSON; it is the argument connecting the access profile to the spend, and the ability to say what changes when the daily volume doubles or the retention requirement moves to three years.