How do you scale pull-based and push-based metrics collection in one estate, and where is the line between them?

## Two different scaling budgets It is tempting to argue that one model "scales better". They do not fail at the same thing, and a lead is expected to name what limits each. | Dimension | Polling tier | Sending tier | | --- | --- | --- | | What limits it | Interval against targets times work per poll | Ingest throughput and per-sender queue memory | | What you tune | Interval, per-target timeout, target assignment | Batch size, flush interval, queue bound, retry policy | | What fails first | Collections overrun the interval | Queues fill and data is dropped at the sender | | What the failure looks like | Gaps in otherwise healthy series | Silence that the backend cannot explain | | Where the cost lands | On the targets, on every cycle | On the ingest tier, in correlated bursts | ## Scaling the polling side The arithmetic is unforgiving but simple. In a 41-service estate averaging 2,847 series per instance, the work per cycle is the number of targets multiplied by the cost of rendering and parsing that payload, and all of it must complete comfortably inside the interval. Three consequences follow. 1. **The per-target timeout must fit inside the interval.** If it does not, collections overlap, resolution silently degrades, and the first symptom is holes in the data for the slowest services rather than an explicit error. 2. **Growth is handled by sharding the target set**, and sharding is where the difficulty is. Assigning targets across collectors is a coordination problem, and once data lives on more than one collector every query needs a layer that fans out and merges - which is a whole component you did not have when one process held everything. 3. **Network topology puts a floor on the number of tiers.** A collector must open connections to everything it polls, so each network domain that cannot be entered from outside needs its own collection tier. That is an operational cost that has nothing to do with data volume. Remember also that the cost falls partly on the targets: every instance spends CPU rendering its current values on every cycle, whether anyone looks at them or not. ## Scaling the sending side Here the questions are queueing questions, and answering them vaguely is the usual failure. - **Batching** amortises connection and framing overhead, at the price of delay and of blast radius: a lost batch loses everything in it, so larger batches mean coarser loss. - **The queue must be bounded, with a stated policy for what happens when it fills** - drop the oldest, drop the newest, or apply back pressure to the application. Unbounded queueing is not reliability; it is an out-of-memory failure moved to a worse moment, and blocking the application means telemetry can take down the service it was meant to observe. - **The delivery guarantee is a choice.** Fire-and-forget loses data quietly; retrying produces duplicates the backend must tolerate. Both are defensible, but the choice must be explicit, and retries need backoff and jitter or a two-minute ingest blip becomes a synchronised stampede from every sender at once. - **Size the ingest tier for the correlated peak.** Senders are not independent: a fleet-wide deployment restarts everything together and a recovering backend gets everyone's backlog simultaneously. Per-tenant quotas and load shedding are what stop one misbehaving service from denying the rest. ## Where the line falls in practice Two properties decide it, and neither is a matter of taste: 1. **Can the collector route to it?** If not - a partner-hosted scheduling gateway, an agent on a court clerk's device, anything behind a boundary nobody will open - it sends. 2. **Does it outlive a collection cycle?** If not - nightly reconciliation, import jobs, anything that finishes in seconds - it sends. Everything else is polled, because the inventory and per-cycle liveness signals are worth more than the convenience of sending. In a 41-service estate that typically lands at roughly seven senders and the rest polled inside two collection tiers, with both intakes normalised into one store. What must be identical across the line: **identity conventions** (service, environment and instance dimensions spelled the same way on both sides, or the halves cannot be queried as one estate), unit and naming conventions, and retention. A dashboard built on one convention will silently cover only half the estate, and nobody notices until the half it omits is the half that broke. ## The part people forget: evidence The two intakes are not equally accountable, and that matters when someone asks for six months of proof rather than a live dashboard. A polling tier records its own failures, so a gap in the data comes with an explanation of why the data is missing. A sending tier's gap is unexplained by construction, and after the fact it cannot be distinguished from a period when nothing happened. If an estate has to demonstrate that a control was running continuously, the pushed half needs a deliberately maintained roster of expected senders plus staleness records kept for the whole retention window - otherwise the honest answer to "prove it was running in March" is that you cannot.