You own a fleet of Redis instances shared by a dozen services and need a latency observability strategy rather than ad-hoc debugging. What signals would you collect, what would you alert on, and where would you accept blind spots?

## The core division: symptom versus explanation A fleet-level strategy fails when it alerts on server internals and hopes users are happy. The disciplined split is: - **Symptom signals** define the SLO and page someone. They come from **clients**: request latency percentiles for the Redis calls the application makes, plus timeout and error rates, plus connection-pool wait time. Only the client sees pool exhaustion, TLS handshakes, DNS, network, and — crucially — time spent queued behind another tenant's expensive command. - **Explanation signals** are server-side and are what you look at once paged. They should be rich, cheap, and never page on their own unless they are a leading indicator of a hard failure. ## What to collect from each instance **Slow commands.** Poll `SLOWLOG GET` on a short interval. Track the highest entry id already exported so each entry is emitted exactly once; entries carry id, timestamp, microseconds, truncated arguments, client address and client name. Raise `slowlog-max-len` well above the 128 default so a burst is not lost between polls, and set `slowlog-log-slower-than` near the service's latency budget rather than the 10ms default. Export as events with `command` and `client_name` labels — that is what makes multi-tenant attribution possible. **Subsystem stalls.** Arm `latency-monitor-threshold` (milliseconds) fleet-wide — 100 is a reasonable standing value — and scrape `LATENCY LATEST`, which returns per event the latest timestamp, latest value, and all-time max. The events that matter operationally: `fork`, `expire-cycle`, `eviction-cycle`/`eviction-del`, `aof-fsync-always` and the AOF write/rewrite family, `command`, and `fast-command`. `fast-command` firing with everything else quiet is the fleet's canary for host-level stalls. **Aggregate command cost.** `INFO commandstats` gives calls, total microseconds, and microseconds per call per command — this catches the case SLOWLOG structurally cannot see: a command that is only mildly expensive but is issued a million times a second. Redis 7 adds `INFO latencystats` with per-command percentiles, which is strictly better for this purpose. **Capacity and background work.** From `INFO`: `used_memory` versus `maxmemory` (headroom is the single most predictive number), `used_memory_rss` and fragmentation ratio, `evicted_keys`, `expired_keys`, `latest_fork_usec`, `rdb_bgsave_in_progress`, `aof_rewrite_in_progress`, `aof_delayed_fsync`, `connected_clients`, `blocked_clients`, `client_recent_max_output_buffer`, `total_net_output_bytes`, `keyspace_hits`/`misses`, and replication offsets/lag. **Host facts.** Swap in use by the process (any value is an incident), CPU steal, and the intrinsic-latency floor measured at provisioning time. If a candidate instance type shows a 3ms floor, that is a procurement decision, not a tuning problem. ## What to alert on Alert on **rates and budgets**, never on single samples: - **Page:** client p99 (or the SLO percentile) breached for a sustained window; error/timeout rate; instance unreachable; replica lag beyond the failover budget; memory headroom below the copy-on-write reserve. - **Ticket / warn:** new slow-log entries per minute above a baseline; `fork` latency above a chosen budget; `evicted_keys` above zero on an instance that is supposed to be sized as a store rather than a cache; `aof_delayed_fsync` climbing; fragmentation ratio drifting; a single client name responsible for a disproportionate share of slow entries. - **Never page on:** one 200ms `LATENCY` sample, or one slow-log entry. They are noisy and self-resolving; the rate is the signal. ## Governance, not just telemetry On a shared fleet, the cheapest latency win is preventing the pathology: - **Ban the O(N)-over-the-keyspace commands** (`KEYS`, `FLUSHALL` in production paths, unbounded `HGETALL`/`SMEMBERS`) using ACL command rules per tenant, so the guardrail is enforced by the server rather than by review. - **Require `CLIENT SETNAME`** in every client library wrapper. Without it, slow-log entries name an IP from an autoscaling group and attribution is guesswork. - **Standardise TTL jitter** in the shared cache client so no team can create an expiry cohort. - **Set a memory headroom target** (commonly leaving room for copy-on-write plus replication buffers) so that swap and eviction cycles are structurally impossible rather than monitored. - **Decide where snapshots run** — typically on a replica — so fork stalls never touch the serving path. ## Blind spots to accept explicitly 1. **Victims are invisible.** A blocking command's queueing cost appears only in client metrics. No server-side signal will ever show it, which is precisely why client percentiles are the SLO. 2. **Attribution is best-effort.** Without client names, and with connection pooling and proxies, mapping a slow command to a team is approximate. 3. **Lossy buffers.** The slow-log ring and the 160-sample-per-event latency history are in-memory and lost on restart; polling frequency bounds what you can recover. 4. **Cost of precision.** `MONITOR` shows everything and costs real throughput; keyspace-level analysis needs `--bigkeys`/`--memkeys` or offline RDB analysis, which are sampling or point-in-time, not continuous. 5. **Cardinality.** Exporting per-key or per-argument labels will destroy your metrics system; keep labels at command and client-name granularity. Stating those limits, and choosing them deliberately, is what makes this a strategy rather than a dashboard.