A runbook claims a database recovery point objective of 5 minutes and a recovery time objective of 30 minutes. How would you determine whether those numbers are actually true, and what continuous telemetry would tell you when they stop being true?

## Two claims, two very different verification methods A recovery-point promise is about a property the system has **right now** - how far behind the surviving copies are - so it can be measured continuously from live telemetry. A recovery-time promise is about behaviour during an event that has not happened, so it can only be established by **inducing the event**. Conflating them is the most common gap in a disaster-recovery programme: teams monitor lag diligently and have never once timed a restore. ## Measuring the recovery point continuously Instrument every path by which data leaves the primary, and alarm on the worst one that a given failure scenario would rely on: - **Replication lag in bytes**, per standby: the difference between the log position generated on the primary and the position flushed or replayed on the standby. Bytes, not seconds, is the honest measure of outstanding data; time-based lag inflates misleadingly on an idle primary because it is derived from the last replayed transaction's timestamp. - **Archive freshness**: the age of the newest successfully archived log segment, and the count of segments waiting to be archived. A failing archive command is the classic silent degradation - backup jobs keep reporting success while the recoverable point drifts backwards by hours, and the primary's disk fills as retained log accumulates. - **Backup age and verification age**: how old the newest backup is, and how long since one was proven restorable. - **Degraded-mode flags**: whether a nominally synchronous standby is currently disconnected and the primary has fallen back to asynchronous commit. That state is a live breach of a zero-loss promise and must page, not merely log. Report these as percentiles over a window. The promise is a ceiling, so p99 or max is the relevant statistic and the average is close to meaningless. ## Measuring the recovery time by exercising it There is no telemetry for this. There are only drills, in increasing fidelity: 1. **Automated restore drill.** On a schedule, restore a real backup into a scratch environment, replay to a target moment, start the engine, and run data assertions. Fully automated, so it can run weekly or nightly and produce a time series of restore duration. This validates the restore-and-replay portion. 2. **Failover drill.** Promote a standby in a staging or, better, production-equivalent topology, including fencing and client repointing, and measure how long until an application client successfully writes. 3. **Game day.** Inject a realistic failure without warning the responders, require them to use the runbook, and time from injection to verified recovery. This is the only exercise that measures detection and human decision time - typically the largest components - and the only one that surfaces the missing credential, the expired certificate, and the step that only one person knows. Instrument the drill with **phase timestamps**: failure injected, failure detected, decision made, fencing complete, data recovery started and finished, log replay finished, first successful client write, latency back to normal. A single total is a scoreboard; the phase breakdown is a work list. If detection is 18 minutes of a 26-minute total, buying faster storage is wasted money. Define the stop condition precisely and in the users' terms: not "the engine opened" but "a representative application transaction committed and read back successfully at acceptable latency". Teams that stop the clock at engine startup systematically under-report. ## Watching for drift Both numbers decay silently: - **Data growth** lengthens fetch, restore, and replay roughly linearly, so a 30-minute restore at 400 GB becomes hours at 3 TB with no change in configuration. Trend measured restore duration against database size and forecast when the promise breaks, then act before it does - more frequent base backups to shorten replay, parallel restore, faster storage, or moving that tier onto a standby-based recovery path. - **Topology and tooling changes** - a new region, a rotated key, an upgraded backup tool, a changed storage class - can break the recovery path while every routine metric stays green. Any such change should invalidate the last proof and trigger a fresh drill. - **Staleness itself is a risk.** Track "time since this cluster's last proven restore" as a first-class metric with a per-tier threshold, and treat breaching it as an incident, because it is the metric that tells you whether the rest of your evidence is still credible. ## What good looks like A dashboard per cluster showing: current and p99 replication byte lag, archive backlog and freshness, newest backup age, newest verified-restore age, last measured restore duration with its phase breakdown, and the declared objectives drawn as lines on the same charts. When the measurement crosses the declared line, either the engineering or the promise has to change - and saying so plainly is exactly the judgement an interviewer is testing.