Why does a database/sql service start up healthy when its database is down, and how do you fix readiness?

## The mechanism behind the symptom `sql.Open` returns a `*sql.DB` without opening a connection. It resolves the driver name against the registry, possibly lets the driver pre-parse the DSN, and returns. No dial, no authentication. So a process configured with a dead host, a wrong password or a nonexistent database starts perfectly, and everything downstream that only asks "did startup return an error?" answers yes. If your health endpoint is the common `func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(200) }`, it is telling the truth about the process and nothing about its dependencies. The orchestrator marks the instance ready, routes traffic to it, and users get the first honest error the system produces: a query failure inside a request. ## Make readiness prove the dependency ```go func (s *Store) ready(w http.ResponseWriter, r *http.Request) { ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second) defer cancel() if err := s.db.PingContext(ctx); err != nil { http.Error(w, "database unavailable", http.StatusServiceUnavailable) return } w.WriteHeader(http.StatusOK) } ``` Three details matter here. **The deadline is yours.** Derive it from the incoming request's context so a cancelled probe stops work, but cap it yourself. Without a cap, a host that black-holes packets leaves the ping hanging and the probe's own timeout — set by someone else, in a manifest you may not own — becomes your failure semantics. A bounded ping fails in a time you chose. **A ping costs a pooled connection.** `PingContext` borrows a connection for the round trip, opening one if none is idle. A probe every second across many replicas is real, permanent load on the database's accept path, and during an incident it competes with the traffic you are trying to recover. If probes are frequent, cache the result for a short interval or run the check on a ticker and have the handler read the last outcome. **Ping proves less than a query.** It shows one connection could be established and is alive. It does not show that the schema is what you expect or that the account can read the tables you need. If those failures matter, a cheap real `SELECT` against a table you actually depend on is a stronger readiness signal — at the cost of a real query per probe. ## Liveness is a different question Liveness asks "is this process wedged and in need of a restart?" Readiness asks "should traffic come here right now?" Wiring a database check into liveness is one of the most reliable ways to convert a dependency outage into a self-inflicted one: the database wobbles, every replica fails liveness, every replica restarts, all of them come back cold and re-dial simultaneously, and the restart storm keeps the database from recovering. Liveness should test the process — that the request loop still schedules and responds. Readiness owns the dependency. ## Should startup fail, or should it start not-ready? This is the judgment call the question is really about, and the answer depends on process shape. **A long-running server.** Prefer to start. Log the failed ping loudly, expose it as not-ready, and keep retrying in the background. The instance then heals by itself the moment the database returns, with no deploy and no operator. Failing hard at boot instead means a database blip during a rollout leaves you with a crash-looping deployment, backoff delays, and possibly a rolled-back release that was fine. The nuance: start-up should still be *loud*. A service that starts, reports not-ready and logs nothing is the same silent failure in a different costume. **A short-lived process — a job that starts, works and exits, perhaps every minute.** The opposite. There is no traffic to withhold and no window in which healing helps: the run either does its work or it does not. Ping once immediately after `sql.Open` with a deadline well inside the schedule interval, and on failure log and exit non-zero so the scheduler records a failed run and your alerting sees it. The failure mode this prevents is the nastiest one on this whole topic: a job that starts every minute, silently processes nothing because its pool never connects, exits zero, and looks perfectly healthy on every dashboard while the data it was supposed to move quietly stops moving. A run that exits non-zero is a page; a run that exits zero having done nothing is discovered days later. ## What to check when you inherit the symptom Confirm the process really did no network work — a startup log with no connection line is the tell. Check whether the readiness handler references the database at all. Check whether the ping is bounded. Then check whether liveness is doing the database's job, because that determines whether your fix stops an outage or starts one.