skip to content

An analyst wants dashboards built directly on a read replica of the normalized production database instead of a modeled mart — what are the real costs?

level: seniorimportance: should knowfreq 50%

answer

  1. the replica fixes contention, not shape
  2. can you reproduce last quarter's number?
  3. where does the metric definition live?
  4. who owns the tables you are querying?
  5. one system means one system's questions

basics

~20 s

A replica gives you the source's shape, not a model: no history once rows are overwritten, no shared metric definitions, many joins per question, and reports that break whenever the application team changes the schema.

solid answer

~50 s

A replica solves exactly one of the problems — resource contention with the primary — and none of the modelling ones. Four costs remain. **No history.** Operational rows are updated in place, so once a customer's tier changes, last quarter's report cannot be reproduced; it will silently re-attribute old orders to the new tier. **No shared definitions.** Every dashboard re-derives "active customer" or "net revenue" from raw status columns, so two dashboards drift apart and nobody can say which is right. **Coupling to a schema you don't own.** The application team owns those tables and will refactor them; each refactor breaks reports without warning, so your BI layer becomes an invisible constraint on their roadmap. **Complexity per question.** Seven-join queries over normalized tables, with soft deletes and operational status codes that only the application team understands. It is a reasonable *start* for one system, low volume and no history requirement — but plan the thin modelled layer before those four bite.

code

text · 7 lines
text
operational row, updated in place on 2024-03-04
  customer 4471: tier = 'GOLD'        (was 'SILVER' until 2024-03-04)

February report, run in February  -> February orders counted as SILVER
February report, re-run today     -> the same orders counted as GOLD

nothing is broken, no error is raised, the number simply changed

go deeper

for a junior

Know that a read replica is a copy of the operational schema, so it carries the source's shape and only its current state — it is not a modelled reporting layer.

for a middle

Explain the concrete failures: overwritten attributes make past reports irreproducible, metric logic gets re-typed per dashboard, and application schema changes break reports without notice.

for a senior

Concede what the replica genuinely fixes, rank the remaining costs by irreversibility with lost history first, and propose the smallest modelled layer that stops the bleeding rather than demanding a full warehouse.

for a principal

Own the organizational consequence: reporting straight off application tables silently converts another team's internal schema into a public contract and constrains their roadmap. Decide where that boundary belongs and who is accountable for it.

## What the replica actually fixes Give the proposal its due first, because a senior answer that just says "no" is a weak one. A read replica genuinely solves real problems: long analytical scans no longer compete with the primary's transactional workload, reporting users cannot write to production tables, and the data is fresh to within replication lag. If the objection to reporting off production was "you'll take the checkout system down", the replica answers it. What the replica does **not** change is the shape and semantics of the data, and that is where every remaining cost lives. ## Cost 1: history you have already lost Operational systems update in place. A customer's tier, an account's owner, a product's category — these are columns that get overwritten, and the previous value is simply gone. That has a specific and nasty consequence: a report re-run today for a period in the past gives a different answer than it gave then, and does so silently. February's revenue-by-tier report attributes February orders to whatever tier the customer holds *now*. This is not a tuning problem you can fix later with a query. Once the source has overwritten the value and nothing captured it, the history does not exist anywhere. Building a modelled layer is partly about deciding, deliberately, which attributes need their changes preserved — and starting that capture before the values you need have already been lost. ## Cost 2: metric definitions that live in dashboards In a normalized source, "revenue" is not a column. It is a rule: which order statuses count, whether cancellations are excluded, how refunds and partial shipments are treated, which currency conversion applies. On a replica, that rule is re-typed into every dashboard's SQL by whoever built it. Two dashboards run at the same instant against the same replica will disagree, and reconciling them means reading two piles of SQL and arbitrating. The modelled layer's real product here is not speed; it is a **single place where a definition lives**, so that changing it changes every consumer at once. ## Cost 3: coupling to a schema you do not own Application tables are internal implementation, not a published contract. The product team will rename columns, split tables, add status values, introduce a nullable column with a meaning your queries do not know about. Every one of those is routine for them and a broken dashboard for you — usually discovered by an executive rather than by a test. The coupling runs both ways, and the second direction is worse: once finance depends on the exact shape of `orders`, the application team can no longer refactor `orders` freely. You have converted their internal schema into an undocumented public API. A modelled layer absorbs source changes in the transformation and keeps the published tables stable. ## Cost 4: the per-question cost of the source shape Normalized sources are built for correct writes, not readable questions. A single business question crosses many tables; soft-delete flags must be remembered in every query or results quietly include deleted rows; status enums encode operational states with meanings only the application team knows; multi-tenant or partition columns must be filtered everywhere. Each of these is a way for an analyst to produce a wrong number that looks reasonable. There is also a scope limit: the replica contains one system. The moment a question needs orders joined to support tickets, marketing spend or subscription events, the replica cannot answer it at all, because conforming entities across systems is exactly what a modelled layer does. ## When it is nonetheless the right call Be honest that the answer is not always "build the warehouse": - One source system, modest volume, and questions that are all about the current state. - No requirement to reproduce a past report, and no regulatory or financial audit trail. - A small team where the same people write the app and the dashboards, so the coupling is internal rather than cross-team. - Early exploration, where you are still discovering which questions matter and premature modelling would model the wrong thing. The failure is not starting on the replica; it is *staying* there past the point where any of those conditions stops holding — usually the first time somebody asks why last quarter's number changed. ## The middle path worth proposing Rather than a binary, propose a thin modelled layer over the replica's extract: land the source tables, snapshot the handful of attributes whose history matters, publish two or three modelled tables with agreed definitions, and point dashboards at those. That captures history from day one — the one cost that is irreversible — and defers the rest until volume or a second source justifies it. ## Answering this in an interview Concede what the replica fixes, then name the four costs in order of irreversibility, putting lost history first. Finish with the conditions under which you would happily start on the replica anyway, and the smallest thing you would build first. That shape — concede, enumerate, qualify, propose — is what "production judgment" looks like in this question.

  • Which of those costs is irreversible, and why does that matter for sequencing?
    Lost history. Every other cost — join complexity, definition drift, schema coupling — can be fixed later by building the model then. But an attribute the source overwrote and nobody captured is gone forever. So even the most minimal first step should start snapshotting the few attributes whose history a business question will eventually need.
  • The team says they will just add a history table in the application database instead. What do you say?
    That can work and is sometimes the right call, but it moves cost onto the write path and onto the application team's roadmap, and it still leaves definitions in dashboards and questions limited to one system. It is a targeted fix for one of four problems, so agree only if history is genuinely the only pressure.
  • How would you make the case to a sceptical engineering lead who sees the warehouse as duplicated data?
    Frame it as decoupling rather than duplication: today their internal schema is an undocumented public API that finance depends on, so they cannot refactor freely. A modelled layer absorbs their changes in one transformation and gives them back the freedom to change their own tables. The duplication buys their autonomy.

Querying an operational replica for history is like asking a whiteboard what it said last month: it is perfectly accurate about right now, and it kept nothing.

saying these in an interview costs you the question

  • Says a read replica makes a modelled layer unnecessary
  • Ignores that source rows are overwritten in place
  • Accepts every dashboard re-deriving its own metric
  • Treats the application schema as a stable contract
  • Names replication lag as the only real drawback

context