skip to content

A distinct-customer figure must tie out exactly for an audit, and the job must finish inside a 20-minute window. Which escape does each requirement rule out?

level: seniorimportance: should knowfreq 44%

answer

  1. requirements veto escapes, not operations
  2. exactness kills the error bound
  3. a deadline kills both extra-read escapes
  4. then negotiate requirement, hardware, or data seen

basics

~20 s

Exactness rules out the bounded-error answer; the wall-clock budget rules out both escapes that re-read the input. Together they leave nothing, so a requirement, the hardware, or the amount of data the operation must see has to change.

solid answer

~50 s

Each requirement forbids a different escape, which is why the pair is worth asking about. - **"It must tie out exactly for an audit"** forbids the **bounded-error answer**. Not because the bound is large, but because a figure someone reconciles against another system cannot carry one: a disagreement smaller than the bound is then unresolvable by inspection. - **"It must finish in 20 minutes"** forbids the escapes that multiply input and output volume — both the **spilled multi-pass run** and the **cheap first traversal**, since each pays the read cost again, and device throughput sets the clock. Stated together they leave nothing, and saying so plainly is the answer. What gives is then a requirement (is the deadline real?), the hardware (enough memory to hold the working set), or the amount of data the operation has to see at all — a narrower column set, a pre-filter, a coarser reporting grain.

go deeper

for a junior

Learn the two simplest vetoes: a figure that must be exact cannot carry an error bound, and a tight deadline is hostile to reading the input more than once.

for a middle

Explain why the deadline forbids two escapes rather than one, and describe the cost as movement of bytes rather than as arithmetic.

for a senior

Map each requirement to the escape it vetoes, say out loud when the set is empty, and move the conversation to the requirement, the hardware, or the working set.

for a principal

Own which requirements are genuinely binding. Whether a figure is audited or merely reported decides the execution shape of the job, and that call is far cheaper made before anything is built.

## Requirements are constraints on the escape, not on the operation The operation itself does not change: an exact count of distinct customers is still one whose answer a later record can move, and it still needs reach over records it cannot hold. What the requirements decide is **which of the three currencies you are allowed to spend** — input and output volume, exactness, or one extra traversal. Reading them as constraints on the escape rather than as complaints about the data is what separates a diagnosis from a shrug. | Stated requirement | Escape it forbids | Why | |---|---|---| | The figure must tie out exactly for an audit | The bounded-error answer | A disagreement smaller than the bound cannot be adjudicated, so the figure stops being evidence | | A hard wall-clock budget | The spilled multi-pass run, and the cheap first traversal | Both pay the read cost again, and device throughput sets the clock | | The input can be read only once | Both multi-read escapes | The second read does not exist to be paid for | | Local storage is tight | The spilled multi-pass run | Intermediates need real headroom, often a multiple of the input | | The answer must be recomputable years later | The bounded-error answer, in practice | Reproducing an approximate figure means pinning the summary's behaviour as well as the data | ## Why exactness is a hard veto and not a preference It is tempting to argue that a bound of a fraction of a percent is smaller than the error already present in the source data, so an approximate figure ought to be acceptable. That argument is about the *data*; the requirement is about the *figure's role*. An audited number exists to be compared with another number produced independently. If the two differ, someone must be able to say which is wrong. With a stated error bound in play, every small difference has two explanations and no way to choose between them, so the reconciliation itself stops working. The veto is structural. This is also why the tolerance conversation, if there is to be one, happens **before** the work is built and with the people who consume the figure — not as a setting chosen by whoever writes the job. ## Why a wall-clock budget hits two escapes, not one Both multi-read escapes pay in the same currency even though they look different. The spilled run re-reads the input and writes intermediates alongside; the cheap first traversal writes nothing but reads the input a second time. In each case the extra work is **movement of bytes**, and on an input far larger than memory that movement, not the folding arithmetic, is what the clock is measuring. A budget that was comfortable for one read is often not comfortable for two, and "we will just read it twice" is exactly the assumption a deadline invalidates. Note the honest asymmetry: the second traversal of the narrowing technique is usually cheaper than the first, because it retains almost nothing. It is not free, and it is not a doubling either — say which, rather than quoting a multiplier. ## When the requirements leave nothing Stating "there is no escape that satisfies both" is a complete and correct answer, and the strongest candidates say it without embarrassment. What follows is a negotiation over the three things that are actually movable: 1. **The requirement.** Is the deadline a real downstream dependency or an inherited habit? Is the figure genuinely audited, or merely reported? One of these is often softer than it was presented. 2. **The hardware.** Renting more memory on one box, priced per hour, removes the need for any escape at all when the working set is within reach of a larger machine. It is an expense, not a defeat. 3. **What the operation has to see.** This is the most under-used lever. Fewer columns loaded, a filter applied at the read so that fewer records ever enter the computation, or a coarser reporting grain can move the working set inside memory, at which point the operation that could not be answered from pieces simply does not need to be. ## The failure mode this question is hunting The weak answer picks an escape and defends it against the requirement that forbids it — arguing the bound is small enough, or that the deadline can slip. The strong answer maps each requirement to the escape it vetoes, notices when the set is empty, and then negotiates on the movable axes instead of on the arithmetic.

  • The team argues the error bound is smaller than the noise already in the source data. Why does that not rescue the approximate answer here?
    Because the objection is about the figure's role, not its accuracy. An audited number is compared against an independently produced one, and any difference has to be attributable. Once a bound is in play, a small disagreement has two explanations — a real discrepancy, or the bound — and no way to choose, so the reconciliation loses its point.
  • Which requirement most often turns out to be the soft one?
    The wall-clock budget, surprisingly often — it is frequently inherited from a schedule slot rather than from a downstream dependency. Exactness, when it is genuinely an audit requirement, almost never moves. Test the deadline first by asking who is waiting for the number and what breaks if it lands later.
  • How does reducing what the operation has to see differ from the three escapes?
    The escapes buy reach over records you cannot hold; reducing the working set means there is nothing to reach for. Loading fewer columns, filtering at the read, or reporting at a coarser grain can bring the data inside memory, and then the operation runs unremarkably with no escape and no charge at all.

saying these in an interview costs you the question

  • Argues an audited figure can use a bounded-error answer if the bound is small
  • Assumes a wall-clock budget only affects the spilled run, not the extra traversal
  • Treats "read it twice" as free because the data is already stored
  • Never considers reducing what the operation has to see
  • Refuses to say plainly that both requirements cannot be met as stated