For a platform where most transformations could run either place, what do you set as the default and what does the escape hatch to a cluster cost?
answer
- decide once, in writing
- follow the data and the staffing
- a capability test, not a taste
- volume never opens the hatch
- the duplicated rule drifts forever
basics
~20 sSet the boundary as a written rule, not per pipeline. Default to whichever side already holds the data and the people, and make the escape hatch a capability test rather than a preference. Owning both sides costs a second skill set, a second deploy path and logic that drifts.
solid answer
~40 sDecide the boundary once, in writing, because the per-pipeline version of this argument is re-litigated forever and settles on whoever argued hardest. The default should follow the data and the staffing: if most inputs already live inside a managed query service and most authors are comfortable with the data rather than with a distributed runtime, the default is a declarative statement. Then write the escape hatch as a **test**, not a taste: a dependency the service cannot load, an input it cannot read, a destination it cannot write, or genuinely iterative logic. What owning both costs is not mainly money — it is a second deploy path, a second on-call surface, a second set of tests, and the standing risk that one business rule ends up implemented in both places and drifts.
go deeper
The takeaway is that this is a written rule, not a fresh argument each time. A platform with no stated default ends up with both arrangements and no reason for either.
Be able to state the default and the exceptions as capability tests: a dependency that cannot load, an input that cannot be read, a destination that cannot be written, genuinely iterative logic.
Price the exception honestly. A second deploy path, a second on-call surface and two differently denominated bills are the visible costs; the business rule implemented in both places is the one that bites later.
Own the default and the conditions for revising it. Name the exception rate at which the policy is simply wrong, and refuse volume as a reason before anyone offers it.
## Why this is policy and not a per-pipeline call Every individual pipeline can be argued either way, and the argument is cheap to start and expensive to finish. Left to case-by-case judgement, the boundary between a **cluster execution engine** — a system you hand a whole program to, which splits its work across many machines — and a **managed query service** — a service that owns its own storage, plan and capacity and takes a declarative statement — is set by whoever is loudest in each review. The result is a platform where both arrangements exist for no articulable reason, which is the worst of the available outcomes because it pays both sets of fixed costs and gets the benefit of neither default. So the deliverable is a rule: a default, a test that opens the exception, and a named owner of the test. ## Choosing the default Two facts about your organisation decide it, and neither is about technology: - **Where the data already lives.** A default that requires most pipelines to move their inputs is the wrong default, whichever side it favours. Movement is the one cost that buys nothing. - **Who is going to write and operate these.** A declarative default lowers the number of engineers who must understand a distributed runtime at all. That is a hiring, on-call and review-throughput consequence, not an aesthetic one. A third fact is worth checking but rarely decides: the two arrangements bill on different meters — machines that exist against work a statement consumed — so any cost comparison has to convert rather than compare. Do the conversion over a representative month before quoting a number. ## Writing the escape hatch as a test The exception must be phrased so that two engineers reading it reach the same answer. Capability tests do that; preferences do not. A workable list: 1. **A dependency the service cannot load** — a decoder, a model, a company library that must run next to the data. 2. **An input the service cannot read** — an opaque payload, media, a format with no reader, an outside system that must be called rather than read. 3. **A destination the service cannot write**, where exporting the result would be comparable in size to the input. 4. **Genuinely iterative logic** with its own termination condition, which resists being stated as a description of a result. Everything that is not on the list — input volume, join count, how long a statement takes, which language the author enjoys — does not open the hatch. Volume in particular must be named and excluded explicitly, because it is the argument the hatch will be attacked with most often. ## What owning both worlds actually costs | Cost | What it looks like | Who pays it | |---|---|---| | **A second skill set** | Someone must understand a distributed runtime, its failure modes and its supply of machines | Hiring, and the review queue | | **A second deploy path** | Packaging, dependency pinning, promotion and rollback for programs, alongside statement deployment | The platform team | | **A second on-call surface** | Machines that can be unhealthy, as well as statements that can fail | Whoever carries the pager | | **Two meters** | Two bills denominated differently, so nobody can answer "what does this pipeline cost" without a conversion | Finance, and every capacity conversation | | **Logic drift** | The same business rule implemented on both sides, diverging quietly until two reports disagree | Whoever is asked which number is right | The last row is the one that gets underestimated. Duplication of a rule is not a one-off cost; it is a permanent obligation to reconcile, and the reconciliation is usually discovered by a stakeholder rather than by a test. ## What would change the default Say this part unprompted — it is what separates a policy from an opinion: - **most new inputs stop being readable** by the service, which flips test 2 from an exception into the common case; - **the exception rate passes some threshold** you set in advance, at which point the default is simply wrong and should be restated rather than routinely overridden; - **the conversion of the two bills reverses** durably, not in one month's spike. And one thing that should *not* change it: a single high-profile pipeline that needed the hatch. The hatch working as designed is evidence for the policy, not against it. ## The shape of a strong answer Name the default, name the test, name the costs of the exception, and name what would make you change your mind. A candidate who only names the default is describing a preference; a candidate who only names the exceptions has no platform at all.
- A team invokes the escape hatch because their statement takes too long. Is that a valid claim?Not on its own. Duration is a cost-and-tuning signal, not a capability one, and the same work on a cluster is not faster by virtue of being a program. Ask what property of the work the service cannot do. If the honest answer is none, the pipeline stays declarative and the conversation moves to the statement itself.
- How do you stop the same business rule being implemented on both sides?Make single ownership part of the rule: any given output is produced in exactly one place, and a second implementation requires naming who reconciles the two and how often. In practice the cheapest enforcement is that the exception request must state which existing statement or program it replaces, not which one it complements.
- What is the honest argument for a cluster default instead?It holds when most inputs arrive in forms no service can read, when destinations sit outside any service, or when the team's work is genuinely code-shaped — model training, media processing, bespoke parsing. In that world a declarative default would force an export on almost every pipeline, and the exception would swallow the rule.
saying these in an interview costs you the question
- Decides the boundary separately for every pipeline and calls it pragmatism.
- Writes the escape hatch as a size threshold in terabytes.
- Treats owning both arrangements as costing only money.
- Lets a team invoke the exception because a statement ran slowly.
- Claims the two bills can be compared without converting the units.
- Sets a default but names nothing that would change it.