An execution engine can interpret a plan operator by operator, or compile the plan into machine code at runtime with adjacent operators fused into one loop. How would you decide which strategy a new engine should adopt, and what does each cost?
answer
- fusion = one loop per pipeline, tuple in registers
- compile latency must amortize
- adaptive: interpret now, switch when compiled
- compute-bound → compile; bandwidth-bound → vectorize
- compilation needs push substrate + observability work
basics
~20 sDecide by workload and engineering budget. Compilation fuses operators into register-resident loops and wins on long, compute-heavy analytical queries, but costs milliseconds of compile time per query plus hard debugging. Vectorized interpretation gets most of the win with precompiled primitives, no warm-up, and far less complexity — the safer default.
solid answer
~50 sBoth strategies attack the same thing: per-tuple interpretation overhead. **Just-in-time compilation** generates code for the plan — typically one loop per pipeline, with filters, projections and probes fused inline so a tuple stays in CPU registers from the scan until a pipeline breaker. It wins most on compute-heavy expressions and long-running queries, where generated code removes both dispatch and intermediate materialization. **Vectorized interpretation** keeps precompiled primitives and passes cache-sized batches, amortizing dispatch and enabling SIMD without generating anything. My decision criteria: query duration versus compile latency (compilation must amortize a few to a few hundred milliseconds, so short OLTP-ish queries must fall back or be interpreted); workload mix (compute-bound favours compilation, memory-bandwidth-bound scans favour vectorization since both end up waiting on memory); engineering capacity, because a code generator plus a runtime compiler is a large, hard-to-debug, hard-to-profile subsystem; and operability, since compiled plans complicate per-operator timings and crash triage. Pragmatic answer for most engines: vectorize first, add adaptive compilation later for the queries that pay for it.
go deeper
Know that some engines interpret the plan while others generate machine code for it at runtime, and that generated code avoids per-row interpretation.
Explain operator fusion into one loop per pipeline and name the obvious cost: compile time per query.
Weigh compile latency against query duration, propose caching and adaptive switching, and raise observability and debugging costs.
Deliver a decision framework — workload profile, latency distribution, executor substrate, team capacity, operability — and a sequencing recommendation with an interpreted fallback retained permanently.
## The problem both approaches solve A plain iterator executor spends most of its cycles on machinery: an indirect call per operator per row, a tree-walking expression interpreter, generic tuple representations, and instruction-cache churn. On analytical workloads this can be an order of magnitude of waste. Vectorization and compilation are the two mature answers. ## What compilation actually does At runtime, after optimization, the engine translates the plan into source or IR (C++, LLVM IR, WebAssembly, bytecode) and compiles it. The important part is not "machine code" in the abstract — it is **operator fusion**. In the data-centric produce/consume model, the plan is cut at pipeline breakers (hash build, sort, aggregation), and every chain of non-blocking operators between breakers becomes **one loop**. A scan → filter → project → hash-probe pipeline compiles into a single loop body where the tuple lives in registers; there is no `next()` call, no intermediate tuple copy, and the compiler can inline the predicate as real arithmetic rather than an interpreted node. Secondary wins: constants (the literal in a predicate, the number of columns, type widths, null-ability) are known at code-generation time, so branches vanish and loops specialize. ## What it costs 1. **Compile latency.** Generating and optimizing code takes from a fraction of a millisecond (bytecode) to tens or hundreds of milliseconds (full LLVM optimization on a large plan). A query that runs in 2 ms is now slower. Mitigations: plan/code caching keyed by plan shape, tiered compilation, and *adaptive execution* — start interpreting immediately, compile in the background, and switch mid-query when the compiled version is ready. 2. **Debuggability and observability.** Stack traces no longer mirror the plan. Per-operator row counts and timings, which operators give away for free in an interpreted engine, need explicit counters injected into generated code — and injecting them can undo the fusion you compiled for. 3. **Engineering weight.** You are shipping a compiler: code generation for every operator and every type combination, a runtime toolchain dependency, memory management for generated code, security review of a runtime code-generation path, and platform portability. 4. **Compile-time correctness risk.** Bugs move from a readable interpreter into generated code, where reproduction is harder. ## What vectorization costs by comparison Vectorization needs a primitive per operation × type, which is a lot of code but *template-generated, precompiled, debuggable, profileable* code. No warm-up, no toolchain at runtime, plan changes cost nothing. Its ceiling is that intermediates still travel through cache-resident vectors instead of registers, and the dispatch between primitives, while amortized, is not zero. ## How the performance actually compares Head-to-head studies of the two designs on the same optimizer land in the same league, with a workload-dependent split: compilation ahead on **compute-intensive** queries with heavy expression evaluation and tight joins; vectorization ahead or equal on **memory-bound scan-heavy** work, where both designs stall on memory bandwidth and the extra register residency buys nothing, and on short queries where compile time cannot amortize. Neither is a magnitude-level winner over the other; both are a magnitude over naive row-at-a-time interpretation. ## A decision framework I would use - **Query duration distribution.** If the p50 query runs in single-digit milliseconds, compilation must be adaptive or cached or it is a regression. If most queries run for seconds, compile cost disappears into the noise. - **Compute versus bandwidth.** Profile a representative workload. If cycles go into expression evaluation, hashing and comparisons, compilation has room. If you are saturating memory bandwidth on scans, vectorize and spend the effort on compression, late materialization and pruning instead. - **Team and lifecycle.** A runtime compiler is a permanent tax on every future operator. Small teams should vectorize. - **Operability requirements.** If you must ship per-operator runtime statistics for user-facing plan diagnostics, factor in the instrumentation cost of compiled pipelines. - **Existing substrate.** Compilation strongly prefers a push-based executor (one loop per pipeline); retrofitting it onto a deeply pull-based engine is a rewrite of the executor, not an addition. ## The pragmatic sequencing Most engines that got here started with a Volcano interpreter, moved to vectorized batches (biggest win per unit of effort, no latency regression, keeps debuggability), and only then added compilation — usually selectively, for expressions first (compiling just the predicate/projection evaluator is a fraction of the work with much of the benefit) and whole pipelines later, behind an adaptive threshold. Framing it as "vectorize by default, compile where measurement proves it pays, and keep an interpreted fallback path forever" is the answer that survives follow-up questions.
- How can an engine get compilation's benefit without paying compile latency on short queries?Adaptive or tiered execution: begin executing with the interpreted or vectorized path immediately, compile the pipeline on a background thread, and switch over at a batch boundary once the compiled code is ready. Short queries finish before the switch and never pay; long queries pay only overlapped time. Caching generated code keyed by plan shape or parameterized statement also removes the cost for repeated queries.
- Why does compilation pair naturally with a push-based executor?Fusion requires one uninterrupted loop over a pipeline so the tuple can stay in registers. Push gives exactly that: the producer owns the loop and each operator contributes inlined body code through consume(). A pull executor returns control after every tuple, which forces state back out of registers into operator objects and reintroduces the per-tuple control transfer that compilation exists to remove.
saying these in an interview costs you the question
- Claiming compiled execution is universally faster; it loses on short queries and is roughly a tie on memory-bandwidth-bound scans.
- Ignoring compile latency entirely, or assuming it is always microseconds.
- Treating compilation and vectorization as mutually exclusive — engines mix them, e.g. compiling expression evaluation inside a vectorized executor.
- Forgetting that per-operator runtime statistics and profiling get materially harder in compiled pipelines.
- Saying compilation removes blocking operators or spills — pipeline breakers still cut the plan into pipelines.