Orchestration Concepts
The tool-agnostic ideas every scheduler implements: dependency graphs, data intervals, idempotent reruns, retries and SLAs, and lineage. Interviewers lean on these when they want to know whether you understand pipelines or only one vendor's UI, and the answers transfer directly between Airflow, Prefect, Dagster and managed services.
on this pageshowhide
explore
- DAGs and Dependency Semantics6 questions
- Scheduling and Data Intervals6 questions
- Idempotency, Backfills and Catchup6 questions
- Retries, SLAs and Alerting6 questions
- Lineage and Pipeline Observability6 questions
- ETL vs ELT and Transformation Placement6 questions
questions
page 2 of 2How would you build end-to-end lineage across a platform where several different tools transform data?
basics
~20 sPick one interchange format and one metadata store, then feed it from three sources: runtime events emitted by each tool, SQL or query-log parsing where no integration exists, and manual edges for opaque hops. The hard part is agreeing dataset naming so the pieces connect.
How do you decide whether a downstream pipeline runs on a clock schedule or on upstream data readiness?
basics
~20 sUse a clock schedule only when the downstream can tolerate acting on whatever data exists at that moment. If correctness depends on the upstream being complete, trigger on an explicit readiness signal instead, because a guessed time offset fails silently when the upstream is late.
In the OpenLineage standard, what does a run event carry and what are facets used for?
basics
~20 sAn OpenLineage run event ties three entities together — a job, one run of it, and the input and output datasets — plus an event type such as START, COMPLETE or FAIL and a timestamp. Facets are optional typed blocks that attach extra metadata to any of them.
When should a pipeline generate its tasks dynamically at runtime instead of declaring a fixed graph?
basics
~20 sOnly when the set of work units is genuinely unknown until the run starts and each unit deserves its own retry and visibility — for example one task per tenant discovered at run time. Otherwise a fixed graph is cheaper to read, compare and operate.
What happens to a pipeline scheduled in a local timezone when that zone's daylight-saving change occurs?
basics
~20 sOne local wall-clock hour disappears in spring and one repeats in autumn, so a schedule inside those hours may be skipped or fire twice, and that day's nominal daily interval is 23 or 25 hours instead of 24.
When a shared upstream pipeline misses its SLA, should downstream pipelines block or run on stale data?
basics
~20 sIt depends on whether the consumer is harmed more by an old answer or a wrong one. Block where output is published externally or joins would mix vintages; proceed with an explicit staleness marker where consumers can tolerate age and see it.
showing 31–36 of 36