A regional feed produces only between 22:00 and 06:00 and shares one job with always-on feeds. How would you design time advancement for both?

## Why this is a design question and not a tuning question The mechanism is settled: a job asserts a **completeness claim** — a timestamp carried with the records meaning *nothing older than this is still expected to arrive* — and it may only assert the minimum across its parallel inputs, so an input that delivers nothing pins everyone. What is *not* settled is who should be paying for that. A feed that is productive eight hours a day and silent sixteen is not a fault to be tuned away; it is a property of the business, and the decision is where in the architecture that property is allowed to land. The four placements, and what each one actually costs: | design | who pays | what it costs them | when it is right | |---|---|---|---| | isolate the quiet feed in its own job | the platform | a second job to run, deploy and watch, plus a downstream join | the feeds are genuinely independent and the joint number is assembled later | | producer emits periodic filler records | the producing team | a change in a system you may not own, and a contract that it keeps emitting | you control the producer, or can ask for it | | exclude on an idle timeout | the quiet feed's consumers | its records are late by construction every morning | its completeness matters less than everyone else's timeliness | | accept the stall | every consumer of the job | sixteen hours of held results each day | one joint number is required and lateness is unacceptable | ## The isolation argument The strongest version of this answer is that mixing a scheduled, bursty, low-traffic input with always-on inputs **in one time domain** is the original mistake. Time is a shared resource in a job: every input contributes to one asserted moment, and the contribution of a silent input is *stop*. Splitting the job gives each population the latency profile it deserves and removes the coupling permanently — no timeout to size, no producer change to negotiate, no lateness manufactured. Its price is real and should be said out loud: - two jobs to operate, deploy and watch instead of one; - two sets of retained state, sized and paid for separately; - a downstream join that must reconcile two time domains advancing at different speeds; - one more place where a schema or a contract change has to be applied twice. ## The filler-record argument The most complete fix, where it is available, is to stop the silence being silence. If the producer emits a small record at a fixed cadence carrying its current moment, the input has evidence to advance on even with nothing to report, so the minimum moves and nothing is ever excluded or made late. Three cautions: - It requires a change in a system the data team frequently does not own, and it becomes a standing contract — if the filler stops, the stall returns, and now silently. - The filler must carry a *moment*, not merely exist. A record with no usable timestamp advances nothing. - Downstream logic has to ignore it in the aggregates, which means everything that reads the feed has to know it exists. ## Why widening the wait is not one of the options A candidate often reaches for "just wait longer". The wait a job holds behind the newest moment it has seen is measured against moments that were actually supplied. A silent input supplies none, so no finite wait advances it — the claim would sit exactly where it is behind a wait of any length. Widening the wait costs every group on every input latency and buys nothing here. Say this explicitly; it is the cleanest demonstration that you understand the mechanism rather than the knob. ## What you owe the consumers Whichever design is chosen, the organisational half is the promise, and it has three parts: 1. **State what a published number means during the quiet window.** If the quiet feed is excluded, the overnight regional figure is provisional until that feed reports; consumers should be told that rather than discovering it. 2. **Say what happens to the records the design makes late** — discarded, sent to a separate channel for reconciliation, or folded into a corrected answer that replaces the published one. Any of the three can be right, and which the downstream sinks can absorb decides it. 3. **Make the silence visible.** A quiet input that is quiet because it failed looks identical to one that is quiet because it is night. Whatever design you pick, something must distinguish "expected silence" from "broken" — otherwise the design that protects the busy feeds also hides the outage of the quiet one. ## What varies between engines Not every runtime gives you all four options. Some expose a direct facility for excluding an idle input; on others the only levers are the producer and the job boundary. A runtime that runs continuous work as a succession of small finite jobs re-evaluates time at those boundaries, which sets the finest granularity any of this can have. And a pipeline that only ever makes a single pass over a finished bounded input has no running claim to stall — which is itself an argument worth making: if the quiet feed is genuinely nightly, the honest design may be to process it as a bounded input each morning rather than to model it as an endless one at all.