When would you schedule a Spark Structured Streaming query with Trigger.AvailableNow instead of running it continuously?

## The four triggers A Structured Streaming query's trigger decides when micro-batches are planned, and the choice is more consequential than it looks because it determines the *shape of the compute bill* as much as the latency. - **Default (no trigger specified).** Spark starts the next micro-batch as soon as the previous one finishes. Latency is as low as micro-batching allows; the cluster is busy continuously. - **`Trigger.ProcessingTime("30 seconds")`.** A batch is started on a fixed interval. If a batch overruns the interval the next one starts immediately after; if it finishes early Spark waits. This is the usual always-on setting because it makes batch size predictable. - **`Trigger.AvailableNow()`.** Process everything available at start, across as many micro-batches as the rate limits imply, then stop the query. Added in Spark 3.3. - **`Trigger.Continuous("1 second")`.** Experimental long-running task model with millisecond latency, at-least-once only, and support for a narrow set of map-like operations. Rarely the right answer in production. `Trigger.Once()` — process everything in exactly one batch and stop — was deprecated in Spark 3.4 in favour of `AvailableNow`. The reason is the failure mode: a single batch ignores `maxFilesPerTrigger` and `maxOffsetsPerTrigger`, so a query that has been paused for a week tries to plan one enormous batch and either takes hours or dies. `AvailableNow` fixes exactly that by keeping the rate limits and simply looping until the backlog at start time is exhausted. ## What scheduling actually buys you The pitch for `AvailableNow` is that you get streaming's *bookkeeping* with batch's *cost profile*. The bookkeeping is the valuable half. The query still writes its offset range to `offsets/N` before processing and its commit after, so a run that dies halfway is replayed over identical input on the next run. It still reads only what arrived since last time, with no `WHERE ingest_date = ...` predicate to get wrong and no manual high-water-mark table to maintain. Stateful operators still work, with state carried in the checkpoint from run to run. Exactly-once against an idempotent sink still holds. You are not giving any of that up by scheduling it — you are only choosing when the compute exists. The cost profile is the other half. An always-on query holds a cluster twenty-four hours a day whether or not data is arriving. Many pipelines feed dashboards refreshed hourly, or downstream tables consumed by a morning report. For those, running for ten minutes each hour is the same result at a fraction of the spend, and it comes with an operational bonus: there is a moment every hour when nothing is running, which makes deploys, config changes and library upgrades ordinary instead of delicate. ## When to stay always-on **Latency.** If a consumer needs results within seconds — fraud scoring, alerting, an operational feed — a scheduled run cannot deliver it, and no trigger interval on a scheduled job closes the gap. **Large state.** A scheduled run reloads the state store from the checkpoint at start. If the query holds tens of gigabytes of session or dedup state, that reload can dominate the run, and running every fifteen minutes means paying it four times an hour. Past a certain state size the always-on query is genuinely cheaper. **Watermark behaviour.** Watermarks advance from event times in data read, and they persist across runs in the checkpoint, so scheduled runs do not lose watermark progress. But append-mode output is still gated by the watermark, and a run that reads a small backlog may not advance it far enough to close windows — meaning output appears in bursts tied to arrival, not to your schedule. Reason about the event-time latency, not the wall-clock cadence. **Downstream expectations.** A topic that other teams consume as a live feed is a contract. Turning it into an hourly batch is a cross-team change, not a cost optimisation. ## The judgment, framed for a lead The honest way to make this call is to separate two questions that are usually conflated: *how fresh does the consumer need the data* and *how do we want to express the computation*. Structured Streaming with `AvailableNow` decouples them — it lets incremental, exactly-once, stateful processing be the default expression for everything, with freshness as a scheduling parameter you can turn up for the handful of pipelines that need it and down for the majority that do not. That is a better platform position than maintaining two codebases, one batch and one streaming, for the same logic and discovering they have drifted. The cost of that position is that every pipeline carries a checkpoint whose lifecycle you must manage — source retention long enough to cover the gap between runs, a documented procedure for rebuilding a checkpoint, and shuffle-partition sizing fixed at launch for stateful queries. Those are real obligations, and they are the ones worth naming when someone proposes making everything a scheduled streaming job. ## Migration note If you inherit jobs using `Trigger.Once`, moving them to `AvailableNow` is usually a one-line change that improves their worst case: the same semantics, but a backlog is drained in bounded batches instead of one unbounded one.