When should an RNN carry its hidden state across mini-batches on a never-ending accelerometer stream?

## The default, and why a stream breaks it Normally each training example is a self-contained sequence: you start from `h_0` (zeros), run the recurrence over the example, compute a loss, and throw the state away. Examples can be shuffled freely because none of them depends on another. A continuous sensor stream has no example boundaries. A wrist accelerometer for fall detection samples all day; a conveyor's vibration telemetry runs until the line stops. You still have to train on finite chunks, so you cut the stream into segments — and every cut throws away the context that preceded it. If a segment is 200 steps and the useful context is 2,000, most segments start blind. **Carrying state** (often called running the layer statefully) is the fix: the final hidden state of segment k becomes the initial state of segment k+1, so context flows across the cut even though the loss is computed chunk by chunk. ## The contract you must honour Carrying state is only meaningful if the next batch really is the continuation of the last one, *row by row*. With a batch of B rows, the setup is: 1. Slice the long stream into **B contiguous sub-streams** (or use B different devices/sessions, one per row). 2. Cut each sub-stream into consecutive segments of equal length. 3. Build batch n from the n-th segment of every sub-stream, and feed batches **in order**. 4. After each batch, keep the final state and use it as the next batch's initial state — **for the same row index**. Break any of these and you are initializing a sequence with a state from unrelated data. The most common way to break it is to shuffle: shuffling is the standard defence against correlated gradients, and carrying state forbids it. That cost is real — consecutive batches are highly correlated, so gradient noise is less independent than usual, and you often compensate with a smaller learning rate or more aggressive averaging. ## Stop the gradient at the handoff If you carry the state as a live part of the computation graph, the graph from batch 1 is still attached at batch 500, and memory grows until the job dies. You carry the state's **values** and cut its gradient path at the boundary, so each batch's backward pass stops at the state it was handed. The state then propagates information forward indefinitely while gradients travel only within a batch. ## The reset policy is the design work "When do I zero the state?" has no default answer on a stream, and getting it wrong is a silent data bug. Reset when the stream is genuinely discontinuous: - the device was removed, powered off, or rebooted; - there is a gap between samples larger than the sampling interval allows; - you move to a different subject, device, or physical asset; - a sub-stream reaches its end and a new one is assigned to that row. Do **not** reset merely because an epoch ended, if the stream itself did not end — and conversely, do not carry a state across a six-hour gap just because the two chunks happen to be adjacent in your file. A stale state is often worse than zeros: it asserts a context that no longer holds. ## Serving looks the same, with a state store Online scoring has the same structure. The service keeps the latest hidden state **per stream**, keyed by device id, updates it as each window of samples arrives, and applies the same reset rules. This is a genuine piece of state infrastructure: it must survive restarts (or be rebuilt), it must not be shared across devices, and it must expire — a state loaded after a long silence should be discarded rather than trusted. Train/serve skew here is nasty and invisible: if training resets at chunk boundaries and serving never does, the model sees state distributions it never trained on. ## The cheaper alternative Often you can avoid carrying state entirely by making segments **overlap**. Take segments with a warm-up prefix — the first K steps rebuild context from real data and their outputs are excluded from the loss and from evaluation. You pay redundant compute on the prefix, but you get back shuffling, restartability, and trivial evaluation. Reach for carried state when the required context is far longer than any segment you can afford to materialize; reach for warm-up prefixes when the context is merely longer than the labelled part. ## How to answer Say when it is needed (context outlives any affordable segment on an unbounded stream), state the row-continuity contract and the loss of shuffling, mention detaching the state, and then spend your last sentences on the reset policy and the per-device state at serving — that is the part that separates someone who has run this in production from someone who has read about it.