An LSTM loses a patient's ventilation flag long before step 400 of a vitals stream — how do you diagnose it?

## Frame the failure as a measurement problem A report that a model *forgets* something is a hypothesis, not a diagnosis. In a vitals stream sampled every five minutes, holding a flag such as *this patient is ventilated* across 400 readings means the fact must survive over thirty hours of input while the emitted state changes at every step. Before touching the architecture, establish which of three things is actually true: the memory is decaying, the gradient never linked the two events, or the model was never asked to. ## Step one: read the gates Instrument a forward pass on a sequence where the flag is set early and matters late, and record all three gate vectors at every step. Two patterns are worth looking for. **The carry regime.** A unit acting as a latch sits with its forget gate near 1 and its input gate near 0 for the whole stretch. Its cell value is flat while the hidden state around it churns, which is exactly the separation the output gate exists to provide. If one or two units look like this and the flag still gets lost, the memory is fine and the problem lies downstream in how that unit is read. **Collapse.** If no unit holds a high forget value over the stretch, the memory is leaking, and the arithmetic is brutal. A forget value of 0.9 retains `0.9^400`, roughly 5e-19 — the stored value is gone within a few dozen steps. Even 0.99 retains about 0.018 after 400 steps. Only values very close to one behave like a latch on this horizon, and the gap between 0.9 and 0.999 is invisible to the eye on a plot but is the whole difference between forgetting and remembering. A related pattern is a forget gate that starts high and drops sharply at a particular kind of event — a unit that has learned to clear itself on a signal that happens to co-occur with something irrelevant. That shows up as a cliff in the trace rather than a slow decay, and it points at the training data rather than the architecture. ## Step two: check the training setup, not the cell The most common real cause is not the architecture at all. **Truncated backward passes.** Long sequences are often trained by cutting the backward pass into windows. If the window is 50 steps and the dependency spans 400, the gradient linking the flag to its use never exists; the model is not failing to learn the dependency, it has never been shown it. Lengthen the window, or carry state forward across windows so the forward pass at least sees the history. **State resets.** If the state is zeroed at every chunk boundary, or sequences are shuffled and re-batched in a way that breaks continuity, the fact is erased by the data pipeline rather than by any gate. **The label does not need it.** If the target at late steps is predictable from local vitals alone, there is no gradient pressure to keep the flag at all, and the model is behaving correctly. Verify the dependency is real before demanding the model represent it. ## Step three: bias the starting point toward remembering With random initialisation every forget gate starts near the middle of its range, so a freshly initialised cell erases most of its memory at every step and long-range gradients are dead before training can teach it otherwise. The standard remedy is to initialise the forget-gate bias to a positive constant such as 1, putting the gate around 0.73 open at the start. The cell begins in a keep-leaning regime, gradients reach further back in the earliest updates, and the model can *learn* to forget from there — which is the easy direction — rather than having to discover remembering from a starting point that destroys the evidence for it. This is initialisation, not a constraint: nothing stops the model from learning small forget values where forgetting is right. ## Step four: consider changing the problem Senior judgment here often means not solving it in the cell. If a fact is known, categorical and persistent, carrying it in learned memory is an odd choice: feed it as an input feature repeated at every step, and the network no longer has to spend a unit and thirty hours of gate discipline preserving something you already know. Coarsening the timestep so the dependency spans forty steps rather than four hundred is the same move from the other side. Reserve learned memory for facts the model must infer. ## What a strong answer sounds like Measure first, and say what you would measure. Quote the decay arithmetic to show why *near* one is not one. Separate architecture causes from pipeline causes, and name the truncation check explicitly. Offer the forget-bias initialisation as a starting-point fix rather than a cure. Finish with the engineering option of supplying the fact directly. That sequence demonstrates having actually debugged one of these rather than having read about them.