How do sequence-to-label, per-step tagging and sequence-to-sequence framings of a recurrent model differ?

## The framing is about output shape, not about the cell A recurrent network reads a sequence one step at a time, updating a hidden state from the previous state and the current input, with the same weights reused at every step. That machinery is identical in all three framings below. What differs is **how many outputs you produce, which hidden states you read them from, and where the loss attaches**. Interviewers open here because a candidate who thinks a recurrent model always produces one label at the end can only build sentence classifiers. ### Sequence-to-label (many-to-one) Input: a sequence of T steps. Output: exactly one prediction for the whole sequence. The classification head is normally attached to the hidden state after the final step, on the reasoning that it has seen everything. A common and often stronger alternative is to compute an output at every step and **pool** them - mean or max over time - before the head. Pooling matters because a single final state is recency-biased: in a long input the early steps have been overwritten many times, and the gradient reaching them is weakest. There is exactly one loss term per example. Typical tasks: deciding whether a support chat needs escalation, classifying the intent of a spoken request, scoring a whole user session as risky or not. ### Per-step tagging (many-to-many, aligned) One output for every input step, in order, one-to-one. The alignment is fixed by the framing - output i belongs to input i - and the model never decides how long its output is. The loss is averaged or summed over the T steps, so every step contributes gradient and every step is supervised. The canonical example is BIO tagging for named entities over news sentences: each token is labelled `B-PER` for the beginning of a person name, `I-PER` for a continuation of one, `O` for outside any entity, with parallel tags for organisations and locations. Part-of-speech tagging, per-frame speech activity detection and per-event anomaly flags in a session all share this shape. A useful contrast: intent classification of exactly the same news sentence is sequence-to-label. Same sentence, same tokens, same recurrent cell - a completely different head, loss and label file. That is the whole point of the distinction. ### Sequence-to-sequence (many-to-many, unaligned) The output is a sequence whose length is not determined by the input length. One recurrence reads the input; a second stage produces the output step by step, each emitted step conditioned on the steps already emitted, until a designated stop symbol is produced. There is no correspondence between output position j and input position j, and the model must learn where to stop. Typical tasks: translation, summarisation, transliteration, generating a free-text explanation of an event. This is the most expensive framing to train and to evaluate. Length is now a model decision, so a whole class of failure - truncated or runaway outputs - exists that the other two framings simply cannot exhibit. ### How to choose Two questions settle it almost always. 1. **Where do the labels live?** If your annotators marked spans inside each example, you have per-step labels and a tagging task. If each example carries one label in one column, you have sequence-to-label. If the target is free-form text of a different length from the input, you need generation. 2. **When is the decision consumed?** A per-step framing gives you a decision you can act on as the sequence unfolds. A sequence-to-label framing gives you one decision, and only once the read-out point is reached. A third consideration is cost. Aligned framings are cheap: one forward pass yields every output. Generation is sequential at inference - you cannot produce step j+1 until you have produced step j - so its serving cost scales with output length in a way tagging does not. ### Common mistakes - **Assuming the output length always equals the input length.** True for tagging, false for the other two. - **Solving a tagging task by classifying each token independently.** That discards exactly the sequential context the recurrence exists to provide; a person name is `B-PER` or `I-PER` depending on what came before it. - **Reaching for generation on an aligned task.** If the output is one label per input step, generating it as free text buys length errors and slower serving for nothing. - **Reading a long sequence-to-label input from the final state alone** when pooling over steps is a one-line change with better behaviour. - **Thinking the framing changes the cell.** It changes the head, the loss and the label file.