Why can a bidirectional recurrent layer not be used in a live captioning system?

## What a bidirectional layer actually is A plain recurrent layer reads a sequence in one direction, carrying a hidden state forward: the state at step t summarises steps 1 through t. A **bidirectional** layer runs two independent recurrences with separate weights over the same input - one left-to-right, one right-to-left - and forms the representation at step t by joining the two states at that position, most commonly by concatenating them. The forward half of that vector summarises everything up to and including t; the backward half summarises everything from t to the end. This is not the same as running one model on the reversed input and averaging two predictions. That would be an ensemble of two causal models, and each member would still only see one side at a time. In a bidirectional layer the two views are joined *before* the head, so every downstream computation sees both sides of every position at once. ## Why that is fatal for streaming Streaming means you must emit a decision for step t shortly after step t arrives, with a bounded delay. A bidirectional layer violates that in a way no engineering can hide: - The backward recurrence starts at the **last** step. It cannot produce the backward state at position 1 until it has consumed positions T, T-1, ..., 2. - Therefore no output at any position is available until the entire sequence has arrived. - The effective latency is the length of the input. For a captioning system that is unbounded - a speaker may talk for minutes. So the constraint is not "bidirectional is slow"; it is that the model's definition of the representation at step t **includes information that does not exist yet** at the moment you are required to answer. That is a causality violation, not a performance problem. ## The offline version of the same bug: reading the future you should not have The more dangerous form of this appears when the sequence *is* fully available at training and evaluation time but the future steps are contaminated. Consider labelling each event in a user session as fraudulent or not, with a bidirectional layer over the session. The representation at the fraudulent event now includes the events that came after it - the chargeback, the account lock, the support contact. The model learns to read the consequence rather than the cause. Offline metrics look outstanding, because validation is scored the same way: on complete sessions, with the aftermath present. In production the model is asked to score an event as it happens, with nothing after it, and it collapses. Nothing in a standard train/validation split catches this, because the leak is symmetric across the split. The check that catches it is not a metric - it is asking, for every input the model consumes, **was this available at the moment of the decision?** ## When bidirectionality is exactly right When the whole input genuinely exists before you must answer, and no step is contaminated by the outcome, bidirectional layers are usually a straight win: - Offline transcription of a recorded meeting, where the file is complete before processing begins. - Named-entity tagging or part-of-speech tagging over complete sentences, where the disambiguating word is frequently to the right of the token being tagged. - Whole-sequence classification, where the read-out concatenates the forward recurrence's state after the last step with the backward recurrence's state after *its* last step - which is position 1 of the input. The cost is roughly double the recurrent computation and parameters of the corresponding unidirectional layer. ## Buying back some right-hand context under a latency budget If accuracy suffers badly without right-hand context, you do not have to choose all or nothing. Process the stream in fixed-length chunks and confine the backward recurrence to within a chunk, or give each position a fixed lookahead of k future steps and emit its decision k steps late. The forward state can still be carried across chunk boundaries. This makes latency a **design parameter** - you pick k and pay exactly k steps of delay - instead of a function of how long the speaker talks. It is a real, deliberate trade, and stating it is what separates a middle answer from a senior one. ## What to say in an interview Name the two recurrences and the join. State that the backward pass begins at the end, so nothing is emitted until the input is complete. Say the latency is the input length, not a fixed budget. Then volunteer the offline trap - a bidirectional model can silently read information that will not exist at scoring time, and your own validation will not tell you.