An irrigation controller's long-poll feed drops some events and repeats others under load; what happens between each response and the next request?

This is the failure that separates people who have read about long polling from people who have run it. It only appears under load, because the gap only matters when something is published inside it. ## The gap is structural One cycle looks like this: the server writes a response, the client reads and processes it, the client issues the next request, the server parks it. Between the first step and the last, **no request of that client is open on the server**. Nothing is listening. The gap is small — a round trip plus whatever the client does with the response — but it exists on every cycle, and at a hundred events a minute something will land in it. This is the concrete difference between long polling and a transport that is genuinely held open: not that long polling is slower, but that it has moments of not being there at all. ## The two failure shapes | symptom | mechanism | what it looks like | |---|---|---| | events missing | the server retained nothing, so a publish during the gap had no reader and no store | gaps that correlate with bursts, worse on slow clients, invisible in the server's own logs because the publish succeeded | | events repeated | the next request asked for everything since a timestamp, and events at that instant qualify again | duplicates clustered at burst boundaries, the same event arriving on two consecutive cycles | Both are silent. The publish path reports success in each case, and only the consumer knows. ## What the server must remember 1. **Assign a monotonic position per channel.** A counter the server controls, increasing on every publish. Not a timestamp. 2. **Return the next position with every response**, including the empty one at hold expiry, so the client always has an exact place to resume from. 3. **Retain a bounded window of published events per channel** — by count, by age, or both — so events published in a gap are still there when the next request arrives. 4. **Answer with everything after the position supplied**, batched into one response, rather than one event per cycle. 5. **Park the request only when the supplied position is already at the head.** If the client is behind, there is nothing to wait for; answer immediately. 6. **Say so explicitly when the supplied position has fallen out of the retained window**, instead of returning the oldest event you still have and implying continuity that is not there. The client can then refetch current state knowingly. ## Why wall-clock time is the wrong key Timestamps look like positions and fail as positions: - **Ties.** Several events can share a millisecond, and then an inclusive boundary repeats them all while an exclusive one skips them all. - **Skew.** If more than one process stamps events, their clocks disagree, and an event can be stamped earlier than one already delivered. - **Non-monotonicity.** Clocks are adjusted, and an adjustment backwards means a later event sorts before an earlier one. A per-channel counter has none of these properties, and it also gives the client something it can log: the exact position it had before a burst went wrong. ## What the client must do - Send back exactly the position the last response returned — not a position it computed itself from the events it saw. - Treat handlers as **idempotent** anyway. Retention windows, redeliveries after a failed write and restarts all make at-least-once the honest expectation, and the cheapest defence is a handler that can process the same position twice without harm. - Keep the position across a restart if the device is expected to resume rather than re-sync, and be explicit about which of those two it is doing. ## The cost of the window Retention is not free, and it is the tuning knob this design gives you. - Too short a window and slow clients fall out of it during a burst, which converts a silent drop into an explicit re-sync — better, but still a cost. - Too long a window and you are storing a growing history for clients that may never come back. - The window only needs to cover the worst realistic gap plus whatever outage you intend to survive, and that is a number you can measure rather than guess: it is the distribution of time between a client's response and its next request. The summary worth saying out loud in an interview: long polling gets you low latency without a held connection, and in exchange it makes delivery the server's bookkeeping problem, because there are moments in every cycle when nobody is listening.