An OpenCV loop on a live RTSP camera keeps showing frames seconds old — why, and how do you fix it?

## The mechanism A `cv2.VideoCapture` on an RTSP or device source sits on top of a backend (usually FFmpeg for network streams, V4L2 or DirectShow/AVFoundation for local cameras) that receives packets in real time. Frames arrive whether or not anyone is consuming them. `read()` is defined as "give me the next frame", and the backend obliges from its buffer. That contract is exactly what you want for a file, where frames have no wall-clock deadline. On a live source it means the capture is a producer running at a fixed rate and your loop is a consumer running at whatever rate it manages. If the consumer is slower, the difference accumulates in the buffer. Latency is then not constant but *growing* — a symptom people describe as "it was fine for the first minute". Eventually the buffer's own limit is reached and frames are dropped somewhere out of your control, or memory grows, depending on the backend. ## Confirming the diagnosis Before fixing anything, measure. Time the body of the loop with a monotonic clock and compare against the stream's nominal frame interval. If mean per-frame processing time exceeds 1/fps, you have the sustained deficit and no amount of buffer tuning will save you — buffer tricks change *which* frames you drop, they cannot make the loop faster. A second useful check is to overlay a wall clock on the source or point the camera at a running timer, which turns the latency into something you can literally read off the screen. Separate the two components of latency while you are at it. There is a fixed pipeline latency (camera encode, network, jitter buffer, decode) that is often 200-500 ms on RTSP and is not your loop's fault, and there is the growing backlog latency that is. Only the second one is fixable in your code. ## Remedy one: ask for a shallow buffer `cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)` requests that the backend keep at most one frame queued. It works on some backend/driver combinations and is silently ignored on others — `set` returns a boolean you can check, but even a True result is not a guarantee of behaviour. Treat it as free insurance rather than a solution. On FFmpeg-backed RTSP, tuning the transport (TCP versus UDP) and the backend's own buffering options via the `OPENCV_FFMPEG_CAPTURE_OPTIONS` environment variable matters more, though that is a deployment knob rather than an API one. ## Remedy two: grab and retrieve `read()` is a convenience for `grab()` followed by `retrieve()`. `grab()` advances the capture to the next frame without decoding it — cheap — while `retrieve()` does the decode and hands you the array. That split lets you deliberately skip: call `grab()` in a short loop to burn through the backlog, then `retrieve()` once for the frame you will process. This is the simplest single-threaded fix and is explicit about which frames you sacrifice. Its weakness is that you have to guess how many to burn. ## Remedy three: a reader thread with a latest-frame slot The robust pattern is to run capture on a daemon thread whose only job is `read()` in a tight loop, storing each frame into a single slot protected by a lock, overwriting whatever was there. The processing loop takes a copy of the slot whenever it is ready for work. The backlog is then discarded by construction: whatever the processing rate, it always gets the newest decoded frame, and latency stays at the fixed pipeline latency instead of growing. Return a `copy()` from the slot so the processing side is not reading an array the reader thread may overwrite mid-computation. This is worth doing even when the loop is fast enough, because it also absorbs jitter — a garbage-collection pause or a slow frame no longer costs you queue depth. ## Remedy four: make the loop fast enough All of the above manage the symptom by throwing frames away, which is often the correct engineering answer for monitoring workloads but is a real loss for anything that must not miss an event. If frames matter, reduce the work: process at a reduced resolution, run the expensive stage every Nth frame with a cheap stage in between, move the expensive stage to a worker with a bounded queue, or use hardware decode so the CPU is not spending its budget on decoding at all. ## Reconnection, the other half of the problem On a real deployment the same loop must survive the stream dropping. `read()` starts returning False, and a naive `while True` either spins at full speed on a dead capture or exits. Production loops detect a run of failed reads, `release()` the capture, back off, and reconstruct it. Bundling that with the reader thread is natural: the thread owns the capture's whole lifecycle, and the processing side only ever sees "latest frame or None".