skip to content

When do you choose Lucas-Kanade over Farneback optical flow in OpenCV?

level: middleimportance: must knowfreq 55%

answer

  1. sparse points versus a full field
  2. status array is not optional
  3. both want grayscale input
  4. brightness constancy is the shared assumption
  5. pyramids buy you larger motion

basics

~20 s

cv2.calcOpticalFlowPyrLK is sparse: you give it specific points and it returns where each moved, plus a status array. cv2.calcOpticalFlowFarneback is dense: it returns a displacement vector for every pixel. Choose sparse when you track a few features cheaply, dense when you need motion everywhere.

solid answer

~50 s

They answer different questions. `cv2.calcOpticalFlowPyrLK(prevGray, nextGray, prevPts, None, ...)` is the sparse pyramidal Lucas-Kanade tracker: you hand it an Nx1x2 float32 array of points — usually from a corner detector — and it returns `(nextPts, status, err)`, where `status` marks which points it managed to follow. Cost scales with the number of points, so a few hundred features tracked per frame is nearly free, which is why it underpins visual odometry and feature-based stabilization. `cv2.calcOpticalFlowFarneback(prevGray, nextGray, None, 0.5, 3, 15, 3, 5, 1.2, 0)` instead returns an HxWx2 float32 field giving a displacement for every pixel — useful for segmenting motion regions, warping frames, or building a motion energy image, but orders of magnitude more expensive. Both take single-channel images, both assume brightness constancy and small displacement between frames, and both use an image pyramid so that motion larger than the window is still recoverable at a coarser level.

code

python · 21 lines
python
import cv2

cap = cv2.VideoCapture("input.mp4")
ok, first = cap.read()
prev = cv2.cvtColor(first, cv2.COLOR_BGR2GRAY)
pts = cv2.goodFeaturesToTrack(prev, maxCorners=200, qualityLevel=0.01, minDistance=10)

while True:
    ok, frame = cap.read()
    if not ok:
        break
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    nxt, status, err = cv2.calcOpticalFlowPyrLK(
        prev, gray, pts, None, winSize=(21, 21), maxLevel=3
    )
    pts = nxt[status.flatten() == 1].reshape(-1, 1, 2)
    if len(pts) < 50:
        pts = cv2.goodFeaturesToTrack(gray, 200, 0.01, 10)
    prev = gray

cap.release()

go deeper

for a junior

Know that one is sparse and one is dense, what each returns, and that both need grayscale input. Be able to name calcOpticalFlowPyrLK and calcOpticalFlowFarneback without hesitating.

for a middle

Explain the return shapes precisely — (nextPts, status, err) versus an HxWx2 float32 field — why status must be filtered, and how the pyramid lets either method survive motion larger than its window.

for a senior

Bring the failure modes from real footage: exposure changes faking global motion, point attrition needing re-seeding, the aperture problem in textureless regions, and downscaling as the standard way to afford dense flow.

for a principal

Own the build-versus-buy call on motion estimation: when classical flow is enough, when a learned flow model or a tracker-by-detection design is worth the dependency, and what the compute budget per camera implies for the fleet's hardware.

## What optical flow gives you Optical flow is the apparent displacement of brightness patterns between two consecutive frames. OpenCV exposes it in two shapes, and knowing which shape you want is most of the interview answer. ## Sparse: calcOpticalFlowPyrLK The signature is `cv2.calcOpticalFlowPyrLK(prevImg, nextImg, prevPts, nextPts, winSize=(21,21), maxLevel=3, criteria=...)` and it returns a three-tuple `(nextPts, status, err)`. - `prevImg` / `nextImg` are consecutive **grayscale** frames (8-bit single channel is the normal case). - `prevPts` is an Nx1x2 array of float32 coordinates. You do not get these from LK itself — they come from a corner detector such as `cv2.goodFeaturesToTrack`, or from wherever your pipeline already has points of interest. - `nextPts` is the estimated new location of each point. - `status` is an Nx1 uint8 array: 1 where the point was found, 0 where tracking failed. **Filtering by status is mandatory.** Points that leave the frame, get occluded, or land in a textureless region come back with garbage coordinates and status 0, and a loop that ignores status accumulates nonsense over a few hundred frames. - `err` holds a per-point error measure you can threshold as a second quality filter. Cost is proportional to the number of points, not the image area. Tracking 300 corners at 1080p is cheap enough for real time on a laptop CPU. ## Dense: calcOpticalFlowFarneback `cv2.calcOpticalFlowFarneback(prev, next, flow, pyr_scale, levels, winsize, iterations, poly_n, poly_sigma, flags)` returns a single array of shape (H, W, 2) and dtype float32: `flow[y, x]` is the (dx, dy) displacement of that pixel. Typical arguments are `0.5, 3, 15, 3, 5, 1.2, 0`. The algorithm fits local polynomial expansions of the image neighbourhood and solves for the displacement that maps one to the other, over an image pyramid. To visualize or threshold it you normally convert to polar form: `mag, ang = cv2.cartToPolar(flow[..., 0], flow[..., 1])`, then map angle to hue and magnitude to value in an HSV image. Cost scales with pixel count, and at full resolution it is far too slow for a 30 fps loop on modest hardware — downscaling the input before computing flow and scaling the vectors back is the standard mitigation. `cv2.DISOpticalFlow` is the faster dense alternative in the main build when Farneback cannot keep up. ## The assumptions both share **Brightness constancy.** Flow assumes a physical point keeps the same intensity between frames. A cloud crossing the sun, an auto-exposure adjustment, a fluorescent flicker, or an IR illuminator switching on violate this and produce large spurious flow across the whole frame — a classic production surprise on outdoor cameras. **Small motion.** The local solution is only valid for displacements within the analysis window. Both functions mitigate this with a pyramid: motion that is 40 pixels at full resolution is 5 pixels three levels up, where the solver is happy, and the estimate is refined downwards. `maxLevel` in LK and `levels`/`pyr_scale` in Farneback control this. Fast motion still defeats a shallow pyramid, and the symptom is flow that snaps to zero or to a neighbouring texture. **Spatial coherence / the aperture problem.** Inside a local window, only the motion component perpendicular to an edge is observable — a featureless edge sliding along itself looks static. This is why LK wants corners, not edges, and why textureless regions in a dense field are unreliable no matter how confident the numbers look. ## Choosing Ask what the consumer of the flow needs. If it needs *the motion of specific things* — track these corners for stabilization, estimate how the camera moved, follow these points on an object — sparse LK is right, and the sparse output is directly consumable. If it needs *motion everywhere* — which regions of the frame are moving, warp this frame to align with the last, produce a motion field for a downstream model — dense flow is the only option, and you pay for it. A practical middle ground worth mentioning: run dense flow on a downscaled frame for region-level decisions, and sparse LK at full resolution for the handful of points where precision matters. And note the maintenance detail for the sparse path — tracked points die off over time, so a long-running loop must periodically re-seed with a fresh corner detection rather than tracking the original set forever. ## Practical shape and dtype notes Most debugging here is shape and dtype. LK wants float32 points with shape (N, 1, 2) and will complain or misbehave with a plain (N, 2) int array. Both functions want single-channel input; passing a BGR frame is the most common error. And after filtering by status you must reshape back to (-1, 1, 2) before the next call, or the following iteration fails.

  • How do you keep a Lucas-Kanade tracker alive over thousands of frames?
    Points attrit — they leave the frame, get occluded, or drift into textureless regions — so you filter by the returned status and error every frame and periodically re-seed. A common pattern is to re-run corner detection whenever the surviving point count falls below a threshold, masking out regions where points already exist so the new ones spread out. A forward-backward consistency check, tracking to the next frame and back, discards drifting points early.
  • An outdoor camera produces huge dense flow across the whole frame for one frame, with nothing actually moving. What happened?
    A global brightness change broke the brightness-constancy assumption — auto-exposure or auto-gain stepping, a cloud crossing the sun, or an IR cut filter switching. Flow interprets the intensity change as motion. Mitigations are locking camera exposure and gain where the hardware allows, normalizing frame intensity before computing flow, and rejecting frames whose global flow magnitude spikes coherently.
  • Farneback at 1080p cannot hit 30 fps. What are your options before giving up on dense flow?
    Downscale the input, compute flow on the smaller pair, and scale the vectors back — flow cost scales with pixel count and region-level decisions rarely need full resolution. Reduce `levels`, `winsize` and `iterations`. Compute flow on every Nth frame pair rather than every frame. Or switch to `cv2.DISOpticalFlow`, which is designed for the speed-accuracy point Farneback misses.

saying these in an interview costs you the question

  • Ignoring the status array and using every returned point
  • Passing BGR frames instead of grayscale
  • Expecting LK to find points for you rather than tracking given ones
  • Assuming dense flow at full resolution can run in a real-time loop
  • Believing flow works under changing exposure or lighting

context

resolving links…