Video & Audio Understanding

## Video is a token problem before it is a vision problem Models do not consume video as video. The pipeline, whether the provider runs it server-side or you run it yourself, decodes the file into a sequence of still frames, and each frame is tokenized exactly the way a photograph would be. An audio track, if included, is tokenized separately. Everything about cost, context pressure and latency follows from that. The governing arithmetic is simple and worth being able to do out loud: image tokens = duration_seconds x frames_per_second x tokens_per_frame ## Working the warehouse example Take a safety review of dock-door CCTV: 8 hours per door, so 28,800 seconds. Using representative native-video figures of about 258 tokens per frame at default resolution and about 66 in a low-resolution mode: - 1 fps, default resolution: 28,800 x 258 is roughly 7.43M tokens. That is several multiples of a 1M-token context window, so the request cannot even be made, let alone paid for. - 0.2 fps (one frame every five seconds), default resolution: 5,760 x 258 is roughly 1.49M tokens. Still over. - 0.2 fps, low resolution: 5,760 x 66 is roughly 380k tokens. This fits, with room for the prompt. - Audio, if you leave the track in at roughly 32 tokens per second, adds about 922k tokens on its own for the same 8 hours. On silent CCTV that is pure waste, which is why stripping or disabling audio is often the single biggest saving. The useful habit is doing this arithmetic before writing any code. It usually kills the naive design in one line. ## The three levers, and what each costs you **Sampling rate.** The dominant term. Halving it halves the bill. It also sets an event-duration floor: anything shorter than the sampling interval may fall entirely between two frames. A three-second near-miss is caught reliably at 1 fps, unreliably at 0.2 fps, and essentially never at 0.05 fps. Choose the rate from the shortest event you must detect, not from the budget. **Per-frame resolution.** Fewer tokens per frame means a coarser image. Coarse frames are fine for gross motion (is anyone in the aisle, is the door open) and useless for small or distant detail (a label, a hand position, a person at the far end of a bay). Lowering resolution is the cheapest lever when the question is about presence and movement rather than fine detail. **Segmentation.** Instead of one giant request, cut the footage into windows, analyse each independently, and combine the results. This is what makes hours of footage tractable at a rate that would otherwise not fit, at the price of losing cross-window context. ## Two common ways to get this wrong The first is confusing file size with token cost. Re-encoding a video at a lower bitrate makes the upload smaller and changes the token bill not at all, because tokens are a function of how many frames get sampled and at what pixel budget, not of how well the codec compressed them. The second is treating the context window as the only constraint. Even when a long request fits, you are paying for every frame on every call, and latency grows with the prompt. A design that fits but costs several dollars per review, repeated across hundreds of cameras and days, is not a viable design. ## What a good answer sounds like State the formula, put real numbers through it, name the rate and resolution you would choose, and then say what that choice gives up. The interviewer is testing whether you reason about the token bill as a first-class design constraint rather than discovering it after the first invoice, and whether you can tie a sampling decision back to the detection requirement it has to satisfy.