Cloud control-plane audit events arrive minutes after the API call - how does that shape a live tune-and-retest session?

## What the lag is Delivery lag is the interval between an API call being made against a cloud control plane and the corresponding audit record becoming searchable in the detection platform. It is not one hop: the provider writes the event into its own audit store on its own cadence, a forwarder or subscription moves it, and the platform indexes it. Each stage batches. The result is a lag measured in minutes that is **variable rather than fixed** - it has a tail, and the tail is where session mistakes live. ## Why it sets the tempo One pass of the loop costs: execute, wait out delivery lag, wait for the rule to run on its schedule, read the result, edit, repeat. If lag runs to roughly ten minutes and a scheduled rule fires every five, a serial one-edit-per-pass loop buys you three or four passes in an hour, and the operator spends most of that hour idle. That arithmetic, not anyone's skill, is what limits how much a session can close. Three practical responses: - **Batch the edits.** Do not spend a pass on a single-character change. Between passes, list every hypothesis you can test at once and encode all of them. - **Pipeline the executions.** Have the operator run several variants of the technique back to back, keeping a written execution log - time, account, identity, technique, which variant. Then observe them together. Execution time is the scarce resource because the operator leaves; queries can be re-run over stored records afterwards. - **Agree a wait threshold before you start.** Not a guess: fire a harmless, uniquely identifiable control-plane call from the operator's identity at the top of the session and time how long it takes to become searchable. That measured number is what everyone in the room commits to waiting before anyone says the word miss. ## The dangerous inference The expensive error is concluding *the rule did not fire, therefore the rule is wrong* two minutes after execution. Absence of an alert has at least three causes: the record has not arrived, the record arrived but the scheduled run has not covered it yet, or the record arrived and the logic failed to match. Only the third is a rule defect, and teams under audience pressure routinely rewrite good logic to fix the first. The discipline is to check the layers in order. Query the raw audit record first - by API action, calling identity and time window. If it is absent, you are waiting on the pipeline. If it is present, check whether the rule has actually run over the window containing that event time. Only then is a non-firing rule a real miss, and only then is editing the logic the right move. ## Event time versus arrival time This is the part that outlives the session. Correlation logic must key on **event time** - the timestamp the control plane recorded for the call - while scheduling and lookback must be sized against **arrival time**. A rule that requires the key-creation record and the policy-attachment record within a short window of each other is fine if the window is over event times; the same rule silently misses in production if the platform evaluates over a stream ordered by arrival, or if the scheduled lookback is shorter than lag plus its tail so late-arriving records fall between successive runs and are never evaluated by any of them. A session is exactly where that gets found, because a session is the one place you watch a record arrive with a stopwatch. Note the observed lag in the session record next to the rule version; the next person to touch the schedule needs it. ## What the tempo costs each side The operator pays in idle time and loses tradecraft variety - they came with a list of techniques and the loop eats their hour. The engineer pays in shipping logic under an audience, which pushes toward whatever change makes the screen go green soonest. Naming both costs at the start, and agreeing what will be finished in the room versus carried out of it, is most of what separates a session that closes gaps from one that produces a demo.