How would you build a workload model from real production traffic and keep it representative over time?

### A model, not a script A performance run is a simulation, and its credibility rests entirely on how well the simulated demand resembles real demand. The deliverable is therefore not a script but a **workload model**: an explicit, reviewable description of what the system is asked to do, from which scripts are generated. Writing it down separately is what makes it arguable — a stakeholder can dispute a mix table, but nobody meaningfully reviews a folder of recorded scripts. ### What the model has to contain **Transaction mix.** The relative frequency of each business operation. Derived from counts over a stated period, not from a workshop. Watch for the trap where the most-discussed operation is rare and a boring one dominates: the ratio between a cheap read and an expensive write often decides everything about the result. **Arrival shape.** How requests arrive, not merely how many: the daily and weekly profile, the burstiness within an hour, and whether arrivals are independent or clustered into sessions. A flat rate equal to the daily mean can be an order of magnitude gentler than the busiest ten minutes. **Session structure.** Which operations occur together and in what order, since a realistic sequence exercises caches, sessions and locks the way one-shot requests never will. **Think time and pacing.** Drawn from the observed distribution of gaps between actions in the same session, with spread. Constant pauses make a simulated population march in lockstep and produce artificial waves. **Data profile.** The distribution of the entities the requests touch. Real data is skewed — a few accounts enormous, most tiny; a few products hot, most cold. Uniform synthetic data hides both the pathological large case and the cache behaviour that real skew produces. **Environmental context.** Concurrent batch work, scheduled jobs, and the state of the datastore, since a run against an empty datastore is not a run against production behaviour. ### Choosing the reference period, and saying which Every model implies a moment in time. A representative busy weekday supports the routine question; a seasonal or campaign peak supports the readiness question; a reconstructed incident supports the regression question. These are different runs with different pass criteria, and the mistake is not choosing one — it is failing to state which one a green result refers to. Anonymise or synthesise the data behind any model derived from real traffic; a workload model must never become a route by which production personal data ends up in a test environment. ### Keeping it alive Models rot. Features ship, a channel takes off, a client library starts batching, an integration partner triples its calls. A mix that was right eighteen months ago will pass runs while the real risk moves elsewhere. Practical countermeasures: - **Own it.** A named owner per model, in the same review path as the service it exercises. - **Re-derive on a cadence.** Recompute from fresh traffic on a schedule, and diff against the current model. A large diff is itself a finding worth discussing. - **Tie it to change.** A new user-facing operation is not done until the model accounts for it — the same discipline as updating a contract or a dashboard. - **Version it.** Store the model with the results it produced, so a comparison across releases can prove that the workload, not just the code, was constant. - **Report drift as invalidity.** A run whose model no longer matches observed traffic is reported as stale, not as passing. ### Pass criteria that survive contact with stakeholders Each run shape needs its own criterion, agreed in advance and expressed in business terms: a percentile of user-visible latency for a named operation and an error-rate limit under the expected mix; a defined *breaking mode* for a run pushed past the target — degraded but recovering, rather than data loss or a failure to recover; bounded drift over a long run for memory, latency and error rate; and a recovery-time expectation after a sudden surge. Criteria stated as a percentile and a limit are debatable and improvable. Criteria stated as *looks fine* are neither. ### A worked example An insurance quote engine has been exercised nightly for a year by a 6-hour run built from a launch-era model: 82% single-driver motor quotes, 18% multi-vehicle. Traffic has since shifted — a partner channel now submits multi-vehicle quotes with an average of 3.4 vehicles each, at 41% of volume, and each one fans out to several rating calls. The nightly run stays green while real evening latency degrades, because the suite is faithfully reproducing a product that no longer exists. Re-deriving the mix from a recent busy week, re-seeding the data pool with the observed distribution of vehicles per quote, and rebuilding the arrival shape from the real evening profile reproduces the degradation on the second run. The subsequent review also caught an off-by-one boundary in the data pool generator, which had capped the vehicles-per-quote field one below the maximum the product allows — so the single most expensive case in production had never once been exercised. Both defects have the same root: the workload was an artefact nobody owned.