ML System Design

The baseline is whatever already decides the number in production - here a lane-and-service-level median transit table refreshed weekly from recent actuals - read on live traffic. A comparator invented for the write-up, such as a network-wide mean, is not the bar.

One row must fix three things: the prediction unit (one account in one billing month), the horizon the label covers (cancels within the next 30 days), and the as-of timestamp that separates feature territory from label territory.

Reserve first what you cannot spend — the caller's network round trip, fixed request handling and explicit headroom — then allocate the remainder across feature lookup, inference and post-processing. The inference slice becomes a ceiling any candidate model must fit.

The offline sweep metric ranks candidate models during training - a ranking score over a frozen retrospective cohort. The online launch metric decides whether the service ships: what the deployed worklist changed in care, read on live shifts.

Shipping the rule first delivers the product immediately and starts the clock on the evidence a model needs: every promise logged with the inputs as they stood at decision time, joined later to the actual delivery. Without that log, a model has no honest training set and no measured bar.

A run record must pin every input and the yardstick: the dataset snapshot identifier, the code commit, the full parameter set, a content digest of the model artifact, and the evaluation set identifier with its exact metric definition. Two runs compare only when that evaluation set and metric are identical.

One verdict on a moderated post carries no error signal of its own. Three verdicts make disagreement visible, route the contested post into an adjudication path, and turn agreement into a running health check on the written guideline.

'Latest' is a moving pointer, not a version. Between the two runs utterances were appended, transcripts corrected and recordings withdrawn, so the second job trains on a different set. Rebuilding needs an immutable snapshot identifier that resolves to fixed bytes.

The negative sampling rate, recorded per stratum and per build. Thinning non-clicks a hundredfold multiplies the odds by about a hundred, and without that factor nobody can convert the model's scores back into real click probabilities.

The step appends. Scheduled steps run at least once, so a retry re-executed work whose rows were already written, and the partition now holds two copies. A step must replace its declared output, not add to it.

A feature read is a key lookup that returns whatever was last written, with no age check. The stale value is well-formed, so the model scores it and the wrong answer shows up only in the output.

Training and dispatch read the same feature with opposite access patterns: a training job scans months of one column across millions of rows, while dispatch fetches every feature for one driver in milliseconds. No single store serves both shapes well.

Event time is when the driving happened; ingestion time is when the trip summary reached the feature pipeline and became readable. Devices buffer and retry, so the two differ by hours or days and records arrive out of order.

A model validates nothing about what its inputs mean: any number in the right slot is scored. Training-serving skew therefore lands as a confidently wrong estimate rather than an exception, and shows up only when real outcomes arrive.

An embedding's coordinates only mean something relative to the encoder weights that produced them. The version stamp lets the serving path prove the vector came from the encoder the value model was trained against; without it, a mismatch is silent.

Batching amortises the accelerator's large fixed per-pass cost over many chunks, so throughput rises several-fold. It costs latency: the first chunk in a batch waits for the batch to fill or the window to expire before any scoring starts.

Feature fetch, at 38 ms against a 25 ms line. The measured total still fits only because the other three stages came in under theirs, and the 10 ms reserve is down to 2 ms.

They differ in when the scorer runs. A nightly job tags the stored library on a schedule, event tagging scores each photo as it arrives, and first-view tagging scores only photos somebody actually opens.

An upload id is unique to one upload, so every lookup misses. A content hash collapses the identical templates and boilerplate clauses that recur across customers onto one key, and repeats are the only thing a prediction cache can serve.

Fall back down a ladder of prediction sources: the personalised scorer, then a cached prediction for this prefix, then a global next-word frequency list, then a constant on-device default. Each rung gives up freshness, then personalisation, then the prefix context.

Everything the model saw and emitted is available immediately: per-feature summary statistics and null rates, the prediction-score histogram, the blocked rate at the current decision cutoff, and traffic volume - each cut by tenant, inbound path and language.

Recent cohorts are still right-censored: a failure resolves the moment it happens while survival is only confirmed at day 30, so dropping unresolved rows over-counts failures. Bucket the metric by prediction date, publish only matured cohorts, and restate as labels land.

It compresses every input the model uses into a single curve the system already produces, so a change in any feature - including ones nobody thought to chart - can show up as a changed shape, for the cost of one series and no labels.

No. The pager belongs to the served forecast error, the effect the operator feels. A feature whose distribution moved without moving error is a diagnostic: attach it to the quality page as context, and let it file a ticket at most.

A rolling reference re-estimates normal from data that already contains the drift, so each week's comparison is against last week's already-moved distribution. Gradual movement never accumulates in the statistic, though a step change still trips it.

Newer training data does not guarantee a better suggester, and a refresh can be worse without failing. The model already serving is the baseline every candidate must clear on a stated rule, so promotion is earned rather than automatic.

A completed play records what the listener tolerated in a context the service chose: autoplay, background listening and the slot given to the track all produce plays. It is a proxy label, not a verdict.

A live pricing model keeps applying the demand pattern it learned from old bookings, and nothing errors when the market moves. Refreshing refits it on recent bookings and cancellations so its rates track today's demand, not last quarter's.

Demoted mail is rarely seen, so it is rarely opened, so the next training set holds almost no positives for that sender class. Each refresh learns a lower score for it, which suppresses exposure further, and the loop tightens.

One impression record per serving request: a request id the client echoes on every later event, each slot's track id, position and rendered flag, the serve-time score and any randomisation propensity, plus the model and feature-spec versions.

A model scored on the rows it was fitted on is being asked to recall, not predict, so that score improves with capacity and understates the errors it will make on the next order. Only rows the fit never saw estimate that.

Shadow mode compares mechanics, not outcomes. The incumbent's queue choice is what agents actually work, so no ticket is ever placed where the candidate said, and resolution time and reroute rate are never produced for the disagreements.

Team-draft interleaving fills the strip by alternating drafts: a coin flip picks who starts, then each policy takes its highest-ranked listing not already placed. Every slot carries a hidden team tag, and a click credits the team that drafted that slot.

Keep the candidate off the critical path. Answer the live request from the incumbent first, hand a copy to a bounded background worker with its own pool and timeout, and let shadow failures only increment a counter.

Because the ramp assignment was not sticky. The same user fell on the candidate side in one planning run and on the incumbent side in another, so both sides queued a send for that day. Deterministic per-user assignment with one claimed send key prevents it.

Scoring every posting is impossible inside the rail's budget: four million postings at ten microseconds each is forty seconds of CPU per request. Retrieval trades exhaustive coverage for a few hundred plausible candidates found in milliseconds.

A typed query is a retrieval constraint applied before any model runs, so the ranker only sees items that already match the stated intent. A browse feed has no such constraint and must manufacture candidates from the viewer's profile and context.

A signal test on the request, not the account's age. Each rung declares a precondition; the funnel serves the first rung whose precondition holds, re-checked on every request, and logs which rung answered.

Cluster the near-duplicates first, then select greedily. Content similarity over title and lead text groups the forty filings into one event cluster; the list-editing pass then picks one representative and, using maximal marginal relevance, scores each further slot on relevance minus similarity to what is already placed.

Inside retrieval, the shortlist comes back full of eligible items. Filtering afterwards spends the shortlist on ineligible ones, so a selective facet leaves too few results to fill the page and wastes the scoring stage's budget.

A per-device velocity counter is a running count of that device's recent login attempts, held in a low-latency key-value store and folded forward by the login event stream, so the inline scorer reads it in one lookup instead of scanning history.

A single cutoff forces every borderline listing into an automatic allow or an automatic removal. The middle band is where the score is genuinely uncertain, so routing it to an analyst buys a cheap decision instead of a costly wrong one.

A settled dispute confirms fraud weeks after the checkout, so recent rows carry no dispute yet. Joining absence-of-dispute to a legitimate training label marks immature fraud as good and deflates the recent fraud rate.

A cascade makes a matching rule an absolute veto and hides the traffic it decided from the model. Rule hits as features make the rule one weighable piece of evidence the model can discount. One is policy, the other is signal.

A precedence order declared in advance - not source order, not whichever rule was authored last. The layer collects the matches, resolves them by that documented ranking, emits exactly one action, and records which branch won.

One hundred instances. Dividing the 12,000-per-second peak by 200 gives 60 instances running flat out; dividing again by the 0.6 utilisation target gives 100, whose 20,000-per-second ceiling leaves peak sitting at 60%.

Six hundredths of a cent. 400 hours at $3 is $1,200 of compute, divided by 20 million embeddings gives $0.00006 each, or $0.06 per thousand. That figure covers marginal compute only.

Per-clip scoring grows with upload volume; the quarterly training run does not. Training is paid once per model version, while scoring is paid again for every clip that arrives, so only the serving half is multiplied by traffic.

Divide the fixed monthly spend of running your own by the per-document saving it earns: crossover volume = fixed monthly cost / (hosted price per document - your own marginal cost per document). Below that volume, buying is the cheaper of the two recurring bills.

The amortised training run, the reserved-but-idle share of the accelerator fleet, vector storage and index writes, and orchestration and monitoring. On the worked catalogue refresh they turn $0.00006 of marginal compute into about $0.0006 fully loaded.

A decision-log row records the decision identifier and timestamp, the snapshot of inputs the model was given, the model version and feature-definition version, the score returned, the threshold in force, and the action taken.

A code revision pins the training instructions, not the values they read: the applications the training query returns, the resolved library versions, the run-time configuration and the seed all moved. Rebuilding a model version means pinning those inputs too.

Every tenant draws on the same finite pool of workers, accelerator time and queue slots. The extra requests wait in the same line, so queueing delay rises for everyone as occupancy climbs. Nothing has to fail for this to happen.

The raw submitted figures survive, because they do not depend on your code. Keep them, keep the vector the model was actually fed as proof of what it saw, and stamp the feature-definition version linking the two.

A model version is a pin set, not a file: the artifact plus the preprocessing and feature-definition versions it reads, its decision threshold and its output contract. Roll back all of them together, by flipping one pointer to a retained, still-loadable version.