Cost & Capacity for ML

Six hundredths of a cent. 400 hours at $3 is $1,200 of compute, divided by 20 million embeddings gives $0.00006 each, or $0.06 per thousand. That figure covers marginal compute only.

Per-clip scoring grows with upload volume; the quarterly training run does not. Training is paid once per model version, while scoring is paid again for every clip that arrives, so only the serving half is multiplied by traffic.

Divide the fixed monthly spend of running your own by the per-document saving it earns: crossover volume = fixed monthly cost / (hosted price per document - your own marginal cost per document). Below that volume, buying is the cheaper of the two recurring bills.

The amortised training run, the reserved-but-idle share of the accelerator fleet, vector storage and index writes, and orchestration and monitoring. On the worked catalogue refresh they turn $0.00006 of marginal compute into about $0.0006 fully loaded.

Serving dominates, by roughly 36 to 1. At 40 ms per clip, 1.9 billion clips need about 21,600 accelerator-hours against the retrain's 600, and the two halves would only be equal at about 54 million clips per retrain interval.

Human maintenance time. Per-sender template rules cost almost nothing to run, so the recurring bill is patch work: templates held multiplied by how often each sender changes its layout. It grows with senders covered, not with documents processed.

Queue wait. Sizing for throughput only checks that arrivals stay under the ceiling; a reading's latency is queue wait plus service time, and mean wait scales with utilisation as u / (1 - u), so a fleet at saturation queues heavily while its throughput still looks fine.

Little's Law. Concurrency equals arrival rate times service time, so eight slots each held for 40 ms turn over 8 / 0.04 = 200 readings per second. The figure is arithmetic from a slot count and a service time, not a machine specification.

Everything the quoted rate hides: a throughput ceiling and a latency distribution you do not set, a model version that can change without any release of yours, and the integration, exception review and degraded path that stay yours whatever the price covers.

Roughly a five-minute backlog. A 20,000-per-second fleet offered 30,000 accumulates 10,000 readings a second, so a four-minute autoscaler lag builds 2.4 million queued readings that take minutes to drain - every one of them scored long after its operator window closed.

The amortised share rises 3.6x, from $0.00025 to $0.0009 per embedding, carrying the fully-loaded figure from $0.0006 to $0.00125. Worse, the successor cannot reuse the old vectors, so the whole catalogue must be re-embedded.

Marginal cost per embedding produced does not move: numerator and denominator fall together, so it stays $0.00006. Cost per listing kept current falls fivefold, and the fully-loaded figure per embedding actually rises.

A bulk re-scan is a third bucket: per-item like serving, but triggered by a policy decision rather than by traffic, and unbounded by any user-facing latency. At 40 ms per clip it costs about 44,400 accelerator-hours, roughly two quarters of live scoring in one pulse.

Because the serving half is paid as provisioned machine-hours, not as forward passes. Capacity held for the evening peak is billed through the quiet hours, so scoring fewer clips overnight consumes less compute without buying back any of the capacity already rented.

Exit cost, the asset you never accumulate, and where the documents are processed. A long commitment locks a rate you can compute and a coupling you cannot: downstream systems written against their fields, no labelled corpus of your own, and customer invoices leaving your boundary for three years.

Three things: extractions and reviewer corrections stored in your own schema so a corpus accrues while you buy, a boundary that keeps exit cost flat, and a written trigger with a date. Without them, build-later is a wish and exit cost grows quietly.

It over-states capacity. Service time rises under concurrency, so if the real figure at eight busy slots is 55 ms the instance sustains 145 readings per second rather than 200, and the fleet built for 60% utilisation is actually running at 83%.

Put it against what a refreshed embedding earns, on the same denominator. Taking the refresh's attributed margin as $30,000 a month over 20 million embeddings, each costs $0.0006 and returns $0.0015 - and that return is very unevenly spread.

The training half's share caps it. At roughly 2,400 of about 88,800 accelerator-hours a year, training is under 3% of compute, so dropping from four retrains to one saves at most about 2% — while the staleness it buys is paid on the serving side and outside the accelerator bill entirely.