Which costs does a per-embedding figure computed from the catalogue re-embedding workers' compute bill alone leave out?

## Two figures, two questions A per-prediction cost is not a single quantity. **Marginal cost** is what producing one more prediction adds to the bill right now. **Fully-loaded cost** is the total cost of having the capability at all, spread over what it produced. On a catalogue image-embedding refresh these differ by roughly an order of magnitude, and an engineer who can only produce one of them will misprice at least one decision. The marginal figure from the worked setting: the bulk re-embedding workers consumed 400 accelerator-hours at a given $3 an hour, producing 20 million embeddings, so `$1,200 / 20,000,000 = $0.00006` each. ## The roll-up to fully loaded | line | monthly | per embedding | why it belongs | |---|---|---|---| | reserved accelerator fleet (1,440 hours) | $4,320 | $0.000216 | billed whether the job runs or not; only 400 hours were consumed | | amortised training run ($90,000 over 18 months) | $5,000 | $0.00025 | no embeddings exist without it | | vector storage and index writes | $1,600 | $0.00008 | every embedding produced is stored and indexed | | orchestration and monitoring | $1,080 | $0.000054 | the job does not run itself | | **total** | **$12,000** | **$0.0006** | ten times the marginal figure | The largest single line is not the compute. It is the **amortised training run**, followed by the **idle share of the reservation** - $4,320 billed against $1,200 consumed. That ordering is typical of a bulk embedding workload and it is why quoting only the compute understates so badly. ## Why the reservation counts even when idle A reservation buys hours, not work. If two accelerators are held for the month, 1,440 accelerator-hours are paid for; the job used 400. The remaining 1,040 hours are a cost of *having the capability available on this cadence*, and they belong in the fully-loaded figure exactly the way an empty seat belongs in an airline's cost per passenger. They are also the lever nobody pulls: the usual response to a cost review is to make the job cheaper, when the honest first move is to notice that two-thirds of the reserved hours produced nothing. ## Using the right one - **Marginal** - incremental decisions inside the current shape of the system: re-embed one more batch tonight, add a category to the rolling window, back-fill a supplier's listings. The fixed lines do not move, so including them argues against work that is genuinely almost free. - **Fully loaded** - decisions about the capability: keep it, cut its cadence, compare it against the margin it earns, or set an internal charge for another team that wants to use it. A figure quoted without its label is the defect. 'Six hundredths of a cent per embedding' sounds like a rounding error and will win an argument it should not; '$0.0006 fully loaded, of which the training run is $0.00025' invites the right next question. ## The denominator moves too Because four of the five lines are fixed within a month, the fully-loaded figure is **inversely sensitive to output**. Halve the embeddings produced without changing the reservation and the fully-loaded unit cost nearly doubles while the bill barely falls. That is not a paradox; it is the definition working correctly, and it is why any claim that an efficiency change reduced cost per prediction has to say which of the two figures moved, and whether the fixed lines were resized to match. ## What to say in the room Give the marginal number, label it, then give the fully-loaded number with its two or three largest lines named. Say which one you would use for the decision on the table. If you are asked for one number, give the fully-loaded one and its window: it is the one that cannot be gamed by pushing spend into a line you chose not to count.