A presence feed re-scores 50,000 members a second in a two-million-member score-ordered set - where does that cost land?

## Where the cost actually goes Three separate costs are in play, and candidates who name only one usually name the wrong one. - **The round trip.** Fifty thousand events a second, sent one at a time, is fifty thousand round trips a second. That cost belongs to the access path and exists for any shape, but it dominates often enough that it must be excluded before blaming the shape. - **The ordered write.** This is the cost the shape adds. The store must determine that the member is already present and place it at the position its new score implies. The work involved grows with how many members the collection already holds rather than staying flat. - **The resident memory.** Two million members each carry bookkeeping beyond the member's own bytes, and this tier's scarce resource is memory. The re-scoring rate does not change this one - member count does. ## Why the comparison with an append is the useful one Appending to a two-ended sequence puts the new element at a known end. Nothing has to be located and nothing has to be positioned relative to what is already stored, so the cost does not care how long the sequence is. A score-ordered write does care. That difference - not an absolute number - is what you carry into a design discussion: - ordering is charged **per write**, not per read; - it is charged **even when no rank read ever happens**; - it is charged **again on every re-score**, because moving a member costs about what placing it did. A presence feed is the worst possible fit for that cost model: its entire traffic is re-scores of members that are already there. ## Spending less 1. **Coarsen the score.** If the ranking only needs `active in the last minute`, round the clock reading to a bucket - say five seconds - and skip the write when the bucket has not changed since the last one this instance sent. Most heartbeats then cost nothing at all, and the ordering quality the feature actually needs is unchanged. 2. **Batch the round trips.** Sending many writes in one round trip removes network cost but not ordering cost. Be explicit about which of the two you are attacking; batching a workload that is ordering-bound buys far less than it looks like it should. 3. **Split the collection.** One collection per region or per tenant gives several smaller sets, each of which is one unit of work on the server, at the price of merging results in the caller when a read spans them. Where those separate keys land across nodes is a distribution question, not a shape question. 4. **Drop the order.** If reads are only `is this member currently present` or `what is this member's last score`, a value of named fields or an unordered member collection answers them without maintaining any order, and the write becomes flat in cost. 5. **Stop writing what nobody reads.** A ranking maintained at full fidelity but read once a minute by one dashboard can often be derived from a sampled or bucketed source instead. ## Check the read pattern before the write path The shape only earns its price when a read genuinely needs position: - a **rank read** - where does this member stand; - a **range read** - give me an ordered slice without transferring everything; - a **count over a band** - how many members lie between two scores. If none of those is in the read path, the collection is paying for order that is never consumed. This is the most common finding when a re-scoring workload is investigated: the order was maintained for a report nobody built. ## What varies between stores - The **internal structure** that holds the order differs between stores, so the constant factor on a re-score differs too. What transfers is the shape of the curve: more than an append, growing with size. - Some stores in this class have **no ordered shape at all**. There the same feature means reading the whole collection, ordering it in the caller and writing it back - a read-modify-write round trip per event, which is far worse for this workload and comes with a lost-update risk under concurrent writers. - Whether the server runs one operation at a time or many in parallel also differs, which changes what a burst of ordered writes does to the latency other callers see. Do not assume either design. ## The premise that never goes away All of this is on a tier that can drop the entry under memory pressure or lose it on a restart, and eviction takes the whole collection rather than its stalest members. A two-million-member set is exactly the kind of entry whose loss is both likely to matter and expensive to rebuild - so the rebuild path is part of the design, not an afterthought.