Scalability & System Design

Multiply users by requests per user for total daily requests, divide by seconds in a day for average rate. Peak is higher because usage bunches at busy hours, so multiply average by 2-3x.

Never size servers off the average - size off the busiest moments plus extra room. Take the peak number, divide by what one server handles, then add a safety margin so a small spike doesn't cause an outage.

Add up what one upload creates (photo plus thumbnails), multiply by uploads per day, then by days you keep them, then multiply again by how many safety copies you store (replication). That total is your storage budget.

Memory is nanosecond-fast, an SSD read is a fraction of a millisecond, a same-datacenter round trip is about half a millisecond, and a round trip to another continent is over 100ms - a cross-region call will dominate response time.

Multiply people watching at once by data used per stream for the total pipe needed. Real traffic isn't evenly spread across servers, so this is a starting point, not the final answer - real measurements and margin are still needed.

A load balancer spreads incoming requests across many servers so no single one gets overloaded and the app stays up if one server dies. Round-robin just takes turns in order; least-connections sends the next request to whichever server currently has the fewest requests in flight.

An L4 load balancer only looks at IP addresses and TCP/UDP ports and just forwards packets, without knowing what's inside. An L7 load balancer reads the actual HTTP request (URL, headers, cookies) and can route based on that content, but doing so costs more CPU and adds latency.

Session affinity means the load balancer sends all of one client's requests to the same backend server, usually to keep server-stored session data (like a login session) working. It's typically done with a cookie identifying the server. If that server goes down, the client loses that session state unless it's shared elsewhere.

Active health checks are the load balancer regularly pinging each server with a test request to see if it's alive. Passive health checks watch real traffic and mark a server unhealthy if its actual responses start failing. Bad thresholds can either kick out healthy servers too fast or take too long to notice a broken one.

Plain modulo hashing (key % number-of-servers) reshuffles almost every key's assigned server whenever a server is added or removed, which is disastrous for caches. Consistent hashing arranges servers and keys on a ring so that adding or removing one server only reassigns a small fraction of keys, not nearly all of them.

Caching means saving a copy of data somewhere fast so you don't have to redo slow work every time. Different layers (CDN, reverse proxy, app memory, database) each keep a copy closer to where it's needed, so fewer requests reach the slow original source.

Cache penetration is traffic for keys that exist in neither the cache nor the database, so nothing ever gets cached and every repeat reaches the database. An ordinary miss loads the row once, and later requests hit.

When a cache runs out of space, it has to throw something away. LRU throws out the item nobody's touched in the longest time. LFU throws out the item used the fewest times overall. TTL just deletes items after a fixed amount of time, whether or not they're popular.

A sharded cache places whole keys, so all of one key's traffic goes to its single owner node. Adding nodes only moves whole keys around, so a key that needs more than one node's capacity stays overloaded wherever it lands.

Store a reserved tombstone value that readers can tell apart from a cache miss, write it only for a confirmed no-row result, and give it a short TTL so a later-created item appears quickly.

A relational database stores typed rows and answers flexible SQL queries with joins and multi-row transactions; a key-value store only gets and puts a value by its key. Choose key-value when every access is a lookup by a known key.

Vertical scaling means giving one database server more CPU/RAM/disk. Horizontal scaling means adding more database servers and spreading the work across them. Vertical scaling is simple but hits a hardware ceiling and stays a single point of failure.

In primary-replica setups, one server accepts writes and copies them to other servers that only serve reads. In multi-primary, several servers can all accept writes and have to sync changes with each other, which can cause conflicts.

Opening a database connection is slow and uses server memory, so apps keep a reusable stash of already-open connections (a pool) instead of creating a new one each time. If the pool is too small, requests wait in line; too big, the database runs out of resources and slows down or crashes.

Order ledger: relational. Sessions: in-memory key-value with expiry. Catalog: document or relational. Device metrics: time-series. Search box: a search engine's index fed from the source of truth. Media files: an object store, with metadata in the database.

Use fast, "good enough" data for low-stakes info like recently viewed items, and exact, up-to-date data for money like a payment balance. Keeping everything instantly perfect everywhere is slow, so you only pay that cost where being wrong actually hurts.

With R=1 and W=1, a write needs one replica and a read needs one replica — not necessarily the same one, so a read can miss the update. Overlapping quorums, like R=2 and W=2 of 3, fix it.

2PC locks both services until both agree to commit — all-or-nothing, but it freezes resources and can hang if the coordinator dies. A Saga commits each step and undoes earlier ones if a later step fails — faster, briefly inconsistent.

With LWW, only the "later" timestamped update survives — the other increment is thrown away, silently losing a count. A CRDT counter instead merges both by adding each replica's own tally, so no increment is lost.

If the network is broken, choose staying available with stale data or refusing requests to stay correct; if it's fine, choose fast-but-stale or slow-but-exact anyway. Favor correctness for a post, favor speed for a like count.

A queue sits between sender and receiver so the sender doesn't wait or fail when the receiver is busy. It stores messages temporarily and the receiver processes them at its own pace, smoothing out traffic bursts.

Backpressure tells the sender 'slow down' when the receiver can't keep up, instead of letting work pile up forever. It caps how much a producer can send, or makes it wait until there's room.

More consumer instances reading the same queue means more messages processed per second — like adding more cashiers. It stops helping once something else, like a shared database or the queue's own partition count, becomes the bottleneck instead.

Splitting order creation and points update into separate steps leaves a window where the order exists but points haven't been added — and if that update fails, nothing automatically undoes the order. You must handle that gap deliberately.

A queue only smooths a spike if the backlog eventually clears — it buys time, not processing power. If incoming work stays faster than the consumer even after the spike ends, the backlog never shrinks, regardless of queue size.

Rate limiting caps how many requests a client can send in a time window. Once they go over it, the server rejects the extra requests with HTTP 429 and tells them when to retry.

Fixed windows reset abruptly at a clock boundary, so a client can max out the limit right before the reset and again right after — nearly double the intended limit in a short burst. Sliding window looks at a rolling time range instead of a fixed clock tick, so it never lets that double-up happen.

Token bucket lets a client save up unused capacity and spend it in a burst. Leaky bucket smooths everything out to a fixed steady rate no matter how bursty the input is.

Two requests can both read the counter before either writes it back, so both see 'under limit' and both proceed — a race condition called a check-then-act bug. The fix is to make the check-and-increment a single atomic operation, e.g. Redis INCR or a Lua script.

Each limit protects a different thing over a different time scale, so each needs its own counter and window; a single number can't answer 'am I under my daily cap AND my per-second burst cap AND is the whole system under its global cap' at once.

Short segments let a player start after fetching a few seconds, switch bitrate at each segment boundary as bandwidth changes, and seek or retry cheaply, while HTTP servers and caches handle small uniform objects instead of one multi-gigabyte file.

A CDN stores copies of files on servers (PoPs) close to users everywhere. The first request has to fetch and cache the file from the real origin server, which is slow. Later requests are served from the nearby copy, which is fast because the data travels a much shorter distance.

The file's cache setting is probably telling browsers and the CDN to keep the old copy for too long, and reusing the same filename gives no signal that anything changed. Fix: give changed files new filenames whenever the content changes (like app.abc123.js) and cache those forever, while keeping the URL that points to the latest version very short-lived.

The player estimates available throughput from recent segment downloads, checks how many seconds are buffered, and picks the highest rung it can sustain with a safety margin, stepping down quickly when the buffer drains and up cautiously to avoid oscillation.

Anycast means many servers around the world share one IP address, and the internet's normal routing (BGP) automatically sends each user to whichever one is 'closest' by network path. The problem: if the network path changes mid-connection, a user can suddenly get routed to a different server than before, breaking anything that depended on talking to the same machine.

A correlation ID is a unique tag attached to a request when it first enters the system. Every service that touches the request writes that same tag into its logs, so later you can search all the logs for that tag and see everything that happened for that one request, even across many machines.

RED tracks Rate, Errors, and Duration of requests a service handles, good for things that answer requests, like an API. USE tracks Utilization, Saturation, and Errors of a resource, like a CPU or a disk, good for things that don't handle requests themselves but can still run out of capacity.

Each unit of work, like one service handling part of a request, creates a 'span' recording what it did and how long it took. All spans for one request share a trace ID, and each span points to its parent span, so a backend like Jaeger can stitch them into a single timeline showing the whole request's path and where time was spent.

Sampling means only keeping some traces, not all of them, to control cost. Head-based sampling decides at the very start of a request, before anything is known about it, usually just by rolling dice, like keep 1%. Tail-based sampling waits until the whole request is finished, then decides based on what actually happened, like keeping it if it was slow or errored.

Metrics systems create a separate stored time series for every unique combination of label values. Adding something like user_id, which has millions of possible values, multiplies the number of series by millions, which overwhelms the system. Logs don't have this problem because each log line is just recorded as its own event, with the user_id as a plain field you search later, not as a new stored series.

Each keystroke can send a request, so autocomplete sees several times the traffic of full search and must answer before the next key. Precomputing ranked suggestions per prefix turns each request into a cheap lookup.

A batch rebuild periodically re-reads the whole catalog and builds a fresh index, so results lag by up to the rebuild interval. Near-real-time indexing applies each change as it commits, giving seconds of lag but needing an always-running pipeline.

A coordinator sends the query to every shard in parallel. Each shard scores its own documents and returns only its local top 10. The coordinator merges those lists and keeps the best 10 overall.

The URL frontier holds every discovered but unfetched URL and decides what to fetch next and when: it orders URLs by priority and spaces out requests to each host, so the crawler stays polite and spends its budget on valuable pages.

A top-k trie gives lookups proportional to prefix length but costs memory; an edge-ngram index reuses search infrastructure and ranks at query time, costing index size and latency; an FST is compact and fast but immutable, rebuilt offline.

Each long-lived connection is pinned to one server for its whole life, so that server holds state: capacity is counted in open connections, pushes must be routed to the right node, and restarts disconnect everyone on it.

Divide the memory budget by per-connection memory to find a node's ceiling, check descriptor, port and CPU limits, then plan well below it: 10 million connections at 500,000 per node is 20 nodes plus spares.

A per-conversation sequence number, assigned by the server when it stores a message, gives every device the same order and makes a missing message visible as a gap. Sender timestamps suffer from clock skew and ties, and they never show that a message is missing.

The client keeps a cursor of the last acknowledged sequence and sends it on reconnect. The server first answers which conversations changed, then returns only the messages after each cursor. Offline sends are retried with client-generated IDs so the server can recognise duplicates.

Derive a deterministic idempotency key from the event id, recipient, channel and template, claim it in a send ledger with a unique constraint before calling the provider, skip keys already marked sent, and pass the same key to providers that accept one.

Each location ping is replaced by a newer one within seconds, so only the latest position matters. Keeping it in memory with a time-to-live makes writes cheap and removes silent drivers automatically once their updates stop.

A geohash turns latitude and longitude into one short string by interleaving their bits, so points in the same cell share a prefix. A prefix range scan on an ordinary sorted index then returns every point in that cell.

Write rate is drivers divided by interval, while staleness is speed times interval. One million drivers every 4 seconds means 250,000 writes per second, and a driver at 50 km/h may be about 56 m from the stored point.

Pick the finest precision whose cells are at least as large as the radius on their shorter side, scan the query's cell plus its eight neighbours, then keep only candidates within the exact distance.

Keep each driver's dispatch state in an authoritative store and move it from available to offered with a conditional write that only one matcher can win. Offers expire, and accepting is another conditional write checked against the offer id.

A full scan touches every stored job on every poll, though almost all of them are far in the future. An index ordered by due time lets a poller read only rows already due, so cost tracks due work rather than total backlog.

Each node's local scheduler fires on its own, so the job runs five times per tick and customers can be billed five times. The fire decision must be made once for the cluster, by an elected runner or a shared per-tick claim.

A job state machine is the fixed set of job states (queued, running, retry-wait, succeeded, dead) and the only transitions allowed between them. Per-attempt run records keep the job row small while preserving who ran each attempt and why it failed.

Make the claim one conditional write: move a job from pending to claimed only if it is still pending, and treat zero rows changed as losing. Reading first and updating afterwards lets both pollers see the same pending row.

An elected leader runs all schedules from one node: simple, but ticks can be missed or doubled around failover. A per-tick lock lets any live node win each tick with no handover, at the cost of every node attempting a claim every tick.

File contents are large, immutable blobs read whole, while names, versions and permissions are tiny records that change often and need transactions and indexed queries. Splitting them lets each store scale and be tuned for its own workload.

Chunking plus content addressing stores identical bytes once, lets an edited file re-upload only the chunks whose hashes changed, and turns each chunk's key into an integrity check on read.

A server-side change journal records every change in order with a monotonic position. Each device stores the last position it applied, so one small request returns exactly what it missed instead of scanning and diffing the whole file tree.

Proxying gigabytes pins app servers to slow byte streams for minutes. Instead the app hands out a short-lived signed upload URL, the phone sends bytes directly to the object store, and app servers handle only small control requests.

Fixed-size blocks cut at fixed offsets, so an insert shifts every later boundary and hash. Content-defined chunking cuts where a rolling hash of nearby bytes matches a pattern, so boundaries follow the content and realign after the edit.

One hundred instances. Dividing the 12,000-per-second peak by 200 gives 60 instances running flat out; dividing again by the 0.6 utilisation target gives 100, whose 20,000-per-second ceiling leaves peak sitting at 60%.

Six hundredths of a cent. 400 hours at $3 is $1,200 of compute, divided by 20 million embeddings gives $0.00006 each, or $0.06 per thousand. That figure covers marginal compute only.

Per-clip scoring grows with upload volume; the quarterly training run does not. Training is paid once per model version, while scoring is paid again for every clip that arrives, so only the serving half is multiplied by traffic.

A run record must pin every input and the yardstick: the dataset snapshot identifier, the code commit, the full parameter set, a content digest of the model artifact, and the evaluation set identifier with its exact metric definition. Two runs compare only when that evaluation set and metric are identical.

One verdict on a moderated post carries no error signal of its own. Three verdicts make disagreement visible, route the contested post into an adjudication path, and turn agreement into a running health check on the written guideline.