Data Management

Save the big file to storage like S3, then send only a small pointer (its location or ID) through the queue. The receiver downloads the real file from storage when it needs it.

Because writing data and reading data back have different needs. CQRS keeps them as two separate models so each can be built, scaled, and hosted the way that suits it best, instead of one design compromising for both jobs.

Build a second, smaller table that maps the field you want to search by (email) to the ID of the matching row in the main table. Look there first, then fetch the real record. It costs extra storage and effort to keep both tables in sync.

A materialized view is a saved copy of a query's result, stored ahead of time. Instead of recalculating joins and aggregates on every request, the app just reads the pre-built copy, which is much faster but can be slightly out of date.

Sharding splits a big table's rows across several separate database servers so no single machine has to hold or serve all the data. Each server (shard) keeps a slice of rows, chosen by a shard key like user_id.

The server hands the client a special temporary link (like a pre-signed URL) that lets it talk straight to the storage service. That link only works for a short time and only for one specific action, like uploading one file.

Deleting the old cache entry is simpler and safer than trying to recompute and overwrite it. But there's a timing gap: another request could reload the old value into the cache right after the delete, right before the write finishes, leaving the cache wrong until it expires.

Producer saves the big report to storage first, gets back a location, then sends a small message with that location through the queue. Consumer reads the message, fetches the report from storage using the location, then processes it.

A separate background process reads new events one by one from the log, in order, and updates a query-friendly copy of the data (a projection) to match. That copy is what queries actually read from.

You can either write to both tables together as one atomic operation when the store allows it, or write to the base table first and update the index table afterward in the background. The first is safer but slower and not always possible; the second is faster but can leave the index briefly out of date.

On-demand refresh recalculates the view only when someone asks, so it's always fresh but slow the first time. Scheduled refresh updates it every so often (like every hour), which is simple but can be stale between runs. Event-driven refresh updates it right when the source data changes, keeping it fresh with less wasted work, but is more complex to build.

Change the key so that one busy device's writes don't all pile onto one shard — for example, combine the device ID with a time bucket or a random suffix so its data spreads across several shards, then merge them back together when reading.

Range sharding groups nearby keys (like consecutive IDs) on the same shard, which is great for range scans but can pile new writes onto one shard. Hash sharding spreads keys evenly using a hash function, avoiding that but breaking range queries. Lookup sharding uses an explicit table to say which shard each key lives on, giving full control at the cost of an extra hop.

It should limit the link to one exact file, one action (like upload-only, not delete), and sometimes the file size or type — not just make the link expire soon.

Short TTL means fresher data but more trips to the database because things expire fast. Long TTL means less database load but data can be wrong for longer. TTL alone doesn't fix staleness — it just puts a ceiling on how long stale data can survive.

Old files in storage are probably never getting deleted after being processed -- orphaned blobs pile up forever. Fix by adding a cleanup step or an automatic expiry rule on the storage bucket so old, unused files get removed.

The read copy of the data hasn't caught up to the write yet — there's a short delay while the update flows through to it. Fixes include showing the user their own change right away without waiting for the read copy, or making that specific read wait until it's caught up.

The order exists but is invisible to anyone searching by email, because its index entry was never created. Fix it by either doing both writes as one atomic operation when possible, or by having a background process that finds and repairs these gaps automatically.

Use a materialized view when the query is expensive and slightly old data is fine, like a dashboard. Use a live join when the data must always be current, like a bank balance. Picking wrong means either a slow, overloaded system or a fast one that shows outdated info to people who needed the latest.

Each shard is a separate database that doesn't know about the others, so it can't run a join or a single all-or-nothing transaction spanning two of them. The application has to query each shard separately and combine results, and cross-shard writes need special handling like two-phase commit or a saga, or the schema should be redesigned so related data lives on the same shard.

If you need to check, transform, or tightly control the file as it arrives, like scanning for viruses before it's stored or enforcing a quota that can change moment to moment, routing through your server lets you do that inline. Direct-to-storage uploads only let you react after the fact.

It gets slower per message (extra round trip to fetch the payload) and more complex (a second system to run, pay for, and secure). In exchange, the queue stays fast and cheap and can handle way more messages per second.

Cloud providers offer managed 'always-on' logs (streaming services) to hold the events, managed databases with built-in 'change feeds' to trigger updates automatically, serverless functions to run the update logic, and managed read stores (like document databases or search services) to hold the fast, query-ready copies.

Every time a product is created or changed, the app now has to write not just to the main table but to all four index tables too — so one logical update turns into up to five separate writes. Add more index tables and every write gets slower and more expensive, even if nothing about that particular field changed.

Use a lock so only one request reloads the data while everyone else waits or gets a slightly old value. Or make one request per process share its in-flight database call with all the others asking for the same thing, so only one query actually runs.

The common mistake is making the storage bucket wide-open or giving every consumer permanent access to everything in it. Better: give each consumer only the narrow permission it needs, ideally a short-lived link scoped to just that one file.

When the service is small, simple, or needs strict instant consistency — the extra moving parts (two models, an event log, projectors) cost more in complexity than they save, and a plain database with normal reads/writes does the job fine.

Using status as the partition key means all orders with the same status pile into one 'bucket,' so writes for that status compete for the same limited capacity even though the rest of the system is fine. Spreading the writes across more buckets — for example by adding some kind of extra prefix or suffix — fixes the bottleneck.

Instead of re-adding up every order every time, you just add or subtract the new order's amount to the existing total. But you also need to handle edited or canceled orders (subtract the old amount first) and make sure you don't double-count an order you've already processed.