A wide-column table used like a queue, with rows inserted and soon deleted, reads more slowly over time even though little live data remains. Why, and how do you redesign it?
answer
- deleted is not gone
- reads walk the markers
- the head of the range
- start after what was consumed
- buckets that expire whole
basics
~20 sEvery delete leaves a tombstone, and a read from the head of the range steps over all unpurged markers, so cost grows with deletes, not live rows. Read from a cursor, delete by bucket, or use a real queue.
solid answer
~50 sEvery consumed row leaves a **tombstone**, which stays until compaction can purge it — in replicated stores not before a grace period. A consumer that reads "the oldest pending rows" starts at the **head** of the partition, exactly where all the markers are, so each read scans thousands of dead entries to find a few live ones. Latency grows with churn, not with live data, and some stores warn or abort a read after it meets too many markers. Redesign options: **start reads after the last consumed position** (store a cursor, and read from it); put a **time bucket** in the key so consumed buckets are never read again and can be dropped **with one range delete or by expiry**; mark rows as processed with a TTL instead of deleting individually; or move the workload to a **message queue or log**, which is built for consume-and-forget.
go deeper
Know that deleted rows leave markers that reads may still have to step over.
Explain why a head-first scan in a queue-like table meets every recent marker and why cost follows deletes, not live rows.
Redesign the table with cursors, time buckets and bulk expiry, and monitor tombstones per read to catch it early.
Be ready to decide when a queue-like workload belongs in a messaging system instead of a wide-column store, and set that as a team rule.
## The symptom A table holds work items: producers insert rows, consumers read "the oldest pending items" and delete each one after processing. The table stays small in live rows, yet: - read latency climbs steadily; - reads start failing or logging warnings about scanning too many deleted cells; - compaction shows many tombstones that it cannot yet drop. ## The cause 1. **Each delete writes a tombstone.** The row does not vanish; a marker hides it. 2. **Markers live for a while.** They are purged only when compaction merges them with the data they hide, and, in stores whose replicas repair each other, only after a grace period. 3. **The read starts at the head.** "Oldest pending" is a scan from the start of the partition or key range — precisely where every consumed item's tombstone sits. 4. **The read steps over every marker.** To return 10 live rows it may read 100,000 dead ones, merging them from memory and files. Cost therefore scales with **recent deletes**, not with live data. Some stores guard against runaway scans by warning after a number of tombstones and aborting beyond a higher limit. ## Redesign options | option | how it helps | caveat | |---|---|---| | **read from a cursor** | store the position of the last consumed item and scan from there | the cursor must be durable and advanced safely | | **time-bucketed partitions** | consumers move to the next bucket; old buckets are never read | needs bucket arithmetic and a lag bound | | **range deletes or expiry per bucket** | one marker or whole-file expiry instead of one marker per row | only possible when the whole bucket is done | | **mark as processed with a TTL** | no read-path scan over deletes if reads filter by state and position | still produces expiries to clean up | | **use a queue or log system** | consume-and-forget is its native operation | another system to run | ## A concrete redesign 1. Key items by `(queue, time bucket)` with the item time as the ordering part. 2. Each consumer keeps a durable **cursor** (bucket plus last item time). 3. Reads scan **from the cursor forward**, never from the head. 4. Items are **not deleted individually**; each bucket carries a TTL, or the whole bucket is removed with **one range delete** once every consumer has passed it. 5. Size buckets so a bucket is fully consumed long before it expires. Now reads never touch dead data, and cleanup costs one operation per bucket. ## Warning signs to monitor - tombstones scanned per read, and warnings from the store about them; - read latency on the specific table rising while its live size stays flat; - partitions or rows where the ratio of markers to live cells is high. ## Interview angle This is a classic production story. Explain why deletes are not free, why the head of a queue-like scan is where the markers pile up, and offer cursors, buckets with bulk expiry, or a real queue.
- Would compacting the table more often fix the problem?Only partly. Compaction cannot purge markers before the grace period in replicated stores, and churn keeps creating new ones. It may lower the backlog but does not change the design flaw of reading from the head.
- Why does a single range delete help compared with deleting each row?One marker covers the whole consumed range, so reads that do touch it skip one entry instead of thousands, and far fewer markers are written and later compacted.
saying these in an interview costs you the question
- Assuming deleted rows cost nothing to read past
- Blaming the slowdown on data volume when live rows are few
- Raising the store's tombstone failure limit as the fix
- Deleting consumed items one by one in a hot, head-read partition
- Expecting compaction to purge markers immediately after each delete