Wide-Column Store Concepts

A wide-column store maps a key plus a column name plus a timestamp to a value. Only the cells a row actually has are stored, so rows can differ in their columns, and rows are stored in key order.

The write is appended to a commit log for durability and inserted into a sorted in-memory buffer, then acknowledged. When the buffer fills it is flushed to disk as a new immutable sorted file, and background compaction later merges those files.

A wide-column store answers efficiently only reads that follow a key and its stored order, and it cannot join. So you list the application's reads first and give each one a key layout that serves it in one contiguous read.

It fits very high write volumes, data that outgrows one machine, a small known set of queries, and time-ordered or sparse data. You give up joins, ad-hoc queries, rich secondary access and transactions spanning many rows.

With a hashed partition key, the hash picks the servers and clustering columns sort rows inside that partition. With one sorted row key, the whole keyspace is kept in byte order and split into contiguous ranges, each served by one server.

Choose a B-tree engine for read-heavy work needing predictable point and range latency, since each key has one place. Choose an LSM engine for sustained heavy writes, since it writes sequentially and compresses well, at the cost of reads checking several files and background compaction.

A partition or row is never split across servers, so one that grows forever becomes slow to read, compact and repair and can hit size limits. Adding a time bucket to the grouping part of the key starts a fresh partition each period.

The key is the only structure that narrows a read, so a filter on another column makes the store read and discard data across partitions. Serve that read with a layout keyed by the field instead.

In the leaderless model any replica accepts a row's writes and the client chooses how many replicas must answer, so reads can be stale. In the range-owner model one server serves each key range, so single-row reads and writes are strongly consistent by default.

If a replica missed a delete and the others have purged the tombstone, repair sees the old data only there and copies it back. Tombstones must outlive the longest gap between successful repairs.

Size-tiered compaction merges groups of similar-sized files into bigger ones, rewriting data rarely but leaving many overlapping files. Leveled compaction keeps each level's files non-overlapping, so reads touch few files and less stale data lingers, at the cost of rewriting data more often.

A time-to-live stamps each cell or column group with an expiry; nothing is deleted at that moment. Expired cells are skipped when read or merged, and their space returns only when compaction rewrites or drops the files holding them.

Every cell carries a timestamp, and when a cell has several writes the one with the highest timestamp wins. Some stores keep older versions readable up to a per-family limit; others show only the newest and discard the rest when files are merged.

A column family is a named group of columns declared in the schema; the columns inside it are created freely by writes. Stores keep a family's data together and apply settings such as version retention per family, so grouping decides what a read touches.

A read checks the memory buffer and then every sorted file that could hold the key, merging entries so the newest version wins. Per-file bloom filters skip files that certainly lack the key, and per-file indexes jump straight to the right block.

Put the fields the read matches exactly, such as the customer id, first in the key and the field it ranges over, such as order time, next, in the order it is read. A field only filters cheaply as part of the key.

It merges each time window's data into one file that is never compacted again, so a fully expired window is dropped as a whole file. Back-dated writes, updates to old data and mixed expiries mix windows and defeat it.

Every delete leaves a tombstone, and a read from the head of the range steps over all unpurged markers, so cost grows with deletes, not live rows. Read from a cursor, delete by bucket, or use a real queue.

The key is the physical address, fixing where each row lives and its order on disk, so a store cannot re-key a table in place. Changing it means building a new table with the new key, backfilling, keeping both in step and cutting reads over.

Write amplification is disk bytes written per byte the application writes; read amplification is work per read; space amplification is disk used per byte of live data. Merging files more eagerly cuts read and space costs but raises writes.

A key cannot be updated, so every layout keyed by the changed field must delete the row at its old key and write it at the new one. That needs the old value, read from the source of truth or carried in the change event.

A range-served store runs the check and the write atomically on the one server that owns the row, costing about a read plus a write. A leaderless store must run a consensus round among the row's replicas, adding several round trips and slowing under contention.

A leaderless store keeps serving from the surviving replicas, as long as enough of them answer. A range-served store makes the failed server's key ranges unavailable until they are reassigned and any unflushed writes are replayed from the log.

Atomicity stops at one row, or one partition in partition-keyed stores. Operations across rows have no isolation and can be partly applied, so put data that must change together under one key, or make multi-row steps idempotent and re-drivable.

Choose by which failure the product tolerates and which operations it needs. Leaderless stores favour write availability and multi-site writes but make conditional updates costly; range-served stores give strong single-row reads and cheap check-and-mutate at the price of brief per-range unavailability.

The memory buffer is lost on a crash, so the commit log is what makes acknowledged writes recoverable; on restart it is replayed into a new buffer. A log segment can be discarded once every buffer holding its writes has been flushed to sorted files.

One row per event is simpler, copes with uneven or unbounded streams and slices any time range. One row per entity and period packs events compactly and reads a period in one fetch, but must stay bounded in size.