Explain Pulsar's segment-centric storage versus Kafka's partition-centric log, and why it matters operationally.

**Definitions first.** - A **partition** in Kafka is an ordered, append-only log of messages identified by offsets. It has a leader broker and follower replicas; physically it is a directory of *segment files* (e.g. `0000…000.log`) — but crucially all those segment files for a partition live together on each broker that hosts that partition. - A **ledger** in Pulsar/BookKeeper is an append-only sequence of *entries* (BookKeeper's name for records). A ledger is the unit of storage placement: when it is created, BookKeeper chooses an *ensemble* of bookies to hold it. **Kafka — partition-centric.** The partition is the atom. The leader broker holds the entire partition log on its disk; replicas hold full copies. Therefore: - The whole partition must fit on one broker's disk (you can't stripe one partition across brokers). - Parallelism is bounded by partition count, and storage is bounded per-partition by one disk. - Moving a partition (reassignment, scaling out, replacing a broker) copies the entire log over the network — a heavy, throughput-affecting operation. **Pulsar — segment-centric.** A topic-partition in Pulsar is a *managed ledger*: a logically ordered chain of ledgers (segments). Pulsar rolls over to a new ledger periodically (by size, time, or on broker ownership change). Each new ledger gets a fresh bookie ensemble chosen by BookKeeper's placement policy. Consequences: - The topic's bytes are spread (striped) across the entire bookie cluster over time. No single bookie holds the whole topic, so a topic can be far larger than any one node's disk. - **Adding capacity is instant and copy-free:** a new bookie starts receiving newly created ledgers immediately; existing data is never moved to 'rebalance'. Contrast Kafka, where adding a broker means reassigning partitions and copying logs. - **Recovery is segment-scoped:** if a bookie dies, only the ledgers whose ensemble included it are under-replicated; BookKeeper's auto-recovery re-replicates just those ledger fragments to restore the *ack quorum*, rather than rebuilding whole partitions. - **Reads/tailing** are served by the broker from a write cache or by fetching entries from bookies; a long historical read can fan out across many bookies (good for parallel IO, but more connections/metadata). **Why it matters — concrete scenarios.** 1. *Sudden retention bump or a huge backlog.* In Kafka a backlogged partition can fill its leader's disk and you cannot spread it; in Pulsar the growing topic keeps allocating new segments on whichever bookies have room. 2. *Fast broker replacement.* Because brokers are stateless and storage is segmented, replacing a Pulsar broker doesn't move data. Replacing a Kafka broker means re-replicating its partitions. 3. *Skew.* Kafka can get hot/large partitions stuck on one broker; Pulsar avoids single-node topic-size skew but can still get *throughput* skew if one topic's traffic concentrates on the brokers that currently own it. **Costs / edge cases.** Segment-centric design multiplies metadata: every ledger has an entry in the metadata store, so very high topic/ledger counts pressure ZooKeeper/etcd. Placement policies (rack-aware, region-aware) must be configured or ensembles can land badly. And because data is striped, a historical replay touches many bookies — beneficial for parallelism but it means read latency depends on the slowest bookie in the ensemble for each entry (mitigated by BookKeeper's *speculative reads*).