Managed Kafka offerings advertise hiding ZooKeeper/KRaft and providing tiered storage. As a principal, how do these abstractions change capacity planning, scaling, and lock-in decisions?

## The two abstractions ### 1. ZooKeeper → KRaft Classically Kafka used **ZooKeeper**, a separate distributed coordination service, to store cluster **metadata** (brokers, topics, partitions, ACLs, controller election). It was an extra system to deploy, secure, and scale, and it **capped practical partition counts** (metadata churn through ZooKeeper limited clusters to tens of thousands of partitions and made controller failover slow). **KRaft** (Kafka Raft, **KIP-500**) moves metadata **into Kafka itself**: a quorum of controllers stores metadata in an internal Raft log. Benefits: one system instead of two, **much higher partition ceilings** (millions), and **faster controller failover/recovery**. Managed offerings **hide whichever they use** — you never operate ZooKeeper or KRaft. ### 2. Tiered storage Normally a partition's full log lives on **broker local disk**, so **retention is bounded by disk size** and rebalancing/scaling must copy large amounts of data. **Tiered storage** (KIP-405; productized by Confluent, MSK, Aiven) keeps only **recent (hot) segments locally** and offloads **older (cold) segments to object storage** (e.g. Amazon S3). Effects: retention is effectively **unbounded and cheap**, broker disks shrink, and **adding/replacing brokers is faster** because less local data must move. ## How this changes capacity planning - **Partitions**: the old ZooKeeper ceiling is gone, so you *can* run far more partitions — but on managed offerings the **new ceiling is a vendor quota and per-partition price**, not a metadata limit. Plan partition counts against **cost and quotas**, not ZooKeeper. - **Retention/storage**: with tiered storage you size **hot tier** for working-set latency and treat cold retention as a storage-cost decision, not a disk-capacity wall. Long-lookback consumers (replays, backfills) become viable. - **Scaling speed**: less local data → faster elastic scaling and rebalances, so capacity can track demand more closely (especially with serverless tiers). ## How this changes scaling/operations - Controller failover and metadata operations are faster/safer (KRaft), so large clusters are more stable. - Broker right-sizing focuses on **throughput and hot-set**, not total retention. - **Cold reads** (consumers reading old, tiered data) hit object storage: higher and more variable latency, and possible **per-read/egress fees** — a new performance and cost consideration absent in pure local-disk Kafka. ## How this changes lock-in These conveniences are largely **vendor-specific**: - Tiered-storage formats and the object-storage integration are **proprietary** per vendor; you cannot trivially lift the tiered data to another platform. - Metadata, IAM, connectors, and stream-processing add-ons bind you further. - Pricing (per-partition, tiered-read, egress) becomes the **governing constraint**, replacing the old technical ZooKeeper limit with a commercial one. ## Principal-level governance 1. Set **partition budgets** per team (cost + quota driven), since the technical cap no longer protects you. 2. Define **retention tiers** and which workloads may use long/infinite retention (replay/backfill vs cost). 3. Account for **cold-read latency/cost** in SLAs for replay-heavy consumers. 4. Make **lock-in explicit**: document exit cost (proprietary tiered data, connector rewrites) and, if portability matters, prefer open formats and keep an extraction plan. 5. Treat KRaft/tiered storage as **risk-reducing but constraint-shifting** — the bottleneck moves from operations to commercial terms.