skip to content

How do you enable tiered storage in a Kafka cluster and turn it on for a specific topic?

level: juniorimportance: must knowfreq 55%

answer

  1. two switches: system.enable (broker) + storage.enable (topic)
  2. KIP-405, GA 3.9
  3. RSM class + RLMM (TopicBased default, __remote_log_metadata)
  4. only closed segments tiered, active stays local
  5. can't easily disable per-topic pre-KIP-950

basics

~10 s

Turn it on cluster-wide with remote.log.storage.system.enable=true on each broker (plus configure a RemoteStorageManager plugin), then per topic set remote.storage.enable=true. After that, old segments are copied to the remote store.

solid answer

~40 s

Tiered storage (KIP-405) is enabled in two layers. First, at the broker level, set remote.log.storage.system.enable=true in server.properties on every broker and configure the plugin: remote.log.storage.manager.class.name points to a RemoteStorageManager (RSM) implementation, and remote.log.metadata.manager.class.name to the RemoteLogMetadataManager (default TopicBasedRemoteLogMetadataManager). The RSM needs backend-specific configs (bucket, region, credentials) under a configurable prefix. Second, per topic, set remote.storage.enable=true (via topic config at create time or kafka-configs --alter). Once enabled, closed log segments are asynchronously uploaded to remote storage by a background thread; only the active segment and recent data governed by local.retention.* stay on local disk. You generally cannot disable remote storage on a topic once enabled (until KIP-950) without recreating it.

go deeper

for a junior

Recall the two switches: broker remote.log.storage.system.enable and topic remote.storage.enable.

for a middle

Knows the plugin classes (RSM, RLMM) and that closed segments are offloaded by a background thread.

for a senior

Explains plugin config prefixes, __remote_log_metadata topic, and the disable limitation / KIP-950.

for a principal

Frames enablement as a rollout: validating the plugin, sizing the metadata topic, and operational reversibility constraints across the fleet.

**What tiered storage is.** Normally a Kafka broker keeps every log segment for a partition on its local disk until retention expires. Tiered storage (introduced as KIP-405, GA in Kafka 3.9) lets the broker offload older, closed segments to a cheaper, larger external store (typically an object store like S3/GCS/Azure Blob, or HDFS) while keeping recent data local. This decouples retention from local disk size. **Two enablement layers — both are required:** 1. **Broker / cluster level** (server.properties, every broker): - `remote.log.storage.system.enable=true` — the master switch that loads the tiered-storage subsystem. Without it, the per-topic flag does nothing. - `remote.log.storage.manager.class.name` — fully-qualified class of the **RemoteStorageManager (RSM)** plugin that actually reads/writes segment data to the backend (e.g. an S3 implementation). Kafka ships an interface, not a production cloud backend, so you supply a plugin (Aiven's tiered-storage-for-apache-kafka, Confluent's, etc.). - `remote.log.metadata.manager.class.name` — the **RemoteLogMetadataManager (RLMM)**; defaults to `TopicBasedRemoteLogMetadataManager`, which stores segment metadata in an internal Kafka topic `__remote_log_metadata`. - Plugin-specific properties live under a configurable prefix (e.g. `remote.log.storage.manager.impl.prefix` / `remote.log.metadata.manager.impl.prefix`) — bucket name, region, endpoint, credentials, chunk size, etc. 2. **Topic level** (`remote.storage.enable=true`): set at creation (`kafka-topics --create --config remote.storage.enable=true`) or later (`kafka-configs --alter --add-config remote.storage.enable=true`). This is what actually marks a topic's partitions for offload. **What happens after enablement.** A background RemoteLogManager thread on the leader picks up **closed** (rolled) segments and uploads them to the RSM, recording metadata via the RLMM. Local disk then only needs to hold the active segment plus whatever `local.retention.ms` / `local.retention.bytes` keep around; the overall topic retention (`retention.ms` / `retention.bytes`) governs how long data lives in the remote store. Consumers reading old offsets are served transparently from remote storage. **Edge cases / gotchas:** - The active (currently-written) segment is never tiered — only rolled segments are eligible. - Compacted topics could not use tiered storage initially; combining compaction + tiering matured later — verify your version. - Disabling remote storage on a topic was historically not supported (you'd recreate the topic); KIP-950 adds a controlled disable path in newer versions. - The cluster switch cannot be safely flipped off while topics still rely on remote data.

  • What is the default RemoteLogMetadataManager and where does it keep metadata?
    TopicBasedRemoteLogMetadataManager, which persists remote-segment metadata in an internal Kafka topic named __remote_log_metadata (replicated, partitioned), so the cluster itself stores the index of remote segments.
  • Why isn't setting remote.storage.enable=true on a topic enough by itself?
    The broker-level master switch remote.log.storage.system.enable=true and a configured RSM/RLMM must be present first; without the subsystem loaded, the topic flag is inert and no offload occurs.

saying these in an interview costs you the question

  • Thinking the per-topic flag alone activates tiering without the broker-level system.enable switch.
  • Claiming Kafka ships a built-in S3 RemoteStorageManager out of the box — you supply/choose a plugin implementation.
  • Believing the active segment gets uploaded — only rolled/closed segments are eligible.
  • Assuming you can freely toggle remote storage off on a topic (historically unsupported pre-KIP-950).

context

resolving links…