Apache Kafka

bootstrap.servers is the initial list of broker host:port pairs a client contacts to discover the full cluster. After the first metadata fetch the client learns all brokers, so the list only needs a few entries for redundancy.

A broker is a single Kafka server that stores topic partition data and serves produce/fetch requests. broker.id is its unique numeric identifier within the cluster; no two brokers may share the same id.

Each partition is an append-only log: records are only added to the end, never changed in place. Kafka splits that log into fixed-size files called segments so old data can be deleted or compacted one whole file at a time instead of editing one giant file.

The log cleaner is a background process that runs compaction: it scans a topic's log and keeps only the latest record per key, deleting older duplicates. You enable it by setting the topic config cleanup.policy=compact.

Kafka writes data to files and lets the OS keep recently used file pages in RAM (the page cache). It avoids a JVM heap cache to dodge GC pressure, double-buffering, and to reuse the OS cache that survives broker restarts.

The active controller is the single elected controller node that is the only one allowed to write cluster metadata (topics, partitions, broker registrations) by appending records to a replicated log. Other nodes follow and replay that log.

KRaft (Kafka Raft) is Kafka's built-in consensus protocol that replaces Apache ZooKeeper for storing cluster metadata. It lets Kafka manage its own metadata internally, so you no longer run a separate ZooKeeper cluster.

Each broker periodically sends a BrokerHeartbeat request to the active controller. The interval is set by broker.heartbeat.interval.ms (default 2000 ms). If heartbeats stop, the controller eventually fences the broker.

Static quorums list every controller in controller.quorum.voters on all nodes, fixed at startup. KIP-853 dynamic quorums (KRaft) let you add or remove controllers at runtime with AddRaftVoter/RemoveRaftVoter instead, using controller.quorum.bootstrap.servers.

The metadata log grows forever as records are appended. Snapshots capture the current state at a point in time so old log records can be deleted, keeping the log small and making new nodes catch up faster.

The producer hashes the key and takes hash % numberOfPartitions. The same key always lands in the same partition, so all records for that key stay ordered together on one partition.

Compaction keeps at least the latest value for each message key, deleting older values for that key. The default delete policy instead drops whole segments once they exceed retention.ms or retention.bytes, regardless of key.

An offset is a number that marks a record's position inside one partition. It starts at 0 and increases by 1 for each new record. Offsets are per-partition, so offset 5 in partition 0 and offset 5 in partition 1 are unrelated records.

Kafka guarantees order only within a single partition: messages are read in the same order they were written. It does NOT guarantee any order across different partitions of a topic.

It can only INCREASE the partition count of an existing topic, never decrease it. To shrink, you must create a new topic with fewer partitions and republish the data; Kafka has no in-place shrink.

acks=0 means the producer never waits for acknowledgement (fastest, can lose data). acks=1 waits only for the leader to write the record. acks=all waits for the leader plus all in-sync replicas, giving the strongest durability but the highest latency.

batch.size caps how many bytes a producer collects per partition before sending; linger.ms tells it to wait up to that many milliseconds for more records to fill a batch. Bigger batches and more lingering raise throughput but add latency.

Retriable errors (like a leader change or timeout) are transient, so the producer automatically resends. Fatal errors (like an unknown topic, message too large, or auth failure) cannot be fixed by resending, so the send fails immediately.

An idempotent producer guarantees that producer retries don't create duplicate messages on a partition. You enable it by setting enable.idempotence=true (the default since Kafka 3.0).

For a keyed record, Kafka hashes the key and maps it to a partition. The same key always lands on the same partition (as long as the partition count is unchanged), which keeps records with that key in order.

Kafka stores message keys and values as raw bytes. key.deserializer and value.deserializer are classes that turn those bytes back into objects. They must match the producer's serializers, or the bytes won't decode correctly.

The Group Coordinator is a broker that manages a consumer group: it handles members joining/leaving, triggers rebalances, and stores offsets. The coordinator is the broker that leads the __consumer_offsets partition the group maps to.

A heartbeat is a small periodic signal a consumer sends to the group coordinator broker to prove it is alive and still part of its consumer group. If heartbeats stop, the broker assumes the consumer died and reassigns its partitions.

Run kafka-consumer-groups.sh --bootstrap-server host:port --describe --group <group>. It lists each partition with CURRENT-OFFSET (committed), LOG-END-OFFSET (latest), and LAG = the difference, plus the owning consumer.

Consumer lag is how far behind a consumer is on a partition: the latest message offset (log-end-offset) minus the offset the consumer has committed. Lag of 0 means fully caught up.

Replication factor (how many copies of each partition), min.insync.replicas (how many copies must be caught up to accept a write), and acks (how many copies the producer waits for: 0, 1, or all). Together they decide how many failures you can survive without losing data.

Kafka promotes one of the in-sync follower replicas to be the new leader. Clients get an error, refresh metadata, find the new leader's broker, and resume producing/consuming there. No manual action is needed.

After a successful write the data is in the broker's RAM (the OS page cache), and with acks=all also in the RAM of other replicas. It isn't necessarily on the physical disk yet — the operating system writes it to disk a bit later.

LEO is the offset just past the last message written to the log. HW is the highest offset that has been replicated to all in-sync replicas. Consumers can only read up to (below) the HW, so they never see un-replicated records.

The ISR is the subset of a partition's replicas (leader plus followers) that are fully caught up with the leader's log. Assigned replicas (AR) are all replicas; ISR is the healthy, up-to-date ones eligible to serve and be elected leader.

It's the read-process-write loop: a consumer reads from an input topic, the app transforms records, and a producer writes results to an output topic. At-least-once can produce duplicates or commit offsets without the output, so the output and the offset commit aren't atomic.

Kafka EOS only guarantees exactly-once within a single Kafka cluster: a read-process-write where the input topic, output topic, and consumer offsets all live in that same cluster. External databases, APIs, or other clusters are outside the transaction, so it can't cover them.

At-most-once: each message is delivered zero or one time (loss possible, no duplicates). At-least-once: delivered one or more times (duplicates possible, no loss) — Kafka's default. Exactly-once: delivered once and only once (no loss, no duplicates).

An idempotent consumer can process the same message more than once with no extra effect. Kafka's at-least-once delivery can redeliver a record after a crash; if processing is idempotent, those duplicates are harmless, so you get effectively exactly-once results.

isolation.level controls whether a consumer sees records from in-progress or aborted transactions. read_uncommitted (default) returns all records; read_committed only returns records from committed transactions, hiding aborted and not-yet-committed ones.

groupByKey() groups by the existing record key with no repartition. groupBy() picks a NEW key, so Streams must repartition (re-shuffle records to topic partitions) so all records with the same new key land on the same task.

A KStream is a record stream where every record is an independent event (insert/append). A KTable is a changelog stream where each record is an update keyed by its key — only the latest value per key matters.

It guarantees each input record affects the application's state and output exactly once, even if the app crashes and restarts — no duplicates and no lost updates. You enable it by setting the config processing.guarantee to exactly_once_v2.

Interactive Queries let your app read directly from the state stores Kafka Streams already maintains, instead of querying an external database. You call KafkaStreams.store(...) to get a read-only store and look up a key with get().

Kafka Streams supports stream-stream joins (over a time window), stream-table joins (enrich a record with the latest table value), and table-table joins (combine two changelog tables). GlobalKTable joins let a stream join a fully-replicated table without matching partitioning.

Debezium is a Kafka Connect source connector that reads a database's transaction log (MySQL binlog, Postgres WAL) and streams every row insert, update, and delete to Kafka as events. Log-based CDC catches all changes, including deletes, with low impact on the database.

A converter serializes/deserializes Connect data to/from bytes on the Kafka topic. key.converter handles the record key; value.converter handles the record value. They are configured independently and can differ.

errors.tolerance decides what Connect does when a record fails processing. none (the default) stops the connector task on the first error; all skips the bad record and keeps going.

Kafka Connect is a framework for streaming data between Kafka and external systems (databases, files, S3) using ready-made connectors, so you configure plugins instead of writing custom producer/consumer code.

Source connectors store their own progress (where they read from the external system) in a special Kafka topic called the offset storage topic. Sink connectors just use a normal Kafka consumer group, so Kafka tracks their offsets like any consumer.

Avro is a compact binary serialization format that stores data with a separate schema. With Kafka it gives small messages, a typed contract for producers and consumers, and safe schema evolution via a Schema Registry.

Schema Registry has BACKWARD (default), FORWARD, FULL, NONE, and a _TRANSITIVE variant of each. They control what schema changes are allowed: BACKWARD lets new consumers read old data, FORWARD lets old consumers read new data, FULL means both, NONE disables checks.

Deserialization turns raw bytes back into objects. A Kafka topic is just bytes from whoever produced them, so a malicious producer can send crafted bytes that exploit the consumer's deserializer — for example triggering code execution or crashing it.

A poison pill is a record the consumer cannot deserialize (corrupt or wrong-format bytes). Deserialization happens inside poll(), so it throws every time, the offset never advances, and the consumer is stuck retrying the same record forever.

Use the format-specific serializer: KafkaProtobufSerializer or KafkaJsonSchemaSerializer instead of KafkaAvroSerializer. Each subject in the registry carries a schemaType (AVRO, PROTOBUF, or JSON) so the registry knows which format the stored schema is.

Kafka's authorizer writes an audit trail via a dedicated logger named kafka.authorizer.logger. Set that log4j logger to DEBUG to see allowed requests too; at INFO you only see denials. Route it to its own file appender.

An ACL (access control list entry) is a rule saying a principal (user) is allowed or denied a specific operation (like Read or Write) on a resource (like a topic), optionally from a specific host.

super.users is a broker config listing principals that are granted full access and bypass all ACL checks. Any principal in the list is always authorized for every operation, no ACLs needed.

No. Open-source Kafka has no built-in encryption of log segment files on disk. Encryption at rest is provided externally: encrypt the broker's volume/disk (LUKS or a cloud KMS-backed encrypted EBS volume), or encrypt the message payload before producing it.

Kafka has four security protocols: PLAINTEXT (no encryption, no auth), SSL (TLS encryption, optional client cert auth), SASL_PLAINTEXT (SASL auth, no encryption), and SASL_SSL (SASL auth plus TLS encryption).

Use kafka-topics.sh with --bootstrap-server. To create: --create --topic name --partitions N --replication-factor R. To inspect: --describe --topic name, which shows partitions, leaders, replicas, and in-sync replicas (ISR).

cleanup.policy sets how old data is removed. delete drops whole log segments once they exceed retention.ms (age) or retention.bytes (size). compact keeps the latest value per key forever. retention.ms/bytes only apply to delete.

A new broker joins the cluster but Kafka never auto-moves existing partitions onto it. You must explicitly reassign replicas to the new broker using kafka-reassign-partitions to scale out.

Storage = ingest rate (bytes/sec) x retention seconds x replication factor. So a 10 MB/s topic kept for 7 days at RF=3 needs roughly 10MB x 604800s x 3 = about 18 TB of disk across the cluster.

Use kafka-configs.sh with --bootstrap-server, --entity-type topics, --entity-name <topic>, and --alter --add-config key=value. To remove an override use --delete-config key. --describe shows current overrides.

It is Kafka's built-in load generator for producers. --num-records sets how many messages to send, --record-size sets each message's size in bytes, and --throughput caps messages per second (-1 means unthrottled, full speed).

UnderReplicatedPartitions counts how many partitions led by this broker have fewer in-sync replicas than the configured replication factor. In a healthy cluster it should be 0. A sustained non-zero value means replicas are falling behind or down.

JMX (Java Management Extensions) is the Java standard for exposing runtime metrics as MBeans. A Kafka broker publishes its internal metrics as JMX MBeans. You enable it by setting JMX_PORT (or jmxremote system properties) and read it with tools like JConsole, jmxterm, or a Prometheus JMX exporter.

num.network.threads handle reading requests off and writing responses onto network sockets; num.io.threads do the actual work (reading/writing the log on disk). Default 3 and 8. Raise them if request-handler/network idle time drops.

Kafka clients expose metrics over JMX. Key producer metrics: record-send-rate, request-latency-avg, buffer-available-bytes. Key consumer metrics: records-consumed-rate, fetch-latency-avg, records-lag-max. They show throughput, latency, and buffering/lag health.

MM2 prefixes replicated topics with the source cluster's alias (e.g. topic 'orders' from cluster A becomes 'A.orders' on cluster B). Because the prefix marks where data came from, MM2 won't replicate a topic back to the cluster it originated from, so records don't loop forever.

In active-passive DR, one Kafka cluster (primary) serves all traffic while a second cluster (standby) only receives a one-way replicated copy. Clients use the standby only after failover. Active-active runs producers/consumers on both clusters at once.

Cluster Linking is a broker-built-in feature that copies topics from a source Kafka cluster to a destination cluster over a direct link, with no separate Connect cluster. The destination gets read-only mirror topics that keep the same message offsets.

MirrorMaker 2 is just a Kafka client. Securing the link means: (1) encrypt traffic in transit with TLS, and (2) authenticate MM2 to each cluster (mTLS or SASL) so only an authorized identity can read the source and write the target.

Normally Kafka consumers read only from the partition leader. Fetch-from-follower (KIP-392) lets a consumer read from a nearby replica (follower) in its own availability zone instead, cutting cross-zone network traffic and the cloud egress cost that comes with it.

No. The Java KafkaConsumer is not thread-safe — only one thread may call it (except wakeup()). The standard model is one consumer instance per thread, each in its own poll loop.

DefaultErrorHandler is the standard component that catches exceptions thrown by your @KafkaListener. It retries the record a configurable number of times with a backoff delay, and if all retries fail it calls a recoverer (by default just logs the failed record).

An event states something that already happened (past tense, e.g. OrderPlaced) and has no specific intended recipient. A command tells one service to do something (imperative, e.g. PlaceOrder) and expects an action. Events go to topics for any consumer; commands target one handler.

Kafka redelivers messages after rebalances or restarts (at-least-once), so the same record can arrive more than once. You record each processed message's id in a store (a DB unique constraint or Redis key) and skip any id you've already seen.

Build a Properties (or Map) of config, then pass it to the constructor: new KafkaProducer<>(props) or new KafkaConsumer<>(props). Producers need bootstrap.servers plus a key and value serializer; consumers need bootstrap.servers plus a key and value deserializer (and usually group.id).

Kafka couples storage and serving: each broker owns its partition data on local disk. Pulsar separates them: brokers are stateless serving nodes, and storage lives in a separate layer (Apache BookKeeper).

librdkafka is a C/C++ implementation of the Kafka client protocol. The official Python, Go, .NET, and Node.js clients are thin language bindings that wrap librdkafka, so they share one battle-tested core instead of each reimplementing Kafka.

Kafka Connect is a framework for moving data between Kafka and external systems without custom code. A sink connector reads records from Kafka topics and writes them to a destination like S3, BigQuery, or Snowflake using ready-made plugins and config.

Azure Event Hubs exposes a Kafka-compatible endpoint on port 9093, so a normal Kafka client can produce and consume by just changing bootstrap.servers and using SASL/SSL auth with a connection string — no Kafka broker is actually run.

They are alternative streaming systems that speak the Kafka wire protocol, so existing Kafka clients work unchanged. Redpanda is a C++ broker with no JVM or ZooKeeper; WarpStream stores data in S3 with stateless brokers.