How should you design Kafka topics for domain events — granularity (one event type per topic vs many), keying, and retention/compaction — and what is a domain event versus an integration event?

## Domain event vs integration event - A **domain event** is a fact that matters **inside a bounded context** — it's part of your domain model: `OrderPlaced`, `PaymentCaptured`. It may be rich and reflect internal structure. - An **integration event** is what you publish **across boundaries** to other services/teams. Best practice is to **not** leak your raw internal domain event; instead publish a **deliberately designed, stable, public** integration event (a curated subset/translation) so internal refactors don't break external consumers. The integration event is a **published API**; the domain event is internal. ## Topic granularity — the central design choice **Option A: one event type per topic** (`order-placed`, `order-cancelled`). - Pros: clean single schema per topic; per-type retention/compaction; consumers subscribe only to what they need; simplest schema-registry story (TopicNameStrategy). - Cons: **no ordering across types** (a cancel could be processed before its place if on separate topics/partitions); topic **proliferation**. **Option B: many related event types on one topic** (an `orders` topic carrying placed/updated/cancelled). - Pros: **ordering preserved** across all events for an entity (same topic, keyed by entity → same partition); fewer topics; a consumer sees the full lifecycle in order. - Cons: heterogeneous schemas on one topic (needs `RecordNameStrategy`/`TopicRecordNameStrategy` so each type has its own schema subject); consumers must filter types they don't care about; retention policy is shared. **Guideline**: group event types that belong to the **same aggregate and need mutual ordering** onto one topic; split unrelated streams. Don't make 'one giant topic for everything' (loses parallelism, forces global filtering) nor 'a topic per field change' (sprawl). ## Keying **Key by the aggregate/entity id** (orderId, customerId). Kafka's default partitioner hashes the key, so all events for one entity land on the **same partition**, and Kafka guarantees **order within a partition** — giving per-entity ordering, which is what business correctness usually needs. Null keys → round-robin → no ordering. Keying also makes **log compaction** meaningful (latest value per key). ## Retention & compaction - `cleanup.policy=delete` + `retention.ms` / `retention.bytes`: time/size-based retention for **transient notifications** (you only care about recent events). - `cleanup.policy=compact`: **log compaction** keeps the **latest** record per key indefinitely — ideal for **state/ECST** events so a consumer can replay the topic and **materialize current state** of every entity (foundation of `KTable`). A **tombstone** (null value) deletes a key. - You can combine (`compact,delete`) for compacted topics that still age out tombstones. ## Partition count Partitions set the **max consumer parallelism** in a group (one partition per consumer at most) and affect ordering granularity. Over-partitioning wastes resources and increases end-to-end latency/rebalance cost; under-partitioning caps throughput. Choose for projected throughput and consumer count, and remember **increasing partitions later changes key→partition mapping** (breaks existing ordering and compaction grouping), so size with foresight. ## Pitfalls - Publishing raw internal domain events as the cross-team contract (brittle). - Splitting lifecycle events across topics when they need mutual ordering. - Null/poorly-chosen keys destroying ordering and breaking compaction. - Forgetting that adding partitions later re-shuffles key placement.