You are choosing a stream processing engine for a Kafka-centric platform. What concrete factors drive the choice between Kafka Streams, Flink, and Spark Structured Streaming, and when would you pick each?

## Decision factors 1. **Operational ownership / org topology** — The biggest factor. Kafka Streams keeps processing inside an application a product team already builds, ships, and on-calls. Flink/Spark are typically a **shared platform** run by a data/platform team. Choosing the wrong boundary creates either bottleneck platform teams or product teams drowning in cluster ops. 2. **Latency** — Kafka Streams and Flink are **true (per-record) streaming**: low, predictable latency. Spark Structured Streaming is historically **micro-batch** (latency ~ trigger interval, often hundreds of ms to seconds); fine for many analytics use cases, less so for tight real-time SLAs (its low-latency modes narrow but don't fully erase this). 3. **State size and complexity** — Flink shines with very large state (RocksDB backend, incremental checkpoints) and complex event-time logic (CEP, flexible lateness). Kafka Streams handles substantial state but is bounded by partition-count parallelism and local-disk restore concerns. Spark handles big aggregations well, especially batch-shaped. 4. **Sources and sinks** — Flink and Spark have broad connector ecosystems (files, S3, JDBC, Iceberg/Delta/Hudi, many message systems) and rich SQL. Kafka Streams is **Kafka-first**; non-Kafka I/O typically goes through Kafka Connect or app code. 5. **Existing investment** — If you already run a Spark/Databricks lakehouse, Spark Structured Streaming unifies batch and streaming with one skillset and storage. If you already run Flink, extend it. 6. **Scaling and upgrades** — Flink's **savepoints** give clean rescale and code-change semantics; Spark restarts from checkpoints; Streams rescales via partition count and consumer-group rebalance. 7. **SQL and analyst access** — Flink SQL and Spark SQL let non-Java users build pipelines; Kafka Streams is code-centric (ksqlDB sits on top for SQL on Kafka but is a separate product). ## When to pick each - **Kafka Streams**: enrichment/aggregation/joins **within a microservice**, Kafka-to-Kafka, low latency, single-team ownership, no appetite for a cluster. E.g., per-order enrichment in an order service. - **Apache Flink**: large-scale, complex, low-latency stateful analytics, multi-source joins, CEP, strict event-time/lateness control, on a platform team's runtime. E.g., real-time fraud detection across many feeds. - **Spark Structured Streaming**: organizations standardized on Spark/lakehouse wanting unified batch+stream and SQL, tolerant of micro-batch latency. E.g., streaming ingestion into Delta/Iceberg tables alongside batch ETL. ## Anti-patterns / edge cases - Standing up a **Flink cluster for one small Kafka-to-Kafka transform** that Kafka Streams would do with zero extra infra. - Forcing **sub-100ms latency** onto Spark micro-batch when Flink or Streams is the right tool. - Using **Kafka Streams for joins across many heterogeneous external systems** it can't natively reach. - Ignoring **who operates the runtime** — the cheapest engine to run is the one your org already operates well. ## The meta-point There is rarely a single 'best' engine; the right answer is contextual. A senior/principal answer names the **factors and trade-offs**, ties them to **org topology and existing platforms**, and gives **concrete picks** rather than declaring one engine universally superior.