When is dropping from the DataFrame API to Spark's RDD API still justified?

## Start from the presumption against it RDDs are not deprecated and are not going away — every DataFrame query still executes as tasks over partitions underneath. But writing RDD code today means giving up, at once: predicate pushdown and column pruning, operator reordering, whole-stage code generation, and the compact binary row layout. In PySpark it additionally means shipping every row into a Python worker process. So the burden of proof sits with the RDD, and "I find the functional API more readable" does not discharge it. ## Cases that genuinely justify it **Data that is not tabular.** Raw binary records, protocol dumps, irregular log formats where your parser is the first step. There is no schema to declare until after your code runs. `sc.binaryFiles` or `wholeTextFiles` plus `mapPartitions` is a reasonable front end; convert to a DataFrame the moment rows have a shape. **Explicit control of physical partitioning.** The RDD API lets you attach a custom `Partitioner` to a key-value RDD and keep that partitioning across operations, so repeated joins on the same key avoid a shuffle. The DataFrame API expresses partitioning through `repartition`, bucketing and the planner's own choices; when you have a very specific co-partitioning scheme the planner will not reproduce, the RDD API is the direct route. **Position-based and multi-dataset operators.** `zipWithIndex`, `zipWithUniqueId` and `zipPartitions` have no direct DataFrame equivalent with the same semantics. (`monotonically_increasing_id()` is *not* a dense sequence; claiming it is, is a common wrong answer.) **Library interop.** GraphX is an RDD-based API. Older `spark.mllib` entry points take RDDs, while `spark.ml` is DataFrame-based. Some third-party connectors still expose RDDs. **Fine-grained per-partition resource control.** Opening one database connection or loading one model per partition. This is a real need — but note `Dataset.mapPartitions` in Scala and `mapInPandas` / `applyInPandas` in PySpark cover it without leaving the structured API. ## What the conversion actually costs `df.rdd` is not a free view. Spark executes the plan up to that point and *materializes* every binary row into a JVM `Row` object — allocation, GC pressure, and the end of code generation from there on. In PySpark it goes further and pickles each row into the Python worker. Coming back with `spark.createDataFrame(rdd, schema)` requires you to restate a schema, and inference over an RDD triggers its own sampling job. A pipeline that bounces between the two APIs pays this toll each way. ## The modern alternatives you should reach for first Before `df.rdd`, check whether one of these does the job: the built-in functions in `pyspark.sql.functions` (there are hundreds, including higher-order functions over arrays and maps such as `transform` and `aggregate`); `pandas_udf` for vectorized scalar logic; `mapInPandas` for arbitrary per-batch Python over an iterator of pandas DataFrames; `applyInPandas` with `groupBy` for per-group logic; `Dataset.mapPartitions` or typed `map` in Scala. Most historical RDD use cases in real pipelines are covered by these, and they keep the rest of the plan optimizable. ## How to talk about it in an interview The strong answer has three moves. First, state the presumption: DataFrames by default because the optimizer needs to see intent. Second, give one or two concrete cases where the RDD API is genuinely the right tool, and be specific — a custom partitioner for repeated co-partitioned joins is a better answer than "low-level control". Third, describe containment: drop down for the smallest step, convert back immediately with an explicit schema, and never let an RDD conversion sit upstream of the joins and aggregations that most need optimizing. A weak answer says "RDDs are for unstructured data and DataFrames for structured data" and stops there. It is not wrong, but it does not show you have measured what the conversion costs or that you know the vectorized alternatives that removed most of the historical reasons to reach for RDDs at all. ## Migration reality On a legacy codebase full of RDD jobs, the pragmatic path is not a rewrite. Push the source read and the final write into the DataFrame API first — that alone recovers pushdown and columnar I/O — then replace shuffle-heavy `reduceByKey`/`join` middles, and leave the genuinely opaque custom logic in `mapPartitions` until it earns a rewrite.