In an ADF Mapping Data Flow, when should you change the Optimize tab away from Use current partitioning?

## What the Optimize tab offers Every transformation node in a Mapping Data Flow - source, join, aggregate, sink and everything in between - has three tabs, and the third is **Optimize**. It offers three choices: - **Use current partitioning** (the default): keep whatever partitioning the incoming stream already has and let the service decide. - **Single partition**: collapse the stream into one partition. - **Set partitioning**: choose a scheme explicitly - *round robin* (with a partition count), *hash* (on chosen columns), *dynamic range*, *fixed range*, or *key* (a column whose distinct values define the partitions). The important cultural point first: the default is the right answer for the overwhelming majority of transformations. Setting partitioning on every node because you once read that partitioning matters is a reliable way to make a flow slower. The Optimize tab is a targeted instrument, not a checklist item. ## The cases that justify an override **Reading a relational source in parallel.** A database source read that is left to a single stream can be the bottleneck of the whole flow. Source-side partitioning splits the read into ranges or hashes so several readers pull concurrently. This needs a column with reasonable distribution - a monotonic key or a well-spread numeric or date column. Partitioning on a column with a handful of distinct values just creates a few enormous partitions. **Co-locating data ahead of an expensive operation.** Joins and aggregates work on matching keys. If the stream is already partitioned by the join or grouping key, matching rows are together and the operation does less movement. Hash or key partitioning on that column before a heavy join can pay for itself. This is also the case where the choice is easiest to get wrong: if the key is skewed, you have simply moved the problem into one very large partition. **Fighting skew with no usable key.** When no column distributes well, round robin with an explicit partition count spreads rows evenly by construction. It destroys locality, so it is the wrong choice immediately before a join on a key, and a reasonable one before independent per-row work. **Controlling the shape of the output.** This is the most concrete and most frequently asked case. A sink writing to a data lake produces files per partition, so partitioning drives the output layout: key partitioning by, say, a date column gives you a folder or file per date, while a stream with many partitions gives you many files. When the requirement is a single output file - a downstream consumer that cannot glob a folder, or a small extract for a partner - the sink's Optimize tab must be set to single partition, which is also what the sink's single-file naming option depends on. ## The cost of single partition Single partition is the setting people reach for casually and regret. Everything funnels through one worker: one partition's worth of memory, no parallelism, and a hard ceiling on throughput. On a small result set that is fine and is exactly what you want for a single-file extract. On a large one it can turn a comfortable flow into a memory failure. If a downstream system merely dislikes many *tiny* files, the better answer is usually to reduce partition count moderately or to partition by a meaningful key, not to collapse to one. ## Measure, then change The data flow monitoring detail for a run shows the transformation graph with row counts and per-stage timings. That is where you find out whether a stage is genuinely slow or whether the activity duration was dominated by cluster acquisition - a distinction that decides whether partitioning is even the right lever. Change one setting, rerun on comparable data, and compare the same stage. Partitioning changes interact, so changing several nodes at once tells you nothing about which one helped. It is also worth remembering what partitioning does not fix. It does not reduce cluster startup time. It does not compensate for reading far more data than you need - a filter pushed into the source query beats any partitioning scheme. And it does not rescue a flow whose real problem is that a fundamentally SQL-shaped transformation is being done on an external engine at all. ## How to answer this in an interview Say that the default is correct by default, name the schemes, then give two or three concrete triggers - parallelising a database read, co-locating a join key, and shaping sink output including the single-file case - and finish with the caution about single partition and the habit of measuring in the monitoring view. That reads as someone who has tuned a real flow rather than someone reciting options.