In Azure Data Factory, why can a Set Variable inside a parallel ForEach produce wrong values?
answer
- the loop does not create fresh state
- one slot, many writers
- the default is not one-at-a-time
- give each iteration its own run instead
basics
~20 sBecause pipeline variables are scoped to the whole pipeline run, not to a loop iteration. A ForEach runs its iterations in parallel by default, so concurrent Set Variable activities overwrite each other and later activities read whichever write landed last.
solid answer
~50 sA ForEach activity iterates over an array from `items` and, unless `isSequential` is true, runs iterations **in parallel** with a bounded batch count (default 20, maximum 50). Variables declared on the pipeline are single-instance state for the whole run — there is no per-iteration copy — so two iterations calling `Set Variable` on the same variable race, and any activity that reads it gets an arbitrary winner. There are three clean fixes. Set `isSequential: true` when the loop really is order-dependent, accepting the loss of parallelism. Move the loop body into a child pipeline invoked by `Execute Pipeline`, passing `@item()` as a **parameter** — parameters are per-run and immutable, so each child run has its own value. Or avoid the variable entirely and reference activity output directly with `@activity('Name').output`, which is scoped to that iteration's activity instance.
code
json · 21 lines{
"name": "ForEachTable",
"type": "ForEach",
"typeProperties": {
"items": "@activity('LookupTables').output.value",
"isSequential": false,
"batchCount": 8,
"activities": [
{
"name": "RememberTable",
"type": "SetVariable",
"typeProperties": { "variableName": "currentTable", "value": "@item().tableName" }
},
{
"name": "CopyOne",
"type": "Copy",
"dependsOn": [{ "activity": "RememberTable", "dependencyConditions": ["Succeeded"] }]
}
]
}
}go deeper
Know that a ForEach loops over an array, that @item() is the current element, and that pipeline variables belong to the run rather than the loop.
Explain the default parallel execution and the batch count bound, and why single-instance variable scope makes concurrent Set Variable calls race silently.
Diagnose the symptom from evidence — identical values across audit rows, no failures — and give the child-pipeline-with-parameters fix, plus how you would bound concurrency against the source.
Own the pattern library: metadata-driven loops that delegate to a parameterised child pipeline, concurrency budgets across pipeline and trigger levels, and reviews that catch shared mutable state before it ships.
## What ForEach actually does The `ForEach` activity takes an `items` expression that must evaluate to an array — typically `@activity('LookupTables').output.value` from a control table, or a pipeline parameter holding a list. Inside the loop, `@item()` is the current element, so `@item().tableName` reads a field of the current row. Two settings shape execution: - `isSequential` — when `false` or absent, iterations run **concurrently**. - `batchCount` — the parallelism bound, defaulting to 20 with a maximum of 50. It is ignored when `isSequential` is true. Parallel-by-default is the part people forget. A loop that behaves perfectly with three test rows starts misbehaving at thirty, which is why this shows up as a production question rather than a design question. ## Variables are pipeline-scoped A pipeline declares `variables` alongside `parameters`. The two are not symmetric: - **Parameters** are supplied when the run starts (by a trigger, a parent pipeline, or a manual invocation) and are **immutable** for the life of the run. Read with `@pipeline().parameters.name`. - **Variables** are mutable during the run, written with `Set Variable` or `Append Variable` and read with `@variables('name')`. There is exactly **one** instance of each variable per pipeline run. That single instance is the whole problem. A `Set Variable` inside a parallel `ForEach` is twenty concurrent writers to one slot. Whatever a downstream activity reads is a race outcome, and — worse — it usually *looks* right in a small test where iterations happen to serialise. The symptom in production is a loop of twenty table loads where nineteen rows in the audit table say the same table name, or where a "current file" variable used to build a sink path sends several iterations to the same destination. It is a data-corruption bug, not a cosmetic one, and it does not raise an error. ## Fix 1 — make the loop sequential Setting `isSequential: true` serialises iterations, so a single shared variable is safe. Use it when the loop genuinely is order-dependent or when the target cannot take concurrent writes. The cost is wall-clock time: twenty five-minute copies become a hundred minutes. ## Fix 2 — push the body into a child pipeline The idiomatic ADF fix is to move the loop body into its own pipeline and call it with `Execute Pipeline`, passing `@item()` (or its fields) as **parameters**. Each child run is a separate pipeline run with its own parameter values and its own variables, so nothing is shared. This also sidesteps ADF's nesting restriction: a `ForEach` cannot directly contain another `ForEach` or an `Until`, so a child pipeline is required for nested iteration anyway. One detail to know: `Execute Pipeline` has a `waitOnCompletion` flag. Leave it `true` if the parent must fail when the child fails; with it `false` the parent fires and forgets, and a child failure will not surface in the parent's status. ```json { "name": "LoadOne", "type": "ExecutePipeline", "typeProperties": { "pipeline": { "referenceName": "LoadOneTable", "type": "PipelineReference" }, "waitOnCompletion": true, "parameters": { "tableName": "@item().tableName" } } } ``` ## Fix 3 — do not use a variable at all Much of what people store in variables is already available as an expression. `@item()` is the loop element. `@activity('CopyOne').output.rowsCopied` is that iteration's own activity output and is *not* shared state. `@pipeline().RunId`, `@pipeline().TriggerTime` and `@pipeline().Pipeline` cover run metadata. Reaching for `Set Variable` to "remember" something inside a loop is usually a sign of importing an imperative-programming habit into a control-flow engine that does not have local scope. Accumulating results is the one case where people legitimately want state. `Append Variable` on an array is atomic per call, so appending inside a parallel loop does not lose entries the way overwriting does — but the resulting order is nondeterministic, so never rely on the array's ordering matching the input. ## What good answers include Name the scope (pipeline-wide, one instance), name the default (parallel, bounded batch count), and give the child-pipeline pattern as the primary fix rather than reaching straight for `isSequential`. Mention that the bug is silent — no failure, just wrong values — because that is what makes it a senior-level war story instead of a syntax question.
- Is Append Variable safe inside a parallel ForEach?Safer than Set Variable, because each append adds an entry rather than overwriting the slot, so entries are not lost. The resulting order is nondeterministic though, so never assume the array matches the input order or that positions line up with iterations.
- Can a ForEach contain another ForEach in Azure Data Factory?Not directly — a ForEach cannot nest another ForEach or an Until activity. The supported pattern is to move the inner loop into its own pipeline and invoke it with Execute Pipeline, passing the outer @item() as a parameter, which also gives the inner loop isolated variables.
- How do you keep a parallel ForEach from overwhelming the source system?Lower batchCount so fewer iterations run at once, or set isSequential when the source cannot take concurrency at all. Combine that with the trigger's own concurrency limit and the pipeline's concurrency setting, so a backfill of many runs does not multiply the parallelism you tuned for a single run.
- What does waitOnCompletion false do on an Execute Pipeline activity?The parent starts the child run and moves on immediately without waiting for its outcome, so a child failure does not fail the parent. Use it only for genuine fire-and-forget work; for a loop whose success matters, leave it true so failures propagate.
A pipeline variable is one whiteboard at the front of the room; twenty people writing their own name on it in parallel does not give twenty names, it gives one arbitrary name.
saying these in an interview costs you the question
- Assumes ForEach iterations run sequentially by default
- Thinks each iteration gets its own copy of a variable
- Uses Set Variable to pass values between loop iterations
- Nests ForEach inside ForEach and expects it to validate
- Blames flaky sources for values that are actually a race