How does TensorFlow's MultiWorkerMirroredStrategy find its peer workers?

## The configuration surface Multi-worker TensorFlow has no separate launcher binary that assigns ranks. Instead each process learns its identity from an environment variable, `TF_CONFIG`, whose value is a JSON object with two keys: - **`cluster`** — a map from task type to a list of `host:port` addresses. For `MultiWorkerMirroredStrategy` this is normally just `{"worker": [...]}`. The list must be **identical on every worker**, including its order, because position in the list *is* the worker's identity. - **`task`** — `{"type": "worker", "index": N}`, where `N` is this process's position in that list. So a two-machine job is the same script started twice with the same `cluster` and `index` 0 on one host and 1 on the other. Whatever orchestrates the job — a scheduler, a shell script, a Kubernetes job template — is responsible for injecting the right value; TensorFlow only reads it. `tf.distribute.cluster_resolver.TFConfigClusterResolver` is the object that parses it, and you can pass a resolver explicitly if your cluster information comes from somewhere else. ## Set it before the strategy exists The strategy constructor parses `TF_CONFIG`, opens the gRPC server for this task, and begins the peer handshake. Setting the variable after `MultiWorkerMirroredStrategy()` has been constructed does nothing. In notebooks, where the variable is often set in a cell after TensorFlow has already been imported and used, this is a frequent source of "it silently ran single-worker" confusion — check `strategy.num_replicas_in_sync` if you are unsure whether the peers were found. ## The handshake blocks Construction does not return until every worker in `cluster` has been reached. That is a deliberate design: collectives require all participants. The operational consequence is that a mistake in a hostname, a firewalled port, or one worker pod that has not been scheduled yet does not produce an error — it produces a hang, on every worker. When debugging a stuck multi-worker start, the first questions are always: is every address in the list resolvable and reachable from every other worker, and has every task actually been started? ## What the chief does By convention the task with `index` 0 in the `worker` list is the **chief**. It is a full training worker like all the others, plus it owns cluster-wide side effects: writing the authoritative checkpoint, writing TensorBoard summaries. This matters because writing from several workers to the same path corrupts it. The TensorFlow multi-worker guidance is that *every* worker must call the save API — because saving involves collective ops and skipping it on non-chiefs would deadlock — but non-chief workers must write to a **unique temporary directory** and delete it afterwards, leaving only the chief's output. Note that a separate `"chief"` task type also exists in `TF_CONFIG` and is used by some other strategies; with `MultiWorkerMirroredStrategy` the usual arrangement is worker 0. ## Collective communication Workers exchange gradients with collective ops. You can hint at the implementation with `tf.distribute.experimental.CommunicationOptions(implementation=tf.distribute.experimental.CommunicationImplementation.NCCL)` — the alternatives are `RING` and `AUTO`, the default. NCCL generally wins on GPU hosts with fast interconnect; `RING` is the fallback where NCCL is unavailable or unreliable. This is a tuning knob, not a correctness one. ## All-or-nothing execution Synchronous training means every worker must execute the same number of steps: each step includes an all-reduce that no worker can complete alone. Consequences worth stating in an interview: - **Gang scheduling.** The job needs all its workers simultaneously; N-1 workers do no useful work. - **The slowest worker sets the pace.** A heterogeneous fleet wastes the fast machines. - **A worker crash stops everything.** There is no in-place recovery. For the last point, the mitigation is `tf.keras.callbacks.BackupAndRestore`, which snapshots training state to `backup_dir` and, when the whole job is restarted, resumes from the beginning of the interrupted epoch rather than from scratch. It is restart-based recovery layered on an external supervisor, not live fault tolerance. ## Data and batching The global batch size now spans every replica on every worker, so a two-worker job with four GPUs each has 8 replicas. Each worker builds its own input pipeline and receives a shard of the data, which is the subject of auto-sharding.