A nightly reconciliation job that worked against a laptop directory now reads an empty input tree on a scheduler-chosen machine — why?
answer
- the path is not the data
- resolved on whichever machine it lands on
- missing, empty, or somebody else's copy
- platforms differ on a missing directory
- an empty input can look like success
basics
~20 sA host path resolves on whichever machine the container lands on, so the path is not the data. On an unprepared machine the directory is missing or empty, and platforms differ on which of those you get.
solid answer
~50 sThe mount named a path, not a dataset. On a laptop that path was the drop directory you filled yourself; on a machine chosen by the scheduler it resolves against that machine's filesystem, where three things can happen: the directory is absent, it exists and is empty, or it exists with a different machine's leftovers. Platforms differ on the absent case — some refuse to start the container, some create an empty directory — and the second behaviour is the dangerous one, because the job then reconciles zero records and writes a perfectly valid settlement file covering nothing. The cause is that host-path storage belongs to a machine while the workload belongs to the scheduler. The fix is to take the inputs from storage the platform attaches wherever the job lands, and to treat an empty input set as an error rather than as a number.
go deeper
Take away the one fact: a host path is resolved on whichever machine runs the container, so moving the workload moves it away from the data. The directory name being the same does not make the contents the same.
Explain the three outcomes on an unprepared machine — absent, empty, or holding something else — and why platforms differ on the absent case. Then say which of the three produces a wrong answer rather than an error.
This is the level the question is aimed at: separate when it read from where it read, refuse the retry, and name the guard that turns a silent zero-record run into a failure. Say where the output is stranded too.
The judgment is about what the platform guarantees to teams. If storage is something each team points at, every incident of this kind is new; if storage is something teams request, placement stops being a correctness input across the whole estate.
## What the mount actually promised A host path mount is a promise about a **path**, not about **data**. It says: at start-up, resolve this directory on the machine this container is running on and show it inside. On a laptop, that promise and the promise you thought you made are the same thing, because there is exactly one machine and you filled the directory yourself. Hand the same spec to a scheduler and the two come apart, because the scheduler chooses the machine and nothing tells it that this workload's data lives on one particular one. ## Three outcomes, and the dangerous one When the container starts on a machine that was never prepared, one of three things happens: 1. **The directory does not exist.** Platforms differ here: some refuse to start the container and you get a loud failure; others create an empty directory at that path so the mount can succeed. The second is quiet. 2. **The directory exists and is empty.** The job starts, finds no input files, and does exactly what it was written to do with an empty input set. 3. **The directory exists with something else in it.** Another workload's leftovers, or an older copy from when this workload last ran here, get reconciled as if they were today's drop. Outcome 2 is the one that costs money. A reconciliation batch that reads zero input files and writes one settlement file covering zero records has not crashed, has not logged an error, and has reported success. Everything downstream treats that file as the day's truth. A crash would have been better. ## Why it is not a timing problem The reflex is to assume the files had not arrived yet and to add a retry or a delay. That diagnosis is wrong and the fix makes it worse: the retry re-reads the same machine's directory, which will still be empty on the next attempt and the one after, and the run now fails slowly instead of quickly. The distinguishing question is not *when* the job read, it is *where* it read. ## The reasoning chain This is a model question rather than a runbook, and the chain is short: 1. Which machine did this run land on, and is it the same one the last successful run used? 2. Does the path exist on that machine, and if so, what created it — the operator who prepared the original machine, or the platform creating an empty directory so the mount could succeed? 3. If the previous run wrote output through the same mount, that output is on the previous machine, so compare the two machines rather than the two runs. 4. Ask whether anything in the platform's storage inventory lists this data at all. If nothing does, no amount of looking at the workload will explain it. ## What follows for the output too The same property runs in the other direction. Anything the job **writes** through a host path is stranded on the machine it ran on: - a replacement elsewhere cannot see it; - two instances on two machines each accumulate their own partial history; - snapshot and backup tooling aimed at the platform's volumes never sees it, because the platform does not know it is data; - restoring it puts it back on one machine, which is the situation you were trying to leave. | Symptom | Host path | Platform-provisioned volume | |---|---|---| | Replaced on another machine | reads that machine's copy of the path | the same storage is attached again | | Missing on arrival | platforms differ: refuse, or create empty | the platform provisions it | | Two instances, two machines | two independent datasets | the platform enforces its writer rules | | Included in storage inventory and snapshots | no | yes | ## The fix, and the workaround The fix is to stop pointing at a machine. Take the inputs from storage the platform attaches wherever the job lands, or from a shared store the job reads over the network, so that placement stops being an input to correctness. Then make the job defend itself: an empty input set should be an explicit error, and a run should assert an expected manifest or a plausible record count before it is allowed to write a settlement file. That guard is worth having regardless of where the storage comes from, because it converts a silent wrong answer into a failed run. The workaround is to constrain placement so the workload only ever runs on the prepared machine. It makes the path resolve, and that is all it does. You have given up the scheduler's ability to replace the workload after that machine fails, so the workload's availability is now that machine's availability, and the data still exists in exactly one place with no supported way to move it. It is a decision with an expiry date, and it should be written down as one.
- What breaks about backups when a workload's state lives on a host path?Nothing knows it is state. The platform's storage inventory does not list it, so snapshot and backup tooling aimed at volumes misses it entirely, and whatever backs up that machine has to be configured by hand. A restore also puts the data back on one machine, which is where the trouble started.
- Would pinning the workload to the prepared machine fix it?It makes the path resolve, and nothing more. The scheduler can no longer replace the workload after that machine fails, so availability is now the machine's availability, and the data still lives in one place with no supported way to move it. Treat it as a dated workaround, not a design.
- Two copies of the batch ran on two machines through the same host path. What is the result?Two independent datasets. Each machine has its own directory, so each copy reconciled whatever its own machine held and wrote its own settlement file. Nothing detected the split, because the platform never knew the two mounts were meant to be the same storage.
saying these in an interview costs you the question
- Calls it a timing problem and adds a retry on the same machine.
- Assumes every machine in the fleet has the same directories prepared.
- Believes every platform refuses to start when a host path is missing.
- Thinks pinning the workload to one machine makes the storage durable.
- Treats a host disk as backed up because it is outside the container.
- Expects a snapshot of the platform's volumes to include host-path data.