skip to content

os.environ or a mounted secret file for a long-running service's API key - how do you choose?

level: principalimportance: should knowfreq 40%

answer

  1. Both hand over plaintext; compare the blast radius
  2. One of them children inherit automatically
  3. One of them the filesystem can protect
  4. Only one can change without a restart
  5. Reload plus retry equals a duplicate submission

basics

~20 s

An environment variable is inherited by child processes, appears wherever the environment is dumped, and is fixed for the process lifetime. A mounted file carries filesystem permissions, is not inherited, and can be re-read after rotation.

solid answer

~50 s

Both are delivery mechanisms, and the deciding question is usually lifetime, not secrecy. `os.environ` is a snapshot of the process environment taken at interpreter start: what you leave in it is handed to every child started with the default environment, appears in anything that serializes the environment, and cannot change without a restart. A mounted file has an owner and mode you can check with `os.stat`, is not inherited, can live on a memory-backed mount, and can be re-read when the writer replaces it, so a long-running bidder picks up a rotated exchange key without restarting. The cost is a filesystem dependency and a reload path that must be right: read atomically, guard the swap, and keep the retry after a reload idempotent, or the bid that failed on the stale key is submitted twice. My default is to support both, with a `NAME_FILE` variable naming the path, and fail closed when neither is present.

code

python · 20 lines
python
import os
import stat
from pathlib import Path


def load_credential():
    path = os.getenv("EXCHANGE_API_KEY_FILE")
    if path:
        mode = stat.S_IMODE(os.stat(path).st_mode)
        if mode & 0o077:
            raise SystemExit(f"{path} is readable by group or other: {mode:o}")
        return bytearray(Path(path).read_bytes().strip())
    value = os.getenv("EXCHANGE_API_KEY")
    if value is None:
        raise SystemExit("no credential: set EXCHANGE_API_KEY or EXCHANGE_API_KEY_FILE")
    del os.environ["EXCHANGE_API_KEY"]   # children no longer inherit it
    return bytearray(value.encode())


print(callable(load_credential))

go deeper

for a junior

Know that both are just ways to hand plaintext to the process, and that neither belongs in source control. Be able to read either one with os.getenv or a file read and to stop if the value is absent.

for a middle

Explain the concrete differences: children inherit os.environ unless you pass an explicit env, a file has permission bits you can verify, and a file can be read as bytes while an environment value arrives as a str.

for a senior

Show the operational side: verifying the file mode, failing closed at startup, deleting the variable after reading it, and a reload path that reads atomically and does not turn one rejected request into a duplicated submission.

for a principal

Own the decision and its blast radius: which delivery the platform standardises on, how rotation happens without restarts, what the idempotency contract is for work that spans a credential change, and where the boundary with the secret store sits.

## Frame the question correctly Neither mechanism is "the secure one". Both hand a plaintext credential to a process that will hold it in memory. What differs is the blast radius of the delivery: who else can see it, what copies it leaves behind, and whether it can change without a restart. For a service that runs for weeks, that last property usually decides the argument. ## What os.environ actually is At interpreter start, CPython reads the process environment into `os.environ`, a mapping that behaves like a dict. Mutating it also calls the C-level setter so that the change is visible to child processes; deleting an entry calls the corresponding unsetter. Two consequences follow. First, **inheritance**. Any child started with the default environment gets the whole thing, including the credential. A service that shells out to a helper, a converter, or a health-check script is handing that credential to code with a different review history. The mitigation is to pass an explicit `env` mapping to `subprocess` rather than the process environment, and to delete the variable from `os.environ` once you have read it. Note the limit of the second step: the copy the kernel exposed at process start does not necessarily change when you unset a variable, so deleting it reduces inheritance, not forensic visibility. Second, **breadth of exposure**. On a typical Linux host another process running as the same user can read the environment of a running process. Crash reporters, application-performance agents and "dump my configuration" endpoints routinely serialize the environment wholesale. Orchestrator inspect output shows variables declared on the image or the deployment object, and a variable baked into an image layer is copied into every registry, cache and backup that image touches. None of these are exotic; they are the normal way environment-delivered secrets escape. Third, **immutability for the process**. The environment is fixed once the process is running. Nothing outside can change it. Rotating the credential means restarting. ## What a mounted file gives you A file has the properties the environment lacks. It is protected by ordinary filesystem permissions, so the process can verify the mode before trusting it and refuse to start if the file is group- or world-readable. It is not inherited by children; a child gets it only if it can read the path. It can be placed on a memory-backed mount so the bytes never reach persistent storage. It can be read as **bytes**, which is the only way to keep the credential out of an immutable `str` and to zero the buffer afterwards. And crucially it can be replaced, which makes rotation without a restart possible. The costs are real. It is a filesystem dependency that can be misconfigured into a missing file or an empty file, so the startup path must fail closed and exit rather than continuing with an empty credential and retrying forever. A writer that is not atomic can be caught mid-write, so the contract must be write-then-rename. The path is easy to get wrong in a way that leaves the file inside a build context or a source tree. And the value must not be logged when the path is logged. ## Rotation is where the design is decided Consider an ad-auction bidder that authenticates to an upstream exchange. The credential is rotated on a schedule. With an environment variable, rotation is a restart, and a restart of a bidder mid-flight drops in-progress auctions. With a mounted file, the process can re-read the path when the previous credential is rejected, or when the file's modification time changes, and continue. That reload path is where a subtle failure lives. The natural implementation is: a request is rejected as unauthorized, so reload the credential and retry the request. If the operation is not idempotent, the retry duplicates it. A bid that the exchange actually accepted before the rejection was generated is now submitted twice, and on a campaign sitting at the 92nd percentile of the account's budget distribution the duplicate is expensive and visible. The fixes are ordinary but must be chosen deliberately: attach an idempotency key so the upstream collapses duplicates, or refuse to retry non-idempotent operations and let the in-flight request fail while the reload happens out of band. Also guard the reload itself so a burst of rejections triggers one re-read rather than one per request. ## The shape I actually ship Support both, with the file winning. Read `NAME` from the environment for local development and small deployments; read `NAME_FILE` for a path, and prefer it when present. The path itself is not a secret, so passing it through the environment is fine. Verify the file's permission bits before reading. Read it as bytes. Delete `NAME` from `os.environ` immediately after reading it so children do not inherit it, and pass an explicit environment to any subprocess. Exit non-zero when neither source yields a credential: a service that starts unauthenticated and hammers an upstream with rejected requests is worse than one that does not start. What is out of scope here is where the secret came from before it reached the container - a broker, a sealed manifest, an operator-managed store - and the policy for how often it rotates. That is a platform decision. This question is only about the last hop into the process, and about which of the two hops leaves fewer copies and permits a restart-free rotation.

  • How do you stop a credential in os.environ from reaching child processes?
    Read it once at startup and delete the entry from `os.environ`, which unsets it in the real process environment so later children do not receive it, and pass an explicit `env` mapping to every `subprocess` call rather than inheriting. Be honest about the limit: on Linux the environment block the kernel exposes for the process start is not rewritten by an unset, so this reduces inheritance rather than erasing evidence.
  • What does a safe credential-reload path look like for a long-running service?
    The writer replaces the file by atomic rename so a reader never sees a partial value. The reader re-reads on an authorization failure or on a modification-time change, under a lock so a burst of failures causes one reload rather than many. The retry that follows must be idempotent, or non-idempotent work must be failed rather than retried, otherwise the reload turns one rejected request into two submitted ones.
  • The mounted file is missing at startup because the mount was misconfigured. What should the process do?
    Exit non-zero with a message naming the expected path. Never fall back to an empty string, a placeholder or an anonymous mode: a service that starts without its credential will send a stream of rejected requests upstream, may trip rate limits or lockouts, and hides the misconfiguration behind a running-but-useless process. Failing to start is the loud, correct outcome.

An environment variable is a note pinned to the door that everyone who walks through carries away a copy of; a mounted file is a note in a locked drawer that can be swapped for a new one while the office stays open.

saying these in an interview costs you the question

  • Claims environment variables are private to the process
  • Believes the orchestrator encrypts variables at runtime
  • Says a file on disk is always weaker than an env var
  • Thinks rotation is done once the file is rewritten
  • Ignores that children inherit the environment by default
  • Falls back to an empty credential when the source is missing

context