In pydantic-settings, a database password moved to a file in secrets_dir, yet production still connects with an old value - what in the source priority explains it?
answer
- secrets sit near the bottom
- environment beats dotenv beats secrets
- a leftover variable shadows the file
- settings_customise_sources reorders
- debug flag lists every source
basics
~10 sBy default pydantic-settings ranks initialiser arguments, then environment variables, then the .env file, then secrets_dir files, then defaults; a leftover password variable or .env entry therefore shadows the new secret file.
solid answer
~40 spydantic-settings resolves each field from its sources in a fixed default order, highest first: CLI arguments (only when `cli_parse_args` is on), initialiser keyword arguments, environment variables, the `.env` file, files in `secrets_dir`, then field defaults. The secret file is almost the weakest source, so an old `APP_DATABASE_PASSWORD` still set in the deployment, or an entry in a `.env` file baked into the image, silently wins. To confirm, set `PYDANTIC_SETTINGS_DEBUG=1` with DEBUG logging, which logs each source's collected values in priority order, secrets included, so only in a trusted session. Fix it by removing the stale variable or by overriding `settings_customise_sources` to return the sources in the order you want, for example secrets above the environment. Also note that a missing `secrets_dir` only produces a warning.
code
python · 24 linesfrom pydantic import SecretStr
from pydantic_settings import BaseSettings, PydanticBaseSettingsSource, SettingsConfigDict
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_prefix="APP_",
env_file=".env",
secrets_dir="/run/secrets",
)
database_password: SecretStr
@classmethod
def settings_customise_sources(
cls,
settings_cls: type[BaseSettings],
init_settings: PydanticBaseSettingsSource,
env_settings: PydanticBaseSettingsSource,
dotenv_settings: PydanticBaseSettingsSource,
file_secret_settings: PydanticBaseSettingsSource,
) -> tuple[PydanticBaseSettingsSource, ...]:
# first item wins: secret files now outrank the environment and .env
return init_settings, file_secret_settings, env_settings, dotenv_settingsgo deeper
Learn the default order: initialiser arguments, environment variables, .env file, secrets directory, defaults, highest first.
Explain deep merging of nested blocks, how settings_customise_sources reorders or drops sources, and that later env_file and secrets_dir entries override earlier ones.
Diagnose a shadowed value methodically: list each source's contribution with the debug flag in a safe session, find the stale variable or baked-in .env, then fix the source rather than the symptom.
Decide which source is authoritative for secrets across services, and encode it in settings_customise_sources or deployment policy so a leftover variable cannot quietly override a rotation.
## The symptom A team moves the database password out of an environment variable and into a **mounted secret file**, `/run/secrets/APP_DATABASE_PASSWORD`, read through `secrets_dir="/run/secrets"`. They rotate the password. The service keeps authenticating with the old one. Nothing errors, because a value was found; it was just the wrong one. ## The default priority order When a field can be filled from several places, pydantic-settings picks the value in this order, **highest priority first**: 1. CLI arguments, but only when `cli_parse_args` is enabled; 2. **keyword arguments** passed to the `Settings` initialiser; 3. **environment variables**; 4. entries in the **dotenv** (`.env`) file; 5. files in the **secrets directory**; 6. the field's **default** value. The documentation states both halves of the trap plainly: environment variables always take priority over values from a dotenv file, and a dotenv file and environment variables always take priority over values loaded from the secrets directory. A secret file is the second weakest source. So the old password is coming from one of two places above it: - an `APP_DATABASE_PASSWORD` **environment variable** still set in the deployment manifest, the image or the shell that starts the process; - an `APP_DATABASE_PASSWORD` **line in a `.env` file** copied into the image. ## How to prove which source supplied the value - Set `PYDANTIC_SETTINGS_DEBUG=1` and enable `DEBUG` logging for the `pydantic_settings` logger. The library then logs, for each settings class, the dictionary every source collected, in priority order (`InitSettingsSource`, `EnvSettingsSource`, `DotEnvSettingsSource`, `SecretsSettingsSource`, `DefaultSettingsSource`), and also which dotenv and secret files were found or skipped. - That output **contains the secret values themselves**. The documentation warns to enable it only in a trusted debugging context and not to leave it on in production. A methodical diagnosis runs in this order: 1. list what each source collected, with the debug flag in a safe session or by inspecting the process environment and the image's files; 2. find the highest source that holds the field, since that one decides the value; 3. remove the stale copy at its origin, the manifest, the image or the `.env` file, rather than patching the value in code; 4. restart and confirm the secret file is now the highest source that holds the field. ## How merging works for nested blocks Sources are combined with a **deep update**, not a whole-field replace. Higher-priority sources override lower ones key by key, and a lower source can still contribute sub-keys the higher source did not set. A `database` block can take `host` from the environment and `name` from the `.env` file. The same rule that helps here also hides the bug: a stale key at a higher level silently overrides a fresh one below it. ## How to fix it | Fix | When it fits | |---|---| | delete the stale variable and the `.env` entry | the secret file should be the only source | | override `settings_customise_sources` | the team wants secret files to outrank the environment by design | | `_env_file=None` when constructing in production | a `.env` meant for development ships in the image | `settings_customise_sources` is a classmethod that receives `init_settings`, `env_settings`, `dotenv_settings` and `file_secret_settings` and returns a tuple; **the first item in the tuple is the highest priority**, and leaving a source out disables it. ```python from pydantic import SecretStr from pydantic_settings import BaseSettings, PydanticBaseSettingsSource, SettingsConfigDict class Settings(BaseSettings): model_config = SettingsConfigDict(env_prefix="APP_", env_file=".env", secrets_dir="/run/secrets") database_password: SecretStr @classmethod def settings_customise_sources(cls, settings_cls, init_settings, env_settings, dotenv_settings, file_secret_settings): return init_settings, file_secret_settings, env_settings, dotenv_settings ``` ## The related traps - **Secret file names follow the variable names.** With `env_prefix="APP_"`, the file is `APP_DATABASE_PASSWORD` (matched ignoring case by default), not `database_password`. Its content is read as UTF-8 and stripped of surrounding whitespace. - **A missing secrets directory is only a warning.** If the mount is absent, the source returns nothing and the field falls through to the next source or its default. A required field then raises `missing`; a field with a default silently keeps it. - **Multiple files: later wins.** `env_file=(".env", ".env.prod")` and a tuple of `secrets_dir` paths both let later entries override earlier ones, the opposite of the tuple returned by `settings_customise_sources`.
- Why does the order rule for env_file tuples look reversed compared with settings_customise_sources?They answer different questions. `env_file=(".env", ".env.prod")` loads files in order and each later file overrides the earlier one, and a tuple of `secrets_dir` paths behaves the same way. The tuple returned by `settings_customise_sources` ranks whole sources, and there the first element has the highest priority.
- How do lower-priority sources still affect a nested database block that the environment already sets?Sources are merged with a deep update. If the environment supplies `database.host` and the `.env` file supplies `database.name`, both survive; only keys present in the higher source are overridden. That is useful for layering, but a stale nested key high up silently shadows a fresh one lower down.
- What happens if the production secrets mount is missing entirely?`SecretsSettingsSource` issues a warning that the directory does not exist and contributes nothing. A required field then fails with a `missing` ValidationError at startup, which is safe; a field with a default quietly keeps the default, which is not. A path that exists but is a file raises `SettingsError`.
Like a stack of transparent sheets laid over a printed form: wherever an upper sheet has ink, you read that ink; a lower sheet shows through only where everything above it is blank. The secret file is near the bottom of the stack.
- Secret files always outrank environment variables because they are more secure.
- The last source returned by settings_customise_sources has the highest priority.
- A missing secrets_dir makes Settings() raise an error at startup.
- Leaving PYDANTIC_SETTINGS_DEBUG on in production is harmless because SecretStr masks the values.
- Each source replaces a whole nested block rather than merging keys.