In an ECS task definition, what is the difference between putting a value in a container's `environment` list and its `secrets` list, and what happens to an already-running task when the referenced secret is rotated?

## Two ways to get a value into a container An ECS container definition can populate environment variables two ways: ```json "environment": [ { "name": "LOG_LEVEL", "value": "info" } ], "secrets": [ { "name": "DB_PASSWORD", "valueFrom": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:prod/db-AbCdEf" } ] ``` Inside the container both appear identically — `LOG_LEVEL` and `DB_PASSWORD` are both environment variables. The difference is entirely about where the value lives before the container starts. ## What `environment` costs you The literal string is part of the task definition document. That has consequences people underestimate: - **It is readable.** `ecs:DescribeTaskDefinition` is a common, low-suspicion permission, frequently granted broadly for deployment tooling and dashboards. Anyone holding it reads the value. - **It is immortal.** Task definitions are immutable revisions and old revisions are retained. A password committed in revision 12 stays readable long after revision 40 removed it, unless the revision is explicitly deregistered and deleted. - **It usually lives in source control**, because the definition is generated from a template checked into a repository. So `environment` is right for configuration — log levels, feature flags, endpoints — and wrong for anything you would not paste into a pull request. ## What `secrets` does A `secrets` entry stores only a reference. `valueFrom` accepts the ARN of a Secrets Manager secret or of an SSM Parameter Store parameter, and for Secrets Manager it can address a single JSON key and a version — the ARN can be suffixed with the JSON key name, the version stage and the version id — so one secret holding a JSON credential blob can populate several distinct environment variables. At task launch the ECS agent resolves each reference and injects the result. Two authorization points matter, and mixing them up is the standard failure: - The **task execution role** is what resolves the secret. It is the role ECS itself assumes to pull the image, write logs and fetch secrets, and it is the one that needs `secretsmanager:GetSecretValue` or `ssm:GetParameters`, plus `kms:Decrypt` when a customer managed key is involved. - The **task role** is what the application's own SDK calls use at runtime, and it is irrelevant to injection. Grant the permission to the task role by mistake and the task fails to start with a ResourceInitializationError before your code runs at all — which is at least a loud failure rather than a silent one. A parallel mechanism exists for the log driver: `logConfiguration.secretOptions` takes the same `{name, valueFrom}` shape, so a third-party log destination's API key is not in the definition either. ## The rotation gap Resolution happens once, at task start. There is no watcher, no refresh, no signal into the running container. If the secret rotates ten minutes later, every running task still holds the previous value, and will until something replaces the task — a new deployment, a scale-out that starts fresh tasks, a health-check failure that recycles one. That produces a specific and confusing incident shape: after a rotation, *newly started* tasks work and old ones fail, so the service is partially broken in proportion to how recently each task launched. It also means "we rotate our database password automatically" is only half true if consumption is by injection — the rotation succeeds and the fleet keeps using the old credential. The options, in increasing order of effort: 1. **Accept it** for values that rotate rarely, and make rotation a deployment: rotate, then force a new deployment (`aws ecs update-service --force-new-deployment`) so tasks pick up the new value. Simple, and adequate for a quarterly rotation. 2. **Read through the SDK** in the application with an in-process cache and refetch on authentication failure. The task role needs the read permission, and the value is never in the environment at all — which also keeps it out of anything that dumps the process environment into a crash report. 3. **Use alternating-users rotation** so the injected credential stays valid until the next deployment, buying time for option 1 to be unhurried. ## A limit worth naming Injection puts the value in the process environment, which is a slightly weaker resting place than a variable your code fetched: the environment is visible to every process in the container, is captured by many crash and diagnostic tools, and is inherited by child processes. `secrets` fixes the *task definition* exposure, not the *process environment* exposure. For most services that tradeoff is entirely reasonable — the point is to be able to say which problem you solved.