AWS SDKs resolve credentials through a default provider chain. What sources does it consult, in roughly what order, and why does that ordering explain a container that ignores its attached role and calls AWS as some other principal?

## What the chain is for The same application binary should run on a laptop, in CI, and on ECS without code changes, and pick up whatever credentials that context provides. The AWS SDKs achieve this with a **default credential provider chain**: an ordered list of providers, each asked in turn, the first one that returns credentials wins, and later providers are never consulted. Nothing about this is magic — it is a plain for-loop, and knowing its order is what makes the surprising cases obvious. ## Roughly the order Exact ordering differs slightly between SDK languages and versions, so state it as a shape rather than a memorised list: 1. **Credentials supplied explicitly in code** when constructing the client. 2. **JVM system properties** (`aws.accessKeyId`, `aws.secretAccessKey`) — Java SDK only. 3. **Environment variables**: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` for temporary credentials. 4. **Web identity token file**: `AWS_WEB_IDENTITY_TOKEN_FILE` plus `AWS_ROLE_ARN`, which triggers `sts:AssumeRoleWithWebIdentity`. This is how EKS IRSA and GitHub Actions OIDC deliver a role. 5. **Shared config and credentials files** (`~/.aws/credentials`, `~/.aws/config`), with `AWS_PROFILE` choosing the profile. A profile here can itself say `role_arn` + `source_profile`, causing an assume-role hop. 6. **Container credentials**: the relative or full URI variable ECS and EKS Pod Identity inject, served by a local agent. 7. **EC2 instance metadata (IMDS)** — the instance profile role, consulted last. The principle behind it: the more explicitly a human stated the credential, the earlier it is tried; the ambient identity the platform attached is the fallback. ## Why the ordering produces the classic bug Because the platform-provided role sits at the bottom, *anything* left above it wins. Real cases: - A `Dockerfile` with `ENV AWS_ACCESS_KEY_ID=...` from a debugging session. The image runs on ECS with a perfectly good task role and never uses it — until the baked key is rotated and every task starts failing at once. - A CI runner that exports keys for one step and leaks them into the whole job's environment, so a later step that should use an OIDC-assumed role runs as the old user. - `AWS_PROFILE=prod` copied into a container's environment from someone's laptop; the file it names does not exist in the image, so the SDK errors out with a profile-not-found instead of falling through to the role. - A `~/.aws/credentials` file baked into a base image, quietly outranking the instance profile on every host that runs it. Each of these looks like an IAM problem — AccessDenied — and is actually a resolution problem. The permission you keep widening is on a role nobody is using. ## Diagnosing it in one command ```bash aws sts get-caller-identity # arn:aws:sts::111122223333:assumed-role/orders-task/abc123 <- the task role, good # arn:aws:iam::111122223333:user/legacy-deploy <- an IAM user, someone left keys around ``` Run it inside the failing container (on ECS, via `aws ecs execute-command`), not on your machine. `aws configure list` complements it by showing which *source* each config value came from (`env`, `shared-credentials-file`, `iam-role`), which pinpoints the shadowing provider. In application code, most SDKs let you log the resolved provider or construct a client with an explicit provider to test a hypothesis. ## Hygiene that prevents it - Never bake credentials or `AWS_PROFILE` into an image; strip them in the Dockerfile if a build stage needs them. - Treat the presence of `AWS_ACCESS_KEY_ID` in a production container as an alarmable condition. - Keep to the default chain in application code. Hard-coding a specific provider makes the binary non-portable across laptop, CI and production, which is the whole point of the chain. - Remember the chain is also *how* the good path works: IRSA, ECS task roles and instance profiles are simply providers late in the same list, which is why correctly configured workloads need zero credential code.