After you set an EC2 instance's metadata options to require IMDSv2, applications in Docker containers on that host can no longer read instance role credentials from 169.254.169.254, although the same request works on the host itself. What is the cause, and what is the fix?

## The symptom You harden a host by setting `HttpTokens=required`. Immediately, applications running in containers on that host start failing to obtain AWS credentials: SDK calls report that no credential provider succeeded, and a manual `curl -X PUT http://169.254.169.254/latest/api/token` from inside the container times out or returns nothing. The same command run in an SSH session on the host returns a token instantly. Nothing about IAM changed, and the instance profile is still attached. ## Why the container is different The missing piece is `HttpPutResponseHopLimit`. It is not a rate limit and not a token lifetime — it is literally the **IP time-to-live written into the token response packet**. TTL is decremented by each router the packet crosses; at zero it is discarded. That is a deliberate defence. If something outside the instance ever managed to have a token request made on its behalf, the response would have to be routed back out of the instance to reach it — and with a TTL of 1 it dies on the first forwarding step. The token cannot leave the box. A container on Docker's default bridge network is on the far side of exactly such a forwarding step. Its packets are routed and NAT-ed by the host's network stack, so from the metadata service's point of view the container is one hop further away than a process on the host. A TTL of 1 gets the response as far as the host and no further. A container using host networking has no extra hop, which is why the failure looks selective and inconsistent across workloads on the same machine. ## Why only the token step breaks The hop limit is applied to the PUT response, which is why the failure appears exactly when you enforce IMDSv2 and not before. While `HttpTokens` was `optional`, containers were reading metadata with plain IMDSv1 GETs, which never involve the token response — so a fleet can run for years with a hop limit of 1 and expose the problem only on the day someone hardens it. That is the shape of the incident: the change looks like an IAM or a permissions problem, and it is a TTL. ## The fix ```bash aws ec2 modify-instance-metadata-options \ --instance-id i-0123456789abcdef0 \ --http-tokens required \ --http-put-response-hop-limit 2 ``` The attribute can be changed on a running instance and takes effect without a restart. Do the same in the `MetadataOptions` block of the launch template, or the next instance the Auto Scaling group creates arrives back at 1 and the incident repeats — which is the second half of the answer an interviewer is listening for. ## Choose 2, not 64 The valid range runs to 64, and it is tempting to set a comfortable number. Do not. Each additional hop the token response may cross is another network element through which a token could be relayed, which erodes precisely the property that made IMDSv2 worth enforcing. Two is the value that makes one layer of container networking work and nothing more. If a workload needs three, that is a signal about its network topology worth understanding rather than accommodating. ## The better fix Raising the hop limit makes the host's instance role reachable from every container on the host — so every container shares the same credentials and the same blast radius. The stronger design is that containers should not be borrowing the host's identity at all: give each workload its own credentials, so that the container's permissions are scoped to what it does, and the host role can shrink to almost nothing. Container orchestrators on AWS all provide a per-workload credential mechanism for exactly this reason. Where that is available, the correct end state is a hop limit of 1, containers that never touch the instance metadata service, and an instance role narrow enough that reaching it would not be interesting. ## Diagnosing it quickly The tell is the asymmetry: metadata works from the host and not from a container, only the PUT fails, and it started the moment `HttpTokens` became `required`. Compare `curl -v -X PUT` from both sides, check `aws ec2 describe-instances` for the instance's `MetadataOptions`, and confirm the container's network mode. If the container is on host networking and still fails, the hop limit is not your problem and you should be looking at whether the container can reach link-local addresses at all.