An application in a private subnet reached S3 through a NAT gateway, and the bucket policy allowed only the NAT gateway's Elastic IP with an aws:SourceIp condition. After you add an S3 gateway VPC endpoint, requests start failing with 403 Access Denied. Why, and what do you change?
answer
- the path changed, not the permissions
- S3 no longer sees the Elastic IP
- the allow's condition simply stops matching
- two keys name the endpoint and the VPC
- widen the policy before flipping the route
basics
~20 sTraffic through a VPC endpoint no longer arrives from the NAT gateway's public Elastic IP, so an aws:SourceIp condition matching that address stops matching and the bucket policy denies the request. Replace the condition with aws:SourceVpce or aws:SourceVpc.
solid answer
~50 sThe endpoint changed the network path, and with it the request's apparent origin. Going through NAT, S3 saw the NAT gateway's public Elastic IP, which is what `aws:SourceIp` was pinned to. Once the gateway endpoint is in the route table, the request travels the private AWS path and S3 no longer sees that public address, so the condition evaluates false and — because the bucket policy only granted access under that condition — the result is a 403. The fix is to express the same intent in endpoint terms: allow when `aws:SourceVpce` equals the endpoint id, or when `aws:SourceVpc` equals the VPC id if several endpoints in that VPC should qualify. During a migration you can allow either the old IP or the new endpoint so the two paths coexist, then drop the IP clause once traffic has moved. This is worth calling out before anyone flips an endpoint on: any resource policy anchored to a NAT address is going to break.
code
json · 16 lines{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowEitherPathDuringMigration",
"Effect": "Allow",
"Principal": { "AWS": "arn:aws:iam::111122223333:role/app-role" },
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::example-bucket/*",
"Condition": {
"StringEqualsIfExists": { "aws:SourceVpce": "vpce-0a1b2c3d4e5f" },
"IpAddressIfExists": { "aws:SourceIp": "203.0.113.10/32" }
}
}
]
}go deeper
Know that a VPC endpoint changes how a request reaches the service, and that a policy allowing only a specific public IP address can stop matching once traffic takes the private path.
Explain that S3 no longer sees the NAT gateway's Elastic IP, so the aws:SourceIp condition evaluates false and the conditional allow no longer applies. Name aws:SourceVpce and aws:SourceVpc as the replacements.
Show the change-management instinct: widen the policy to accept both paths, flip the route, verify with CloudTrail data events, and only then remove the IP condition. Be able to diagnose the 403 from a log line rather than by guessing.
Own the data-perimeter standard. Decide that network provenance is expressed with VPC and endpoint condition keys rather than NAT addresses, that it is enforced as explicit denies with named exemptions, and that endpoint rollouts are audited against resource policies before they are enabled.
## What actually changed Nothing about the caller's IAM permissions changed, and nothing about the bucket changed. What changed is the *path*, and AWS resource policies can condition on properties of the path. Before the endpoint, the instance had no public address, so its S3 requests were source-NATed by the NAT gateway and arrived at S3 carrying the NAT gateway's Elastic IP. A bucket policy written as "allow `s3:GetObject` only when `aws:SourceIp` is 203.0.113.10" is really saying "only when the request came out of that NAT gateway", and it worked. After the gateway endpoint is associated with the route table, longest-prefix matching sends S3-bound traffic to the endpoint rather than to the NAT gateway. The request now reaches S3 over the private AWS network. It is no longer presented with the Elastic IP, so the `aws:SourceIp` condition no longer matches. Since the only `Allow` in the bucket policy carried that condition, the request is left with no applicable allow and S3 returns `403 Access Denied` — even though the instance's IAM role is untouched and perfectly adequate. ## The condition keys that replace it AWS exposes the endpoint path through global condition keys designed for exactly this: - `aws:SourceVpce` — matches a specific VPC endpoint id (`vpce-...`). Use it to say "only through *this* endpoint". - `aws:SourceVpc` — matches the VPC id. Broader: any endpoint in that VPC qualifies, which is convenient when you have several or when endpoint ids are recreated by automation. - `aws:VpcSourceIp` — the *private* IP of the requester as seen through the endpoint, for the rare case where you need per-host granularity. A corrected bucket policy statement looks like this: ```json { "Effect": "Deny", "Principal": "*", "Action": "s3:*", "Resource": ["arn:aws:s3:::example-bucket", "arn:aws:s3:::example-bucket/*"], "Condition": { "StringNotEquals": { "aws:SourceVpce": "vpce-0a1b2c3d4e5f" } } } ``` Note the shape: expressing the network restriction as an explicit `Deny` with `StringNotEquals` is usually the stronger construction, because a deny cannot be overridden by any allow elsewhere, whereas a conditional allow silently stops granting the moment the condition drifts — which is precisely how the outage started. ## Doing the migration without an outage The safe sequence is to widen the policy *before* changing the path. Add the endpoint condition alongside the existing IP condition so both the NAT path and the endpoint path satisfy the policy, then associate the endpoint with the route table, then verify — CloudTrail data events on the bucket, or S3 server access logs, will show requests arriving with the VPC endpoint id populated — and only then remove the `aws:SourceIp` clause. Reversing that order guarantees a window where neither path is permitted. ## The wider lesson This failure generalises well beyond S3. Anywhere an organisation has pinned access to a NAT gateway's Elastic IP — a bucket policy, a KMS key policy, an API Gateway resource policy, even a third-party allow-list — introducing an endpoint invalidates the assumption. It is also why network-path allow-lists based on public IPs are a weak control inside AWS: the address belongs to a NAT gateway that anything in the subnet can use, it changes if the NAT is rebuilt, and it disappears when the traffic goes private. `aws:SourceVpce` and `aws:SourceVpc` are stronger because they identify a network the account actually controls. Two further traps are worth naming. First, these conditions are only meaningful for requests that *do* traverse the endpoint; a request coming from the console, a CI runner, or another Region will never satisfy them, so a blanket deny can lock out legitimate administrative access — teams usually exempt specific principals or roles from the deny. Second, an endpoint policy on the endpoint itself is a separate control from the bucket policy: it restricts what may pass through the endpoint, but it can never grant anything, so it is not a substitute for fixing the resource policy. ## Confirming the diagnosis quickly If you are handed the 403 cold, the fastest confirmation is CloudTrail: the S3 data event for the denied call records the VPC endpoint id when the request came through an endpoint, and the source IP field shows the private address rather than the Elastic IP. Seeing a `vpce-` id in an event that is being denied by a policy conditioned on `aws:SourceIp` is the whole answer in one line.
- Would an endpoint policy on the gateway endpoint have fixed this instead?No. An endpoint policy can only restrict what passes through the endpoint; it never grants access. The 403 comes from the bucket policy having no applicable allow once the IP condition stopped matching, and only editing that resource policy — or the caller's IAM policy — can restore the grant.
- Why is an explicit Deny with StringNotEquals on aws:SourceVpce often preferred to a conditional Allow?An explicit deny is final: nothing elsewhere can override it, so the restriction holds even if a broad allow is added later by another team. A conditional allow only grants under the condition, which means any drift in the path silently removes access instead of blocking the wrong path — a fragile control that fails as an outage.
- How would you confirm from logs that requests are now arriving through the endpoint?CloudTrail data events for the bucket record the VPC endpoint id for requests that traversed an endpoint, and the source address appears as the caller's private IP rather than the NAT gateway's Elastic IP. S3 server access logs likewise show the endpoint context, so a before-and-after comparison makes the path change unambiguous.
saying these in an interview costs you the question
- Blames the instance role's IAM permissions
- Thinks the endpoint should be granting access itself
- Assumes the NAT Elastic IP is still the source
- Removes the old IP condition before the endpoint works
- Treats a public-IP allow-list as a strong control