An EC2 instance's VPC Flow Logs show an ACCEPT record for an inbound TCP flow and a REJECT record for the same addresses with the ports reversed. Which layer is dropping the traffic, and how would you confirm it?

## Reading the record pair A VPC Flow Log record covers one direction of one flow over an aggregation interval, and includes the source and destination addresses, the source and destination ports, the protocol, packet and byte counts, and an `action` of `ACCEPT` or `REJECT`. For a single TCP connection into an instance you therefore expect two records: ``` ... 203.0.113.9 10.0.7.20 51314 443 6 ... ACCEPT OK ... 10.0.7.20 203.0.113.9 443 51314 6 ... REJECT OK ``` Same pair of addresses, ports swapped. The first is the request arriving; the second is the reply leaving. `ACCEPT` then `REJECT` is a specific, diagnosable fingerprint. ## Why the security group is exonerated Security groups track connections. Once an inbound rule admits a flow, the response is permitted outbound irrespective of the group's egress rules — the check simply is not made against the rule set. So a security group can produce an inbound `REJECT` (no matching allow rule), but it cannot produce an outbound `REJECT` for a flow it just accepted. That single property is what turns the record pair into a conclusion rather than a guess. Network ACLs keep no state. The reply is a fresh packet evaluated against the subnet's outbound rules in ascending rule-number order, first match wins, falling through to the implicit final deny. Because the reply's destination is the client's ephemeral port — high and effectively random — an outbound rule written over the service port never matches it. That is the overwhelmingly common cause of this exact fingerprint. ## The confirmation walk 1. **Fix the direction first.** Check which address is the instance. If the REJECT has the instance as *source*, the reply is being dropped and the subnet's egress rules are the suspect. If the instance is the *destination* on a REJECT with no matching ACCEPT anywhere, the request itself never got in and you are looking at an inbound problem instead. 2. **Read the NACL for that subnet, outbound, in number order.** Find the first rule that matches the reply's protocol, destination address and destination port. If none does, the implicit deny is your answer. Watch for a low-numbered `deny` shadowing a correct allow further down — first match wins, so rule 90 beats rule 100 even when rule 100 is exactly right. 3. **Check both subnets when the flow crosses one.** A packet between two subnets in the same VPC is evaluated by the source subnet's egress rules *and* the destination subnet's ingress rules. Asymmetric NACLs are why a flow works in one direction only. 4. **Fix and re-observe.** Widen the egress rule to the ephemeral range `1024-65535`, then confirm the REJECT records stop rather than assuming they did. ## What flow logs will not tell you The `action` field says `REJECT`; it does not name the rule, the NACL or the security group that produced it. The layer has to be inferred, which is exactly why the stateful-versus-stateless reasoning above is the whole skill. Several flows are never logged at all, and treating their absence as evidence sends you down the wrong path: traffic to and from the instance metadata endpoint at `169.254.169.254`, traffic to and from the Amazon-provided DNS server, DHCP traffic, and traffic to the reserved VPC router address. If you are debugging "my instance cannot reach the metadata service", flow logs will show you nothing, and the cause is more likely IMDS configuration than a packet filter. Records are also aggregated over an interval — one minute or ten depending on how the flow log was created — so a record's appearance lags the event, and a very short flow shows up as a single record rather than a live packet trace. Flow logs answer "was this permitted", not "what happened in the last two seconds". ## The other pairings worth memorising - **Inbound REJECT, nothing else.** Either the security group has no matching inbound allow, or the subnet's NACL denies inbound. Compare the security group's rules against the destination port first — that is the cheaper check. - **No records at all for the flow.** The packet never reached the ENI. Look at routing, at whether the client is even resolving to the right address, and at the load balancer or endpoint in front of the instance — not at the filters. - **ACCEPT in both directions but the client still fails.** The filters are innocent; the problem is above the network — the application, TLS, or a timeout somewhere in the path. ## Why interviewers like this one It cannot be answered by reciting "security groups are stateful, NACLs are stateless". It requires using that fact as an inference rule against real evidence, which is the difference between having read the documentation and having debugged a VPC at two in the morning.