Connections to a service on a Linux host are disappearing with no reply and you suspect the local iptables rules. Using iptables itself, how do you determine which rule is responsible?
answer
- the kernel already counts for you
- make the numbers describe one attempt
- a target that logs and lets the packet continue
- bound the logging before you insert it
- check which address family the client used
basics
~20 sEvery iptables rule carries kernel-maintained packet and byte counters. Zero them with iptables -Z, reproduce the failure, then list with -L -n -v --line-numbers and see which rule's counters moved. Insert a rate-limited LOG rule above the suspect to capture the packets themselves.
solid answer
~60 sI start with the counters, because they are free and already there. `iptables -Z` zeroes them, then I reproduce the failure from the client and run `iptables -L -n -v --line-numbers`: whichever rule's `pkts` column moved is the one handling that traffic. If nothing moved but the chain header counters did, the packets fell through every rule and the default policy dropped them, which is a different fix. When I need to see the packets rather than a count, I insert a `-j LOG` rule immediately above the suspect with `-I`, give it `--log-prefix` so it is greppable, and rate-limit it with `-m limit`; LOG is non-terminating, so the packet carries on to the next rule and behaviour is unchanged. The log lines land in the kernel ring buffer, so `journalctl -k -f` or `dmesg -w`. Two things I always check: whether the client is arriving over IPv6, in which case `iptables` shows me nothing relevant and I need `ip6tables`, and whether the symptom is a hang or an instant error — a hang points at DROP, an immediate error at REJECT or at nothing being listened on. And I remove the LOG rule when I am done.
code
bash · 5 linesiptables -Z
iptables -L INPUT -n -v --line-numbers
iptables -I INPUT 1 -p tcp --dport 8443 -m limit --limit 5/min -j LOG --log-prefix "fw-8443: "
journalctl -k -f
iptables -D INPUT -p tcp --dport 8443 -m limit --limit 5/min -j LOG --log-prefix "fw-8443: "go deeper
Know that every iptables rule has packet and byte counters and that iptables -L -n -v shows them; that alone answers most "is this rule doing anything" questions.
Explain how zeroing with -Z turns totals into a clean per-attempt delta, that LOG is non-terminating so it can be inserted safely on a live chain, and where the log lines end up.
Show a whole method: read the symptom shape, check the address family, zero and reproduce, distinguish a firing rule from the policy line, add bounded logging only when counters are insufficient, then clean up and persist the fix.
Own the tradeoff between ad-hoc diagnosis on a box and standing visibility across a fleet, and decide where firewall drop evidence should live so an incident does not depend on someone SSH-ing in to add a LOG rule.
## Counters are the cheapest evidence you have The kernel maintains a packet count and a byte count for every iptables rule, continuously and for free. `iptables -L -n -v` shows them in the `pkts` and `bytes` columns — this is the main reason `-v` is not optional in practice. A rule whose counters move is handling traffic; a rule at `0 0` is not being reached at all. The raw totals are usually useless because they cover everything since boot. The technique that makes them decisive is zeroing: ```bash iptables -Z # zero every counter in the filter table # reproduce the failure from the client, once iptables -L -n -v --line-numbers # read the deltas ``` Now every non-zero count describes exactly the traffic you just generated. `iptables -Z INPUT` zeroes a single chain, and `iptables -Z INPUT 3` zeroes just rule 3, which is handy when you want to watch one suspect without disturbing a longer investigation. ## Read the chain header too Under `-v` the chain header carries its own counters: `Chain INPUT (policy DROP 12 packets, 900 bytes)`. That number is packets that traversed the whole chain, matched nothing terminating, and were disposed of by the default policy. It is a genuinely different diagnosis from a specific rule firing: no rule is wrong, a rule is *missing*. People spend a long time hunting for "the rule that is blocking it" when the answer is that nothing accepts it and the policy is doing its job. ## When counters are not enough: the LOG target Counters tell you how many packets, not which. When you need the packets — source address, port, flags, interface — insert a logging rule directly above the suspect: ```bash iptables -I INPUT 5 -p tcp --dport 8443 -m limit --limit 5/min \ -j LOG --log-prefix "fw-8443: " --log-level info ``` Three properties make this safe: - **`LOG` is non-terminating.** The packet is logged and then continues to the next rule, so inserting the rule does not change what happens to the traffic. This is what lets you probe a live system. - **`--log-prefix` makes the lines greppable.** Keep it short and distinctive; you will be filtering on it. - **`-m limit --limit 5/min` bounds the damage.** An unrated LOG rule on a busy chain can flood the kernel log, fill a disk, and evict the very lines you were trying to read. Never insert one without a limit on a production host. The messages go to the kernel ring buffer, so read them with `journalctl -k -f` or `dmesg -w`. `NFLOG` is the alternative target when you want the packets delivered to a userspace collector rather than the kernel log. Remove the rule when you are finished. A forgotten LOG rule is noise for the next person, and the `-m limit` on it silently disguises how much traffic is really matching. ## The two checks people skip **Is the client actually on IPv4?** `iptables` governs IPv4 only. If the host has an AAAA record and the client prefers IPv6, the entire IPv4 ruleset is irrelevant to the failure and `ip6tables -L -n -v` is where the evidence lives. A wide-open-looking `iptables` listing has convinced many people the firewall is innocent while `ip6tables` was quietly dropping everything. **Is it even the filter rules?** `iptables -t nat -L -n -v` and `iptables -t mangle -L -n -v` list the other tables, each with its own counters, and a packet redirected somewhere unexpected produces the same "no reply" symptom as a dropped one. ## The symptom itself narrows the search Before any of this, the shape of the failure tells you something: - **Hangs until the client's own timeout** is the signature of a silent `DROP`. Nothing is sent back, so the client retransmits until it gives up. - **An immediate error** points at `REJECT`, which actively sends a refusal — `--reject-with icmp-port-unreachable` by default for most cases, or `--reject-with tcp-reset` — or at a host with no process listening on that port at all, which is not a firewall problem. That distinction is worth stating out loud in an interview, because it is the difference between "the firewall is dropping it" and "the firewall never saw it". ## Putting it together The sequence that resolves this quickly: confirm which address family the client uses; note whether the failure hangs or errors; zero the counters; reproduce once; read `-L -n -v --line-numbers` and find the rule or the policy line that moved; add a rate-limited LOG above the suspect if you need packet detail; fix; remove the LOG rule; and re-save the ruleset so the fix survives a reboot.
- The chain header shows the policy counters climbing while every individual rule stays at zero. What does that mean?That the packets traversed the entire chain, matched nothing terminating, and were disposed of by the default policy. No rule is misconfigured — an accept rule is missing. It is a materially different fix from removing or reordering an over-broad block, and reading the header line is what distinguishes the two in seconds.
- Why must a LOG rule added to a production chain be rate-limited?Because it logs every matching packet into the kernel ring buffer. On a busy chain that floods the log, can fill the filesystem holding the journal, and evicts the earlier lines you were trying to read. `-m limit --limit 5/min` caps it while still giving you samples. The same rule left in place afterwards also hides how much traffic really matches.
- A client hangs until timeout rather than failing immediately. What does that tell you before you look at any rule?That something is discarding the packets silently rather than refusing them, which is the signature of a DROP target — or of the packets never arriving at all. An immediate refusal points instead at REJECT, which sends an explicit error back, or at a host where nothing is listening on that port. The symptom halves the search space before you list a single rule.
saying these in an interview costs you the question
- Reads raw counters without zeroing and misattributes old traffic
- Adds a LOG rule with no rate limit on a busy host
- Thinks LOG drops or accepts the packet it matches
- Concludes the firewall is innocent without checking ip6tables
- Leaves diagnostic LOG rules in the ruleset afterwards