skip to content

Where in an ECMP leaf-spine fabric can a stateful firewall pair not stand, and why?

level: seniorimportance: should knowfreq 40%

answer

  1. state only works if you see both halves
  2. equal-cost paths do not promise symmetry
  3. the underlay sees tunnels, not workloads
  4. adopting mid-stream packets is the loophole
  5. one path between zones, or no proof

basics

~20 s

Anywhere both directions are not guaranteed to cross the same member. Equal-cost paths let the reply take another route, and on a VXLAN underlay the box sees tunnels rather than workloads, so it drops valid replies or is bypassed.

solid answer

~50 s

A stateful control only works where it sees both halves of a flow: the reply is permitted by the entry the request created, not by a rule. Two positions in a leaf-spine fabric break that. First, an active/active pair reached over equal-cost paths: the request can hash to one member and the reply to the other, which holds no entry and drops good traffic — and the usual field fix, letting the box adopt packets for flows it never saw start, is exactly the loophole that lets an uninvited sender build state for a conversation the firewall never validated. Second, anywhere on the underlay, where traffic is encapsulated: the box sees outer UDP between tunnel endpoints with thousands of workload flows collapsed inside, and can express no per-workload policy at all. The fix is to make the insertion point the only path between the zones; the price is a chokepoint sized for all of it.

go deeper

for a junior

Remember the core requirement: a stateful firewall must see both directions of a flow, because the reply is allowed by the entry the request created and not by a rule.

for a middle

Explain how equal-cost paths can send request and reply different ways, and what a firewall does with a packet in the middle of a flow it never saw begin.

for a senior

Show that you would establish path symmetry before inserting anything, and that you can name the security cost of the loose state setting that makes asymmetry stop hurting. Be able to describe how you would prove no bypass path exists.

for a principal

Own the trade explicitly: forcing every inter-zone flow through one insertion point spends the fabric's path diversity and creates a failure domain, and that is the price of a boundary you can actually evidence.

## The requirement nobody writes down A stateful firewall's whole value rests on one assumption: **it sees both directions of every flow it filters**. The reply is not permitted by a rule, it is permitted by the entry the request created. Take away either direction and the control does not degrade gracefully, it inverts: either it drops traffic that is perfectly legitimate, or it is configured loose enough to stop dropping and is then no longer enforcing what you think. A leaf-spine fabric is built to violate that assumption. Equal-cost multipath exists precisely so that traffic can take any of several equivalent routes, chosen per flow by a hash. Nothing in that design promises that the hash computed for the request and the hash computed for the reply select the same intermediate device, because the two directions present different tuples to the hash. Add more than one services leaf, or an active/active pair, and the two directions can land on different firewall members. ## Position one: split across an active/active pair Request hashes to member A, which creates an entry. Reply hashes to member B, which has no entry and sees an unsolicited mid-stream packet. Correct behaviour is to drop it, and the application sees a hang. The field fix that gets reached for is to relax the state check so the box will adopt a flow from a packet in the middle of it. That makes the symptom disappear and quietly removes the guarantee you deployed the firewall for: any sender who can put a plausible mid-stream packet on the wire now gets an entry created for a conversation the firewall never watched begin, and once that entry exists the return direction is open too. The setting that made the outage stop is the setting that made the boundary porous. The legitimate fix is **session synchronisation** between the members, so either one can serve a flow the other created. That works, and it costs: a replication channel sized for your session-creation rate, both members sized for the whole table rather than half of it, and a sync lag during which very short flows can finish before their state has replicated. ## Position two: on the underlay When workload traffic is carried in an overlay, the tunnel endpoints wrap it before it hits the fabric. A device sitting on the underlay sees outer UDP between two tunnel endpoints — VXLAN's assigned destination port is 4789 — and the entropy that spreads it across paths lives in the outer source port the ingress endpoint derives from the inner flow. Every workload conversation between the same pair of endpoints therefore looks like one conversation. The box will happily build state for that, and the state describes the tunnel, not the workloads. No per-workload policy is expressible, the entry count bears no relation to the number of real flows, and a segmentation claim made from that position is not true. The stateful control has to sit where traffic is decapsulated — at the insertion point in the overlay, not in the path between the switches. ## Position three: a return path that skips you The subtlest failure is not asymmetry, it is a shortcut. If the database zone has any route back to the server zone that does not cross the pair, then some traffic crosses the boundary in one direction only. Sometimes the firewall drops the half it sees and you find out immediately. Sometimes the whole conversation takes the shortcut and the firewall never sees it at all — no drop, no log, no alert, and a boundary that exists only on the diagram. ## Proving it, and paying for it The conclusion is that **denial is a property of the topology, not of the rule base**. "Prove it denies" is answered by showing that no path exists between the two zones that avoids the pair — which means reading the routing back from the database zone, not only forward from the server zone — and then testing it: originate forbidden traffic from a host in the server zone and confirm both that it fails and that the firewall logged the denial. A block with no matching log almost always means the traffic went a way you did not know about. Making the insertion point the only path is what buys you that proof, and it is not free. You have deliberately removed the fabric's path diversity for inter-zone traffic and created a chokepoint that must be sized for all of it: bandwidth, session-creation rate and table size, with headroom for one member carrying everything. You have also created a single failure domain between two zones that used to have many paths between them. That is the real trade in this question — you are spending the fabric's redundancy to buy an enforceable boundary, and you should be able to say so out loud rather than pretend the firewall was free to insert.

  • Session synchronisation between the two members fixes the asymmetry. What does it cost?
    You take on a state-replication channel sized for your session-creation rate, and each member must be sized for the entire table rather than half of it, because either may end up owning every flow. It also couples the pair: a member that falls behind adopts flows it only partly knows, and very short flows can complete before their state has replicated at all.
  • How do you prove the pair actually denies, rather than assuming it?
    Denial is a property of the topology. Show that every path between the two zones crosses the pair, reading the routing back from the far zone rather than only forward from the near one, then originate traffic the policy forbids and confirm both that it fails and that the firewall logged the denial. A block with no matching log usually means the traffic took a path nobody had documented.
  • Why can a firewall sitting on the underlay not enforce workload-level policy?
    Encapsulated traffic presents an outer header between two tunnel endpoints, so the firewall sees UDP from one endpoint to another and every workload conversation between that pair looks identical. It can build state, but the state describes the tunnel rather than the workloads, so no per-workload rule is expressible without decapsulating first.

saying these in an interview costs you the question

  • Assumes equal-cost paths keep both directions together
  • Enables mid-stream session pickup to make asymmetry go away
  • Treats a box on the underlay as workload segmentation
  • Verifies the boundary from the forward path only
  • Ignores return routes that bypass the insertion point

context