skip to content

What distinguishes north-south from east-west traffic in a data centre, and why does the mix shape the network's topology?

level: juniorimportance: should knowfreq 45%

answer

  1. which side of the edge
  2. client to server vs server to server
  3. replication, migration, service calls
  4. trees sized for the trunk
  5. many equal paths between racks

basics

~20 s

North-south traffic crosses the data centre's edge, between outside clients and inside servers; east-west traffic stays inside, server to server. Tree designs were sized for north-south; heavy east-west traffic favours a spine-leaf fabric with many equal paths.

solid answer

~40 s

**North-south** traffic enters or leaves the data centre: a client request arriving from the internet or the campus, the response going back, a backup leaving for another site. **East-west** traffic never leaves: a web tier calling an API tier, a database replicating to its replica, storage nodes rebuilding a copy, a virtual machine migrating between hosts. The mix matters because a three-tier tree was built for the trunk: bandwidth grows toward the core and the edge, and traffic between two access blocks has to climb through shared upper layers where oversubscription stacks up. When most bytes go server to server, you want every rack to reach every other rack over the same short path with many links in parallel, which is what a spine-leaf (folded Clos) fabric gives you.

go deeper

for a junior

Recall the definitions by the edge: crossing the data centre's edge is north-south, staying inside is east-west, and give one concrete example of each.

for a middle

Explain why a three-tier tree handles east-west traffic badly: long paths through shared upper layers, stacked oversubscription, and spanning tree idling redundant uplinks.

for a senior

Show you would measure the traffic mix before choosing a topology, and connect heavy east-west load to equal-length paths, ECMP across spines and rack-local placement.

for a principal

Frame the direction mix as the input that decides the topology and its cost: a north-south site can buy cheap oversubscription, an east-west site pays for a wide fabric.

## The two directions Engineers describe data-centre traffic by where it goes relative to the site's edge. Picture the network drawn as a tree, with the internet and wide-area links at the top and the servers at the bottom. | Direction | What it is | Typical examples | |---|---|---| | **North-south** | Traffic that crosses the data centre's edge, entering or leaving | A browser's request to a web front end, the response, a download of software updates, a backup sent to another site | | **East-west** | Traffic that starts and ends inside the data centre | Service-to-service calls, database replication, storage rebuilds, virtual-machine live migration, batch jobs shuffling data between nodes | The labels come from that picture: north-south flows run up and down the drawing, east-west flows run sideways between racks. RFC 7938, an Informational RFC describing routing in large data centres, uses exactly these terms and records the shift: traffic used to be mostly north-south, and many large data centres now carry significant server-to-server traffic that never egresses the site. ## Why the mix changed A single user request rarely touches one server any more. It typically fans out: - the front end calls several back-end services, each of which may call others; - each write is copied to replicas on other racks so that losing a rack loses no data; - distributed storage and analytics move large blocks between nodes; - virtual machines and containers are rescheduled and moved between hosts. One small north-south request can therefore cause many times its own volume in east-west traffic. The ratio depends entirely on the workload, so the honest design step is to measure it, not to assume it. ## What a tree does to east-west traffic The traditional design has three layers: **access** switches where servers attach, **distribution** (or aggregation) switches that join groups of access switches, and a **core** that joins the distribution blocks and the exit. Each layer up has more bandwidth and more ports, because it was expected to carry the trunk of traffic heading out. East-west traffic between two servers in different access blocks takes the long way: access, distribution, core, distribution, access. Three problems follow: 1. **Oversubscription stacks.** Each layer is usually built with less upward bandwidth than downward, so the ratios multiply along the path. 2. **Path length varies.** Two servers in one block cross fewer switches than two servers in different blocks, so latency depends on placement. 3. **Scaling up hits a wall.** RFC 7938 notes that growing a tree means buying ever-larger upper-tier switches, and at some point no device with enough port density exists. If the access-distribution links are Layer 2, spanning tree also blocks redundant uplinks to keep the topology loop-free, so half of the bandwidth that was bought may sit idle. ## What a spine-leaf fabric does instead A **spine-leaf** fabric, which RFC 7938 calls a folded Clos, connects every leaf (top-of-rack) switch to every spine switch and nothing else. Any rack reaches any other rack through exactly one spine, so: - every inter-rack path has the same length; - there are as many equal-cost paths as there are spines, and routing spreads flows across all of them; - adding a spine adds bandwidth for east-west traffic, and adding a leaf adds server ports, without replacing anything. North-south traffic still exists in a fabric: it leaves through a dedicated set of border switches attached like any other leaf. It simply stops being the traffic the whole topology is built around. ## Reading a design question When an interviewer asks you to design a data-centre network, the traffic direction is the first thing to pin down. A site that mostly serves static content to the outside can live with a tree and heavy oversubscription. A site running microservices, distributed databases or large compute jobs is dominated by east-west traffic, and its topology should give every rack a wide, uniform path to every other rack. Measuring the mix on an existing network is straightforward: - compare the traffic counters on the border links with the counters on the uplinks between racks; - sample flows at the top-of-rack switches and classify each by whether its destination is inside the site's address space; - look at peaks, not averages, because replication and batch jobs are bursty. The answer is often surprising: the bytes leaving the site are a small fraction of the bytes moving between racks.

  • Is a call from a web tier to a database in the same data centre north-south or east-west?
    East-west. Both ends sit inside the data centre, so the traffic never crosses the edge, even though it was triggered by a north-south request from a client. That is the typical pattern: one inbound request produces several internal calls, replication and logging traffic, which is why east-west volume often exceeds north-south volume.
  • Does east-west traffic between two servers in the same rack load the fabric?
    No. Two servers on the same leaf switch exchange frames through that leaf alone; the spine layer never sees them. Only east-west traffic between racks crosses the uplinks and the spines, which is why placement (keeping chatty services rack-local) changes how much fabric bandwidth a workload uses.

saying these in an interview costs you the question

  • A user's request to a web server is east-west because both are inside the company.
  • East-west traffic always stays inside one rack, so it never touches the fabric.
  • A tree design scales east-west bandwidth indefinitely by buying a bigger core switch.
  • Only internet uplinks need sizing; server-to-server traffic never needs planning.