When a TCP server shows thousands of connections stuck in CLOSE-WAIT while the clients calling it show thousands in TIME-WAIT, what does each pile mean, and which one is a bug?

## Read the state, then the side A connection's teardown state says who has closed and who has not. Two facts from RFC 9293, the current TCP specification, settle this question: - `CLOSE-WAIT` "represents waiting for a connection termination request **from the local user**". The peer has sent `FIN`, the stack has acknowledged it, and the stack will not send its own `FIN` until the application closes. - `TIME-WAIT` is held by the endpoint that **closed actively**, the one whose `FIN` went first, and it MUST last 2×MSL before the connection is deleted. So the two piles in the question are on different sides for different reasons, and only one of them is a defect. ## CLOSE-WAIT: the stack is waiting for the application RFC 9293 has no timer that ends `CLOSE-WAIT`. The user timeout applies to data that has not been acknowledged, and keep-alive probing is optional. A connection in `CLOSE-WAIT` leaves it when the application closes it (moving to `LAST-ACK`), when the application aborts it, or when an error such as an incoming `RST` ends it. If the application has lost track of it, it can stay until the process exits. Each one holds a file descriptor, a connection record and its buffers. A count that climbs steadily and never drains is therefore almost always an **application bug**. Typical causes: 1. **A missing close on an error or timeout path.** The happy path closes; the exception path returns without doing so. 2. **A pool that forgets connections.** A connection pool removes a broken connection from its books but never closes the socket underneath it. 3. **Threads blocked elsewhere.** The code that would close the connection is waiting on a lock, a slow downstream call or a full queue, so it never reaches its close. 4. **Ignoring end of stream.** The reader never checks for a zero-byte read, so the peer's `FIN` goes unnoticed. To diagnose it, group the stuck connections by remote address and port, match them to the code that owns that kind of connection, and check whether the count follows error rates or traffic. Restarting the process empties the pile, because the operating system closes every descriptor on exit, but that hides the leak rather than fixing it. ## The mirror on the client: FIN-WAIT-2 Every server connection stuck in `CLOSE-WAIT` has a peer that sent the first `FIN`, got it acknowledged and is now in `FIN-WAIT-2`, waiting for a `FIN` that will not come. By the specification a connection can sit in `FIN-WAIT-2` indefinitely, because it is also the legitimate state of a half-closed connection. Stacks that time out a `FIN-WAIT-2` connection no application still owns are making an implementation choice; after that, the client has no record at all. So the clients' `TIME-WAIT` entries do **not** belong to the leaked connections. They belong to connections the server did close properly, where the clients had closed first. ## TIME-WAIT: arithmetic, not a leak Every active close parks one four-tuple in `TIME-WAIT` for its full duration. At steady state: `entries in TIME-WAIT ≈ active closes per second × TIME-WAIT duration` | Close rate | TIME-WAIT duration | Entries at steady state | |---|---|---| | 500 per second | 60 s (an implementation choice; Linux, for example) | 30,000 | | 500 per second | 240 s (RFC 9293: 2×MSL with MSL = 2 minutes) | 120,000 | | 5 per second | 60 s | 300 | The count tracks **connection churn**, not failure. A `TIME-WAIT` entry holds a small record and blocks reuse of that exact four-tuple; it holds no application resource. It becomes a problem mainly when one client opens connections to the same server address and port so fast that it runs short of four-tuples, which is a port-allocation question owned by the sockets topic. ## What actually reduces each pile | Pile | Where it lands | Real fix | What does not fix it | |---|---|---|---| | `CLOSE-WAIT` | the side that received the first `FIN` | close every connection on every code path | restarting, raising descriptor limits | | `FIN-WAIT-2` | the peer of a leaked `CLOSE-WAIT` | fix the leak on the other side | anything done on this side alone | | `TIME-WAIT` | the active closer | open fewer connections by reusing each for many exchanges; choose deliberately which side closes first | aborting with `RST`; shrinking the wait far below 2×MSL | ## Fixes that make things worse - **Aborting to skip TIME-WAIT.** An abort discards queued data, gives the peer a reset error instead of an orderly end, and throws away the protection TIME-WAIT exists to give against old duplicate segments. - **Moving TIME-WAIT instead of reducing it.** Making the server close first moves the entries to the server; that may be the right place, but the total is unchanged. - **Treating the TIME-WAIT count as the incident.** On the clients here it is the healthy half of the picture. The incident is the `CLOSE-WAIT` pile on the server.