Why does the docker CLI report "client version is too new" against a host's daemon, and what does DOCKER_API_VERSION do?

### Two packages, two versions, one wire protocol Docker ships the client and the engine as separate packages — `docker-ce-cli` and `docker-ce` — and they communicate over a **versioned REST API**. Every request the CLI makes is sent to a path that carries the version, e.g. `/v1.43/containers/json`. That is the whole reason "skew" exists as a category: a host can perfectly well have a 2024-era CLI and a 2019-era daemon, because they were installed at different times, or because the CLI is a local one talking to a different host's engine. `docker version` is the diagnostic. It prints a Client block and a Server block, each with its own `Version:` (the product version, e.g. 27.x) and `API version:` (e.g. 1.47), and the server block additionally reports a **minimum API version** it will still accept. Read those three numbers before theorising. ### Negotiation is the normal path Since the API 1.25 era the CLI has performed **version negotiation**: on its first call it asks the daemon for its supported version and then downgrades itself to that version for the rest of the session. This is why the common case — a newer CLI against an older engine — usually just works, and it is why a candidate who says "the versions must match exactly" is wrong. What the client cannot do is negotiate *upwards*: an old client against a new daemon simply keeps speaking its old version, and the daemon accepts it as long as it is at or above the daemon's minimum. So the failure has only a few real causes: 1. **Negotiation was bypassed.** Something set `DOCKER_API_VERSION` in the environment — a shell profile, a CI image, a Makefile — and the client now insists on that exact version instead of asking. If the value is higher than the daemon supports, you get the "too new" refusal even though an un-pinned client would have worked. 2. **The daemon is below the client's floor.** A very old engine may not reach the minimum version a modern client will speak at all. 3. **The daemon is newer than the client, and the client is too old.** Modern engines refuse API versions below their advertised minimum, so an ancient client is turned away by a fresh daemon. The error text is the giveaway: ``` Error response from daemon: client version 1.44 is too new. Maximum supported API version is 1.41 ``` Note *who* is speaking: `Error response from daemon` means the request reached dockerd and dockerd rejected it. That already rules out socket permissions and a dead service. ### What DOCKER_API_VERSION actually does `DOCKER_API_VERSION` is an environment variable read by the client. Setting it overrides negotiation and forces every request to that version: ``` DOCKER_API_VERSION=1.41 docker ps ``` That is the correct emergency unblock: scoped to one command or one job, it costs nothing and gets the pipeline moving. Two consequences are worth naming in an interview. First, pinning **down** removes capability: fields and flags introduced after that version are not available, so a command may run but return less than you expect, or refuse an option outright. Second, an *unnecessary* pin is itself a bug — a stale `DOCKER_API_VERSION` exported in a base image is one of the classic ways a working host starts failing after an unrelated engine upgrade, because the pin outlived the daemon it was written for. ### A concrete shape of the ticket A CI job builds an invoice-rendering worker image. It runs fine on 37 of the 41 build hosts and fails on the remaining 4 with the "too new" message. The cause is almost always uneven installs: the job's container carries a current `docker-ce-cli` and mounts the host socket, while those 4 hosts were provisioned earlier and never had `docker-ce` upgraded. The fast fix is to export `DOCKER_API_VERSION` for that job. The durable fix is to bring those hosts' engine packages onto the same version as the rest and to stop hosts drifting in the first place — pin the package version in provisioning so every host lands on the same engine, and upgrade them as a fleet rather than individually. ### How to reason about it under pressure The useful mental model is: **the CLI adapts, the daemon does not.** Whenever you see a version complaint, ask (a) what does `docker version` say for client API, server API and server minimum; (b) is `DOCKER_API_VERSION` set anywhere in this environment (`env | grep DOCKER`); (c) is the CLI local or is it a client inside a container talking to a mounted socket. Those three answers explain essentially every instance. The general rule for a fleet is that the **daemon should be at least as new as the newest client that talks to it** — running old engines under new clients is the configuration that generates these tickets.