skip to content

Container image pulls dominate cold-start time across your fleet. Which levers do you pull, and what does each cost?

level: principalimportance: should knowfreq 38%

answer

  1. make it a number before making it a project
  2. four families, not one trick
  3. the fastest transfer is the one already done
  4. every fix adds a dependency somewhere
  5. a fleet starting at once is a burst you sized for

basics

~20 s

Measure pull time as a first-class signal, then pick among four levers: move fewer bytes, move them a shorter distance, move them before they are needed, or start before they all arrive. Each trades a different cost — staleness, a dependency, coupling, or money.

solid answer

~50 s

Start by making it measurable — p95 seconds from host creation to first container running, split into transfer and extraction — because every lever trades a different cost. Then work four families. **Fewer bytes**: an image-size budget per service, owned by the teams building the images. **Shorter distance**: serve the bytes from a copy in the same region as the hosts, the biggest single win for a spread fleet, at the price of an availability dependency on the start-up path. **Earlier**: bake images into the machine image, or pull on boot before the host takes work — this removes pull time from the critical path but adds a pipeline and staleness. **Sooner than complete**: lazy-pull formats, which leave running workloads depending on the registry. Also own the fleet defaults for download concurrency and layer codec, and size for the burst: 47 hosts each pulling 1.87 GB is roughly 88 GB at once.

code

bash · 8 lines
bash
#!/usr/bin/env bash
set -euo pipefail
for ref in \
  registry.example.com/tileserver:1.9 \
  registry.example.com/tile-indexer:4.2; do
  docker pull "$ref"
done
touch /run/images-warmed

go deeper

for a junior

You will not be asked to design this. Understanding that a brand-new host must fetch the whole image, while a reused host does not, is the piece worth carrying into the conversation.

for a middle

Be ready to name the concrete levers and what each one attacks: bytes, distance, timing, or overlap. Being able to say which half of the pull each addresses is what distinguishes a considered answer from a list.

for a senior

Expect to be pushed on operations: what breaks when the nearby copy is down, how you pre-pull without knowing what a host will run, and how you avoid a simultaneous fleet launch overwhelming whatever serves the bytes.

for a principal

Own the tradeoff explicitly and pick a debt: staleness from baking, an availability dependency from caching, runtime coupling from lazy pulling, or money from warm capacity. Say which one your organisation is best placed to carry, and what measurement would change your mind.

### First, decide whether it is worth solving Pull time only matters where it lands on a critical path: capacity that arrives late during a traffic spike, a deployment that takes minutes longer than it should, a batch job whose runtime is dwarfed by its start-up, or an incident where recovery is gated on new hosts coming up. Quantify that. If new hosts are created twice a day and take four minutes, this is not where your engineering time goes. If they are created hourly in response to load, a 4m12s cold pull is a capacity problem wearing a distribution costume. Make it a signal: p95 seconds from host creation to first container running, split into transfer and extraction, tracked per image. Without that split, teams argue about levers that address different halves of the problem. ### Lever 1 — fewer bytes The cheapest byte is the one not shipped. This is real leverage but it is owned by the people who build the images, not by the platform: an image-size budget per service, visible in the pipeline, with the platform providing the measurement rather than the fix. Treat it as governance — a number teams see and are accountable for — and leave the build-side technique to them. One platform-side variant does belong to you: standardising base images across services so that hosts running one workload have already materialised the layers of the next. Reuse is by digest, so uniformity is worth real minutes on a mixed host. ### Lever 2 — shorter distance If your hosts are in one region and the bytes are served from another, distance is the whole problem, and no local setting fixes it. Serving the image from a copy near the hosts — a replicated repository in the same region, or a cache in the same network — is usually the single largest improvement available and the one with the best effort-to-benefit ratio. What it costs: another piece of infrastructure on the start-up path. If hosts cannot start without it, its availability target is now your capacity's availability target, and you need a defined fallback to the upstream registry. It also costs money and someone's ownership. ### Lever 3 — earlier: get the bytes there before they are needed The fastest pull is one that already happened. - **Bake the image into the machine image.** Hosts boot with the layers already on disk, and the pull at start-up prints `Already exists` throughout. Cost: a machine-image build pipeline, and staleness — every application release drifts from the baked content until the next bake, so this works best for the stable lower layers rather than the top application layer. - **Pre-pull at boot, before the host takes work.** The host fetches the images it will need while it is still warming, so the pull overlaps with the rest of its start-up instead of blocking the first workload. Cost: you must know what the host will run, and hosts that take work early get no benefit. - **Keep hosts alive longer, or hold warm capacity.** Fewer cold hosts is fewer cold pulls. Cost: paying for idle capacity — often the honest, boring answer, and frequently cheaper than the engineering alternatives. ### Lever 4 — start before the transfer finishes Lazy-pull image formats let a container start while its content is still arriving. This is the most technically appealing option and the one with the most coupling: it puts the registry on the critical path of *running* workloads, adds a per-node runtime prerequisite, and adds an index or conversion step to every build. Reach for it when pre-warming is impossible — very large images, hosts that must start immediately, images you do not control — rather than as a first move. ### Fleet defaults you should own explicitly - **Download concurrency.** The daemon's default of 3 parallel layer transfers is conservative for hosts far from their registry on a fat link; a considered fleet default of six to ten is reasonable. Set it as configuration written at host provisioning time, and remember it multiplies across every host that starts at once. - **Layer codec.** A faster codec such as zstd attacks extraction rather than transfer, but every engine that must pull the image has to understand it. That is a fleet-wide compatibility floor, so it is a platform decision, not a per-team one. - **The thundering herd.** 47 hosts launching together and each pulling a 1.87 GB image is roughly 88 GB of near-simultaneous egress. Whatever serves those bytes must be sized for the burst, or your scale-out event becomes a self-inflicted outage of your own image distribution. Stagger, cache locally, or pre-bake. ### How to sequence it Measure, then: standardise base images and set a size budget (cheap, slow-acting); put a copy of the bytes near the hosts (biggest single win); pre-pull or bake for the workloads whose cold start actually hurts (removes the cost rather than shrinking it); and only then consider lazy pulling for the residue. Re-check the p95 after each step — several of these levers make the others look unnecessary, which is a good outcome, not a wasted analysis. The judgement being tested is not knowing the list. It is knowing that pre-warming trades staleness, caching trades an availability dependency, lazy pulling trades a runtime coupling, and warm capacity trades money — and being able to say which of those your organisation would rather owe.

  • Baking images into the machine image removes pull time entirely. Why is it not always the answer?
    It trades pull time for staleness and pipeline cost. Every application release drifts from the baked content until the next machine-image build, so the top application layer still has to be fetched, and you now maintain a second build pipeline with its own rollout. It works best for stable lower layers shared across services, not for the layer that changes every deploy.
  • How do you keep a nearby copy of the images from becoming a single point of failure for capacity?
    Give it an explicit availability target and a defined fallback to the upstream registry, so a host that cannot reach the local copy still starts, slowly, rather than not at all. Test that fallback deliberately. And size it for the burst case — a simultaneous fleet-wide launch — because the failure you will actually see is saturation during scale-out, not a crash.
  • Which single metric would you put in front of leadership to justify this work?
    p95 seconds from host creation to first container serving, tracked over time and split into transfer and extraction, with the business consequence attached — how much of a scale-out or a recovery is spent waiting. A distribution of that number across images also tells teams which of their images is the outlier, which does more than any platform mandate.
  • When is doing nothing the right call?
    When hosts are long-lived and rarely replaced, so cold pulls are rare and never on a latency-sensitive path; or when the cheapest fix is a small amount of warm capacity you can simply pay for. Distribution engineering has ongoing operational cost, and buying the minutes back with idle capacity is often the smaller, more reversible bill.

saying these in an interview costs you the question

  • Jumps to a caching layer without measuring anything
  • Treats image size as the only lever
  • Ignores that new infrastructure becomes a start-up dependency
  • Forgets the burst load a fleet-wide launch creates
  • Assumes lazy pulling is a free win
  • Never considers simply holding warm capacity

context

resolving links…