In GitLab Runner's config.toml, what is the difference between the global `concurrent` setting and a runner entry's `limit`?

## The shape of config.toml ```toml concurrent = 10 check_interval = 3 [[runners]] name = "docker-general" url = "https://gitlab.example.com/" token = "glrt-EXAMPLE" executor = "docker" limit = 8 request_concurrency = 2 [runners.docker] image = "alpine:3.20" [[runners]] name = "deploy-protected" url = "https://gitlab.example.com/" token = "glrt-EXAMPLE2" executor = "shell" limit = 1 ``` One process, one file, two registrations. Understanding which setting lives at which indentation level is most of the answer. ## concurrent — the process ceiling `concurrent` is a top-level key. It is the maximum number of jobs this runner *process* will have in flight at any moment, summed across every `[[runners]]` entry. If it is 10 and both entries above are busy, the process stops asking GitLab for work once ten jobs are running, no matter how much either entry's own `limit` would allow. The default in a fresh installation is `1`. That single line explains a large fraction of "our runner is so slow" reports: an eight-core machine dutifully processing one job at a time while a queue builds up. ## limit — the per-registration ceiling `limit` lives inside a `[[runners]]` block and caps that registration alone. It exists because different registrations have different costs and different risks: - A deployment runner set to `limit = 1` serialises production deploys, so two pipelines cannot race each other onto the same environment. - A memory-hungry integration-test runner might be held at 2 while a lint runner on the same host is allowed 8. - For autoscaling executors, `limit` bounds how many machines or pods can exist for that registration — it becomes a spend control, not just a parallelism one. The effective parallelism for one entry is `min(limit, concurrent - jobs running elsewhere)`. Raising `limit` without raising `concurrent` accomplishes nothing. ## request_concurrency — a different axis entirely `request_concurrency` (default 1) is how many job *requests* the runner will make to GitLab in parallel. It affects how quickly a runner can fill empty slots when a burst of jobs appears, not how many it can execute. On a runner with a high `concurrent` value and a bursty queue, leaving it at 1 makes slot-filling serial and the fleet looks underused; raising it too far adds API load for no benefit. Related is `check_interval`, the seconds between polls for new work — lowering it reduces pickup latency at the cost of more requests. ## Sizing it honestly The number to set is a function of the host, not the backlog: - **CPU.** Most CI jobs are compile- or test-bound and will happily use every core. Over-committing turns a two-minute job into a six-minute one for everybody, which looks like a broken runner rather than a saturated one. - **Memory.** The binding constraint more often than CPU: a JVM or bundler job with a large heap times four concurrent jobs exceeds the box, and the kernel's OOM killer starts terminating builds — which surfaces as random, unreproducible job failures. - **Disk and I/O.** Each concurrent job has its own build directory, cache extraction and image layers. Parallel jobs contend for I/O and for disk space; a full disk fails every job on the host at once. - **Executor.** With the Kubernetes executor, `concurrent` is a limit on simultaneous pods and the real capacity question moves to the cluster's resource requests and autoscaler. ## Symptoms to recognise - Jobs pending while the runner sits at low CPU: `concurrent` too low, or `request_concurrency` throttling slot-filling. - Every job slow and timing out together: `concurrent` too high for the host, everything contending. - Random OOM kills or "no space left on device" appearing only under load: concurrency exceeding memory or disk, not a code problem. - Two deploys colliding on one environment: a deployment runner without `limit = 1`, and no resource-group serialisation in the pipeline.