skip to content

A bank's GitLab Self-Managed must go from 18.6 to 19.4 with minimal downtime: which versions must it stop at, and what decides whether zero downtime is possible?

level: seniorimportance: should knowfreq 14%

answer

  1. a fixed cadence of required stops
  2. x.2, x.5, x.8, x.11
  3. wait for background migrations
  4. one minor at a time for zero downtime

basics

~20 s

The path is 18.8, 18.11, 19.2, then 19.4, each at its latest patch, with background migrations finished before every hop. Zero downtime needs a multi-node Linux package install with high availability, upgraded one minor release at a time.

solid answer

~40 s

Since GitLab 17.5, required upgrade stops fall at `x.2`, `x.5`, `x.8` and `x.11`, so 18.6 to 19.4 goes 18.8, 18.11, 19.2, 19.4, each at the latest patch. Between hops, batched background migrations from the previous upgrade must show **Finished** (**Admin** > **Monitoring** > **Background migrations**, or `gitlab-rake gitlab:background_migrations:list` from 18.9), or the next upgrade stops with an error. The 19.0 step needs PostgreSQL 17: the 18.11 package upgrades a single-node install's bundled PostgreSQL automatically, but a replicated or external database must be upgraded by hand, outside the zero-downtime procedure. Zero downtime is only for a multi-node Linux package install with load balancers and high availability for each component, upgraded **one minor release at a time** (ten hops here), stateful back ends first and Rails and Sidekiq last. A single node always takes downtime.

code

bash · 9 lines
bash
# Single-node Linux package instance, 18.6 -> 19.4 (Ubuntu shown)
sudo gitlab-rake gitlab:background_migrations:status   # 18.8 and earlier
sudo gitlab-rake gitlab:check
sudo apt update && sudo apt install gitlab-ee=18.8.<patch>-ee.0
# wait until every background migration shows Finished, then:
sudo apt install gitlab-ee=18.11.<patch>-ee.0
sudo apt install gitlab-ee=19.2.<patch>-ee.0
sudo gitlab-rake gitlab:background_migrations:list     # 18.9 and later
sudo apt install gitlab-ee=19.4.<patch>-ee.0

go deeper

for a junior

Know that GitLab cannot jump straight to the newest version: required stops at x.2, x.5, x.8 and x.11 since 17.5, each at its latest patch.

for a middle

Lay out the 18.6 to 19.4 path, why background migrations must finish between hops, and where to check them in the Admin area or with Rake.

for a senior

Judge zero-downtime feasibility from the topology, plan the 19.0 prerequisites such as PostgreSQL 17, and rehearse the order and migration timing on a clone.

for a principal

Weigh four short outages against ten rolling hops, and set an upgrade cadence that keeps the instance close enough to current that paths stay short.

## Required upgrade stops A **required upgrade stop** is a GitLab version you must install before going further, because it carries database changes later versions assume. Since **GitLab 17.5** the stops are on a fixed cadence, `x.2`, `x.5`, `x.8` and `x.11`: for GitLab 18 that is 18.2, 18.5, 18.8 and 18.11, and for GitLab 19 it is 19.2, 19.5, 19.8 and 19.11. From **18.6 to 19.4** the path is therefore: 1. **18.8**, the latest 18.8 patch release. 2. **18.11**, the latest 18.11 patch release, which is also the last GitLab 18 release. 3. **19.2**, the latest 19.2 patch release. 4. **19.4**, the target. Use the latest patch of each minor release, not `.0`. Package managers install the newest package by default, so on a path with stops you pin each version explicitly, as in `gitlab-ee=<version>-ee.0`. ## Between every hop - **Background migrations.** Some data changes run as batched background migrations in Sidekiq after an upgrade. All must have the status **Finished** before the next upgrade; otherwise migrations fail with `Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active'`. Check under **Admin** > **Monitoring** > **Background migrations**, or with `sudo gitlab-rake gitlab:background_migrations:list` from 18.9 (`...:status` on 18.8 and earlier). More Sidekiq capacity for the `background_migration` queue shortens the wait. - **Health checks.** Run `sudo gitlab-rake gitlab:check` before and after, and confirm sign-in, clone, push, runners and the registry. - **Rollback plan.** The package backs up the database before installing a newer version, but keep your own full backup plus `gitlab.rb` and `gitlab-secrets.json`; a restore must go onto the same GitLab version that made the backup. ## Prerequisites the 19.0 hop adds | Requirement for 19.0 | What to do before leaving 18.11 | |---|---| | PostgreSQL 17 minimum | Single-node Linux package: the 18.11 upgrade moves the bundled database to 17.7. Replicated or external PostgreSQL: upgrade it yourself | | Redis 6 support removed | Move an external Redis 6 to Redis 7 | | Bundled Mattermost removed | Remove every `mattermost[...]` key from `gitlab.rb`, or reconfigure aborts | | Ubuntu 20.04 and SUSE packages discontinued | Move the servers to a supported operating system on 18.11 | | Helm chart 10.0 | Bundled PostgreSQL, Redis and object storage removed; migrate to external services | ## When zero downtime is possible GitLab's zero-downtime procedure applies to a **multi-node Linux package installation** that has: - an external load balancer health-checking the Rails nodes on `/-/readiness`, and an internal load balancer for the database pooling and Praefect tiers; - high availability for the database, Redis and service-discovery tiers; anything not highly available must be upgraded separately, with downtime. It then imposes two constraints: 1. **One minor release at a time.** 18.6 to 18.7, then 18.8, and so on, ten hops to reach 19.4, each waiting for its background migrations. Skipping minors can leave the schema broken, so a multi-minor jump means downtime. 2. **Back to front order.** The stateful back ends (service discovery, PostgreSQL, the connection poolers, Redis) go first, then Gitaly and Praefect, which reload gracefully, then Rails and Sidekiq. One Rails deploy node runs regular migrations with `SKIP_POST_DEPLOYMENT_MIGRATIONS=true`, the remaining Rails nodes follow, the deploy node then runs the post-deployment migrations, and the Sidekiq nodes are upgraded after that. `/etc/gitlab/skip-auto-reconfigure` on each node stops the package from running reconfigure, and so migrations, on its own. Two gaps remain even then: the Linux package offers no high availability for the **Praefect database**, so it needs a third-party database to avoid downtime, and **PostgreSQL major version upgrades** are a separate process the procedure does not cover. A **single-node** instance must always take downtime; users see **Deploy in progress** or a `502`. A Helm chart instance follows the chart's own upgrade documentation. ## The trade-off For a bank on a multi-node, highly available install, the choice is four short downtime windows (18.8, 18.11, 19.2, 19.4) or ten rolling hops with no planned outage but far more operator time. Rehearse whichever you pick on a clone of production; it also measures how long the background migrations take.

  • The 18.11 upgrade moved a single-node instance's PostgreSQL to 17.7 automatically. Why did a three-node replicated database cluster on the same release not get upgraded?
    GitLab's notes say automatic PostgreSQL version upgrades only apply to single-node Linux package instances. A replicated cluster built with the Linux package, a Geo site or an external database must be upgraded to PostgreSQL 17 manually before GitLab 19.0, and that major upgrade is outside the zero-downtime procedure.
  • An upgrade from 19.2 to 19.4 fails with "Expected batched background migration ... to be marked as 'finished', but it is 'active'". What happened and what do you do?
    The previous hop's batched background migrations had not finished. Let them complete, watching **Admin** > **Monitoring** > **Background migrations** or `gitlab-rake gitlab:background_migrations:list`, finalize a stuck one with the documented Rake task if needed, then retry. Adding Sidekiq capacity for the `background_migration` queue shortens the wait.
  • Why does a zero-downtime upgrade still need a third-party database for Gitaly Cluster (Praefect)?
    Praefect keeps its own PostgreSQL database, and the Linux package offers no high availability for it, only for the main GitLab database. Upgrading or restarting that single database interrupts Praefect, so avoiding downtime needs a highly available database from outside the package.

  • 18.6 can go straight to 19.4 if you install the newest package
  • Only the last minor of each major, 18.11, is a required stop
  • Background migrations can be left running across the next upgrade
  • A single-node instance can be upgraded with zero downtime using maintenance mode
  • Zero-downtime upgrades can jump directly between required stops

Assumes GitLab 19.4: since 17.5 required upgrade stops fall at x.2, x.5, x.8 and x.11 (18.8, 18.11 and 19.2 on this path), and 19.0 made PostgreSQL 17 the minimum.