skip to content

In GitLab Self-Managed, how does Gitaly Cluster (Praefect) differ from sharded Gitaly, and why does Gitaly on Kubernetes run unclustered?

level: seniorimportance: should knowfreq 11%

answer

  1. one node per repository versus replicas
  2. a router in front with its own database
  3. seconds to fail over, under a minute lost
  4. pod rotation is not a reload

basics

~20 s

Sharded Gitaly stores each repository on exactly one Gitaly node, a single point of failure. Gitaly Cluster puts Praefect in front of several Gitaly nodes, replicating writes and failing over automatically; on Kubernetes it is beta, so production Gitaly pods run sharded.

solid answer

~50 s

With **sharded Gitaly**, each repository storage is served by one Gitaly node, and new repositories are spread across storages by configured weights; lose a node and its repositories are unavailable until it returns. **Gitaly Cluster (Praefect)** presents a *virtual storage*: Praefect nodes route requests, replicate writes to the other Gitaly nodes (by default to every storage in the virtual storage) and fail over automatically, targeting an RPO under one minute and an RTO under ten seconds for a single node failure. The minimum recommended setup is three Praefect nodes, three Gitaly nodes, a load balancer and a PostgreSQL database for Praefect's metadata, which the Linux package cannot make highly available. On Kubernetes, sharded Gitaly has been generally available since 18.11, but Gitaly Cluster there is a beta since 19.1, so repository failover means Gitaly Cluster on VMs, as in cloud native hybrid.

go deeper

for a junior

Recall that Gitaly stores the repositories, and that sharded Gitaly keeps each repository on one node while Gitaly Cluster keeps several copies.

for a middle

Explain Praefect's role, the virtual storage, the replication factor and the minimum node counts, including the separate Praefect database.

for a senior

Reason about failure: the RPO and RTO a single node failure gives, snapshot pitfalls, the unprotected Praefect database, and why Kubernetes Gitaly stays sharded.

for a principal

Decide whether repository failover is worth Gitaly Cluster's operating cost, and whether that pushes the platform to VMs or hybrid rather than all-Kubernetes.

## What Gitaly does **Gitaly** is the GitLab service that stores Git repositories and answers every Git operation the web interface, the API and `git` clients need. It handles only repository data; the database, uploads and the rest live elsewhere. Because Git work is many small reads and writes, GitLab strongly recommends fast local SSDs for Gitaly nodes. ## Sharded Gitaly In the default layout, GitLab has one or more **repository storages**, each served by a single Gitaly node. - Every repository lives on **exactly one** storage, chosen at creation time by configured weights. - Scaling means adding storages and moving repositories between them. - A failed node makes **its** repositories unavailable until it is back; other storages keep working. - Recovery is by backup and restore, or by repairing the node. ## Gitaly Cluster (Praefect) **Gitaly Cluster (Praefect)** puts a routing tier, **Praefect**, in front of several Gitaly nodes and presents them to GitLab as one **virtual storage**. | | Sharded Gitaly | Gitaly Cluster (Praefect) | |---|---|---| | Copies of a repository | One | Replication factor; by default one per storage in the virtual storage | | Node failure | Repositories on that node are down | Praefect fails over to a replica automatically | | Reads | From the one node | Distributed across replicas | | Extra components | None | Praefect nodes, a load balancer, a PostgreSQL database for Praefect | | Backups | Backup and restore Rake tasks | Backup and restore Rake tasks only; snapshots are unsupported | The documented objectives for a **single node failure** are an **RPO under one minute** and an **RTO under ten seconds**. Writes replicate **asynchronously**, so writes not yet copied to the newly promoted primary are lost; **strong consistency** prevents that loss in some circumstances. Each Praefect node health-checks every second and fails over after ten consecutive failures. The minimum recommended cluster is: 1. One load balancer in front of Praefect. 2. Three Praefect nodes. 3. Three Gitaly nodes, one primary and two secondaries; an odd number gives transactions a tie-breaker. 4. One PostgreSQL database for Praefect's replication metadata. The costs are real. The Linux package offers no high availability for the **Praefect database**, so a third-party database is needed to remove that single point. Snapshot backups can leave Praefect's metadata out of step with the disks, so only the official backup and restore Rake tasks are supported, and a single node must never be restored from a snapshot into a running cluster. GitLab's technical support for Gitaly Cluster is limited to Premium and Ultimate customers. One documented race also bites CI: a post-receive hook can fire before a write has reached every replica, so a pipeline that fetches from a lagging replica fails with `couldn't find remote ref`; retrying the job or its fetch stage is the documented workaround. ## Why Gitaly on Kubernetes runs unclustered On a VM, the Linux package upgrades Gitaly by replacing the binary in place and **gracefully reloading** it. In Kubernetes a pod rotates by shutting down completely and starting a new pod, during upgrades, node maintenance or eviction, so non-clustered Gitaly interrupts the repositories that pod serves; client retries hide some of it. - Gitaly on Kubernetes became **generally available in GitLab 18.11**, for **sharded, non-clustered** Gitaly only. Each Gitaly pod is a single point of failure for its repositories. - **Gitaly Cluster on Kubernetes** was introduced as a **beta in GitLab 19.1** and is not part of the Cloud Native reference architecture. - A team that needs automatic repository failover therefore runs Gitaly Cluster on **virtual machines**, with stateless components in Kubernetes if it wants them: the cloud native hybrid layout. ## Choosing between them 1. **Sharded Gitaly with tested backups** when a storage being down until it is repaired or restored is acceptable; it is simpler and needs no extra tier. 2. **Gitaly Cluster on virtual machines** when repositories must survive a node failure within seconds, with a plan for the Praefect database's own availability. 3. **Sharded Gitaly in Kubernetes** when the platform is all-Kubernetes and per-pod interruptions during rotation are acceptable; Gitaly Cluster there is a beta, not yet generally available. ## Not the same as Geo **Geo** is a different kind of redundancy: whole GitLab sites in several locations, eventually consistent, with manual failover for disaster recovery. Gitaly Cluster stays in one location, needs sub-second latency between nodes, fails over automatically and protects only Git data. Many large installations use both.

  • Why does GitLab tell you never to restore one Gitaly Cluster (Praefect) node from a disk snapshot while the cluster runs?
    The cluster tracks replica state in Praefect's database. A node restored from a snapshot is behind, and the cluster cannot reconcile its data with the other nodes. Restore with the official backup and restore Rake tasks; if snapshots are unavoidable, shut GitLab down, snapshot every Gitaly node at once and dump the Praefect database.
  • When would you add Geo on top of Gitaly Cluster rather than rely on Gitaly Cluster alone?
    When the risk is losing the whole site or region. Gitaly Cluster protects Git data inside one location against a node failure, failing over automatically. Geo replicates the entire GitLab instance, Git data included, to another site for disaster recovery, with manual failover and eventual consistency.

Sharded Gitaly is a library system where each book sits on a shelf in exactly one branch: if that branch closes for the day, its books are unavailable. Gitaly Cluster keeps copies of every book in three branches, with a front desk (Praefect) that sends each reader to a branch that is open.

  • Sharded Gitaly keeps a second copy of each repository on another shard
  • Gitaly Cluster replicates synchronously, so no write is ever lost
  • Gitaly Cluster on Kubernetes is generally available in 19.4
  • Snapshot backups are the recommended way to back up Gitaly Cluster
  • Gitaly Cluster and Geo are interchangeable ways to get disaster recovery

Assumes GitLab 19.4: Gitaly on Kubernetes became generally available in 18.11 for sharded Gitaly only, and Gitaly Cluster on Kubernetes has been a beta since 19.1.