On GitLab.com, a group's jobs on hosted runners stop mid-month while its own group runners keep working; what happened, and what becomes of pending and running jobs?
answer
- looks like a runner outage
- namespace quota at zero
- instance runners stop taking jobs
- a small grace overage
- your own runners unaffected
basics
~20 sThe top-level namespace used up its compute quota. Instance runners stop taking its new jobs, pending and retried ones are dropped, running jobs continue until usage is 1,000 minutes over quota, and the group's own runners carry on.
solid answer
~50 sThe pattern points to quota enforcement, not an outage. When a namespace's compute quota for the month is used, GitLab's instance runners, which on GitLab.com are the hosted runners, **stop processing new jobs** for it. In pipelines already started, any pending or retried job that needs an instance runner is **dropped**; jobs already running continue until the namespace is **1,000 compute minutes over quota**, then they are dropped too. Group and project runners are not subject to the quota, which is why they keep working. Owners get a banner and an email when less than 25% and less than 5% remains and when it hits zero. The fixes are buying additional compute minutes, which are used after the included quota and roll over, waiting for the reset on the first of the month, reducing usage, or moving work to the team's own runners.
go deeper
Recall that GitLab.com hosted runners draw on a monthly compute quota per top-level namespace, and that when it runs out hosted jobs stop while your own runners keep working.
Explain enforcement step by step: new jobs refused, pending and retried jobs dropped, running jobs allowed a 1,000-minute grace overage, and warnings to owners at 25%, 5% and zero.
Show the diagnosis: telling quota enforcement apart from tag routing and normal hosted start-up delay, reading the Usage quotas page, and choosing between buying minutes, cutting usage and moving jobs to your own runners.
Treat the quota as an operational control: who receives the owner warnings, whether critical pipelines should depend on a metered pool at all, and how purchased minutes and own runners are budgeted.
## The symptom and its cause A team on GitLab.com sees jobs that normally run on GitLab-hosted runners vanish or never start, partway through a month, while the jobs routed to the runners it operates itself keep running. Because nothing errors on the runner side, it looks like a **runner availability problem**. GitLab's documentation says exactly that: enforcement of the compute quota can look like a runner problem rather than a quota problem. The cause is the **compute quota** of the job's **top-level namespace**. Every job on an instance runner adds `job duration / 60 * cost factor` to that namespace's monthly usage. On GitLab.com the instance runners are the hosted runners, and the quota is enabled by default. ## What enforcement does When the quota for the current month is used up: 1. **Instance runners stop processing new jobs** for the namespace. 2. In pipelines that have already started, any **pending** job, and any **retried** job, that must be processed by instance runners is **dropped**. 3. Jobs **already running** on instance runners continue until the namespace's usage is **1,000 compute minutes over quota**. After that grace period, any jobs still running are dropped as well. 4. **Group and project runners are not affected** and continue processing jobs, because the compute quota applies only to admin-managed instance runners. | Job state when the quota hits zero | Runs on | Outcome | |---|---|---| | Not yet started (pending) | Hosted runner | Dropped | | Retried | Hosted runner | Dropped | | Running | Hosted runner | Continues until 1,000 minutes over quota, then dropped | | Any | The group's or project's own runner | Unaffected | ## The warnings that came first Namespace owners are warned through an in-app banner and an email when the remaining compute minutes drop below **25%** of the quota, below **5%**, and when they are **completely used**. If nobody acted, check who the namespace owners are: the warnings go to them, not to everyone who runs pipelines. The group's **Usage quotas** page, on its **Pipelines** tab, shows usage and remaining minutes, including purchased additional minutes, and lists the month's projects by consumption. Only the top-level group has it, and you need the Owner role to view it. ## Ways out - **Buy additional compute minutes.** They are used only after the included monthly quota runs out, roll over to the next month if unused, stay with the namespace when the subscription tier changes, and cannot be transferred from one group to another. - **Wait for the monthly reset.** Usage returns to `0` on the first day of each month and the quota is back to the monthly allocation. - **Reduce usage.** Find the projects at the top of the usage list and the jobs that changed machine type, went parallel or run more often. - **Run jobs on your own runners.** Group and project runners are outside the quota, at the price of operating them. ## Self-Managed is different On GitLab Self-Managed the compute quota is **disabled by default**: the instance-wide default is `0`, which means unlimited. An administrator can set a default quota for all namespaces, override it for a single group, and reset a namespace's usage. An administrator can also **Reset compute usage** for a namespace that has used its allowance, which restores a namespace that has run out without waiting for the month to end. When a quota is set, notifications and enforcement work the same way as on GitLab.com. On the Usage quotas page, a namespace without a limit shows **Unlimited**, and one where instance runners are not enabled shows **Not supported**. ## How it fits the rest of the picture Two other things can make a job wait or disappear, and telling them apart is the diagnosis. A job whose tags match no available runner sits pending; that is a routing question for the material on runner configuration. A job that waits a minute or two for a hosted VM is normal, since GitLab's objective is that 90% of hosted jobs start within 120 seconds. A namespace at its quota is the case where the **hosted** runners alone stop taking jobs while the team's own runners carry on.
- On GitLab.com, a group bought additional compute minutes and used only half by month end. What happens to the rest?They roll over to the next month. Purchased minutes are a one-time purchase consumed only after the included quota runs out each month, so unused ones carry forward rather than resetting. They also cannot be moved to another group, so buying for the right top-level group matters.
- On GitLab Self-Managed, why does a team never hit a compute quota even though it uses instance runners heavily?Because the instance default is `0`, which means unlimited; quotas are off until an administrator sets a default for all namespaces or a quota for a specific group. Once one is set, notifications and enforcement behave as on GitLab.com.
- On GitLab.com, a running job on a hosted runner was dropped after the quota hit zero, but another kept running for a while. Why?Running jobs are allowed to continue until the namespace is 1,000 compute minutes over quota. The job that survived longer was still running inside that grace overage; once the namespace crossed it, any jobs still running on instance runners were dropped.
- When the quota runs out, every runner the group uses stops, including its own.
- Running jobs are killed the moment the quota reaches zero.
- Purchased compute minutes expire at the monthly reset like the included quota.
- Jobs that cannot run are paused and resume automatically next month.
- Self-Managed instances apply the same default quota as GitLab.com Free.