On a cgroup v2 Linux host, moving a PID into a cgroup that has controllers listed in its cgroup.subtree_control fails with EBUSY. What rule is the kernel enforcing, and how does a supervisor like systemd shape its tree around it?

## What the rule says In the cgroup v2 unified hierarchy, mounted at `/sys/fs/cgroup`, a non-root cgroup may **either** contain processes **or** distribute resources to its children — not both. The kernel documentation calls this the "no internal process" constraint. It is enforced at both write points: - Writing a PID into `cgroup.procs` of a cgroup whose `cgroup.subtree_control` is non-empty → `EBUSY`. - Writing a controller into `cgroup.subtree_control` of a cgroup that already has member processes → `EBUSY`. The root cgroup is exempt, because there has to be somewhere for the processes that nobody has classified — kernel threads and early boot processes — to live. ## Why the kernel insists on it A controller distributes a resource among the entities immediately below a cgroup. If a cgroup could have both children *and* its own loose processes, the controller would have to decide how a process competes against a whole subtree. cgroup v1 allowed this and the semantics were both inconsistent between controllers and hard to reason about, so v2 removed the ambiguity by construction: at every level, a controller sees only cgroups. The practical effect is a shape rule. Your tree becomes inner nodes that carry policy and leaves that carry processes. ## The three files that matter - `cgroup.controllers` — read-only; lists the controllers this cgroup **may** use, because its parent enabled them. - `cgroup.subtree_control` — read-write; lists the controllers this cgroup passes **down to its children**. You may only enable something that appears in `cgroup.controllers`. - `cgroup.procs` — the member processes. Writing a PID moves that process here. Controllers are therefore enabled top-down, one level at a time: ```bash cat /sys/fs/cgroup/cgroup.controllers # cpuset cpu io memory hugetlb pids rdma misc echo "+cpu +memory" > /sys/fs/cgroup/cgroup.subtree_control mkdir -p /sys/fs/cgroup/app/worker echo "+cpu +memory" > /sys/fs/cgroup/app/cgroup.subtree_control echo $$ > /sys/fs/cgroup/app/worker/cgroup.procs # leaf: OK echo $$ > /sys/fs/cgroup/app/cgroup.procs # inner node: EBUSY ``` Disabling a controller in a parent's `subtree_control` also removes its interface files from the children, so limits set there are lost — enabling and disabling controllers is not a free operation on a live tree. ## The threaded exception There is one nuance worth knowing so you do not overstate the rule. A cgroup can be switched to *threaded* mode by writing `threaded` to its `cgroup.type`, which creates a threaded subtree where individual threads rather than whole processes are the unit of membership and `cgroup.threads` becomes the file you write. Inside such a subtree the constraint applies differently, because thread-granular controllers (the CPU controller in particular) must be able to arbitrate between threads of one process. Mentioning that you know the exception exists is usually enough; the domain-cgroup rule is what the question is really about. ## How supervisors build around it Because processes must live in leaves, any manager of many workloads ends up with a two-layer shape: grouping nodes that carry limits, and per-workload leaves that carry PIDs. systemd's cgroup layout is exactly this. Slices are the inner nodes — the root slice, and below it the slices for system services, for user sessions, and for machines — and services and scopes are the leaves where processes actually sit. Setting a limit on a slice constrains everything beneath it; setting one on a service constrains that unit. The second half of the picture is **delegation**. On a systemd host, systemd is the single writer of the cgroup tree, and two agents writing the same tree fight — one creates a cgroup, the other reaps it as unmanaged. When another manager legitimately needs to build its own subtree (a session manager, a container manager), it is *delegated* one: the unit declares `Delegate=`, and systemd hands over ownership of that subtree, changing the ownership of the directory, `cgroup.procs`, `cgroup.threads` and `cgroup.subtree_control` so an unprivileged manager can write them. The kernel's `nsdelegate` mount option reinforces that boundary by treating a cgroup namespace root as a delegation boundary that migrations may not cross. The operational lesson: on a systemd host, do not `mkdir` cgroups directly under the root and expect them to survive. Ask for a delegated subtree, or express the limit through the supervisor. ## What a good answer sounds like Name the constraint, show that you know it fires in both directions with `EBUSY`, explain *why* — resource distribution has to be unambiguous — note the root exemption, and land on the shape it forces: policy on inner nodes, processes in leaves, with delegation as the way two managers coexist.