A workload's ServiceAccount is getting HTTP 403 Forbidden from the Kubernetes API server. How do you determine exactly which permission is missing and which binding should grant it?

## The error already contains the answer Kubernetes 403 messages are unusually good. A typical one: ``` Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:orders:worker" cannot list resource "secrets" in API group "" in the namespace "orders" ``` Four facts fall out: the **identity** (`system:serviceaccount:<ns>:<name>` for a workload; a username or `system:anonymous` for a human or a broken auth path), the **verb**, the **resource with its API group**, and the **namespace** (absent for cluster-scoped resources). Any RBAC debugging that starts by guessing has skipped reading this line. If the identity is wrong — `system:anonymous`, or `system:serviceaccount:orders:default` when you expected `worker` — the problem is authentication or the pod spec, not RBAC. The usual cause is a Deployment that never set `serviceAccountName`, so the pod got the namespace's `default` ServiceAccount, which has essentially no permissions. ## Reproducing with impersonation `--as` makes the API server evaluate the request as another subject: ``` kubectl auth can-i list secrets -n orders \ --as system:serviceaccount:orders:worker ``` It returns `yes` or `no` without performing the action. Add `--as-group` to test group-based bindings. Impersonation itself requires the `impersonate` verb on `users`/`groups`/`serviceaccounts`, which `cluster-admin` has; that is also why `impersonate` is a permission to hand out very sparingly, since holding it means being able to act as anyone. The higher-value command is the listing form: ``` kubectl auth can-i --list -n orders \ --as system:serviceaccount:orders:worker ``` This prints the subject's entire effective rule set in that namespace — every resource, verb and resourceName. Comparing it against what the workload actually calls is the fastest way to spot both the missing grant and any surplus you should remove. Since Kubernetes 1.27 there is also `kubectl auth whoami`, which reports the identity the current kubeconfig authenticates as — useful when the question is "who am I, actually?" rather than "what may I do?". ## Locating the bindings There is no built-in reverse index from subject to bindings, so you enumerate: ``` kubectl get rolebindings,clusterrolebindings -A -o json \ | jq -r '.items[] | select(.subjects[]?.name=="worker") | "\(.kind) \(.metadata.namespace // "-")/\(.metadata.name) -> \(.roleRef.kind)/\(.roleRef.name)"' ``` Then `kubectl describe clusterrole <name>` or `describe role` to read the rules. Remember permissions **union** across all bindings and there is no deny, so the presence of one narrow binding does not exclude a broader one elsewhere. When something has *more* access than expected, it is because a second binding exists — commonly a ClusterRoleBinding to `edit` or an aggregated ClusterRole that grew a new label. ## Common causes, ranked 1. **Wrong ServiceAccount** — `serviceAccountName` missing from the pod template; the pod runs as `default`. 2. **Missing subresource** — `pods` granted, `pods/log` or `pods/exec` not. They are separate resource strings. 3. **Wrong apiGroup** — `deployments` under `""` instead of `apps`; a CRD's group misspelled. 4. **get without list** — `kubectl get <resource>` with no name is a `list`; controllers and informers additionally need `watch`. 5. **resourceNames plus list** — narrowing by name cannot filter a collection, so the list is denied outright. 6. **Binding in the wrong namespace** — a RoleBinding grants only in its own namespace; the SA may live elsewhere. A RoleBinding in namespace A *can* name a ServiceAccount from namespace B, and the grant applies in A. 7. **Cluster-scoped resource under a RoleBinding** — nodes, PVs and namespaces need a ClusterRoleBinding; the rules are inert otherwise. 8. **Cached token or stale binding** — `roleRef` is immutable, so an "updated" binding may in fact be a stale one that was never recreated. ## Fixing it well Grant the minimum: the exact verbs, the exact resource including subresource, in a `Role` bound by a `RoleBinding` in the workload's namespace. Reach for a ClusterRole only when the workload genuinely needs the permission in multiple namespaces or on cluster-scoped objects. For Secrets, narrow with `resourceNames` to the specific objects — remembering that this only works with `get`, not `list`. Then verify the same way you diagnosed: re-run `kubectl auth can-i --list` as the ServiceAccount and confirm the new rule appears and nothing broader crept in. Restart the workload only if it caches its permission failures; the API server picks up RBAC changes within seconds, no restart required. ## Do not The wrong fix is binding `cluster-admin` "temporarily" to unblock a deploy. Those bindings are never removed, and they are exactly what an attacker who compromises the pod inherits. If you must unblock fast, bind the built-in `view` or `edit` ClusterRole scoped by a RoleBinding to that one namespace, and file the narrowing as follow-up work.