skip to content

A contractor leaving the fraud-rules engine team still holds a Kubernetes client certificate valid for 290 more days. How do you actually cut off their cluster access?

level: seniorimportance: must knowfreq 58%

answer

  1. authentication has no revocation check
  2. cut it at authorization
  3. groups are frozen in the subject
  4. superuser group needs a new CA
  5. tokens they minted outlive them

basics

~20 s

Kubernetes checks no CRL for client certificates, so the certificate keeps authenticating. Remove every RBAC binding matching its CN, re-key any group in its O field, rotate the CA if it holds system:masters, and hunt for credentials they minted.

solid answer

~40 s

You cannot revoke the certificate itself: the API server only checks that it chains to the client CA and is unexpired, and deleting the CSR object changes nothing. So cut access at authorization. Delete or edit every RoleBinding and ClusterRoleBinding naming the certificate's CN. Groups from its O values cannot be removed for one person, so bind the team to a new group, reissue their certificates and drop the old binding. If the certificate carries `system:masters`, RBAC is bypassed and only rotating the trusted CA helps. Then look for side doors in the audit logs: ServiceAccount tokens or certificates they minted, impersonation rights, secrets they read. Finally, never reuse the username, issue short `expirationSeconds`, and move people to OIDC.

go deeper

for a junior

Know that Kubernetes client certificates cannot be revoked and that access is removed by deleting RBAC bindings.

for a middle

Explain why CN bindings can be removed per person while O groups and system:masters cannot, and why deleting the CSR does nothing.

for a senior

Walk the full sequence: bindings, group re-keying, CA rotation for superusers, minted tokens and certificates, secret rotation, audit evidence, and broken automation.

for a principal

Turn the incident into policy: short-lived certificates, no groups in personal certificates, no username reuse, and humans moved to an identity provider with break-glass kept separate.

## Why this is the part people get wrong When a person with a Kubernetes **client certificate** leaves, the instinct is to "revoke the certificate". Kubernetes gives you nothing to revoke it with: - The API server's client-certificate authenticator checks only that the certificate **chains to the client CA and is within its validity dates**. It consults **no CRL and no OCSP responder**. - Deleting the `CertificateSigningRequest` object does nothing to the issued certificate; approved CSR objects are garbage-collected after an hour anyway. - There is no User object to disable. So a certificate signed with the default 8760h lifetime 75 days ago still **authenticates** for another 290 days. Revocation must happen at **authorization**, or by removing the trust anchor. ## Step 1: find what the identity can do The certificate subject decides which RBAC subjects apply. Decode the contractor's certificate (or find it in your issuance records) and note the CN and every O: | Subject part | RBAC subject it matches | How to cut it off | |---|---|---| | `CN=dev.kowalski` | `User` `dev.kowalski` | Delete or edit every RoleBinding and ClusterRoleBinding naming that user | | `O=fraud-rules-oncall` | `Group` `fraud-rules-oncall` | Cannot be removed for one person; the whole group binding must go | | `O=system:masters` | the superuser group | No RBAC change helps; authorization is bypassed | List bindings that mention the user or group, for example with `kubectl get rolebindings,clusterrolebindings -A -o json` filtered on `.subjects`. `kubectl auth can-i --list --as=dev.kowalski --as-group=fraud-rules-oncall` shows the effective result. ## Step 2: handle groups baked into the certificate A group written into a certificate is **frozen** until expiry. If the fraud-rules engine team's access comes from a binding on `fraud-rules-oncall`, you must: 1. Create a new group name (for example `fraud-rules-oncall-v2`) and bind it to the same roles. 2. Reissue certificates for the remaining team members with the new O value. 3. Delete the bindings for the old group once everyone has moved. If the certificate carries `system:masters`, which the default-on `CertificateSubjectRestriction` admission plugin prevents for the `kubernetes.io/kube-apiserver-client` signer but which a hand-signed certificate or kubeadm's `super-admin.conf` can hold, the only fix is to **rotate the CA** that the API server trusts. That is a cluster-lifecycle operation, so plan it, don't improvise it. ## Step 3: close the side doors Removing bindings stops new actions, but the contractor may have created credentials that outlive the certificate: - **ServiceAccount tokens** they minted with `kubectl create token --duration=...`, or long-lived token Secrets they created. - **Other certificates** they requested through the CSR API while they had create rights. - **Impersonation** or `escalate`/`bind` rights that let them grant themselves access again. - **Secrets they read**, such as database passwords for the fraud-rules engine, which now need rotating. - **Automation** running as their identity. One example: a script on a build host that performs a 13-minute node drain across the spot pool of the 38-node cluster using their certificate, which will now fail with 403. Audit logs are the evidence source for all of this: filter on their username for the last 75 days. ## Step 4: fix the process so it never recurs - **Never reuse a username.** A new hire given the same CN inherits every binding the old certificate still matches, and the old certificate matches the new hire's bindings too. - **Issue short lifetimes.** Set `expirationSeconds` so certificates last hours or days, not the one-year default. - **Do not put team groups in personal certificates.** Bind users by name, or better, move human access to an OIDC identity provider where disabling the account stops new tokens and short token lifetimes bound the delay. - **Keep certificates for break-glass only**, stored and audited separately. - As a stopgap, a network allowlist on the API endpoint can block the contractor's source, but it is not an identity control. ## What a strong answer sounds like "I can't revoke it; the API server has no CRL. I remove every binding that matches the CN, re-key any group baked into the certificate, rotate the CA if it holds system:masters, hunt for tokens and certificates they minted, rotate secrets they touched, then shorten lifetimes and move people to OIDC."

  • Why is reusing the departed contractor's username for a new hire dangerous?
    Kubernetes identifies certificate users only by the CN string. If a new hire gets the same username, every binding created for them also matches the old, still-valid certificate, silently re-granting the contractor access. Treat usernames as never reusable, or use identifiers from an identity provider that are unique and immutable.
  • Could you block the contractor with a deny rule instead of editing bindings?
    Not with RBAC, which only has allow rules. A deny needs an authorization webhook, or another authorizer placed before RBAC, that returns an explicit deny for that username; most clusters don't run one. Admission policies do not help either, because they see only write requests, so reads such as `get secrets` would still succeed.
  • Which credentials might the contractor have created that survive all binding changes?
    ServiceAccount tokens requested with `kubectl create token` and a long `--duration`, legacy token Secrets, additional client certificates obtained through the CSR API, and copies of kubeconfigs for shared ServiceAccounts. Those authenticate as other identities, so removing the contractor's own bindings does not touch them; revoke them separately and rotate any secret values they could read.

It is like a building badge that the door readers never check against a lost-badge list: you cannot cancel the card, so you remove its name from every door's access list and rekey any door it opens by team membership.

saying these in an interview costs you the question

  • Deleting the CertificateSigningRequest object revokes the issued certificate.
  • The API server checks a certificate revocation list for client certificates.
  • Removing the user's RoleBinding also removes access granted through their certificate's groups.
  • RBAC can express a deny rule for one specific user.
  • A certificate carrying system:masters can be restricted by editing ClusterRoleBindings.
  • Once their bindings are gone, nothing they created can still reach the cluster.