skip to content

Every deploy pipeline in your estate runs helm upgrade with a cluster-admin credential. How would you cut that back?

level: principalimportance: should knowfreq 36%

answer

  1. The credential is the whole blast radius
  2. Inventory the charts before changing anything
  3. Most charts are purely namespaced
  4. Rare and reviewed beats broad and routine

basics

~20 s

Measure what the charts actually write, then tier the install paths: per-namespace identities for application charts, and a rare, reviewed privileged path for the cluster-scoped ones. Helm offers no least-privilege switch — the identity is the only control.

solid answer

~50 s

Because Helm applies as the caller, the credential is the entire blast radius and there is nothing else to tune. I would start by measuring rather than legislating: render every chart in the estate offline and classify its kinds by scope, which usually shows that the large majority are purely namespaced and only a handful — operators, ingress controllers, anything with CRDs or webhooks — are not. Then split the estate into two install paths: application charts installed by a per-team, per-namespace identity, and cluster-scoped charts installed rarely through a privileged path with its own review. Move the CRDs out first, since Helm installs them once and never upgrades them, so `--skip-crds` retires that requirement permanently. Set an authoring rule that new in-house charts must be installable by a namespace-scoped identity, and treat exceptions as a reviewed list rather than a default.

go deeper

for a junior

Understand the starting point: a pipeline that installs charts with a cluster-wide credential can change anything in the cluster, because Helm applies as whoever runs it and adds no restriction of its own.

for a middle

Be able to explain why most charts do not need cluster scope at all, and which parts of an install do — CRDs, namespace creation, cluster-scoped identity objects — so a scoped identity can be derived from a render.

for a senior

Show how you would migrate a real estate without breaking it: pilot one team, discover the delayed failures like history pruning, and keep a working path for the charts that genuinely need privilege.

for a principal

Own the tradeoff and the endpoint. Decide which charts may contain cluster-scoped objects, who approves the privileged path, what the split costs teams in velocity, and how you will state — in numbers — that the estate is better off than before.

## Why this is an identity problem and not a Helm problem Helm has no in-cluster component and no privilege model of its own. A release is created by whoever ran the command, so "what can this pipeline break?" and "what does this credential permit?" are the same question. There is no mode to turn on, no policy file inside Helm, no per-chart sandbox. That is clarifying: every lever is either the identity you hand the runner, or the shape of the charts you install. ## Measure before you legislate The common failure is to announce least privilege and then spend a quarter debugging pipelines. Start by rendering: `helm template` every chart the estate installs, collect the distinct kinds, and split them into namespaced and cluster-scoped. Typically you find something like 40 charts of which 34 render nothing cluster-scoped, 4 carry only CRDs, and 2 are genuinely cluster-level software with webhooks and cluster-scoped identity objects. That inventory turns an intimidating programme into a small list of real exceptions, and it is derived from artefacts rather than from what teams believe their charts contain. Also measure the writes Helm makes on its own behalf — release records in each release namespace, plus `--create-namespace` where pipelines provision their own environments — because those are the requirements that make a carefully scoped identity fail on its first run. ## Tier the install paths The target shape is two or three paths, not one: - **Application path.** One deploy identity per team-namespace, permitted only what that team's charts render plus release storage in that namespace. A compromised runner reaches one namespace. This is where the overwhelming majority of installs live and it should be entirely self-service. - **Platform path.** Cluster-scoped charts — operators, CRD bundles, cluster-wide controllers — installed by a deliberately rare, reviewed path. Rare is the security property: a privileged capability used twice a month with change review is a different risk from the same power sitting in every pipeline. - **Bootstrap path.** The one-time installs of CRDs and cluster singletons, ideally retired to `--skip-crds` afterwards so the requirement does not recur. Where an in-cluster reconciler such as Argo CD or Flux already owns a chart, the pipeline should not be installing it at all; who that reconciler applies as is a separate design question, but the estate-level point is that a chart should have exactly one installer. ## Make chart shape a rule The long-term fix is authoring policy. In-house charts must be installable by a namespace-scoped identity: cluster-scoped objects either live in a separate platform chart or sit behind a values toggle that is off by default and can be pointed at a pre-existing identity by name. Names for any cluster-scoped object must include the namespace, because those objects share one namespace-free name space across the cluster and truncation to 63 characters makes collisions between similar release names entirely possible. Third-party charts are the hard case: many bundle CRDs, cluster-scoped identity objects and webhook configuration in one chart, and the only honest choices are to install them through the platform path, or to render them and split the output yourself — at the cost of owning that split at every version bump. ## Sequence the migration and be honest about cost Run both paths in parallel before switching. Give one team a scoped identity, let their pipeline fail in a non-production namespace, and fix the gaps — release storage, namespace creation, a CRD they had forgotten — before you touch anyone else. Expect to hit failures that appear weeks later rather than at cutover: the eleventh upgrade that must prune a release record at the default `--history-max` of 10, or the chart version that adds a CRD Helm will never install on upgrade. The costs are real and you should name them rather than discover them. Splitting charts means two-step installs with ordering to think about; upgrades to CRD schemas need the privileged path again; the platform team becomes a dependency for the exceptions; and a team that previously shipped anything now has a request queue for the rare cluster-scoped change. Weigh that against the alternative you have today, which is that any job on any runner can do anything to any cluster. ## Know when you are done Pick measurable statements rather than a feeling: how many identities can create cluster-scoped identity objects, how many pipelines can write outside their own namespace, how often the privileged path is used and whether each use has a reviewed change behind it. If the answer to the first two is a small, named, reviewed set and the third is a handful of audited runs a month, the programme has landed — even though a residual privileged path still exists, because it always will.

  • A third-party chart bundles CRDs, cluster-scoped identity objects and webhook configuration in one package. What do you do?
    Install it through the platform path rather than bending the application path around it — it is cluster-level software and pretending otherwise gives every team's pipeline cluster reach. If it must be self-service, render it and split the output into a privileged half and a namespaced half, and accept that you now own that split at every version bump, including whatever the vendor changes about the CRDs. I would only take that on for a chart that is upgraded rarely.
  • How would you convince a team that already ships fine with a broad credential?
    Not with principle, with a concrete failure story and a small cost. Show what a compromised job or a malicious chart could do with today's credential — every namespace in every cluster — and then show the migration is mostly free for them, because their chart renders nothing cluster-scoped. Reserve the argument for the few charts that genuinely need the privileged path, and make that path fast enough that going around it is not tempting.
  • What is left exposed after the programme lands?
    The privileged path itself, which still exists and still holds broad power; the release records in each namespace, which contain the values used and are readable by anyone with namespace Secret access; and anything a chart legitimately renders inside its own namespace, which a hostile chart can still abuse. Narrowing the installer bounds the damage — it does not review the charts, and those are separate controls.

saying these in an interview costs you the question

  • Bans cluster-admin without inventorying the charts
  • Assumes every chart needs cluster-scoped permission
  • Treats a single shared deploy identity as least privilege
  • Ignores release-record writes when scoping identities
  • Says a policy engine replaces narrowing the installer
  • Promises no privileged install path will remain

context