Your team hand-maintains around two hundred Argo CD Application manifests across several clusters. How would you decide whether to replace them with ApplicationSet generators, and what risks does that introduce?
answer
- derived sets versus authored ones
- one template, many generated Applications
- the generator's input becomes the deploy trigger
- children are owned, so deletion cascades
- less typing, more concentrated blast radius
basics
~20 sReplace hand-written Applications when the set is derived from data you already have — a list of clusters, directories in a repo, or repositories in an org. ApplicationSet generates Applications from that data plus one template, which removes the copy-paste but concentrates blast radius in a single object.
solid answer
~50 sThe deciding question is whether the two hundred manifests are *derived* or *authored*. If they differ only by values you could enumerate — cluster name, team, directory — they are derived, and an ApplicationSet with a generator plus one `template` replaces them: `list`, `cluster`, `git` (directory or file), `matrix` and `merge` for combining, `scmProvider` and `pullRequest` for repository-driven sets. If instead each app has genuinely bespoke settings, a generator becomes a template full of conditionals and you have made things worse. The risk is concentration: one edit to the template now rewrites every generated Application at once, and deleting the ApplicationSet cascades to delete its children. Mitigations are `spec.syncPolicy.applicationsSync` (`create-only`, `create-update`, `create-delete`) to restrict what the controller may do, `preserveResourcesOnDeletion: true` so removing an Application does not tear down its workloads, and progressive rollout via `spec.strategy` where available. Keep the ApplicationSet itself in Git and review changes to it like production code, because it is.
code
yaml · 37 linesapiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: team-apps
namespace: argocd
spec:
goTemplate: true
generators:
- matrix:
generators:
- git:
repoURL: https://github.com/acme/manifests.git
revision: main
directories:
- path: apps/*
- clusters:
selector:
matchLabels:
env: prod
template:
metadata:
name: '{{.path.basename}}-{{.name}}'
spec:
project: platform
source:
repoURL: https://github.com/acme/manifests.git
targetRevision: main
path: '{{.path.path}}'
destination:
server: '{{.server}}'
namespace: '{{.path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncPolicy:
preserveResourcesOnDeletion: truego deeper
Know that ApplicationSet is a separate Argo CD resource that generates Application objects from a template plus a source of parameters, so you do not hand-write one manifest per app or cluster.
Name the common generators — list, clusters, git directory and file, matrix, merge, pull request — and explain that generated Applications behave exactly like hand-written ones once created.
Discuss the operational consequences: cascading deletion, preserveResourcesOnDeletion, applicationsSync policies, and how a generator's input silently becomes a deployment trigger.
Own the tradeoff itself — when repetition justifies concentrating the estate into one high-privilege object, how template changes are reviewed and rolled out progressively, and where hand-written Applications should remain.
## The real question is not tooling Two hundred Application manifests are a smell only if they are *repetitive*. The test to apply: pick three of them at random and diff. If the differences are a name, a path, a destination and maybe an image tag, the set is derived from data and hand-maintenance is pure transcription cost. If each one carries genuinely different sync policy, ignore rules, project and source layout, then they are authored artefacts and templating them will produce a generator template riddled with conditionals — harder to read than the two hundred files were. ## What ApplicationSet is `ApplicationSet` is a second Argo CD CRD, handled by its own controller, that produces `Application` objects. It has two halves: - `spec.generators` — one or more sources of parameters. - `spec.template` — an `Application` skeleton with placeholders filled from those parameters. ```yaml apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: checkout-per-cluster namespace: argocd spec: goTemplate: true generators: - clusters: selector: matchLabels: env: prod template: metadata: name: 'checkout-{{.name}}' spec: project: payments source: repoURL: https://github.com/acme/manifests.git path: apps/checkout/overlays/prod targetRevision: v1.8.3 destination: server: '{{.server}}' namespace: checkout syncPolicy: automated: prune: true selfHeal: true ``` The generated objects are ordinary Applications: everything you know about `syncPolicy`, `ignoreDifferences` and sync waves still applies. ## Choosing a generator - **list** — an explicit array of parameter sets. Honest and readable when the set is small and genuinely hand-picked. - **clusters** — one Application per cluster registered with Argo CD, filterable by label. The natural fit for "this platform component belongs on every production cluster". - **git** — directory mode creates one Application per directory matching a pattern (the "app of apps by convention" model: adding a folder adds an app); file mode reads config files in the repo and uses their contents as parameters, which keeps the data explicit and reviewable. - **matrix** / **merge** — combine two generators, most often *apps × clusters*, or overlay per-cluster overrides on a base list. - **scmProvider** / **pullRequest** — enumerate repositories in an organisation, or open pull requests, the latter being how ephemeral preview environments are built and torn down automatically. Git *file* generators tend to age better than directory generators at scale, because the parameters are visible data in a reviewed file rather than an emergent property of the folder tree. ## The risks you must name **Concentration of blast radius.** A typo in `template` no longer breaks one app; it rewrites all of them on the next reconcile. This is the central tradeoff — you traded two hundred small, independently reviewable objects for one large, powerful one. **Cascading deletion.** Generated Applications are owned by the ApplicationSet. Delete or narrow the generator and the Applications go, and with `prune: true` in the template, their workloads go too. `spec.syncPolicy.preserveResourcesOnDeletion: true` keeps the deployed resources when a generated Application disappears; `applicationsSync: create-only` or `create-update` restricts the controller from deleting at all. On a production ApplicationSet, at least one of those should be set before you rely on it. **Silent set changes.** With a `clusters` generator, registering a new cluster deploys to it immediately; with a `git` directory generator, adding a folder does. That is the feature, and it is also how something reaches production without anyone reviewing a deploy. Decide deliberately which generator's input is a reviewed artefact. **All-at-once rollout.** Changing `targetRevision` in the template updates every generated Application simultaneously. Where supported, `spec.strategy` progressive syncs let you roll the change through labelled groups of Applications in steps rather than all at once; otherwise, stage the change by splitting into more than one ApplicationSet. ## A defensible migration 1. Start with the clearest repetitive family — one platform component across all clusters — not the whole estate. 2. Generate into a non-production project first and diff the produced Applications against the hand-written ones. 3. Set `preserveResourcesOnDeletion` and a restrictive `applicationsSync` before pointing it at anything that matters. 4. Put the ApplicationSet in Git, require review on it, and treat template changes with the same gravity as a change to every service it generates — because that is what they are. 5. Leave genuinely bespoke Applications hand-written. A hybrid is the normal end state, not a failure. The principal-level judgment is knowing that ApplicationSet moves work rather than removing it: less transcription, more concentrated risk, and a new question about who is allowed to change the generator's input.
- What happens to running workloads if someone deletes the ApplicationSet?By default the generated Applications are deleted as owned children, and any of them with prune enabled takes its workloads down with it. Setting `spec.syncPolicy.preserveResourcesOnDeletion: true` keeps the deployed resources alive, and an `applicationsSync` policy of `create-only` or `create-update` stops the controller deleting generated Applications at all.
- When is a git directory generator worse than a git file generator?When the set of applications becomes an emergent property of the folder tree: adding a directory silently deploys something. A file generator puts the parameters in an explicit, reviewable file, so the same change is a visible diff with an approver. Directory generators are fine for internal or preview environments where speed beats ceremony.
- How would you roll a template change out to two hundred generated Applications without changing everything at once?Use progressive sync via `spec.strategy` where your version supports it, stepping through labelled groups of Applications with health gates between steps. Where it is not available, split the estate into several ApplicationSets by environment or tier and change them in sequence, which also gives you a natural place to stop.
- Would you keep any Applications hand-written after adopting ApplicationSet?Yes — the ones that are genuinely bespoke. Forcing an application with unusual sync policy, ignore rules or source layout through a shared template produces a conditional-heavy generator that is harder to reason about than the file it replaced. A hybrid estate is the normal outcome, and the boundary is repetition, not principle.
saying these in an interview costs you the question
- Templating applications whose real differences are not enumerable
- Assuming generated Applications survive deleting the ApplicationSet
- Forgetting a template change rewrites every generated app at once
- Treating a cluster or directory generator's input as unreviewed
- Believing ApplicationSet replaces AppProject-level restrictions