On a single-node Kubernetes dev cluster you clone a clinical-records portal from namespace records-dev into records-review, and the copy misbehaves; which namespace-scoping mistakes do you check, and how do you fix them?
answer
- what in the bundle has no namespace
- manifest namespace beats context
- volume already claimed elsewhere
- copied Secrets go stale
- fully qualified names point home
basics
~20 sCheck for manifests that hard-code metadata.namespace, cluster-scoped objects shared by both copies, a PersistentVolume already bound to the old claim, Secrets that exist only in records-dev, and fully qualified Service names still pointing at records-dev.
solid answer
~50 sI split the bundle with `kubectl api-resources --namespaced=false` and check five things. First, any manifest with `metadata.namespace: records-dev` is applied back into records-dev when no `-n` is given, and kubectl refuses when `-n records-review` disagrees, so I strip it and pass the namespace at apply time. Second, cluster-scoped objects such as a ClusterRole are shared, so the second apply updates the first copy's object; I rename them per copy or move them to a shared bundle. Third, a hand-made PersistentVolume already bound to records-dev's claim leaves the new PVC `Pending`. Fourth, pods fail on Secrets like `portal-tls` that exist only in records-dev, and a one-off copy expires within the certificate's 18-hour validity, so each namespace needs its own issued copy. Fifth, a `DB_HOST` of `records-db.records-dev.svc.cluster.local` quietly sends the review copy to the dev database; I use the short name and add NetworkPolicy.
code
bash · 4 lineskubectl api-resources --namespaced=false -o name
kubectl get pv
kubectl get events -n records-review --sort-by=.lastTimestamp
kubectl get pods -n records-reviewgo deeper
Recall that only namespaced objects are separated by a new namespace, and that pods only see Secrets and claims in their own namespace.
Explain how kubectl chooses a namespace when a manifest sets one, how a PersistentVolume's claimRef blocks a second claim, and why short Service names resolve locally.
Walk the triage in order: split the bundle by scope, read the new namespace's events, and replace Secret copies and fully qualified names with per-namespace sources.
Decide how environments are templated, who owns cluster-scoped objects, and how credentials and data stay separated when several copies share one cluster.
## The scenario A developer runs a clinical-records portal on a **single-node** laptop cluster (a kind or minikube style setup). It lives in namespace `records-dev`. To review a branch, they copy the manifest directory, create `records-review`, apply the copy there, and see a mess: some pods never start, the dev copy changed unexpectedly, and the review portal shows dev patient data. Every one of those symptoms is a scoping mistake. A namespace separates **names of namespaced objects**; everything else in the bundle is still shared or still pointing at the original. ## Step 1: sort the bundle by scope Before touching anything, find which objects in the bundle are cluster-scoped: ```bash kubectl api-resources --namespaced=false -o name grep -h '^kind:' manifests/*.yaml | sort | uniq -c kubectl get events -n records-review --sort-by=.lastTimestamp ``` Then work in a fixed order: 1. Compare the kinds in the bundle against the cluster-scoped list; every match is shared between the two copies. 2. Read the review namespace's events, which point at most of the start-up failures below. 3. Check `kubectl get pv` for volumes already bound to `records-dev` claims. 4. Grep the configuration for `records-dev`, which finds both hard-coded namespaces and fully qualified Service names. ## Step 2: the five usual mistakes | Symptom | Cause | Fix | |---|---|---| | Dev copy changed after "applying to review" | Manifests hard-code `metadata.namespace: records-dev` | Remove the field; pass `-n` or set it in the overlay | | Dev portal's permissions changed | A shared cluster-scoped object was re-applied | Unique names per copy, or a separate shared bundle | | Review PVC stuck `Pending` | Hand-made PersistentVolume already bound to the dev claim | One PV per copy, or dynamic provisioning | | Review pods in `CreateContainerConfigError` or `ContainerCreating` | Secret exists only in `records-dev` | Issue the Secret in `records-review` from its source | | Review portal shows dev records | `DB_HOST` names `records-db.records-dev.svc.cluster.local` | Use the short name `records-db`; add NetworkPolicy | ### 1. Hard-coded namespaces kubectl picks the namespace in a fixed way. If a manifest sets `metadata.namespace` and you pass **no** `-n`, the manifest wins over the context, so the "review" apply quietly updates `records-dev`. If you pass `-n records-review`, kubectl refuses with an error saying the object's namespace does not match. Either way, the fix is to strip the field from reusable manifests and supply the namespace at apply time (a manifest-packaging tool's namespace setting achieves the same). ### 2. Cluster-scoped objects are shared A `ClusterRole` named `records-portal-reader`, a `PriorityClass`, an `IngressClass` or a `CustomResourceDefinition` has no namespace. Applying the review copy does not create a second one; it **updates the existing object**. If the review branch changed the ClusterRole's rules, the dev portal's permissions changed too. Either give per-copy names (`records-portal-reader-review`) or keep cluster-scoped objects in a separately owned bundle that both copies use as-is. Bindings that grant access within one namespace belong in each copy; how Roles and ClusterRoles are bound is an RBAC topic. ### 3. PersistentVolumes bind to one claim Laptop clusters often use a hand-written `hostPath` PersistentVolume, say `records-uploads-pv`, with claims that ask for `storageClassName: manual`, a class with no provisioner behind it. It is cluster-scoped, and its `spec.claimRef` already names `records-dev/portal-uploads`. The review namespace's `portal-uploads` claim has the same name but a different namespace, so it cannot bind, and it stays `Pending`. `kubectl get pv` shows the binding in its `CLAIM` column as `records-dev/portal-uploads`. Create a second PV for the review copy, or use a StorageClass with dynamic provisioning so every claim gets its own volume. ### 4. Secrets do not follow The TLS Secret `portal-tls` and the database credential exist only in `records-dev`. Pod references are resolved in the pod's own namespace, so review pods fail to start. The tempting fix is to export the Secret with `kubectl get secret portal-tls -n records-dev -o yaml`, change its namespace, strip the server-set fields and apply it to `records-review`. That copy is a snapshot. The portal's certificate is reissued with an **18-hour validity**, and the renewal rewrites `portal-tls` in `records-dev` only, so the review copy expires at most 18 hours after it was issued. The durable fix is to have whatever issues the certificate issue it into `records-review` too, and to create database credentials per namespace. For clinical data, the review copy should get its **own** credential and its own database. ### 5. Fully qualified names still point home A config value of `records-db.records-dev.svc.cluster.local` is the same address from every namespace, so the review portal reads the dev database, and possibly real clinical records. Reusable config should use the short name `records-db`, which resolves to the Service in the pod's own namespace. Then add a NetworkPolicy so that only `records-dev` pods can reach the dev database: a namespace alone never blocks that connection. ## Step 3: make the next clone safe - Keep namespace-agnostic manifests; inject the namespace once per environment. - Keep cluster-scoped objects in a separate, separately owned bundle. - Generate Secrets per namespace from their source, never by copying. - Use short Service names for in-namespace dependencies, and fully qualified names only for deliberate cross-namespace calls. - Pair each environment namespace with RBAC, quotas and network rules; the namespace is the handle they attach to.
- Why does kubectl apply -n records-review fail on a manifest that sets metadata.namespace to records-dev?When the namespace comes from the `-n` flag, kubectl requires every namespaced object in the input either to have no namespace or to match. A mismatch fails with an error saying the object's namespace does not match the one given, which guards against editing another namespace by accident. Without `-n`, the manifest's own namespace is used, which is how the dev copy got overwritten.
- How would you give each cloned namespace its own TLS certificate instead of copying the Secret?Declare the certificate request in each namespace's manifests, so the issuing controller writes and renews a `portal-tls` Secret in every namespace itself. Each copy then follows its own 18-hour renewal instead of expiring as a snapshot. If a Secret genuinely must be shared, use a controller that syncs it from a source of truth on every change.
saying these in an interview costs you the question
- Applying the bundle to a new namespace creates new ClusterRoles
- kubectl -n always overrides a manifest's own namespace
- A copied Secret stays valid because Secrets never change
- A PVC with the same name will bind to the existing volume
- A separate namespace stops the review copy reaching the dev database