In Salt, what is the difference between grains and pillars, and which of the two should hold a database password?

## Two data planes pointing in opposite directions Salt has two ways to attach data to a minion, and the whole distinction is direction and ownership. **Grains flow up.** They are facts about the host, gathered by the minion itself when `salt-minion` starts: operating system and version, kernel, CPU architecture and count, memory, FQDN, IP addresses, virtualisation type. You can add your own static grains in `/etc/salt/grains`, in the minion config under a `grains:` key, or at runtime with `grains.setval`. Inspect them with: ```bash salt 'web1' grains.items salt 'web1' grains.get os ``` **Pillars flow down.** They are data the **master** owns, rendered from files under `pillar_roots` (conventionally `/srv/pillar`), and delivered to minions individually over the return port. Which minion gets which data is decided by the pillar `top.sls`, using the same matchers as ordinary targeting: ```yaml base: 'web*': - webserver 'os:Ubuntu': - match: grain - apt ``` Inspect with `salt 'web1' pillar.items`, and read a nested key in a state or template with `salt['pillar.get']('db:password')`. ## Using each in states Grains answer "what am I?", so they drive branching: ```yaml {% if grains['os_family'] == 'Debian' %} nginx_pkg: pkg.installed: - name: nginx-full {% endif %} ``` Pillars answer "what should I be configured with?", so they carry the values: ```yaml /etc/myapp/app.conf: file.managed: - source: salt://myapp/app.conf.jinja - template: jinja - mode: '0640' - context: db_password: {{ salt['pillar.get']('db:password') }} ``` The same SLS file then serves every environment; only the pillar changes. ## Why the password goes in pillar Three reasons, and interviewers want at least two of them. 1. **Ownership.** Grains are generated on the minion and can be set on the minion. A host that has been tampered with can claim any grain it likes — including a grain you use for targeting. Pillar is compiled on the master; the minion has no way to assert what its pillar should be. 2. **Scoping.** Pillar is targeted. Only minions matched by the pillar `top.sls` receive that data, and each minion's payload is encrypted for it on the way down. Put the secret in a grain and it is on the host in plain view, sitting in `grains.items` for anyone who can run a command against it. 3. **Layering.** Because the master owns pillar, you can plug in stronger sources: the GPG renderer keeps values encrypted at rest in the pillar files and decrypts them on the master at render time, and `ext_pillar` pulls from an external system entirely. That said, be honest about the limit: once rendered, the value is in memory on the minion and typically written into a config file. Pillar controls *distribution*, not what happens after delivery — so keep managed files' modes tight and never pass a secret as an argument to `cmd.run`, where it lands in the job cache and the logs. ## Refreshing A minion caches its pillar. Edit `/srv/pillar/...` on the master and existing minions do not magically see it; you refresh explicitly: ```bash salt '*' saltutil.refresh_pillar ``` Do not confuse this with `saltutil.sync_all`, which syncs custom modules, states, grains and returners to minions — a different kind of payload entirely. Grains refresh with `saltutil.refresh_grains`, and a minion restart re-collects them anyway. ## The mental shortcut "Grains are what the host tells you; pillar is what you tell the host." Target and branch on grains, configure and secret with pillar. Where the two overlap — for example, you can target minions by pillar with `-I 'role:db'` — prefer pillar for anything you would be unhappy for a compromised host to control, and grains for anything the host is genuinely the authority on.