What does conftest's --combine flag change about the input a rule sees?
answer
- one evaluation instead of many
- input becomes an array of wrappers
- each element carries path and contents
- per-file rules go silent, not red
- only way to catch a missing file
basics
~20 sWithout it, each file is evaluated on its own and the input is that file's document. With --combine, all files are merged into one evaluation where the input is an array of elements carrying a path and the file's contents, so rules must iterate rather than address fields directly.
solid answer
~50 sNormally conftest evaluates file by file: one input document per file, results attributed to that file. `--combine` performs a single evaluation over everything at once, and the input becomes an array whose elements each hold the file's `path` and its parsed `contents`. That is the point - it is the only way to write an invariant that spans files, such as a pipeline job in one file consuming a value defined in another, or a workload in one manifest referring to an object declared elsewhere in the directory. The cost is that any rule written for the per-file shape stops matching: its field references become undefined, so it reports nothing and the run goes green. Failure messages also lose their file attribution unless you deliberately include each element's `path` in the message. In practice, keep combined rules in their own namespace and run conftest twice rather than trying to make one rule set work in both modes.
go deeper
Know that conftest normally checks one file at a time, and that there is a mode which looks at all the files together for questions that span them.
Describe the shape change precisely - an array of elements holding path and contents - and explain why an unadapted rule goes silent rather than erroring in that mode.
Show the operational discipline: separate namespaces per mode, two invocations, paths in every combined message, and a diagnosis reflex when a rule suddenly reports nothing.
Decide how much policy is allowed to depend on cross-file state, given that combined rules are harder to attribute, harder to test and easy to leave silently inert.
## Two evaluation modes conftest has exactly two ways of feeding a policy: **Per file (the default).** Each input file is parsed and evaluated on its own. `input` *is* that file's document, so a rule addresses fields directly - the kind, the spec, the instruction array. Results come back grouped by file, so a failure names the file that caused it without any effort from the author. **Combined (`--combine`).** All the input files are parsed and evaluated **once, together**. `input` is now an array; each element carries the file's `path` and its parsed `contents`. A rule that wants a field must first iterate the array and reach into `contents`. ## Why the combined mode exists Some invariants are simply not visible in one file. A CI job defined in one file might consume a value declared in another. A workload manifest might name a configuration or account object that is supposed to be declared in the same directory, and the interesting question is whether it *is*. A repository might be required to hold at least one of something - the absence of a file is a property of the set, not of any member of it. None of those can be decided by looking at documents one at a time, and per-file evaluation cannot express them. That last case deserves emphasis: **per-file evaluation can never catch a missing file.** If the rule only runs when a file is present, a repository that simply does not have the file passes trivially. ## The trap Switching a run to `--combine` does not adapt existing rules; it changes what they are looking at. A per-file rule that references a top-level field now finds nothing there, because the top level is an array of wrappers. In this evaluation model a reference to a field that does not exist is **undefined**, and undefined is not false - the rule body simply produces no result. So the rule does not error, does not fail, and does not warn. It reports nothing, the run exits zero, and the pipeline reports success while enforcing less than it did the day before. The symmetric mistake is running combined-shaped rules without the flag, where the same silence results for the same reason. Either way, the diagnostic is the same: a rule that suddenly reports nothing at all, on inputs that used to trigger it, is almost always looking at the wrong shape. ## Attribution and message quality Under per-file evaluation, "which file was wrong" is answered by the grouping. Under `--combine` there is one evaluation, so a message that says only "a container passes a plaintext secret" leaves the developer to search the whole directory. Each array element carries its `path` precisely so the author can put it in the message. Write the path into every combined-mode message as a matter of habit; a gate whose message does not say where to look is a gate people learn to route around. The counts change too. A combined run is one evaluation, so the summary reflects that rather than per-file totals - worth knowing if anything downstream parses those numbers. ## How to organise it The workable arrangement is to treat the two modes as two rule sets: - Put per-file rules in their own namespace and combined rules in another. - Invoke conftest twice - once normally over the files, once with `--combine` and the combined namespace - and let CI fail if either exits non-zero. - Never mix: a namespace that contains both shapes will always have half of itself silently inert, whichever way it is invoked. And keep the combined set small. Most useful policy is genuinely per-file, and per-file rules are easier to write, easier to attribute and cheaper to evaluate. Reach for `--combine` when the question you are asking is honestly about the set of files - a cross-file reference, a required file that must exist, a uniqueness constraint - and not as a default "in case a rule needs it later".
- How do you keep failure messages useful in combined mode?Include the offending element's `path` in the message text. The array element carries it precisely for this reason, and without it a combined-mode failure tells a developer only that something in the directory is wrong. Make it a review rule for the combined namespace: no message ships without the path in it.
- Can one namespace hold both per-file and combined rules?It can hold them, but half of it will always be inert - whichever shape does not match the invocation produces undefined references and therefore no results. Keep the two in separate namespaces and run the tool twice, so each rule set is always evaluated against the shape it was written for.
- Give a check that per-file evaluation fundamentally cannot perform.Asserting that a required file exists at all. A per-file rule only runs when its input is present, so a directory that omits the file passes trivially. Combined evaluation sees the whole set of paths at once, so it can assert that some element matches the required path - or that none does.
Per-file evaluation is marking each exam paper alone; combined evaluation is spreading the whole class's papers on one desk to spot two identical ones. Different question, and a marking scheme written for one is useless for the other.
saying these in an interview costs you the question
- Thinks --combine concatenates files into one document
- Expects per-file rules to keep working once combined
- Reads a green combined run as proof rules matched
- Omits the path, leaving failures unattributable
- Uses --combine by default for every rule