In an InSpec profile run, what is the difference between a failed control and a skipped one?
answer
- ran and lost versus never ran
- no evidence, not a pass
- only_if, unsupported platform, waiver
- a missing file fails, it does not skip
- report the skip count beside the pass rate
basics
~20 sA failed control ran and the host did not satisfy it. A skipped control never ran at all - wrong platform, a guard, or a waiver - so it yields no evidence about that requirement, and it is not a pass.
solid answer
~50 sA **failure** means the control executed and at least one assertion inside it was false: the host is in a state the benchmark forbids. A **skip** means the control never executed, so nothing at all is known about that requirement on that host. Skips come from a platform the profile does not support, an `only_if` guard evaluating false, a `skip_control` in a wrapper profile, or a waiver with `run: false`. What does not cause a skip is a missing resource - a control describing a file that does not exist fails, because the matcher evaluated and the answer was no. Practically, that means you never compute compliance as passed over total. Report passed, failed and skipped separately, with a reason for each skip, because a skip is an evidence gap and folding it into the green number overstates what the run actually established.
go deeper
Be ready to read a profile report out loud and say what each result state means. Know that skipped is not a pass and that a control against a missing file fails rather than skipping.
Explain the four sources of a skip - unsupported platform, only_if guard, skip_control in a wrapper, waiver with run false - and why impact only sets severity rather than deciding whether a control runs.
Show how you would build a fleet report that no one can misread: passed, failed and skipped as separate counts, a reason attached to every skip, and a rule that unassessed never renders as compliant.
Own the argument about which number the organisation is measured on. A pass percentage that absorbs skips will always drift upward, so the metric definition itself is a control you should be willing to defend to an auditor.
### What a profile run is An **InSpec profile** is a directory of Ruby files under `controls/` plus an `inspec.yml` that names the profile, its version, the platforms it `supports`, its `inputs`, and any profiles it `depends` on. Each file holds **controls**, and each control holds one or more `describe` blocks that point a **resource** (`sshd_config`, `file`, `mount`, `package`, `service`) at the running machine and assert something about it. You run the profile against a live target — locally on the box, or over SSH/WinRM from elsewhere — and it reports per-control results. Nothing is prevented: the profile inspects a system that is already running, so every outcome is *detective and reported*. ### Passed, failed, skipped - **Passed** — the control ran and every assertion inside it held. - **Failed** — the control ran and at least one assertion did not hold. The host is in a state the benchmark says it should not be in. - **Skipped** — the control **did not run at all**. No assertion was evaluated, so the report says nothing about that requirement on that host. The distinction that matters for reporting is between *"we looked and it was wrong"* and *"we never looked"*. A skipped control is an evidence gap wearing a neutral colour, and the single most common reporting sin in compliance-as-code is folding skips into the pass column so a dashboard can show a bigger green number. ### Where skips come from 1. **Platform mismatch.** `inspec.yml` (or a control's own `supports`) declares which platform families a profile applies to. Run a Linux benchmark against a Windows host and controls that cannot apply are skipped rather than failed. 2. **An `only_if` guard.** A control wrapped in `only_if { package('nginx').installed? }` is skipped when the guard is false. This is the deliberate, honest way to express *not applicable here*. 3. **An explicit skip in a wrapper profile.** A profile that inherits an upstream benchmark can `skip_control 'some-id'`, and that control then reports as skipped for everyone running the wrapper. 4. **A waiver.** A waiver file entry with `run: false` suppresses the control and marks it waived, with the justification and expiry recorded alongside the result. What does **not** produce a skip is a resource that is absent. `describe file('/etc/foo.conf') do its('mode') { should cmp '0600' } end` on a host where that file does not exist **fails** — the matcher had something to evaluate and the answer was no. Authors who expect a graceful skip there are surprised by a wall of red on hosts where the component was never installed; the fix is an `only_if` guard, not a lower `impact`. ### Impact is severity, not selection Every control may declare `impact`, a float from 0.0 to 1.0 that reporters map to severity bands (informational through critical). Impact does not decide whether a control runs, and lowering it does not turn a failure into a pass — it only changes how loudly the failure is ranked. Downstream tooling may filter or prioritise on it; the result itself is unchanged. ### The same distinction in the other runners The vocabulary differs but the idea is identical: | Runner | "we looked and it was wrong" | "we never looked" | | --- | --- | --- | | InSpec | failed | skipped | | oscap over XCCDF | `fail` | `notapplicable`, `notchecked`, `notselected` | | kube-bench | `[FAIL]` | `[WARN]` (a manual check) / `[INFO]` | XCCDF's result vocabulary is the most explicit of the three: it separates *the rule does not apply to this target* (`notapplicable`) from *no check was available or run* (`notchecked`) from *the profile did not select this rule* (`notselected`). kube-bench's `[WARN]` is the trap in its own output — it usually marks a check a human has to verify by hand, and a team that counts WARN as PASS is reporting compliance it has not established. ### Why it comes up in interviews Reading a report is the first thing anyone operating a runner fleet actually does, and the first mistake anyone makes is arithmetic: `passed / total`. The defensible number is `passed / (passed + failed)` reported *next to* an explicit skip count, with the reason each skip happened. An auditor asking whether a control operated all quarter is asking exactly this: not "was it green", but "did it run, on which hosts, and what did it see". A run where three quarters of the controls skipped because the profile targeted the wrong platform family is a green run that proves nothing.
- A control describes a config file that is not present on the host. Does it fail or skip?It fails. The matcher ran and the assertion was false, so InSpec reports a failure rather than a skip. If the control genuinely does not apply to that host - the component is not installed at all - guard it with `only_if` so it reports as skipped for a stated reason, instead of producing a red result that everyone learns to ignore.
- Why does an auditor care about the skipped count rather than just the failures?Because a skip means the control did not operate on that host. An auditor asking whether a control ran all quarter cannot accept a green summary that hides two hundred controls which never executed. Skips must be enumerated with reasons - unsupported platform, not applicable, waived with justification and expiry - so the report distinguishes 'we checked and it was fine' from 'we never checked'.
- How is a skipped control different from one filtered out with the --controls flag?A control filtered out at the command line is not part of that run at all - it does not appear in the report, so nothing records that it was omitted. A skipped control does appear, marked skipped, which is why exclusions you intend to keep belong in the profile or a waiver rather than in a run-time flag that leaves no trace.
A skipped control is a question left blank on an exam, not a question answered correctly - the score is only honest if you say how many you left blank.
saying these in an interview costs you the question
- Counts skipped controls as passes in the compliance summary
- Thinks a control for a missing file skips itself
- Assumes a green run means every control was evaluated
- Reports only a pass percentage with no skip count
- Believes a failing control blocks the change that caused it