A control report says 98% of database instances are encrypted at rest — what does that number hide?
answer
- the ratio has two halves
- who decided the bottom number
- unreached is not the same as failed
- pass, fail, unassessed, not applicable
- coverage and pass rate are separate figures
basics
~20 sA pass rate hides its denominator. The 98% counts only instances the check actually enumerated; anything in an account, region or project the tooling never reached sits outside both numbers, so unseen systems are invisible rather than failing.
solid answer
~40 sThat number is a ratio over whatever set the check could see, and that set is rarely the whole estate. If the collector was configured with twelve of nineteen cloud accounts, instances in the other seven are not part of the 2% that failed — they are not in the denominator at all, so a system nobody enumerated reads exactly like a system that passed. I would ask three things: how the denominator was built, which enumeration it came from, and how many systems were reached but could not be evaluated. An honest report carries at least three buckets — pass, fail and unassessed — plus an explicit not-applicable set with a recorded reason. Coverage of the estate and pass rate within that coverage are two different numbers and belong side by side.
code
json · 10 lines{
"control": "managed database instances encrypted at rest",
"accounts_in_org_directory": 19,
"accounts_collected": 12,
"instances_evaluated": 214,
"pass": 210,
"fail": 4,
"unassessed": 0,
"not_applicable": 0
}go deeper
Be ready to say that a compliance percentage is a fraction and to name what is in its denominator. The point to land is that a system nobody checked looks identical to a system that passed.
Explain how the evaluated set actually gets built and the mundane ways it shrinks — an uncollected account, a region never queried, a missing credential. Distinguish fail, unassessed and not-applicable and say who owns each.
Show that you read these reports adversarially: ask how the population was derived, whether it came from a source independent of the checking tool, and what the unassessed bucket contains before you discuss the failures.
Own the reporting standard itself. Decide that coverage and pass rate are always published as a pair, that unassessed is never folded into either, and defend that when leadership asks for a single green number.
A compliance percentage is a fraction, and nearly all of the interesting argument lives in the bottom half of it. ## What 98% is a ratio of When a check reports that 98% of database instances are encrypted at rest, it is reporting `passing / evaluated`. `evaluated` is not the estate; it is the set of resources the collector managed to enumerate on that run. That set is produced by a configuration somebody wrote: a list of accounts, subscriptions or projects, a list of regions, a credential per account, and a set of API calls. Every one of those is a place where part of the estate can drop out silently. The crucial property is that a missing system does not appear as a failure. It appears as nothing. If seven of nineteen accounts were never collected, the instances inside them are absent from the numerator *and* the denominator, and the ratio is unchanged. This is why an under-covered estate tends to report a *better* number than a well-covered one: the systems least likely to be enumerated — a new account, a team's experimental project, a region opened for one customer — are exactly the systems least likely to be configured correctly. ## Unreached is not the same as failed, and neither is not-applicable Three different states get flattened into a binary and they mean different things: - **Fail** — the check ran against the resource and the resource did not satisfy the control. This is a remediation task with a technical owner. - **Unassessed** — the resource is in scope, but the check could not evaluate it. Collection failed, the credential was missing, the API errored, the region was not queried. This is an *access and ownership* task, not a remediation task, and it is a finding in its own right. - **Not applicable** — the control genuinely does not apply. A managed database that stores no regulated data may still be in scope for an encryption control; a queue service is not a database instance at all. Not-applicable must be a recorded decision with a reason, not a default for anything awkward. Folding unassessed into pass is the single most common distortion, because it is what happens by accident: the report counts what it saw, and what it did not see contributes nothing. Folding it into fail is also wrong — it asserts something about systems you have no information on and it pollutes the remediation queue. ## Two numbers, not one The fix is to stop reporting one figure. A control's status over an estate needs a pair: - **Coverage** — of the population the control applies to, what fraction was actually assessed on this run. - **Pass rate within coverage** — of what was assessed, what fraction satisfied the control. Read together, 92% pass over 60% coverage is a very different situation from 92% pass over 99% coverage, even though a single blended headline could make them look similar. They also drive different work: coverage is closed by getting enumeration and credentials into places they are not, and pass rate is closed by changing configuration. ## How the denominator quietly goes wrong Typical causes, all mundane: a new account created outside the platform's provisioning path so the collector's role was never deployed into it; a region enabled for one workload and never added to the query list; a resource type that the collector recognises under one API but not another; an inventory record that was written once at build time and never updated; a tag-based scope filter where untagged resources silently fall outside scope. None of these produce an error anyone reads — they produce a slightly smaller denominator. ## What to ask when someone hands you the number Where did the population come from, and was it built independently of the tool that reported on it? How many accounts or projects did the run attempt, and how many did it complete? What is in the unassessed bucket and who owns it? What is marked not-applicable and who approved that? If those questions have no answers, the percentage describes the tool's configuration, not the organisation's posture.
- Where should a system the collector could not evaluate appear in the report?In its own unassessed bucket, counted and owned. It is not a pass, because nothing was verified, and it is not a fail, because no violation was observed. It also needs a reason code — credential missing, API error, region not queried — since that reason is what tells you who fixes it.
- How is not-applicable different from unassessed?Not-applicable is a decision: someone determined the control cannot apply to this resource and recorded why. Unassessed is an absence of information: the resource may well be in scope and nobody knows its state. Not-applicable legitimately leaves the denominator; unassessed must stay visible inside it.
- Coverage went from 99% to 74% after a discovery exercise. Did posture get worse?No — visibility got better. The estate was always that size; the report was previously describing a smaller slice of it. Expect the pass rate to move too, since newly discovered systems tend to be the least well configured. The right headline for that quarter is the growth of the denominator, not the dip in the ratio.
A survey that reaches 200 of 1,000 households and reports 98% satisfaction is describing 200 households, not the town — and the 800 it never reached are not the unhappy 2%.
saying these in an interview costs you the question
- Treats a high pass rate as proof the control holds
- Assumes the collector sees the whole estate
- Counts unassessed systems as passing
- Says missing systems would have shown up as failures
- Reports one percentage with no coverage figure
- Uses not-applicable as a bucket for anything hard to check