In-house detections yield ten times more cases per rule than your managed vendor pack - how do you act on that?
answer
- the comparison is confounded first
- in-house content is selected for relevance
- breadth punishes an average
- count what only that source caught
- the lever is the contract, not the rule
basics
~20 sTreat the gap as a confounded comparison before treating it as a vendor failure: in-house rules exist because someone already saw the problem here, and a pack written for every customer carries content for platforms you do not run.
solid answer
~50 sFirst I make the comparison legitimate: normalise per rule-day live, and drop rules whose log source or target assets this estate does not have. Then I change the unit for the pack: a broad pack's value is not average yield, it is the cases it **uniquely originated**, the ones no other rule in the portfolio would have raised. A pack that scores badly on mean yield and originated eleven cases nobody else would have caught is earning its money. What I do with the finding is constrained by the fact that I cannot edit the pack: I can scope it, I can put per-rule evidence into the service review and the renewal, and I can move my own authoring effort to where the pack's unique originations are zero. What I will not do is present the ten-to-one headline unnormalised, or mass-disable the tail to make the average look better.
go deeper
Know that detection content comes from several places - a vendor pack, community rules, your own team - and that comparing them fairly means asking whether each rule ever had the telemetry to fire on.
Be ready to list the confounders in a by-source comparison: differing rule ages, scope sizes, absent log sources, and the fact that in-house rules were written for problems already seen here.
Show that you would change the unit for broad content, measuring uniquely originated cases and severity rather than mean yield per rule, and that you can defend the normalisation when the vendor pushes back.
Own the decision this feeds: where in-house authoring headcount goes, what the managed agreement is contracted to deliver, and how the finding is framed upward so it survives contact with the provider.
## Why the headline is not yet a finding Comparing yield across content sources - a managed vendor pack, imported community rules, in-house authored content - is one of the few genuinely useful cuts of a detection portfolio, and it is also the one most likely to produce a confident wrong answer. Four confounders sit under a ten-to-one ratio: **Survivorship.** In-house rules exist because someone in this SOC observed a problem here and wrote content for it. They are selected for local relevance before the first firing. A vendor pack is written once for every customer, so it necessarily contains rules for platforms, services and versions this estate does not run. **Denominator inflation.** If 700 of 900 rules are the pack and 120 are in-house, mean yield per rule punishes the pack for breadth. Rules whose log source was never onboarded, or whose target platform is absent from the asset inventory, should be excluded from the denominator entirely - they were never given the chance to fire. **Age and scope.** Pack rules ship in bulk on the day of onboarding; in-house rules trickle in over years, scoped tightly to the systems that prompted them. Per rule-day live and per asset in scope are the minimum corrections. **Purpose.** Some pack content exists to be rare. A rule intended to fire once every three years, on a technique with no other coverage, has an atrocious mean yield and may be the most valuable line in the file. ## Change the unit for a broad pack Mean yield per rule is the wrong measurement for content whose job is breadth. The unit that answers the real question is **uniquely originated cases**: cases the pack opened that no other rule in the portfolio would have raised, in a window long enough to matter. A short list of alternatives, all more informative than the mean: - unique originations per source, and their escalation rate - the share of confirmed incidents in which the source contributed any signal at all - coverage of behaviours where the source is the *only* content watching - cost per confirmed case, which is where a high-volume, low-yield source shows its real price That reframing changes the conversation from 'your pack is worse than ours' to 'here are the eleven cases only your pack raised, and here are the four hundred rules of yours that have never had telemetry to run on.' ## Acting on it when you cannot edit the content The constraint that makes this a leadership problem rather than an engineering one is that a managed pack is somebody else's artefact. The levers that remain are organisational: 1. **Scope, since you cannot rewrite.** Disabling pack rules whose required telemetry does not exist in this estate is a denominator correction, not a coverage loss - but it must be recorded as a deliberate, reviewed decision with the list attached, because next year's onboarding of that log source should re-enable them. 2. **Make yield a term of the relationship.** Per-rule evidence - firings, cases, unique originations, telemetry availability - belongs in the quarterly service review and, if the numbers hold, in the renewal. A provider who receives disposition feedback can improve their content; one who receives only a complaint cannot. 3. **Aim your own authoring at the holes.** The most defensible use of in-house headcount is behaviours where the pack has originated nothing, not rewriting rules the pack already covers adequately. That is what the by-source cut is actually for. 4. **Ask what the pack is contracted to deliver.** Many managed agreements promise content *coverage* and are silent on outcomes. If yield matters to you, it has to appear in the agreement in a measurable form, and this analysis is the evidence that makes that negotiable. ## The two ways to get this wrong The first is presenting the unnormalised ten-to-one figure upward. It is a satisfying slide, it will be believed, and it will not survive the vendor's first rebuttal - at which point every subsequent number your team publishes is discounted. The second is quietly disabling the pack's silent tail so the average improves. That is optimising the metric rather than the estate, and the blindness it creates is invisible precisely because silent rules were producing nothing to miss. Any removal has to be argued on what you accept not seeing, separately from the measurement that surfaced it.
- Which single metric would you take to the vendor's quarterly service review?Uniquely originated cases per quarter, with the telemetry-availability breakdown beside it. It answers what their content caught that nothing else would have, and it pre-empts the fair objection that much of their pack never had data to run on. Mean yield per rule is easy to compute and easy to dismiss; unique originations are hard to argue with in either direction.
- The pack's mean yield is poor but it originated the two most serious cases of the year. What follows?That mean yield was measuring the wrong thing. Detection value is not evenly distributed across firings - a source that is quiet for eleven months and catches the intrusion that mattered has earned its place, and averaging it against a rule that closes forty benign cases a week is an arithmetic mistake dressed as analysis. I would report severity-weighted originations alongside the mean.
- Is disabling pack rules whose log source you never onboarded a coverage loss?Not in the present tense - a rule with no data was contributing nothing. It becomes a loss the day that source is onboarded and nobody re-enables the rules that were waiting for it. So the action is only safe if it is recorded as a reviewable decision with the rule list attached to the telemetry backlog, rather than performed as a cleanup that disappears into the console's history.
saying these in an interview costs you the question
- Presents the raw ten-to-one ratio as a vendor failure
- Compares sources without excluding rules whose telemetry is absent
- Judges a broad content pack on mean yield per rule
- Disables the silent tail to improve the average
- Ignores that in-house rules exist because someone already saw the problem