skip to content

How do you keep a corpus of federated sign-in and provisioning fixtures honest over the years a service provider runs?

level: principalimportance: should knowfreq 20%

answer

  1. two kinds, two decay modes
  2. loudly wrong versus quietly wrong
  3. recorded proves the world, generated proves the rules
  4. re-capture on a stated cadence
  5. absorbed divergences get named fixtures

basics

~20 s

Hold both kinds deliberately. Recorded messages prove you can still parse what real counterparties actually send; generated ones stay fresh and produce the malformed cases nobody sends. Each catches a defect class the other cannot, and both need an owner and a refresh path.

solid answer

~50 s

A recorded message replayed byte for byte is the only evidence that your parser still handles what a real venue operator's identity provider or provisioning client emits — including the divergences you decided to absorb. But it pins every timestamp and every signature, so it stops being accepted as soon as a bound it carries passes, and it can only ever be a well-formed message. A generated fixture is minted by your own stand-in at the test's clock position, so it never expires and can express every malformation; but it is generated from your own model of the protocol, so it drifts towards what you believe rather than what arrives. Keep both, name the defect class each owns, re-capture the recorded set on a stated cadence, and treat divergences you absorb as named fixtures with a reason attached.

code

json · 6 lines
json
{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
  "Operations": [
    { "op": "replace", "path": "active", "value": false }
  ]
}

go deeper

for a junior

Recall that there are two ways to get a test message: capture a real one, or have your own stand-in produce one. They are useful for different reasons and neither replaces the other.

for a middle

Explain what each kind cannot do — a captured message can never be malformed in the way you need, and a generated one can only be as accurate as your model of the specification.

for a senior

Show the operational habits: immutable captures, parse-versus-verify split, redaction, a re-capture trigger, and a provisioning set covering the repeated write, the supported subset and the conflict answer.

for a principal

Make the standing call on how much fidelity is worth its maintenance cost, who owns the refresh, and the rule that a divergence you absorb is always a named fixture with a written reason rather than a quietly tolerant parser.

## Why this is a years-long problem and not a test-writing problem Once the rota tool has a dozen venue operators, its fixture corpus is a standing asset: it encodes what every counterparty sends, which divergences you decided to live with, and which malformed inputs you promise to refuse. Nobody writes it in one sitting and nobody maintains it by accident. Left alone it decays in two opposite directions at once, and the two kinds of fixture decay differently. ## Two kinds, two defect classes | | Recorded, replayed byte for byte | Generated by your own stand-in | |---|---|---| | Proves | your parser and verifier still handle what real counterparties emit | your rejection rules still fire, at any clock position | | Cannot express | anything nobody actually sends — a wrong key, a stripped signature, a wrong-scope signature | a divergence you have never modelled | | Ages by | carrying absolute bounds that pass and signatures you cannot re-mint | drifting towards your model of the specification | | Fails as | a red build on a calendar date | a green build that is wrong about the world | | Owner of the truth | the counterparty | you | The asymmetry is the point. A recorded corpus fails **loudly and wrongly**; a generated corpus fails **quietly and wrongly**. The first is annoying, the second is dangerous, and a team that has been annoyed once usually deletes the recorded set and keeps only the generated one — which is exactly backwards, because the generated set is the one that cannot tell you the world changed. ## Keeping the recorded half honest - **Capture with consent and scrub.** A real message names a real person and carries real attributes. Redact before it enters the repository, and keep the redaction mechanical so it is not forgotten under pressure. - **Store what the signature covered.** Once captured, a recorded message is immutable: any edit breaks its signature, and a test that then passes is passing for the wrong reason. - **Split parse from verify.** Because a recorded message will eventually fail its own validity bounds, run it through the parse-and-shape assertions permanently, and through the full trust decision only at a clock position pinned to its capture date. - **Re-capture on a cadence and on change.** A stated cadence, plus a capture whenever a counterparty announces a change or an onboarding surfaces a new shape. ## Keeping the generated half honest - Mint every fixture from the stand-in at the test's clock position so nothing expires. - Derive it from the specification, not from the parser — a generator written against your own reader agrees with your own bugs. - Let each deliberate divergence be a named fixture with a written reason. "This venue's client omits a field the specification makes optional and we accept it" is a decision; a silently tolerant parser is a defect. ## The provisioning half is the same problem, one channel over Sign-in is not the only federated surface. A provisioning client writes to your `/Users` and `/Groups` endpoints, and it does so with its own habits: the subset of `filter` expressions it actually sends, the paging parameters it actually uses, the `externalId` it mints and expects back, and a `PATCH` document with `op` `replace` on `active` to deactivate a leaver. Three fixture-shaped commitments live here: 1. **Replay of the same write.** The identical operation arriving twice must land once and answer the same way — a retried deactivation is the normal case, not an anomaly. 2. **The subset you accept.** Clients send a fraction of the specification. Which fraction you support is a decision, and each supported shape is a fixture; an unsupported one should be refused with a clear `scimType` rather than silently ignored. 3. **The conflict.** A create that collides with an existing resource answers `409`, and that path needs a fixture too, because it is the one a retry storm produces. ## The judgment call The question a lead actually answers is how much fidelity is worth paying for. Every recorded fixture is a maintenance liability; every generated one is a statement of belief. The honest position is a small recorded set that covers the counterparties you have and the divergences you absorbed, a large generated set that carries the negative cases, a named owner for the refresh, and a rule that a divergence is never absorbed silently. The failure to avoid is a corpus that nobody re-captures, whose red builds are fixed by widening a tolerance, and whose green builds are a description of what the team believed three years ago.

  • Why is a corpus of only generated fixtures the more dangerous of the two failure modes?
    Because it fails green. A generated set is minted from your own model, so when a counterparty starts sending something you never modelled, nothing in the suite notices — the build stays clean and the breakage arrives as a support ticket from one venue. A recorded set that expires at least fails loudly, and a loud failure gets fixed.
  • A provisioning client sends the same deactivation twice. What does the fixture have to assert?
    That the second arrival is not a second event: the account ends in the same state, the answer is the same, and no duplicate side effect fires. A `PATCH` setting `active` to false is the shape a retry most often carries, so the replayed-write fixture belongs in the corpus permanently rather than being treated as an edge case.
  • How do you stop the recorded set turning into a graveyard nobody trusts?
    Give it a stated refresh cadence and a trigger — re-capture when a counterparty announces a change or an onboarding turns up a new shape — and run recorded messages through parse-and-shape assertions permanently while reserving the full trust decision for a clock position pinned to the capture. A set with no owner and no cadence is deleted within a year.

saying these in an interview costs you the question

  • Deletes the recorded set because it keeps expiring and keeps only generated ones.
  • Edits a captured message's fields to refresh it, breaking the signature it carried.
  • Generates fixtures from the parser, so the suite agrees with its own bugs.
  • Keeps real personal attributes in checked-in capture files.
  • Thinks a recorded fixture proves the rejection rules are still correct.
  • Treats a repeated provisioning write as an anomaly rather than the normal case.