A sign-off check on a deeply nested component reaches 300 ms at p99; what in a relation-tuple check causes that, and how do you bound it?

seniorshould knowfreq 30%

answer

it walks, it does not look up
levels add, hops multiply
round trips, not CPU
a denial cannot short-circuit
declare a maximum traversal depth

basics

~20 s

A check expands rules against stored tuples: each level adds sequential lookups and each hop multiplies, so the cost is round trips. Bound it with a declared maximum depth, flatter chains, and wide relations reached in one hop.

solid answer

~50 s

A check is not a row lookup. It expands the schema's rules against stored tuples, and each level is another set of reads: a tuple-following rewrite must read the tupleset, then ask the same question again at every object it reaches. The cost is therefore fan-out multiplied across depth, paid in round trips rather than CPU, which is why it shows up as latency rather than load. Two schema shapes cause it — a long chain (component to airframe to fleet to organisation to parent organisation) and a wide relation somewhere in the middle of that chain. Bound it by declaring an explicit maximum traversal depth so a runaway expansion fails deterministically, by flattening chains whose intermediate objects carry no independent meaning, and by arranging the schema so the wide set is reached in one hop. Then own the consequence: the store is on every read path, so its p99 is yours.

A check is not a row lookup. It expands the schema's rules against stored tuples, and each level is another set of reads: a tuple-following rewrite must read the tupleset, then ask the same question again at every object it reaches. The cost is therefore fan-out multiplied across depth, paid in round trips rather than CPU, which is why it shows up as latency rather than load. Two schema shapes cause it — a **long chain** (component to airframe to fleet to organisation to parent organisation) and a **wide relation** somewhere in the middle of that chain. Bound it by declaring an explicit maximum traversal depth so a runaway expansion fails deterministically, by flattening chains whose intermediate objects carry no independent meaning, and by arranging the schema so the wide set is reached in one hop. Then own the consequence: the store is on every read path, so its p99 is yours.

code

pseudocode · 10 lines

pseudocode

check(component:hyd-pump-77, signoff, user:15):
  rule: component.signoff = follow(airframe) -> airframe.inspector
  L1  read component:hyd-pump-77#airframe@*      -> 1 object   (airframe:G-ABCD)
  rule: airframe.inspector = direct | follow(holder) -> org.member
  L2  read airframe:G-ABCD#inspector@user:15     -> miss
  L3  read airframe:G-ABCD#holder@*              -> 1 object   (org:northfield)
  L4  read org:northfield#member@user:15         -> hit, return true

// four sequential levels; cost is round trips, not CPU.
// widen L3 to 40 holders and L4 runs up to 40 times before it can answer false.

go deeper

for a junior

Understand that a check walks a path rather than reading a row, so a rule that is one sentence in the schema can be several lookups when it runs.

for a middle

Be able to trace one check level by level for a concrete rule and point at which step multiplies. Knowing that the cost is round trips is the part that changes how you would fix it.

for a senior

Bring a diagnosis: measure depth and fan-out per relation rather than average latency, know why denials are slower than allows, and say what your endpoint does when the expansion does not finish.

for a principal

The decision you own is that authorization now depends on a network call in nearly every request, so its tail is your tail and its outage is a refusal. Declare the depth limit and the failure policy before an incident chooses them for you.

What a check actually does

When your endpoint asks may engineer 15 sign off hydraulic pump 77, the store does not look up a row. It expands the schema's rules for component.signoff, discovers a rewrite that follows the component's airframe tuple, reads that tupleset, and then asks the equivalent question about inspector on the airframe it reached. That relation is itself a union, one arm of which follows holder to an organisation and asks about member there.

Each of those steps is a read against stored tuples, and the steps are sequential because each one's result decides where the next one goes.

Where the cost comes from

The shape of the cost is fan-out across depth, measured in round trips:

every rewrite in the path adds a level;
every tuple-following hop multiplies, because the same sub-question is asked at each object in the tupleset;
every level is at least one lookup, so latency accumulates additively down the deepest path and the total work grows with the product of the widths.

This is why the symptom is tail latency rather than saturated CPU. A p50 that looks fine and a p99 of 300 ms is the signature of a small number of objects whose path is deep, wide, or both — a component on an airframe held by an organisation with thousands of members and a parent organisation above it.

The two shapes that hurt

shape	what it looks like	why it hurts
long chain	component to airframe to fleet to organisation to parent organisation	every link is a sequential level, and latency is the sum
wide relation	one organisation with several thousand members sitting mid-path	the sub-question is asked across a large tupleset before anything can short-circuit
both together	a deep chain whose middle link is wide	the product, and the case that produces your tail

Note the short-circuit works in your favour when the granting arm is found early, which is why the same relation can be fast for one subject and slow for another. Averages hide this completely.

Bounding it

Declare a maximum traversal depth. A bounded expansion fails deterministically instead of running until a timeout. Exceeding it is a schema defect, not a user error, so it should alarm rather than appear as an ordinary denial.
Flatten chains with meaningless intermediates. If an object exists in the path only to connect two others, and nothing in the product ever grants or revokes at that level, removing it removes a level from every check that passes through it.
Keep the wide set one hop from the check. A relation with thousands of subjects is tolerable when it is reached directly and short-circuits early; it is expensive when it sits three levels down and every path crosses it.
Measure per relation, not per endpoint. Record fan-out and depth for each relation your schema declares. The aggregate latency of check is not actionable; the signoff relation on components of airframes held by large organisations is four levels deep and crosses a 4,000-member set is.

What happens when the limit or the timeout is hit

This is where the decision point and the enforcement point must be kept apart. The store reports that it could not complete the expansion; it has not decided anything. Your endpoint decides what that means, and for an authorization question that means deny. But a depth-exceeded error and a genuine denial must not be logged as the same event, because one is a user being correctly refused and the other is your schema telling you it has grown a path nobody designed.

The availability consequence you now own

The deeper point is that moving authorization into a relation-tuple store puts a network dependency on the read path of nearly every request. Its tail latency becomes yours, and its unavailability looks to your users like total refusal rather than like a slow page, because the safe response to an unanswered authorization question is to refuse. That is a defensible design, but it is a decision with a bill, and it should be made explicitly rather than discovered during the first outage.

The practical consequence is that the relations on your hottest paths deserve the same scrutiny as your hottest queries: know their depth, know their widest link, and know what your endpoint does when the answer does not arrive.

## What a check actually does When your endpoint asks *may engineer 15 sign off hydraulic pump 77*, the store does not look up a row. It expands the schema's rules for `component.signoff`, discovers a rewrite that follows the component's `airframe` tuple, reads that tupleset, and then asks the equivalent question about `inspector` on the airframe it reached. That relation is itself a union, one arm of which follows `holder` to an organisation and asks about `member` there. Each of those steps is a read against stored tuples, and the steps are sequential because each one's result decides where the next one goes. ## Where the cost comes from The shape of the cost is **fan-out across depth, measured in round trips**: - every rewrite in the path adds a **level**; - every tuple-following hop **multiplies**, because the same sub-question is asked at each object in the tupleset; - every level is at least one lookup, so latency accumulates additively down the deepest path and the total work grows with the product of the widths. This is why the symptom is tail latency rather than saturated CPU. A p50 that looks fine and a p99 of 300 ms is the signature of a small number of objects whose path is deep, wide, or both — a component on an airframe held by an organisation with thousands of members and a parent organisation above it. ## The two shapes that hurt | shape | what it looks like | why it hurts | |---|---|---| | long chain | component to airframe to fleet to organisation to parent organisation | every link is a sequential level, and latency is the sum | | wide relation | one organisation with several thousand members sitting mid-path | the sub-question is asked across a large tupleset before anything can short-circuit | | both together | a deep chain whose middle link is wide | the product, and the case that produces your tail | Note the short-circuit works in your favour when the granting arm is found early, which is why the same relation can be fast for one subject and slow for another. Averages hide this completely. ## Bounding it 1. **Declare a maximum traversal depth.** A bounded expansion fails deterministically instead of running until a timeout. Exceeding it is a schema defect, not a user error, so it should alarm rather than appear as an ordinary denial. 2. **Flatten chains with meaningless intermediates.** If an object exists in the path only to connect two others, and nothing in the product ever grants or revokes at that level, removing it removes a level from every check that passes through it. 3. **Keep the wide set one hop from the check.** A relation with thousands of subjects is tolerable when it is reached directly and short-circuits early; it is expensive when it sits three levels down and every path crosses it. 4. **Measure per relation, not per endpoint.** Record fan-out and depth for each relation your schema declares. The aggregate latency of `check` is not actionable; *the `signoff` relation on components of airframes held by large organisations is four levels deep and crosses a 4,000-member set* is. ## What happens when the limit or the timeout is hit This is where the decision point and the enforcement point must be kept apart. The store reports that it could not complete the expansion; it has not decided anything. Your endpoint decides what that means, and for an authorization question that means deny. But a depth-exceeded error and a genuine denial must not be logged as the same event, because one is a user being correctly refused and the other is your schema telling you it has grown a path nobody designed. ## The availability consequence you now own The deeper point is that moving authorization into a relation-tuple store puts a network dependency on the read path of nearly every request. Its tail latency becomes yours, and its unavailability looks to your users like total refusal rather than like a slow page, because the safe response to an unanswered authorization question is to refuse. That is a defensible design, but it is a decision with a bill, and it should be made explicitly rather than discovered during the first outage. The practical consequence is that the relations on your hottest paths deserve the same scrutiny as your hottest queries: know their depth, know their widest link, and know what your endpoint does when the answer does not arrive.

Why is a denial often slower than an allow in a relation-tuple check?
Because an allow can stop at the first granting arm, while a denial has to exhaust every arm of every union and every object in each tupleset before it can say no. Any latency budget set from allow traffic will be wrong for the requests that are refused.
What should happen when a check exceeds the declared traversal depth?
The endpoint denies, and the event is recorded separately from ordinary denials and alarmed on. A depth-exceeded result means the schema has grown a path nobody designed — often a cycle or a new intermediate object — and it will not show up anywhere else.
Is a deep chain always worse than a wide relation?
No. Depth adds sequential latency and is felt on every request through that path; width adds work that a lucky short-circuit may skip entirely. A shallow chain across one very wide set is often faster in practice than a five-level chain across small ones.

saying these in an interview costs you the question

Describes check cost as CPU rather than sequential lookups
Reports average check latency and calls the relation healthy
Treats a depth-exceeded error as an ordinary access denial
Assumes a denial costs the same as an allow
Adds intermediate objects to the chain purely for tidiness
Ignores that the store is now on nearly every read path

A sign-off check on a deeply nested component reaches 300 ms at p99; what in a relation-tuple check causes that, and how do you bound it?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

What a check actually does

Where the cost comes from

The two shapes that hurt

Bounding it

What happens when the limit or the timeout is hit

The availability consequence you now own

▾common follow-ups3 questions

▾red flags6 flags

What a check actually does

Where the cost comes from

The two shapes that hurt

Bounding it

What happens when the limit or the timeout is hit

The availability consequence you now own

answer

go deeper