skip to content

A just-in-time compiler specialises a call site that has only ever seen one receiver type - what happens when a second type arrives?

level: seniorimportance: should knowfreq 44%

answer

  1. optimise for what was observed
  2. an assumption needs a check
  3. every check needs an exit
  4. rebuild the frame, resume interpreted
  5. recompile against the widened profile

basics

~20 s

The specialised code carries a guard that tests the assumption on every execution. When the guard fails, the runtime deoptimises: it reconstructs interpreter state for the running frame, resumes there, and later recompiles the method against the widened profile.

solid answer

~50 s

Specialisation is a bet placed on observed behaviour, not a proof, so the compiled code must include a way to lose the bet safely. At a call site that has only ever seen one receiver type, the compiler emits a cheap check - compare the incoming type against the one it specialised for - followed by the specialised body. If the check fails, control leaves the compiled code through a **deoptimisation** exit: the runtime rebuilds the interpreter's view of the current activation, mapping live values back into locals and choosing a resume point, and execution continues interpreted from exactly that point. Nothing is recomputed and no result changes; only speed does. The call site is then recorded as having seen more than one type, and when the method is recompiled it is compiled with that wider knowledge instead of the assumption that just failed.

code

pseudocode · 10 lines
pseudocode
# compiled form of a call site that has only ever observed one receiver type
compiled_call_site(receiver, args, frame):
    if receiver.type != SPECULATED_TYPE:
        record_guard_failure(this_site)
        rebuild_interpreter_frame(frame, resume_point)
        continue_in_interpreter(frame)      # does NOT come back to compiled code

    # guard held: the target is known, so its body sits here as straight-line code
    result = <specialised body for SPECULATED_TYPE>
    return result

go deeper

for a junior

Recall that a compiler working inside a running program may optimise for what it has seen so far, and must therefore include a check so that behaviour stays correct if what it has seen changes.

for a middle

Explain the three parts: the assumption drawn from the profile, the cheap guard that tests it on every execution, and the exit that reconstructs a lower-tier frame and resumes there rather than restarting the method.

for a senior

Demonstrate that you can recognise repeated invalidation in production - unsettled throughput, ongoing compilation CPU, a tail unrelated to input - and that you would look for hot code whose observed behaviour genuinely varies.

for a principal

Treat profile stability as a design property of the system. Decide where polymorphic and hot-stable paths should be kept apart, and what the organisation gives up in flexibility to keep the hottest code predictable enough to speculate on.

## Speculation: optimising for what happened, not what is provable An optimising compiler inside a running program has an advantage a compiler working from source text alone does not: it can see what the program **actually did**. If a call site has been executed a hundred thousand times and every receiver had the same type, the compiler can proceed as though that will continue - resolving the target directly and opening up the transformations that follow from a known target. The crucial point is that this is an **assumption**, not a proof. Nothing in the program forbids a different type from arriving on the next call. So a speculative optimisation is only admissible if it comes with two things: a **guard** that detects the assumption failing, and an **exit** that recovers correct execution when it does. ## The guard The guard is a comparison and a branch placed before the specialised body. Two properties make it worth having: - **It is cheap.** One compare against a constant, with a branch that is almost never taken and is therefore predicted well. - **It is cheaper than what it replaces.** A general call site must look up the target each time and cannot see inside it; a guarded site knows the target statically, so the body can be treated as straight-line code alongside its caller and further optimised as one region. A guard that fails frequently is a bad trade twice over: the check costs something, and the exit costs a great deal. This is why runtimes track how often a guard has failed and stop speculating there once it has failed enough. ## What deoptimisation has to reconstruct When the guard fails, execution cannot simply continue - the specialised code below it is only valid under the assumption that just proved false. It also cannot restart the method, because side effects may already have happened. The runtime therefore performs a transfer that reconstructs the state a lower tier would have had at this point: - **The local values**, which may be spread across processor registers and stack slots chosen by the compiler, are written back into the slots an interpreter frame expects. - **The resume point** is identified: the exact position in the method, recorded by the compiler for each place a guard can fail, at which interpretation should pick up. - **Frames that were merged** by inlining are separated again. If the specialised region absorbed several calls, the single compiled frame must be expanded into the several frames the lower tier would have had. Execution then continues in the lower tier from that point, and the method finishes normally. Correctness is preserved exactly; what is lost is the speed of the compiled form for the remainder of this activation. ## After the exit | Step | What the runtime does | |---|---| | At the failing guard | transfers the running activation to a lower tier and records the failure | | Immediately after | updates the site's profile - it has now seen more than one type | | For the compiled code | marks it unusable for the invalidated assumption, so later calls do not walk back into the same trap | | Later, if still hot | recompiles with the widened profile: two targets tested in sequence, or an ordinary dispatch | A site that has seen many distinct types is usually abandoned as a speculation opportunity altogether; specialising it would mean a chain of checks that is worse than dispatching once. ## The failure mode to recognise The pathological pattern is a cycle: compile with an assumption, run briefly, invalidate it, deoptimise, recompile, invalidate again. Its signature in production is distinctive: - throughput that rises and falls repeatedly instead of settling; - sustained CPU in compilation work long after startup; - a latency distribution with a heavy tail that does not correspond to input size. The usual cause is a workload whose behaviour genuinely changes - an input mix that shifts, a code path that only appears once a feature flag is turned on, a warm-up that exercised a narrower set of types than production does. The fix is normally in the program rather than the runtime: keep genuinely polymorphic sites separate from hot monomorphic ones so that one site's variability does not poison the profile of the code around it. ## Why this design is worth its complexity Without speculation, a compiler must be conservative about everything the program text leaves open, and for a language with late binding that is almost everything interesting. Speculation converts *observed* regularity into *assumed* regularity, and the guard plus the deoptimisation exit is the mechanism that makes an assumption safe to act on. Ecosystems differ in how far they push this and in how much of the mechanism they expose, but the shape - bet, check, exit, relearn - is common to all of them.

  • Why is the guard cheaper than the dispatch it replaces, given that both inspect the receiver?
    The dispatch has to produce a target and then call through it, which blocks everything downstream from being reasoned about. The guard produces no value: it is one compare and a branch that is nearly always predicted correctly, and because the target is then known statically, the body can be treated as part of the surrounding region and optimised with it. The saving is mostly what the known target unlocks, not the comparison itself.
  • What does a deoptimisation storm look like from the outside, and where do you look first?
    Throughput that never settles, CPU spent on compilation long after startup, and tail latency unrelated to input size. Look first for hot code whose behaviour genuinely varies - a shared helper used from both a monomorphic hot path and a polymorphic one, or a path that only appears under certain inputs - and consider separating the variable site from the stable one.
  • Why can execution not simply restart the method from the top after a guard fails?
    Because the activation may already have had observable effects - values written, other calls made - and repeating them would change program behaviour. The transfer therefore reconstructs state at the precise point the guard failed, using the mapping the compiler recorded for that point, so execution continues rather than repeats.

It is a standing order placed on the assumption that the same supplier always delivers: cheap while it holds, but the paperwork must include a clause for the day a different lorry turns up, and that clause has to unwind the order without losing the goods already received.

saying these in an interview costs you the question

  • Thinks the compiler proves the type instead of assuming it
  • Believes a failed assumption produces a wrong result
  • Says deoptimisation aborts the request or raises an error
  • Assumes compiled code, once produced, is never abandoned
  • Thinks the guard costs about as much as the dispatch it removed
  • Expects a site with many observed types to keep being specialised