skip to content

How can a GraphQL server's parsed-document cache exhaust memory, and how do you bound it safely?

level: seniorimportance: should knowfreq 41%

answer

  1. Who decides how many keys exist
  2. The caller supplies the key material
  3. Trees retain more than their source text
  4. Cap entries, and check size before parsing
  5. Hit ratio falling is the early signal

basics

~20 s

Its keys come from caller-supplied document text, so anyone sending unique text grows it without limit until the process runs out of heap. Bound it with a capped evicting cache, a size check before parsing, and hashed keys.

solid answer

~50 s

The danger is that the **keyspace is controlled by callers**: a server caching parsed documents under the request text will happily store an entry for every distinct string it is sent. A client concatenating values per request, or an attacker sending padded junk, turns that into unbounded growth, and a parsed syntax tree retains several times the bytes of its source. The fixes are layered. Cap the cache by entries or bytes with an eviction policy, so the worst case is thrashing rather than an out-of-memory kill. Reject oversized documents by length **before** parsing, since the parse itself allocates. Key on a fixed-width hash so a large document does not also retain a full copy of itself as its key. Scope entries to a schema generation. Then alert on the hit ratio, because a healthy ratio near 100% falling is the signal that something is minting fresh text.

code

pseudocode · 16 lines
pseudocode
MAX_DOCUMENT_BYTES = 51_200
MAX_CACHE_ENTRIES   = 5_000

documentCache = evictingCache(maxEntries = MAX_CACHE_ENTRIES)

function prepare(documentText, schema):
    if byteLength(documentText) > MAX_DOCUMENT_BYTES:
        return requestError("document too large")   # before parsing, not after

    key = (hash(documentText), schema.generation)
    entry = documentCache.get(key)
    if entry == null:
        entry = parseAndValidate(documentText, schema)  # cache failures too
        documentCache.put(key, entry)                   # eviction is bounded here
    metrics.record("document_cache.hit", entry.wasCached)
    return entry

go deeper

for a junior

Know that a server keeps parsed documents in memory and that the entries come from text callers send, so the cache needs a size limit like any other in-memory store.

for a middle

Explain why the keyspace is caller-controlled, and name the concrete bounds: a capped evicting cache, a document size check before parsing, and hashed keys instead of full text.

for a senior

An interviewer expects the incident narrative and the diagnosis: hit ratio as the early alarm, entry-count gauges to prove the cap is real, and the post-collection heap floor to distinguish retention from load.

for a principal

Own the choice between defending an open text endpoint forever and collapsing the keyspace by accepting only known documents. Argue it in terms of client-team coordination cost, not just server memory.

## Why this cache is different from most caches Most caches you add have keys you generate: a record id, a tenant plus a date, a computed tuple. The keyspace is bounded by your own data. A parsed-document cache is not like that. Its key is derived from **text the caller sent**. Anyone who can reach the endpoint decides how many distinct keys exist, and how large each stored value is. That single property is what makes an unbounded implementation a memory-exhaustion vector rather than a tuning oversight, and it is the answer an interviewer is listening for. There are two populations that fill it. The accidental one is a client assembling document text per request, which mints one single-use entry per value and does so at exactly the rate of legitimate traffic. The deliberate one is simpler: send syntactically valid documents padded with a unique comment or an unused fragment, so each is a fresh key, and let the server store them all. Neither needs authentication if parsing happens before authorisation, which it usually does - the server cannot know who you are asking for until it knows what you asked. ## A worked incident A real-estate listings graph runs three instances behind a load balancer at a peak of 1,200 requests per minute. A new agent-facing page ships that builds its document text by concatenating the listing id and the viewer's saved-search id. The document cache is the implementation default: unbounded. Each entry is a syntax tree for a fairly large document plus its validation verdict; source text is about 3.4 KiB and the retained tree is several times that. Over eleven hours the cache reaches roughly 214,000 entries on the busiest instance, heap sits high, garbage-collection pauses stretch, and the instance is eventually killed and restarted by the platform. The alert that fires is not a memory alert. It is a support ticket: a subscription that silently stopped. Long-lived subscription connections terminated with the process, and clients that did not reconnect simply stopped receiving updates - no error, no empty result, just a stream that ended. Queries kept working because they retried. That mismatch - a memory fault surfacing as missing realtime updates - is a good thing to be able to narrate, because it is exactly how this failure presents. ## Bounding it **Cap the cache.** Use a fixed maximum number of entries, or better a byte budget, with an eviction policy. The point is not to make the cache good; it is to make the worst case *thrashing* - a low hit ratio and wasted CPU - instead of a process death. Pick the cap from the number of distinct documents your clients legitimately have, with headroom, not from a round number. **Cap the document size before parsing.** Check the length of the request body or the document string and reject anything beyond a stated limit with a request error. This has to happen before parsing, because parsing is itself the allocation you are trying to avoid; a size check after parse protects nothing. **Key on a hash.** Storing the full text as the key means a large document is retained twice. A fixed-width digest bounds key memory and makes cache accounting predictable. **Decide about invalid documents deliberately.** Caching only successful parses means a repeated junk document is re-parsed every time - CPU burn without memory growth. Caching failures too bounds the CPU but lets junk occupy entries. Either is defensible; caching failures in the same capped cache is usually the better trade, because the cap is what actually protects you. **Scope entries to the schema.** Include a schema generation in the key or clear on reload, or a document validated against a removed field keeps a stale clean verdict. ## The structural fix All of the above is damage limitation on a design where arbitrary text reaches the parser. The structural answer is that the endpoint accepts only documents it already knows - clients send an identifier for a registered document and anything unrecognised is rejected before parsing. That collapses the keyspace to the set your build produced, which is bounded by construction. It also imposes real coordination on client teams, which is why it is a decision to argue for rather than a default to assume; the mechanics of those schemes belong to their own topics. ## What to watch - **Cache hit ratio.** Near 100% is healthy. A sustained fall is the earliest signal and usually means a client started building text per request. - **Entry count and retained bytes**, exported as gauges. If the count only ever rises, the cache is unbounded in practice whatever the config claims. - **Distinct document hashes per named operation.** Any operation whose distinct-text count grows with traffic is being concatenated somewhere. - **Post-collection heap floor**, not peak heap. A floor that climbs across collections is retention, and a cache that only grows is the first place to look.

  • Should the cache store documents that failed to parse or validate?
    It is a trade, and either answer is defensible if you name it. Caching only successes means a repeated bad document is re-parsed every time, which burns CPU but adds no memory. Caching failures too stops that work but lets junk occupy entries. Since the entry cap is what actually protects the process, caching failures inside the same capped, evicting cache is usually the better choice - the cap bounds both populations at once.
  • Why must the document size limit be enforced before parsing rather than after?
    Because parsing is the allocation you are defending against. A megabyte of deeply nested selections becomes a far larger syntax tree during the parse itself, so a check that runs on the parsed result has already paid the cost it was meant to prevent. Measure the raw body or document string and reject it as a request error first. The same reasoning is why depth and cost limits, which run on the parsed tree, are not a substitute for a size check.
  • Heap keeps climbing but the document cache is capped. How do you tell whether this cache is the cause?
    Export entry count and retained bytes as gauges and check whether they are actually plateauing at the cap - a cap that is configured but not applied on the path in use is common. Then look at the post-collection heap floor rather than peak heap: a floor rising across collections is retention. If the cache gauges are flat while the floor climbs, the cache is exonerated and the retention is elsewhere.

saying these in an interview costs you the question

  • Treats the keyspace as bounded by the schema
  • Enforces a document size limit after parsing
  • Relies on a depth limit to bound cache memory
  • Assumes only authenticated callers can fill the cache
  • Stores full document text as the cache key
  • Watches peak heap instead of the post-collection floor

context

resolving links…