An open-world game hitches whenever streamed meshes and textures are copied to the GPU; how would you redesign the upload path, its heap types and its residency handling?
answer
- no allocations on the render thread
- upload, device-local, readback
- a mapped ring and a copy queue
- fence values gate reuse
- stay under the OS budget
basics
~20 sSub-allocate from large pre-made heaps, stage data through a mapped upload ring, copy on a separate queue into device-local memory, and gate use and reuse with fences. Cap bytes per frame and evict least-recently-used data under the budget.
solid answer
~40 sHitches come from creating OS-level allocations mid-frame, uploading synchronously on the graphics queue, copying in bursts and crossing the memory budget so the OS pages resources. The fix: reserve large **device-local heaps** and **sub-allocate** resources inside them; write data into a persistently mapped **upload heap** used as a ring; record copies from the ring to device-local memory on a **copy queue** where one exists; have the graphics queue wait on the copy's signal before first use, and reuse ring space only after its fence value passes. Cap bytes per frame. A **residency manager** tracks last use per resource and evicts least-recently-used data no in-flight frame references, keeping the total under the budget the OS reports.
code
pseudocode · 22 linesUPLOAD_BUDGET_PER_FRAME = 8 MiB # the scenario's figure
function tryUpload(request):
ring.releaseRegionsUpTo(copyFence.completedValue)
size = alignUp(request.byteCount, request.alignment)
if bytesThisFrame + size > UPLOAD_BUDGET_PER_FRAME:
return DEFER # retry next frame
offset = ring.allocate(size) # NONE while the space is still being copied
if offset == NONE:
return DEFER
writeBytes(ring.mapped + offset, request.data)
recordCopy(copyCommands, ring.buffer, offset, request.destination)
ring.tagRegion(offset, size, nextCopyFenceValue)
bytesThisFrame = bytesThisFrame + size
return QUEUED
function endFrameUploads():
submit(copyQueue, copyCommands)
signal(copyQueue, copyFence, nextCopyFenceValue)
graphicsQueueWaitBeforeFirstUse(copyFence, nextCopyFenceValue)
nextCopyFenceValue = nextCopyFenceValue + 1
bytesThisFrame = 0go deeper
Recall that GPU memory comes in types the CPU can write, the GPU reads fastest, or the CPU reads back, and that uploads go through a staging copy.
Explain the staging ring, the copy into device-local memory and the fence that says when ring space and the uploaded resource are safe to use.
Diagnose streaming hitches from allocation calls, synchronous copies, bursts and over-budget paging, and design sub-allocation, a copy queue and LRU residency to remove each.
Set the GPU memory split per platform tier, including how much headroom residency keeps, and decide whether unified-memory platforms get a different upload path.
## Heap types GPU memory is offered as **heaps** with different CPU and GPU access: | Heap type | CPU access | GPU access | Used for | |---|---|---|---| | Device-local | usually none | fastest | textures, meshes, render targets | | Upload | write, usually write-combined | reads, often across the bus | staging, per-frame constants | | Readback | read, cached | writes, often across the bus | screenshots, GPU results for the CPU | On a discrete card, device-local memory sits on the card, while upload and readback memory are system memory the GPU reaches over the bus. On **unified-memory** devices (integrated GPUs, many consoles and phones) all of it is one physical pool and the types differ mainly in caching; some platforms also expose a CPU-visible slice of device-local memory. Write-combined memory is fast for the CPU to write sequentially and very slow for it to read. Why not have the CPU write straight into device-local memory? For buffers on platforms that expose a CPU-visible slice of it, writing directly can skip the copy, but that slice is often small and still slow for the CPU to read. For textures, the GPU's tiled layout makes a copy through the GPU the normal path, so a staging step remains. ## Why streaming hitches The usual causes: 1. **One OS-level allocation per resource**, made on the render thread. Large allocations are slow kernel calls, and many drivers limit or penalize the number of separate allocations. 2. **Synchronous uploads**: a copy is recorded on the graphics queue and the CPU waits for it, idling both sides. 3. **Bursts**: a whole district's data copied in one frame, starving that frame of bandwidth. 4. **Over-budget memory**: the game allocates past what the OS grants it, and on systems that virtualize video memory the OS pages resources between video and system memory, stalling whichever frame touches them. ## A non-blocking upload path - **Sub-allocate.** Reserve large heaps up front and place resources inside them with an allocator that respects each resource's alignment, so creating a texture becomes choosing an offset. - **Persistent staging ring.** Keep one mapped buffer in an upload heap and use it as a ring. The streaming thread writes decoded data into it, and a copy is recorded from the ring offset to the device-local destination. The copy also converts textures into the GPU's tiled layout. - **Copy queue.** Submit copies on a dedicated copy queue where the GPU has one, so they overlap rendering. - **Fences and cross-queue waits.** The copy queue signals a value when a batch completes. The graphics queue waits on it before the first frame that reads the resource, and the CPU reuses that ring region only once it sees the value. On some APIs a resource used on two queues must also be handed from one queue to the other explicitly. - **Per-frame byte budget.** Cap the bytes uploaded each frame so the cost stays level; requests over the cap wait a frame. ## Residency under a budget The OS reports how much video memory the game may use. It is usually smaller than the card's total, because the OS and other programs share the card, and it can change while the game runs. A **residency manager** keeps the resident total under it: - record the frame each streamed resource was last used; - when over budget, evict the least recently used resources that no frame in flight still references, checked against fence values; - make a resource resident again before recording the frame that uses it, never in the middle of that frame; - keep headroom so a burst of requests does not cross the line. Device-local memory for short-lived render targets can also be **aliased**: two targets used in disjoint parts of the frame share one region. ## Applying it to the open-world scenario With these pieces in place the hitch disappears: no allocation calls on the render thread, copies overlapping rendering on their own queue, a level upload cost per frame and no OS paging. Garbage or flickering textures, the other common symptom on this path, come from a missing fence wait: sampling before the copy queue's signal, or overwriting ring space a copy is still reading. Which assets to request, and in what priority, is a streaming-policy question for the asset pipeline; this path makes sure each requested byte reaches the GPU without stalling a frame.
- Why sub-allocate from large heaps instead of creating a GPU allocation per texture?Each separate allocation is a slow call into the OS and driver, and many drivers limit or penalize how many exist. Reserving a few large heaps and placing resources at aligned offsets turns creation into bookkeeping, and the heaps can be made resident or evicted as units.
- Why is reading back from an upload heap on the CPU a mistake?Upload heaps are usually write-combined: writes are buffered and streamed out, but reads bypass the cache and are very slow. Data the CPU must read belongs in a readback heap, which is cached for CPU reads.
- Every streamed texture should get its own allocation from the OS.
- Once the copy queue has the command, the graphics queue can read the data immediately.
- Residency can be ignored until the card's physical memory is full.
- The CPU can read an upload heap as cheaply as ordinary memory.
- Unified-memory devices have exactly the same heap trade-offs as discrete cards.