skip to content

What must hold before you convert &buf[off] of an mmap'd Go []byte into a *record?

level: middleimportance: should knowfreq 30%

answer

  1. three conditions, not one
  2. the address must be a multiple of something
  3. the struct must not run off the end
  4. padding is the compiler's choice
  5. no strings or slices in a mapped record

basics

~20 s

The address must satisfy the record type's alignment, the whole struct must fit inside the mapped region from that offset, and the struct's field layout with its padding must match the on-disk record exactly. Otherwise the conversion is undefined.

solid answer

~50 s

Three things. **Alignment**: the computed address has to satisfy the alignment the struct requires — a leading `uint64` field means the address must be eight-byte aligned, so both the region's base and the record stride have to cooperate. **Extent**: the struct must lie entirely within the mapping from that offset, since converting a pointer that runs off the end of the object it belongs to is undefined. **Layout**: the file format has to match how this compiler lays the struct out, padding included, so a struct mapped over bytes usually needs explicit blank padding fields and must contain no Go pointers, strings or slices. Because the language does not promise a particular struct layout, this is a bet on the toolchain, not a guarantee. Build the package's tests with `-race` so checkptr instrumentation validates the conversions, and keep `encoding/binary` decoding as the fallback when the win is not measurable.

code

go · 11 lines
go
type record struct {
	id    uint64
	score float32
	_     [4]byte // explicit tail padding: the on-disk record is 16 bytes
}

// r.buf is the mapped region; the 16-byte stride keeps every record
// 8-byte aligned, which is what the leading uint64 requires.
func (r *reader) at(i int) *record {
	return (*record)(unsafe.Pointer(&r.buf[i*16]))
}

go deeper

for a junior

Know that pointing a struct pointer at raw bytes is possible in Go and that it is a promise about alignment and layout you are making on the compiler's behalf.

for a middle

Be able to list the three conditions — alignment of the computed address, the struct fitting inside the region, and layout including padding matching the format — and say what goes wrong when each is violated.

for a senior

Demonstrate the operational half: a size-and-offset test guarding the layout, length validation before any pointer is handed out, and the package's tests run with -race so checkptr sees the conversions.

for a principal

Decide whether the technique is allowed in a library other teams import at all, and require the safe decoder plus a benchmark that shows the difference before the fast path is approved.

## The shape of the trick A library that reads a large search index maps the file into memory and wants to hand out records without decoding them. Each record is a fixed number of bytes, so record `i` starts at a known offset, and the tempting move is: ``` return (*record)(unsafe.Pointer(&buf[i*recSize])) ``` One pointer conversion replaces a decode and an allocation per record. It is a real technique, it is fast, and it is defined only when three separate conditions hold. ## Condition one: alignment Every Go type has an alignment requirement, and the compiler generates loads assuming it is met. A struct whose first field is a `uint64` requires an eight-byte-aligned address on the usual targets. `&buf[off]` is just the base of the backing array plus `off` — nothing in the type system checks that the sum is aligned. So two things must cooperate. The base address of the mapped region has to be sufficiently aligned; a mapping returned by the operating system is page-aligned, which is generous, but a plain heap `[]byte` you sliced from somewhere else is not guaranteed to be. And the stride has to preserve it: if records are 12 bytes apart, every other record starts on a 4-byte boundary and the conversion is misaligned half the time. On x86 a misaligned load usually just works, which is the worst possible outcome, because the same binary on another architecture faults or the compiler is free to assume the alignment held. When the instrumentation is on, this is reported as a misaligned pointer conversion. ## Condition two: the struct fits The pointer you produce has to describe memory that is entirely inside the object you started from. If a record straddles the end of the mapping, reading through the resulting `*record` reads past it — a fault if the next page is unmapped, and silent garbage if it is not. In practice that means validating the file length against the record count before you hand out any pointers, not per access. For a Go-heap `[]byte`, the checkptr instrumentation catches the case where a converted pointer straddles more than one allocation. For memory that came from a mapping the runtime did not allocate, it has less to work with, so this condition is on you. ## Condition three: layout equality This is the one that quietly ruins file formats. Go inserts padding between fields to satisfy each field's alignment and pads the struct's tail so that arrays of it stay aligned. The language specification does not promise a particular layout; the compiler chooses one. A struct that happens to match a 16-byte on-disk record today is an assumption about this toolchain on this architecture. The defensive form is to write the padding out explicitly — a blank `_ [4]byte` field where the compiler would have inserted four bytes anyway — so that the intent is visible and any change to the format shows up as a diff rather than as a silent shift of every field after it. Even then, keep a test that asserts the struct's size and the offsets of its fields against the format's spec constants, so a compiler that ever laid it out differently fails the build instead of returning wrong data. Two further restrictions belong here. The mapped record type must contain no Go pointers — no `string`, no slice, no `map`, no `*T` — because memory outside the Go heap is not a place the runtime will find or maintain references for you; store offsets instead and resolve them yourself. And the byte order baked into a direct struct cast is the machine's own, so a format that must be portable across architectures needs an explicit decoder rather than a cast. ## What you get and what it costs The payoff is real: no copy, no allocation, and the operating system's page cache doing the paging. The cost is that this package now depends on properties the language does not promise, and it forfeits the compatibility guarantee that ordinary Go code enjoys. The honest engineering answer is to make the safe version first — `encoding/binary` decoding into a struct, or field-by-field reads — benchmark the two with allocation counts, and only keep the conversion if the difference matters at the scale you actually run at. If you keep it, confine it to one small function like the accessor above, comment which documented `unsafe` pattern it relies on, and run that package's tests with `-race` so the checkptr instrumentation gets a chance to reject a misaligned or out-of-object conversion during CI instead of in production.

  • Why must a struct mapped over foreign memory contain no strings, slices or pointers?
    Because those fields would have to hold real Go addresses, and nothing in the mapped bytes is a valid one. Storing a Go pointer into memory the runtime did not allocate also gets you no help keeping the target alive. Use fixed-width integers and byte arrays, and resolve references yourself from offsets.
  • How would you prove the struct still matches the format after a toolchain upgrade?
    Keep a test that asserts the struct's total size and each field's offset against the constants from the format specification. It is cheap, it runs on every build, and it converts a silent misread into a failing test the moment the compiler lays the type out differently.
  • What does building the package with -race add here?
    It enables the compiler's checkptr instrumentation as well as race detection. Every unsafe.Pointer conversion in that package is then validated at runtime, so a misaligned conversion or one whose result lies outside a valid allocation aborts loudly instead of returning plausible-looking garbage.

saying these in an interview costs you the question

  • Assumes any byte offset can be cast to a struct pointer
  • Says unaligned loads are fine because x86 tolerates them
  • Trusts Go's struct layout to match a file format by luck
  • Puts a string or slice field in a mapped record
  • Never validates the file length against the record count