A C++ binary file reader memcpys an 18-byte on-disk header into its FileHeader struct and gets a wrong entry_count; why, and how should it read the header?
answer
- sizeof is 24, the file has 18
- padding moves entry_count to 8
- trivially copyable allows memcpy
- read each field at its file offset
- endianness is a second layout
basics
~20 sThe struct has padding the file lacks, so entry_count sits at offset 8 in memory but 6 on disk, and a whole-struct memcpy shifts the bytes. Copy each field from its file offset, fix byte order, and assert layout assumptions.
solid answer
~50 s`FileHeader { char magic[4]; uint16_t version; uint32_t entry_count; uint64_t index_offset; }` holds 18 bytes of data, but with 4- and 8-byte alignment the compiler pads it to 24, with `entry_count` at offset 8 and `index_offset` at 16. The file is packed: `entry_count` at 6. Copying the 18 file bytes over the struct put bytes 8 to 11 of the file into `entry_count`, so it read 4194304 instead of 3. `memcpy` itself is legal, because `FileHeader` is **trivially copyable**, and copying the underlying bytes of such an object preserves its value. The bug is assuming the struct's layout is the file's. The fix: read each field from its **file offset** with `memcpy` into a value, which has no alignment requirement on the source, and convert byte order with `std::endian` (C++20) and `std::byteswap` (C++23). `#pragma pack(1)` makes the struct match, but it is a compiler extension and leaves members misaligned.
code
cpp · 37 lines#include <bit>
#include <cstdint>
#include <cstdio>
#include <cstring>
#include <type_traits>
// On disk, little-endian, no padding: magic[4] version:u16 entry_count:u32 index_offset:u64
struct FileHeader {
char magic[4];
std::uint16_t version;
std::uint32_t entry_count;
std::uint64_t index_offset;
};
static_assert(std::is_trivially_copyable_v<FileHeader>);
template <class T> T read_le(const unsigned char* p) {
T v;
std::memcpy(&v, p, sizeof v); // no alignment requirement on p
if constexpr (std::endian::native == std::endian::big) v = std::byteswap(v);
return v;
}
FileHeader parse_header(const unsigned char* bytes) { // 18 bytes from the file
FileHeader h{};
std::memcpy(h.magic, bytes, 4);
h.version = read_le<std::uint16_t>(bytes + 4);
h.entry_count = read_le<std::uint32_t>(bytes + 6); // offset 6 on disk, 8 in the struct
h.index_offset = read_le<std::uint64_t>(bytes + 10);
return h;
}
int main() {
const unsigned char file[18] = {'K','A','T','A', 2,0, 3,0,0,0, 0x40,0,0,0,0,0,0,0};
std::printf("sizeof(FileHeader) = %zu, bytes on disk = %zu\n", sizeof(FileHeader), sizeof file);
FileHeader wrong;
std::memcpy(&wrong, file, sizeof file); // 18 bytes over a padded 24-byte layout
std::printf("memcpy whole: entry_count=%u\n", wrong.entry_count);
FileHeader h = parse_header(file);
std::printf("field by field: version=%u entry_count=%u index_offset=%llu\n",
h.version, h.entry_count, static_cast<unsigned long long>(h.index_offset));
}go deeper
Know that a struct can contain padding, so its size may differ from the bytes a file stores, and never copy a file record into a struct without checking.
Explain trivially copyable as the rule that allows memcpy, compute the padded offsets, and read fields from documented file offsets.
Design a portable reader: field-by-field memcpy, explicit endianness with std::endian and std::byteswap, validation, and static_assert on any layout the code relies on.
Set the team rule for binary formats, such as a schema with explicit offsets or generated readers instead of mapped structs, and weigh packing against portability and alignment faults.
## Two layouts that disagree A file format fixes its header byte by byte. Suppose it says: 4 bytes of magic, a 16-bit version, a 32-bit entry count and a 64-bit index offset, little-endian, no gaps. That is 18 bytes. The natural C++ struct declares the same fields in the same order, but the compiler lays it out by **alignment**, not by the file: | Field | Offset on disk | Offset in `FileHeader` (Apple clang 21, arm64) | |---|---|---| | `magic[4]` | 0 | 0 | | `version` (`uint16_t`) | 4 | 4 | | `entry_count` (`uint32_t`) | 6 | 8, after 2 bytes of padding | | `index_offset` (`uint64_t`) | 10 | 16, after 4 bytes of padding | | total | 18 | `sizeof` 24 | `std::memcpy(&header, file_bytes, 18)` copies byte 6 of the file to offset 6 of the struct, which is padding, and fills `entry_count` from file bytes 8 to 11, which belong to `index_offset`. In the code example that produced `entry_count = 4194304` instead of 3. Nothing crashed; the number was simply wrong. ## When copying bytes into an object is allowed The standard gives **trivially copyable** types a byte-level contract: the underlying bytes of such an object may be copied into an array of `char`, `unsigned char` or `std::byte`, for example with `std::memcpy`, and copying them back, or copying one object's bytes into another of the same type, gives the same value. A type is trivially copyable when it has at least one eligible copy or move operation, all of them trivial, and a trivial, non-deleted destructor. `FileHeader` qualifies, and `static_assert(std::is_trivially_copyable_v<FileHeader>);` keeps it that way. A `std::string` member, a virtual function or a user-provided copy constructor would break the contract, and then `memcpy` into the object is undefined behaviour. **Standard-layout** is a different property: it constrains how members and bases are arranged (same access control for all non-static data members, no virtual functions or virtual bases, no reference members, among other rules), and it is what makes `offsetof` guaranteed and what C interoperability relies on. Neither property says the layout matches **your file**; only the ABI decides that. ## Reading the header correctly 1. **Field by field.** Read each value from its documented file offset with `std::memcpy` into a local of the right type, as the code example's `read_le` does. `memcpy` has no alignment requirement on its source, so a `uint32_t` at file offset 6 is fine. 2. **Fix byte order.** Files and protocols fix an endianness. Compare `std::endian::native` (C++20) with the file's, and swap with `std::byteswap` (C++23) when they differ. 3. **Validate.** Check the magic, the version and that offsets and counts fit the file size before trusting them. 4. **Pin assumptions.** If code does rely on the struct's layout, assert it: `static_assert(sizeof(FileHeader) == 24);` and `static_assert(offsetof(FileHeader, entry_count) == 8);`. ## Why not just pack the struct? `#pragma pack(push, 1)` (documented by GCC, accepted by Clang) or a packed attribute removes the padding so that `sizeof` becomes 18 and a single `memcpy` works. The costs: - It is a **compiler extension**, not standard C++. - Members become **misaligned**: `entry_count` sits at offset 6. Accessing it by name through the struct works, because the compiler knows the member is packed and emits code that tolerates it, but a `uint32_t*` or `uint32_t&` taken to it describes a misaligned object; using it is undefined behaviour, and some architectures fault on such loads. GCC documents `-Waddress-of-packed-member`, enabled by default, for taking such an address; Apple clang 21 did not warn on `&h.entry_count` in this C++ test. - It does nothing about **endianness**. Packing is reasonable for a private, single-platform format with tests that assert offsets; the field-by-field reader is the portable default. ## Pointing a struct at a buffer `reinterpret_cast<const FileHeader*>(buffer.data())` has the same layout problem plus two more: the buffer offset may not meet `alignof(FileHeader)`, and whether a `FileHeader` object exists there is a question of object lifetime that C++23's `std::start_lifetime_as` was added to answer. Copying into a local avoids all three.
- Why is memcpy into a local safe when dereferencing a cast pointer into the buffer is not?`std::memcpy` reads bytes with no alignment requirement and writes them into a properly aligned object whose lifetime has already begun, and the trivially copyable rule says the copied bytes give that object a value. A `reinterpret_cast` pointer into a byte buffer may be misaligned for the type and may not point to an object of that type at all. Compilers usually turn a fixed-size `memcpy` into a single load, so the safe version costs nothing.
- What makes a type trivially copyable, and why does a std::string member break it?It needs at least one eligible copy or move constructor or assignment operator, every eligible one trivial, and a trivial, non-deleted destructor. A `std::string` member has a non-trivial copy constructor and destructor, so the enclosing class's become non-trivial too. Its bytes include a pointer to heap storage or inline characters that only its own copy constructor knows how to duplicate.
- A struct whose fields match the file's order has the file's layout.
- memcpy into any struct is fine as long as the byte count is right.
- #pragma pack is standard C++ and has no side effects.
- Trivially copyable means the struct has no padding.
- Reading a uint32_t through a misaligned pointer is always fine on modern CPUs.
Assumes C++23 for std::byteswap; std::endian is C++20, and the trivially copyable memcpy rule holds in every standard since C++11.