skip to content

Why must a gzip.Writer be closed before the file is valid, and what breaks if its Close error is dropped?

level: juniorimportance: must knowfreq 48%

answer

  1. the last bytes are still in memory
  2. eight bytes at the very end
  3. a checksum and a length
  4. only one call writes the trailer
  5. a dropped Close error is a lost write

basics

~20 s

A gzip.Writer buffers compressed data and writes the gzip trailer, a CRC32 plus the uncompressed length, only on Close. Skip Close and the file is truncated; ignore Close's error and a failed final write vanishes silently.

solid answer

~50 s

`gzip.NewWriter` returns a writer that compresses into an internal deflate buffer, so bytes you hand to `Write` are not on disk yet. `Close` emits the last deflate block and then the 8-byte gzip trailer: the CRC32 of the uncompressed data and its length mod 2^32. Without that trailer the member is unterminated, and any reader gets `io.ErrUnexpectedEOF` instead of a clean end. `Close` also returns an error, and it is the one that reports a failed final write, so `defer zw.Close()` on its own throws away the only signal that the archive is bad. It does not close the underlying `io.Writer`: you close the `gzip.Writer` first, then the file, and if there is a `bufio.Writer` in between you flush that too. To surface the error, use a named result and assign from the deferred close.

code

go · 17 lines
go
func writeGz(path string, src io.Reader) (err error) {
	f, err := os.Create(path)
	if err != nil {
		return err
	}
	defer func() {
		if cerr := f.Close(); err == nil {
			err = cerr
		}
	}()
	zw := gzip.NewWriter(f)
	if _, err = io.Copy(zw, src); err != nil {
		return err
	}
	// Close flushes the last block and writes the CRC32 + size trailer.
	return zw.Close()
}

go deeper

for a junior

Be ready to state that compressed output is buffered and that Close is what finishes the file, and to write the two closes in the right order: compressor first, then the file.

for a middle

Explain the member layout: header, deflate blocks, then an 8-byte trailer with a CRC32 and the uncompressed size, and say why that trailer can only be written at the end.

for a senior

Show how you surface a deferred Close error through a named result, and name the symptom on the reading side: io.ErrUnexpectedEOF for truncation, gzip.ErrChecksum for corruption.

for a principal

Own the rule that any code writing archives must report close errors, and that a pipeline which silently produces truncated artefacts is a review-gate problem, not an individual mistake.

## What a gzip member looks like on the wire A gzip stream (RFC 1952 calls one a *member*) has three parts: a 10-byte header with the magic bytes, a compression-method byte and optional fields such as `ModTime` and `Name`; a body of DEFLATE-compressed blocks; and an 8-byte **trailer** holding the CRC32 of the *uncompressed* bytes and ISIZE, the uncompressed length modulo 2^32. That layout explains the whole question. The CRC and the length cannot be known until the producer has seen every byte, so they can only be written at the very end — and in Go the only thing that writes them is `(*gzip.Writer).Close`. ## What Write, Flush and Close each do `gzip.NewWriter(w)` wraps `w` and starts a DEFLATE compressor over it. Each call to `Write` feeds bytes into that compressor, which accumulates them into blocks; a block is only emitted once the compressor has enough input to encode it well. So after your last `Write` returns, some of your data is still living in the compressor's internal window, not in `w`. `Flush` pushes the pending compressed bytes out and ends the current block so a reader can decompress everything written so far, but it does **not** finish the member — no trailer, and you may keep writing. `Close` is the terminator: it flushes, closes the DEFLATE stream, and appends the trailer. After `Close` the member is complete and the writer must not be written to again (though `Reset` makes it reusable for a new destination). ## What a truncated member does to the reader If you never call `Close`, the file ends mid-stream. `gzip.NewReader` will happily open it, because the header is intact, and decompression will return real data for a while — then `Read` fails with `io.ErrUnexpectedEOF`. If the trailer is present but the data was corrupted, the reader fails with `gzip.ErrChecksum` instead. Note the reading-side counterpart: the checksum is only verified once the stream is consumed to `io.EOF`, so a reader that stops early never learns the file was damaged. This is exactly the failure that pages someone at 3 a.m. for a log-shipping agent: the rotated `.gz` files it uploaded look plausible, most of the lines are there, and the last chunk of every file is missing. ## Why the Close error matters `Close` performs I/O, so it can fail — a full disk, a broken pipe, a closed socket. That failure lands on `Close` and nowhere else, because the earlier `Write` calls succeeded into a buffer. `defer zw.Close()` discards it, and the function returns `nil` while the archive is broken. The idiom is a named result plus a deferred closure: ```go func writeGz(path string, src io.Reader) (err error) { f, err := os.Create(path) if err != nil { return err } defer func() { if cerr := f.Close(); err == nil { err = cerr } }() zw := gzip.NewWriter(f) if _, err = io.Copy(zw, src); err != nil { return err } return zw.Close() } ``` Because `err` is a named result, `return zw.Close()` assigns it before the deferred function runs, so the deferred close only overwrites a `nil` error and never masks a real one. ## Ordering and layering `(*gzip.Writer).Close` does not close the `io.Writer` underneath it — the doc is explicit about that, and it is why you own two closes. The order is bottom-up in construction and top-down in shutdown: close the `gzip.Writer`, then the file. Reversed, the trailer is written to an already-closed file and you get an error you then have to handle anyway. A `bufio.Writer` between the compressor and the file adds a third step: `zw.Close()`, then `bw.Flush()`, then `f.Close()`. `bufio.Writer` has no `Close`, so nothing will flush it for you, and forgetting it truncates the file just as surely as skipping the gzip trailer. ## What to say in an interview The short version: compression is buffered, the trailer carries a checksum that can only be computed at the end, `Close` is the only thing that writes it, and `Close` returns the error for the final write. Everything else follows.

  • Does closing a gzip.Writer also close the file underneath it?
    No. `(*gzip.Writer).Close` finishes the gzip member and returns; the underlying `io.Writer` is untouched, so you close it yourself. The order is `gzip.Writer` first, then the file, because the trailer still has to be written. If a `bufio.Writer` sits in between, flush it after closing the compressor and before closing the file.
  • What does a reader see when the trailer is missing entirely?
    `gzip.NewReader` succeeds, since the header is intact, and `Read` returns real decompressed data until the stream runs out mid-block — then it returns `io.ErrUnexpectedEOF`. A trailer that is present but does not match the data gives `gzip.ErrChecksum` instead, and that check only runs if the caller reads all the way to `io.EOF`.
  • Is it safe to call Close twice on a gzip.Writer?
    Yes. The writer records that it is already closed, so a second call writes no second trailer and simply returns — nil, or the error stored from an earlier failure. That means `defer zw.Close()` alongside an explicit `zw.Close()` will not corrupt the file, but it is still poor style: the deferred call discards the result that mattered, so keep one authoritative close whose error you return.

Writing gzip is like sealing a parcel: the label with the weight and the checksum can only go on once everything is inside, and nobody checks whether the tape actually stuck unless you look.

saying these in an interview costs you the question

  • Says defer zw.Close() is enough, without checking its error
  • Thinks closing a gzip.Writer also closes the underlying file
  • Believes each Write lands on disk immediately
  • Assumes a missing trailer is harmless because most bytes are there
  • Closes the file before the gzip.Writer