A bufio.Writer wraps an output file and the last records never appear on disk - why, and what fixes it?
answer
- the bytes are still in memory
- a bufio.Writer has no Close method
- closing the file does not reach the wrapper
- deferred calls run last-in-first-out
- Flush is also where the write error appears
basics
~20 sA bufio.Writer holds bytes in memory until its buffer fills, so whatever is still buffered when the program ends is simply lost. Call Flush before the file is closed, and check the error Flush returns.
solid answer
~40 s`bufio.NewWriter` gives you a 4 KiB in-memory buffer. Each `Write` or `WriteString` copies into that buffer and only touches the underlying file when the buffer fills or when you call `Flush`. A `bufio.Writer` has no `Close` method and the `*os.File` knows nothing about the wrapper, so closing the file does not push the tail out - those bytes are dropped when the process exits. The fix is an explicit `w.Flush()` whose error you actually return, with `defer f.Close()` registered first so it runs after the flush (deferred calls run last-in-first-out). Flush is also where a failed underlying write first surfaces: `defer w.Flush()` throws that error away, which is how a disk-full failure becomes a silently short file.
code
go · 12 linesf, err := os.Create(path)
if err != nil {
return err
}
defer f.Close() // registered first, so it runs last
w := bufio.NewWriter(f)
for _, rec := range records {
if _, err := w.WriteString(rec + "\n"); err != nil {
return err
}
}
return w.Flush() // the tail leaves here, and so does any write errorgo deeper
Be ready to say in one sentence what Flush does: it pushes bytes the writer is still holding into the file underneath. Know that closing the file does not do it for you and that the wrapper has no Close.
Explain the mechanics: a 4 KiB default buffer, writes that only reach the file when it fills, deferred calls running last-in-first-out, and Flush being the first place a failed underlying write shows up.
Show the shape you actually ship - an explicit Flush whose error is returned, a Close ordered after it, and a clear statement of what Flush guarantees versus what fsync guarantees. Expect to be asked about concurrent writers.
Treat the buffer size as a data-loss budget rather than a performance knob: how many records the team can afford to lose on a hard kill, whether the answer is a smaller buffer, a timed flush or an fsync, and what throughput each choice costs.
## What a bufio.Writer actually is `bufio.NewWriter(w)` returns a `*bufio.Writer` that owns a byte slice - 4096 bytes by default, or whatever you ask for with `bufio.NewWriterSize`. It exists to turn many small writes into few large ones. Writing a 60-byte log line straight to an `*os.File` is a `write` syscall per line; routed through a `bufio.Writer`, roughly seventy lines are copied into memory and leave as one syscall. That is the whole trade: **your data now lives in user-space memory that nothing else knows about**. ## Why the tail disappears Three facts combine into the bug: 1. `Write` and `WriteString` return `nil` as soon as the bytes are copied into the buffer. A successful return says nothing about the file. 2. `bufio.Writer` has **no `Close` method**. There is nothing for a `defer` to close. 3. The `*os.File` underneath has no idea it is being buffered. `f.Close()` closes a file descriptor; it cannot reach into a wrapper it has never heard of. So a program that writes 10,000 records and returns has flushed only the full buffers along the way. The final partial buffer - up to 4 KiB of records - is in a slice that goes out of scope, and the file ends mid-record. It looks exactly like a truncated file, which is why people go looking for a disk problem or a producer problem rather than at the missing `Flush`. ## The correct shape ```go f, err := os.Create(path) if err != nil { return err } defer f.Close() w := bufio.NewWriter(f) for _, rec := range records { if _, err := w.WriteString(rec + "\n"); err != nil { return err } } return w.Flush() ``` Two details matter. First, **order**: `defer f.Close()` is registered before the flush happens, and deferred calls run last-in-first-out, so the file is still open when `Flush` runs. If you write `defer w.Flush()` after `defer f.Close()`, the flush runs first - correct order, but the error is discarded. Second, **the error**: returning `w.Flush()` directly is the cheapest way to make a write failure visible. ## Where write errors live Because the underlying write is deferred, so is the error. If the disk fills while a full buffer is being pushed out, that error is recorded on the writer and returned by **every later** `Write`, `WriteString` and `Flush` call. The writer never retries and never silently drops the error, but it also never tells you at the point the offending record was produced. One check on the final `Flush` is enough to detect the failure; attributing it to a specific record is not possible without flushing more often. ## Flush is not durability `Flush` moves bytes from the `bufio.Writer` into the underlying writer's `Write` method. For an `*os.File` that means a `write` syscall, which hands the bytes to the kernel's page cache. If the machine loses power a second later, the data can still be gone. Durability is `f.Sync()` (an fsync), it is far more expensive than `Flush`, and its error deserves the same check. A candidate who says "Flush guarantees the data is on disk" has conflated two different layers. ## What to do instead of forgetting - Write the `Flush` at the same moment you write the `bufio.NewWriter` - never later. - Prefer `return w.Flush()` (or `if err := w.Flush(); err != nil {...}`) over `defer w.Flush()`. - If you must defer it, defer a closure that captures a named result: `defer func() { if err == nil { err = w.Flush() } }()`. - Treat the buffer size as the amount of output you are willing to lose if the process is killed. The same reasoning applies to any buffered sink in the standard library - a `csv.Writer` and a compressing writer each hold state that only their own `Flush`/`Close` releases. `bufio.Writer` is just the one you meet first.
- Does bufio.Writer.Flush guarantee the records are durable on disk?No. Flush only copies the buffered bytes into the underlying writer's Write method, which for an *os.File is a write syscall into the kernel's page cache. A power loss can still lose them. Durability needs f.Sync(), which is much more expensive and whose error also has to be checked.
- What happens to later writes after an underlying write fails part-way through a Flush?The bufio.Writer remembers the first error and returns it from every subsequent Write, WriteString and Flush. Nothing is retried and nothing is silently swallowed, but the failure is reported far from the record that triggered it, so a single check on the final Flush is what makes it visible at all.
- Is a bufio.Writer safe to share between goroutines?No. It has no internal locking, so concurrent Write or Flush calls race on the buffer and its index. Give each goroutine its own writer, or serialise every Write and Flush behind one mutex - including a flush triggered from a timer.
It is a shipping crate, not a conveyor belt: records pile up inside until the crate is full or you deliberately send it. Locking the warehouse door does not ship a half-full crate.
saying these in an interview costs you the question
- Says closing the underlying file flushes the bufio.Writer
- Believes the buffer is flushed automatically at program exit
- Uses defer w.Flush() and never notices the discarded error
- Claims Flush makes the data durable on disk
- Thinks a successful Write means the bytes reached the file