skip to content

Why does a Node.js log-tailing service lose its early lines when it calls readline.createInterface() on fs.createReadStream(), awaits a queue connection, and only then iterates with for await?

level: seniorimportance: should knowfreq 35%

answer

  1. who started the flow
  2. consumption begins at construction
  3. 'line' events with no listener
  4. connect first, then create

basics

~20 s

readline.createInterface() starts consuming its input immediately, switching the file stream to flowing and emitting 'line' events. The for await iterator only listens once the loop starts, so lines emitted during the await are lost. Create the interface after the await.

solid answer

~40 s

`readline.createInterface()` attaches a `'data'` listener to its input and calls `resume()` as soon as it is constructed, so the file stream switches to **flowing** mode and the interface starts emitting `'line'` events. Its async iterator subscribes to `'line'` only when `for await` asks for it, so every line parsed during the `await` goes to nobody. The readline docs warn about exactly this: asynchronous operations between creating the interface and iterating it may result in missed lines. On a small file it is worse: the interface can emit `'close'` before the loop starts, and the loop then never finishes. The fix is ordering: finish the async setup first, then create the interface and iterate in the same synchronous span; a raw stream left with no consumer would have waited.

code

javascript · 17 lines
javascript
import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';
import { setTimeout as sleep } from 'node:timers/promises';

// Broken: the interface starts reading as soon as it is created.
const early = createInterface({ input: createReadStream('./app.log'), crlfDelay: Infinity });
await sleep(50); // stands in for connecting to a queue
// Lines parsed during the sleep had no 'line' listener; on a small file
// 'close' has already fired and a for await over `early` would never finish.
early.close();

// Fixed: finish the async setup, then create and iterate with no await between.
await sleep(50);
const rl = createInterface({ input: createReadStream('./app.log'), crlfDelay: Infinity });
let count = 0;
for await (const line of rl) count++;
console.log(`counted ${count} lines`);

go deeper

for a junior

Remember that readline starts reading as soon as it is created, so do async setup before creating it.

for a middle

Explain that the interface switches its input to flowing mode and emits 'line' events that need a listener at the moment they fire.

for a senior

Diagnose silent data loss and hung loops from consumers attached after an await, and fix them by reordering rather than adding buffers.

for a principal

Encode the rule in shared helpers: functions that return ready-to-iterate sources so callers cannot insert an await between creation and consumption.

## The symptom A **log-tailing service** reads a log file line by line and forwards each line to a queue. The code looks reasonable: ```js const rl = createInterface({ input: createReadStream(path), crlfDelay: Infinity }); await queue.connect(); // a few milliseconds of async setup for await (const line of rl) { await queue.send(line); } ``` On large files the first part of every file is missing. On small files the loop sometimes processes **nothing** and never finishes. No error is logged. ## Root cause: someone else started the flow A Node `Readable` produces nothing until a consumer is attached: with `readableFlowing === null` it simply waits. The trouble is *who* counts as a consumer. 1. `readline.createInterface({ input })` attaches a `'data'` listener to `input` and calls `input.resume()` inside its constructor. The file stream is now in **flowing mode**. 2. As chunks arrive, the interface splits them into lines and emits a `'line'` event for each one. 3. The `for await` loop has not started yet, so nothing is listening for `'line'`. The interface's async iterator subscribes to `'line'` (and `'close'`) only when `rl[Symbol.asyncIterator]()` is first called, which is when the loop begins. 4. Every line emitted during the `await` is delivered to zero listeners and is gone. Flowing mode is push; there is no replay. The readline documentation says this directly: `readline.createInterface()` will start to consume the input stream once invoked, and asynchronous operations between interface creation and asynchronous iteration may result in missed lines. ## Why small files hang instead If the file is small enough to be read entirely during the `await`, the interface has already seen the end of the input, emitted its last `'line'` events **and** emitted `'close'`. The iterator created afterwards waits for a `'line'` or `'close'` that will never come. In a script, the process exits with an unsettled top-level await warning; in a server, the loop just sits there. ## Fixes - **Order the work**: complete all async setup first, then create the interface and start the loop with no `await` in between. - **Keep construction and iteration together**: a helper that takes a path and returns the async iterable, creating the interface inside, prevents a caller from inserting an `await`. - **Leave raw streams alone until you are ready**: a plain `fs.createReadStream()` with no listeners waits safely in `readableFlowing === null`; it is the readline wrapper (or any early `'data'` listener or `resume()`) that starts the clock. - **Do not mix styles**: a second `'data'` listener added for metrics is another way to start flowing before the real consumer is ready. ## Confirming the diagnosis 1. Compare counts: lines in the file against lines the loop processed. A deficit concentrated at the **start** of each file points at a late consumer, not at the queue. 2. Log `input.readableFlowing` right after `createInterface()`: it is already `true`, before your loop exists. 3. Move the `await` above `createInterface()` and re-run on the same file; if the counts now match, the ordering was the cause. 4. Try a tiny file: if the loop never finishes, the interface closed before the iterator was created. ## The general pattern | Situation | Data lost? | Why | |---|---|---| | Raw Readable, nothing attached, `await` before consuming | no | the stream stays at `readableFlowing === null` and produces nothing | | `resume()` called, no `'data'` listener | yes | flowing with no consumer drops chunks | | `'data'` listener for metrics, real consumer added after an `await` | yes, for the consumer | chunks before the attach went only to the first listener | | `readline.createInterface()`, then `await`, then `for await` | yes | the interface flows its input and emits unheard `'line'` events | ## Related production notes - `fs.createReadStream()` reads to the end of the file **as it is when the reader reaches it** and then ends; following a growing file needs a reopen from a saved byte offset (the `start` option) or a watch, which the file-system lesson covers. - The readline docs also note that the async iterator is slower than the `'line'` event API, so a very high-volume tailer may prefer `rl.on('line', ...)`, attached synchronously right after construction. - Chunk boundaries never split a line for you to handle: readline buffers partial lines across chunks and decodes UTF-8 across chunk boundaries.

  • Would the same await be safe if you consumed the raw fs stream with for await instead of readline?
    Yes, as long as nothing else touched the stream first. A Readable with no consumer stays at `readableFlowing === null` and produces nothing, so `for await` started later still receives every chunk. You then get `Buffer` chunks rather than lines, and must split lines and handle multi-byte characters across chunk boundaries yourself.
  • Why might a high-volume tailer use rl.on('line') rather than for await over the interface?
    The readline docs state that async iteration performance is not on par with the `'line'` event API. An event listener attached in the same synchronous block as `createInterface()` misses nothing; you then pause and resume the interface yourself if each line needs slow asynchronous work.

  • readline buffers every line until the for await loop starts.
  • The missing lines mean fs.createReadStream skipped bytes at the start.
  • Raising highWaterMark on the file stream stops lines being dropped.
  • A Readable always holds its data until a for await loop asks for it.
  • A metrics 'data' listener beside the real consumer cannot change what it receives.