Getting Data In and Out

A reader over a file with no declared types guesses each column's type from the rows it inspects, then holds every value in that column to that one representation. Different rows next month can produce a different guess.

One piece is a batch of whole records handed back per turn of the loop, not an arbitrary byte range: cuts must fall on record boundaries, and the line naming the columns belongs to the first turn only.

A self-describing binary columnar file carries its own type declaration, so the read applies it. Delimited text carries characters only, so something outside the file decides what each field means — on every read, and possibly differently the next time.

Compare rows in the table against a count you knew independently: lines in the file minus the line that names the columns, or the count the producer stated. Equal is reassurance; unequal is the alarm the read never raised.

The types are not guaranteed. Text stores characters only, so the reader must decide afresh what each field is. A file that carries its own type declarations restores them; flat text has nowhere to put them.

A reader that guesses decides per turn, and each turn sees only its own rows, so a late awkward value or a blank field flips the decision. Hand every turn the same type declaration up front.

Three dispositions: stop the read, drop the record — in silence or into a warning nobody reads — or hand it back damaged with the missing fields padded. Only the third keeps the row count, which is why it hides best.

Readers ship a built-in list of spellings — an empty field, a dash, a question mark, short abbreviations for unavailable — and map any field matching one to the absent-value marker. A real value spelled the same way is destroyed, silently.

Readers differ: some fix a type from a bounded prefix of the file, some scan the whole column first, some decide afresh for every batch of rows, and some refuse to guess at all. Which rule applies decides whether a late contradicting value is ever seen.

Balance a fixed cost paid once per turn, which is divided over the rows in the batch, against a peak that scales with the batch. Too small is slow, too large stops bounding anything; measure, do not quote a number.

On a declared columnar layout the unread columns are never read from disk at all. On delimited text every byte of every line still crosses the parser to find the separators; only converting and keeping the unwanted columns is avoided.

The values come back; the declaration does not. Text writes out the values the codes stood for, so the reader sees ordinary characters, and any list rebuilt afterwards holds only what this file happened to contain.

Each trip wrote the table's row labels out as an ordinary leading column, and each read brought them back as plain data. Two trips, two columns. Tables with no row-identity concept never show it.

A reader meeting a value that contradicts a column's type can reject the read, coerce the value into the type, or widen the whole column to text; some also drop the record. In an unattended nightly job you want rejection, because a warning is a non-signal.

Naming them at the read avoids converting and keeping 176 columns on every turn. Whether it also avoids reading their bytes depends on the shape: a columnar layout never fetches them, while delimited text still reads and splits every line.

At the combination step. Every piece is still referenced by the list while the joined result is allocated alongside it, so the peak is at least the finished table plus all the pieces.

Delimited text takes new records by concatenation, so appending is nearly free. A finalised typed file generally cannot be extended byte by byte, so each addition becomes another file — and every later read then pays a fixed cost per file.

Two different faults. A separator inside a field's own text pushes every later value one position right within that line. A line break inside a value makes one record out of two lines, so rows fall below the line count.

A shorter table, with nothing to say so. Delimited text carries no record count, so a file cut off mid-write reads like a shorter complete file; the last partial line either parses wrongly or meets the malformed-line disposition.

The file holds what the writer printed, not the value, so the reload re-parses a printed decimal. Whether that is lossless depends on how many digits the writer prints. Write, read straight back, and compare.

File size is the difference that decides least. Decide on who must be able to open the file without your code, how much of it a typical read touches, whether types must be guaranteed rather than re-established, and how the dataset is produced.