A bash script does `body=$(cat report.txt)` on a file that ends with a blank line, and a later `printf '%s' "$body" | wc -c` reports fewer bytes than the file. What does command substitution do to the captured output, and how would you capture it byte-for-byte?
answer
- something is removed from the tail
- not one of them — all of them
- leading and interior ones survive
- add a character the shell will not eat
- ${var%x} takes the sentinel back off
basics
~20 sCommand substitution strips every trailing newline from the captured output, not just the last one, so a file ending in blank lines loses them. To keep them, append a sentinel inside the substitution and remove it afterwards: body=$(cat report.txt; printf x); body=${body%x}.
solid answer
~50 s`$( )` removes **all** trailing newline characters from the command's standard output before substituting. That is deliberate and almost always what you want — it is why `dir=$(pwd)` gives you a clean path rather than a path with a newline glued on. It bites when the trailing whitespace is data: hashing a file's contents, reproducing a payload byte-for-byte, or preserving blank lines at the end of a document. The standard workaround is a sentinel: run `body=$(cat report.txt; printf x)` so the output no longer ends in newlines, then strip the sentinel with the parameter expansion `body=${body%x}`. Note that only *trailing* newlines are affected — leading and interior newlines survive intact — and that NUL bytes cannot be carried in a shell variable at all, so binary data needs a temp file or an encoding rather than a variable.
code
bash · 11 linesprintf 'line1\n\n\n' > report.txt
# default: every trailing newline is stripped
plain=$(cat report.txt)
printf '%s' "$plain" | wc -c # 5
# sentinel preserves them byte-for-byte
kept=$(cat report.txt; printf x)
kept=${kept%x}
printf '%s' "$kept" | wc -c # 8
wc -c < report.txt # 8go deeper
Know that $(cmd) gives you the output without the newline on the end, and that this is why cd "$(pwd)/x" works. Being able to state the stripping rule out loud is enough at this level.
Be ready to say all trailing newlines are removed, explain why that default is right, and write the sentinel-plus-${var%x} workaround from memory when the trailing whitespace is real data.
Show that you recognise the failure mode in review: checksums, signed payloads and file-rewriting scripts that silently lose a final blank line. Also flag the memory cost of slurping large output into one string instead of streaming it.
Own the boundary call — once a script is carrying byte-exact or binary payloads through shell variables, the shell has stopped being the right tool. Decide when to move that work to a real language rather than accumulating encode/decode workarounds.
## The rule When bash performs a command substitution, it runs the command, captures its standard output, and then **deletes all trailing newline characters** from that output before substituting the result. "All" is the part people get wrong: it is not one newline, it is every newline at the end of the stream. ```bash printf 'a\n\n\n' > f x=$(cat f) printf '%s' "$x" | wc -c # 1 — only the 'a' survives wc -c < f # 3 ``` Leading newlines and newlines in the middle are untouched. Only the tail is trimmed. ## Why bash does this Almost every command ends its output with a newline, because that is how a line is terminated on a POSIX system. If substitution preserved it, the overwhelmingly common cases would all be broken: ```bash dir=$(pwd) cd "$dir/subdir" # would be "/home/me\n/subdir" without stripping name=$(basename "$f") echo "file: $name." # the period would land on the next line ``` So the stripping is a convenience that makes the 99% case work. It is not a bug, and there is no shell option that turns it off. ## When it actually hurts The cases where it matters are the ones where trailing whitespace is *content* rather than *formatting*: - Reproducing a file's exact bytes — a checksum computed over `"$body"` will not match a checksum of the file. - Building an HTTP request or signature payload where the spec says the body ends with a newline. - Preserving a document's shape when a script rewrites it, where the final blank line silently disappears on every run. - Round-tripping heredoc content through a variable. ## The sentinel trick The portable fix is to make sure the captured stream does not *end* in a newline, then remove what you added: ```bash body=$(cat report.txt; printf x) # output now ends in 'x', nothing to strip body=${body%x} # remove exactly one trailing 'x' ``` `${body%x}` is a parameter expansion that deletes the shortest match of `x` from the end. Because the sentinel is the very last character and cannot itself be stripped by the substitution, every original newline survives. Some people write the sentinel as `echo x`, which works identically here; `printf x` is used above to be explicit that no extra newline is added — though in this construct an extra newline would simply be stripped anyway. ## Things this does not fix A shell variable **cannot hold a NUL byte** (`\0`). Command substitution discards NULs (recent bash versions warn about them). So the sentinel trick makes a variable faithful for text, not for binary. If you need binary fidelity, keep the data in a file, or encode it: ```bash blob=$(base64 -w0 < image.png) # safe to carry in a variable printf '%s' "$blob" | base64 -d > copy.png ``` (`-w0` is GNU coreutils; BSD and macOS `base64` do not accept `-w`.) ## The other half of the behaviour Stripping is only one of two things that happen to the result. The substituted text is an ordinary expansion result, so if you leave it unquoted it is word-split and glob-expanded — which is why `"$(cmd)"` is written with quotes. That splitting is a separate mechanism from the newline stripping: quoting stops the splitting but does **not** bring the trailing newlines back. Both happen, and an answer that only mentions quoting is incomplete. ## Related capture gotchas worth naming - Only **stdout** is captured; stderr flows on to the terminal. `out=$(cmd 2>&1)` captures both. - The command runs in a subshell, so variables it sets are gone afterwards. - If the command produces a lot of output, you are holding all of it in memory as a single string — streaming into a file or a `while read` loop is the better shape past a few megabytes. ## The short answer to give "`$( )` strips *all* trailing newlines, always, and there is no switch to turn it off. If the trailing newlines are data, append a sentinel character inside the substitution and strip it with `${var%x}` afterwards — and remember variables cannot hold NUL at all."
- Does quoting the substitution — `"$(cat f)"` — preserve the trailing newlines?No. Quoting and stripping are separate mechanisms. The newlines are removed while bash builds the substitution result, before quoting has any say. Quotes only stop the result from being word-split and glob-expanded afterwards. You still need the sentinel trick if the trailing newlines are data — and you should quote as well, for the splitting.
- Can you store arbitrary binary data in a variable using this technique?No. A bash variable cannot contain a NUL byte, and command substitution discards NULs from the captured stream. The sentinel trick fixes newline fidelity for text only. For binary, keep the bytes in a file, pass a file descriptor, or encode with something like base64 and decode on the way out.
- If I only need a file's contents in a variable, is `$(cat file)` the best way to do it?In bash there is a cheaper spelling: `$(<file)`, which lets bash read the file itself instead of running an external `cat`. It has exactly the same trailing-newline behaviour, so the sentinel trick still applies if you need the tail preserved. It is a bash and ksh extension, not POSIX, so a strict `/bin/sh` script keeps `cat`.
saying these in an interview costs you the question
- Thinks only the final newline is stripped
- Believes quoting the substitution preserves trailing newlines
- Claims a shell option can disable the stripping
- Says leading and interior newlines are stripped too
- Assumes a variable can hold arbitrary binary including NUL