A bash script fails only on the CI machine and you cannot reproduce it locally. What does `set -x` add to the script's output, where does that output go, and how do you enable it for one section only — or for a single run without editing the file at all?
answer
- shows what ran, not what you wrote
- expanded values, before execution
- goes to stderr, not stdout
- set +x turns it back off
- bash -x needs no file edit
basics
~20 sset -x makes bash print every command to stderr just before running it, after expansion, prefixed by PS4 (default "+ "). Wrap a region in set -x and set +x to scope it, or run bash -x script.sh to trace one run without touching the file.
solid answer
~50 s`set -x` turns on xtrace: before bash runs each simple command it writes that command to **stderr**, prefixed by the value of `PS4` (default `+ `). The key point is that the trace is printed **after expansion** — you see `+ rm -rf /var/tmp/build 42` with the variables, globs and command substitutions already resolved, which is exactly the gap between what the script says and what it does. Because it goes to stderr, `./script.sh > out.log` does not capture it and `2>/dev/null` silently throws it away. To scope it, put `set -x` before the suspicious region and `set +x` after it — the setting is shell-global, not block- or function-scoped, so if a helper turns it off it stays off for the caller too. To trace a run without editing anything, invoke the script as `bash -x ./script.sh`. `set -v` is the sibling option: it echoes each input line as bash *reads* it, before expansion.
go deeper
Know that set -x prints each command before bash runs it, that the output is on stderr, and that set +x turns it off again. Be able to say you would run bash -x ./script.sh to trace a failing run.
Explain that the trace is emitted after expansion, so it reveals word splitting, glob results and substituted values, and that the PS4 prefix's repeated first character marks nesting. Show that you know options are shell-global, not block-scoped.
Talk about operating it: gating tracing behind a DEBUG environment variable, redirecting stderr so the trace does not corrupt machine-parsed stdout, the log volume a traced loop produces, and the fact that expanded commands print credentials in clear.
Frame it as a policy question — what every script your organisation ships should support: a documented switch to turn tracing on for one run, a rule that traces never land in a shared log unredacted, and where structured logging from the script beats a raw shell trace.
## The problem xtrace solves A shell script is a program whose text and its runtime meaning can differ enormously, because every line goes through quote removal, parameter expansion, command substitution, word splitting and pathname expansion before a command is executed. Reading the source tells you what you *wrote*; `set -x` tells you what bash actually *ran*. ## What it prints, and when With `set -x` (equivalently `set -o xtrace`) in effect, bash writes each simple command, `for`/`case`/`select` word list and arithmetic `for` expression to standard error immediately before executing it. The line is preceded by the expanded value of the shell variable `PS4`, whose default value is `+ `. The critical property is *when* it prints: after expansion, before execution. Given ```bash file="my report.txt" set -x rm $file ``` the trace reads `+ rm my report.txt`, and you can immediately see that `rm` received two arguments because `$file` was unquoted. Nothing about the source line hints at that; the trace does. The first character of `PS4` is repeated to show nesting: a command running one level deeper — inside a command substitution or a subshell — is prefixed `++` rather than `+`. That is how you tell a command in `$( )` apart from one at the top level of the script. ## Where the output goes xtrace is written to **stderr**, not stdout. Three consequences bite people constantly: - `./deploy.sh > deploy.log` captures the script's normal output and leaves the trace on the terminal. - `./deploy.sh 2>/dev/null` throws the whole trace away. - If the script's own diagnostics also go to stderr, the two interleave, which is what makes the trace hard to read in a CI log. To capture everything in order, redirect both: `bash -x ./deploy.sh > run.log 2>&1`. ## Turning it on and off `set -x` enables it, `set +x` disables it — the `+` form of a `set` flag always turns the option off. Shell options are properties of the shell, not of a block, so enabling xtrace inside a function leaves it enabled after the function returns. If you want a helper to leave the caller's setting untouched, save and restore it yourself. The special parameter `$-` holds the currently enabled option flags: ```bash noisy_step() { local had_x=$- set -x tar -czf "$out" "$dir" case $had_x in *x*) ;; *) set +x ;; esac } ``` ## Tracing without editing the file Often you cannot or should not edit the script — it is baked into a container image, or owned by another team. Two ways in: - Invoke the interpreter explicitly: `bash -x ./script.sh args...`. This runs the whole script traced. - Export `SHELLOPTS` with `xtrace` in it. `SHELLOPTS` is a read-only, colon-separated list of enabled `set` options; if it is present in the environment when bash starts, those options are enabled before anything else runs, so `export SHELLOPTS=xtrace` propagates tracing into child bash scripts too. That is a blunt instrument — it traces *everything*, including scripts you did not mean to trace. ## `set -v`, the other half `set -v` (verbose) echoes each line of input as the shell reads it — the raw source, before any expansion. Used alone it is rarely what you want; used together (`bash -xv`) you see each source line followed by the expanded commands it produced, which is useful when a line expands into something unrecognisable. ## Costs and cautions xtrace is not free and not neutral: - **Volume.** A loop over a thousand files produces thousands of trace lines, which can dominate a CI log or fill a disk. - **Secrets.** Because commands are printed *expanded*, a token in a variable is printed in clear. A script that traces `curl -H "Authorization: Bearer $TOKEN"` puts the token in the log. - **Ordering.** Trace goes to stderr while the program's output usually goes to stdout; when they are redirected to different places, or buffered differently, the apparent ordering can mislead. So the normal pattern is not "always on": gate it behind a flag — `[[ ${DEBUG:-0} == 1 ]] && set -x` — so an operator can turn on tracing for one run without a code change, and the default run stays quiet.
- Why does redirecting a traced script's stdout to a file leave the trace on your terminal?Because xtrace is written to standard error, not standard output. `> run.log` only redirects fd 1, so the trace keeps going to fd 2. Capture both with `> run.log 2>&1`, or route the trace to its own descriptor so the two streams never interleave.
- A helper function calls `set +x` at the end. Why does the rest of the script stop tracing too?Shell options are global to the shell, not scoped to a function or block, so `set +x` inside a function disables xtrace for everything that runs after it returns. A well-behaved helper saves the incoming state — `local had_x=$-` — and only turns tracing off again if it was off when it was entered.
- How would you trace a script that is baked into a container image and that you cannot modify?Override the entrypoint to run the interpreter explicitly: `bash -x /app/entrypoint.sh`. If the script is invoked indirectly by something else, `export SHELLOPTS=xtrace` in the container environment enables xtrace in every bash that starts, because bash reads that variable at startup. The second option is blunt and traces unrelated scripts too.
saying these in an interview costs you the question
- Thinks set -x prints source lines exactly as written
- Expects the trace on stdout, so > log captures it
- Confuses set -x with set -e or set -v
- Believes set -x inside a function is scoped to it
- Leaves tracing on permanently, printing tokens into logs