When parsing a 40 GB PostgreSQL log with pgBadger, how do the -j and -J options differ, and when does each fail to help?
answer
- chunks of one file versus whole files
- seeking needs an uncompressed file
- a small loss at chunk edges
- temporary binaries merged at the end
basics
~20 s-j N splits one log file into N byte ranges parsed by forked processes, which needs a seekable, uncompressed, non-CSV local file; -J N parses N whole files at once, which helps only when there are many files.
solid answer
~40 sWith `-j N`, pgBadger divides each file into N chunks by size, forks N processes that seek to their chunk, writes each one's statistics to a temporary binary file, then merges them. It cannot split a compressed file, `csvlog`, stdin or a remote or `journalctl` source, so there it falls back to one process, and at chunk edges it may truncate or miss up to N queries per file. With `-J N`, each process takes one whole file, so nothing is lost at boundaries, but a single 40 GB file gets one core. pgBadger's benchmark favours `-j` for a few large files and `-J` for hundreds of small files on 8 or more cores; the two can be combined.
go deeper
Remember the two letters: lowercase -j splits one file across cores, uppercase -J parses several files at once.
Explain chunking by byte offset, the temporary binaries merged at the end, and why compressed, csvlog and remote sources cannot be split.
Choose for the workload: rotate smaller so -J loses nothing, decompress when -j is needed, size the temp directory, parse off-host, and recognise a parallel flag that silently did nothing.
Fold parsing into the reporting pipeline design: rotation policy, central parsing capacity and incremental runs sized so nightly reports finish before the team reads them.
## Why parallelism matters here pgBadger is a Perl script, and parsing a full-detail PostgreSQL log is CPU-bound: every line is matched against the log format, statements are normalised and counted. A busy server logging every statement can produce tens of gigabytes a day, and a single process may need hours. pgBadger has **two** multiprocess modes, and choosing the wrong one gives little or no speed-up. ## -j: split one file into chunks `-j N` (`--jobs`) parallelises **within a file**: 1. pgBadger divides the file size by N to get the chunk size. 2. It finds the start and end offsets of each chunk. 3. It **forks** N processes; each **seeks** to its start offset and parses until its end offset. 4. Each child writes its statistics to a **temporary binary file** named like `tmp_pgbadgerXXXX.bin` in the temporary directory (`--tempdir`, default the system temp directory). 5. The parent waits for all children, loads the binaries and builds the report. The cost of cutting a file at arbitrary byte offsets is that at each chunk boundary a multi-line entry can be split. pgBadger's documentation is explicit: it may **truncate or omit up to N queries per log file**, which is insignificant among millions of queries, and most often the effect is a query counted twice with one copy truncated. Because step 3 depends on seeking, `-j` does **not** split: - **compressed files** (gzip, bzip2, xz, zip, lz4, zstd): you cannot seek into a compressed stream; - **`csvlog`**: the CSV parser always reads a file as one stream; - **stdin**, **remote files** (`ssh://`, `http(s)://`, `-r`) and **`journalctl`** or `--command` sources; - **Windows**, where pgBadger disables forking altogether. In those cases pgBadger silently parses the file in one process. ## -J: one whole file per process `-J N` (`--Jobs`) parallelises **across files**: up to N log files are parsed at once, each by one process from start to end. Nothing is cut, so **no queries are lost** at boundaries. The obvious limit: a single huge file is still parsed by one core. Because it does not seek, the per-file mode is the one that can still help with a directory of rotated, compressed logs. ## What pgBadger's own benchmark shows | Workload, 8 CPUs | `-j 8` | `-J 8` | |---|---|---| | one 9.5 GB file (1 CPU: 1h41m) | 15m58 | 34m45 | | 200 files of 10 MB (1 CPU: 20m15) | 4m20 | 2m40 | Its recommendation: use **`-j`** unless you have **hundreds of small files** and at least **8 CPUs**, where `-J` wins. The two modes can be **combined**. ## Applying it to a 40 GB log For one 40 GB `stderr` log on a 16-core reporting host: ```bash # keep the file uncompressed and local so it can be split pgbadger -j 12 --tempdir /srv/pgbadger-tmp \ -o /var/www/pg_reports/orders-2026-10-05.html \ /srv/logs/postgresql-2026-10-05.log ``` Practical points an interviewer listens for: - **Rotate smaller**: hourly or size-based rotation (for example `log_rotation_age` and `log_rotation_size`) turns one 40 GB file into many files, which `-J` then handles with no boundary loss. - **Decompress first** if the archive is compressed and you need `-j`; otherwise use `-J` across the compressed files. - **Temporary space**: the forked processes write binaries to the temp directory and pgBadger removes them at the end; point `--tempdir` at a disk with room, and do not delete those files while a run is active. - **Run off the database host**: parallel parsing consumes CPU and memory that the database should keep. - **Combine with incremental mode** so each night parses only one day's new lines instead of the whole history. - **`--no-fork`** (13.1) disables forking entirely; it exists for debugging, not for speed. ## Why the distinction matters The classic failure is to add `-j 16` to a cron job that reads `postgresql.log.1.gz` or a `csvlog` file and conclude that pgBadger "does not scale". The flag was accepted, but the source could not be split, so it ran on one core. Knowing which mode needs seeking, and which only needs many files, is the whole answer.
- Why does pgBadger accept an accuracy loss with -j, and when would you avoid it?Splitting at byte offsets can cut a multi-line entry, so up to N queries per file may be truncated or missed; among millions of statements that is noise. Avoid it when the report must account for every statement, such as an audit-style count: use -J across rotated files, or one process.
- Two pgBadger cron jobs on the same reporting host start overlapping after logs grow. What protects the output, and what do you change?pgBadger writes a PID file (default pgbadger.pid in /tmp) and a second run exits with code 3. Fix the cause: parallel parsing, smaller rotated files with -J, or incremental mode so each run parses only new lines.
- -j parallelises across files and -J within a file
- -j speeds up parsing a gzip-compressed log
- -j N guarantees exactly the same counts as a single process
- -J splits a single large file across cores
- Parallel parsing works the same way on csvlog files