skip to content

Checkpoints, Journal & Compression

A checkpoint writes a consistent snapshot on a timer and the journal holds what a restart must replay, while block and prefix compression trade CPU for disk. Expect to say what survives a crash.

part ofMongoDBread it →
on this pageshow

explore

questions

6

In MongoDB's WiredTiger engine, what do a checkpoint and the journal each do, and how do they work together for durability?

level: middleimportance: must knowfreq 45%

answer

  1. one is a recovery point
  2. the other covers the gap
  3. 60 seconds against 100 milliseconds
  4. old journal files can go after a checkpoint
  5. a replica set journals less

basics

~20 s

A checkpoint durably writes a consistent snapshot to the data files, by default about every 60 seconds; the journal is a write-ahead log synced about every 100 ms. Recovery starts at the last checkpoint and replays the journal.

solid answer

~50 s

A checkpoint is WiredTiger writing a snapshot's data consistently across all data files, which makes those files a valid recovery point. `storage.syncPeriodSecs` defaults to 60 seconds, and the server's option comment measures that from the end of the previous checkpoint. The old checkpoint stays valid until metadata is atomically updated to reference the new one. The journal is a write-ahead log under `dbPath/journal`: records are buffered in memory and synced every `storage.journal.commitIntervalMs`, 100 ms by default, or sooner on a `j: true` write. After a crash, MongoDB opens the last checkpoint and applies the journal from that point. Checkpoints bound replay time and let old journal files be removed; the journal narrows the loss window from a checkpoint interval to milliseconds. On a replica set member the journal covers the oplog rather than replicated collections, and replaying the oplog rebuilds the rest.

go deeper

for a junior

Recall the two roles: a checkpoint is a consistent point on disk, and the journal records what happened since. Know the defaults of 60 seconds and 100 milliseconds.

for a middle

Walk through a restart step by step, and explain why a crash in the middle of a checkpoint is safe: the previous checkpoint stays valid until metadata switches atomically.

for a senior

Show you know the journal's scope differs by deployment: a standalone journals everything, while a replica set member journals the oplog and relies on oplog replay for the rest.

for a principal

Frame both intervals as things to leave alone. They trade recovery time, disk and small loss windows, and the durability a team really cares about is expressed through write concern.

## Two mechanisms, one durability story WiredTiger keeps recently changed data in memory, because writing every change to its final place in the data files would be far too slow. That leaves two questions: what is on disk that is known to be consistent, and how are the changes since then protected? A **checkpoint** answers the first and the **journal** answers the second. ## What a checkpoint is A checkpoint is a durable, consistent view of the data. WiredTiger takes a snapshot and writes all the data in it to disk in a consistent way across all of the data files. Some details matter when you explain it: - **It is not a full copy.** Dirty pages are written; clean pages are skipped. - **It has a default interval.** `storage.syncPeriodSecs` defaults to 60 seconds. The comment beside the option in the 9.0 source calls it the delay before triggering the next checkpoint after the previous one completes, so the gap between starts is 60 seconds plus however long the checkpoint took. - **The switch is atomic.** While a new checkpoint is being written, the previous one remains valid. The new one takes over when WiredTiger's metadata is atomically updated to reference it, and only then are pages from older checkpoints freed. - **A crash mid-checkpoint is safe.** MongoDB recovers from the last complete checkpoint. Checkpoints alone would give a consistent database after a crash, but one that had lost everything since the last checkpoint: up to a minute or more of writes. ## What the journal holds The journal is WiredTiger's **write-ahead log**, stored in `WiredTigerLog.<sequence>` files in a `journal` directory under `storage.dbPath`. It is not the MongoDB log file that holds diagnostic messages. - A journal record describes one client-initiated write, together with the internal changes it caused, such as index updates. - Records go into an in-memory buffer first. - WiredTiger syncs that buffer to disk every `storage.journal.commitIntervalMs` (100 ms by default), when a write includes or implies `j: true`, and when it creates a new journal file, which happens about every 100 MB. - On a secondary it also syncs after each batch of oplog entries it applies. - Records are compressed with `snappy` by default. ## How they combine at restart On restart after an unclean shutdown, recovery runs before the server becomes available: 1. MongoDB looks in the data files for the identifier of the last checkpoint. 2. It finds the matching record in the journal files. 3. It applies the operations recorded in the journal since that point. The relationship also runs the other way. Journal files older than the most recently completed checkpoint are removed, so checkpoints limit both how much journal must be replayed and how much disk it occupies. WiredTiger's own documentation gives exactly these two reasons for checkpointing when a log is enabled. | | Checkpoint | Journal | |---|---|---| | Format | Consistent data files | Sequential log records | | Default frequency | About 60 seconds | About 100 ms | | Setting | `storage.syncPeriodSecs` | `storage.journal.commitIntervalMs` | | Job in recovery | The starting point | The replay since that point | ## Standalone against replica set member The docs describe one journal record per client write, and that picture fits a standalone. On a replica set member the source is narrower. `WiredTigerUtil::useTableLogging` turns table logging off for replicated collections and keeps it on for the `local` database, including the oplog (`local.replset.minvalid` is the exception). A replica set member therefore relies on this sequence: - The journal protects the oplog entries. - A stable checkpoint holds the collections up to a timestamp. - Replaying the oplog past that timestamp restores the rest, which belongs with replication rather than with storage. An answer that says the journal holds every collection write on a replica set is repeating a simplification. ## Telling them apart from similar things - The **oplog** is a replicated collection of operations for other members to apply. The journal is a local storage-engine log. - `fsync` asks for pending writes to be flushed to disk, and is not the periodic checkpoint schedule. - `j: true` asks the server to acknowledge a write only after it reaches the on-disk journal. What it promises an application is a write concern question; for storage, it simply triggers an immediate sync.

  • With WiredTiger on MongoDB, why keep checkpoints at all if the journal records every change?
    Without them, journal files could never be removed and recovery would have to replay everything. Log files are only removed once a checkpoint completes, and each checkpoint limits the records a restart must process. MongoDB's docs warn that setting `storage.syncPeriodSecs` to 0 lets the journal eventually consume all available disk space.
  • Why does the documented procedure for changing WiredTiger's journal compressor in MongoDB start from a clean shutdown?
    After an unclean shutdown, recovery has to read the existing journal files, and the manual says you must use the old compressor settings for that. So the procedure stops `mongod` cleanly, changes `journalCompressor` and restarts. If an unclean shutdown happens midway, start with the old setting, let it recover, then retry.
  • How do you see journal activity on a running mongod?
    `db.serverStatus().wiredTiger.log` returns statistics on the journal. The files themselves are `WiredTigerLog.<sequence>` under `dbPath/journal`, along with pre-allocated log files WiredTiger keeps ready for use.

A writer saves a clean copy of a long document once a minute and, between saves, notes each edit in a bound logbook several times a second. After a blackout she opens the last clean copy and reapplies the logbook. Only the edits she had not yet written down are gone.

  • An acknowledged write is on disk only once the next checkpoint has run.
  • The journal is the same thing as MongoDB's diagnostic log file.
  • storage.syncPeriodSecs controls how often the journal is synced.
  • A crash during a checkpoint corrupts the data files until you run repair.
  • The journal and the oplog are the same log under two names.
  • A checkpoint rewrites a complete copy of the database every minute.
open as a page

On a standalone MongoDB mongod that loses power 40 seconds after its last checkpoint, which w: 1 writes survive restart: one acknowledged 5 seconds earlier, or one acknowledged 20 ms earlier?

level: seniorimportance: must knowfreq 40%

basics

~20 s

The write from 5 seconds earlier survives, because a journal sync covered it and restart replays it. The write from 20 ms earlier may be lost: w: 1 acknowledges from memory, ahead of the next 100 ms sync.

open as a page

In a MongoDB deployment on WiredTiger, what is compressed by default, and with which compression method?

level: juniorimportance: should knowfreq 28%

basics

~10 s

Three things: collection data gets snappy block compression (time-series collections get zstd), indexes get prefix compression, and the journal gets snappy. Block compression shrinks data only on disk; prefix compression also saves memory.

open as a page

On MongoDB 9.0, can you turn off WiredTiger journaling to speed up writes, and what do storage.journal.commitIntervalMs and storage.syncPeriodSecs actually trade away?

level: middleimportance: should knowfreq 33%

basics

~20 s

No: journaling has been always on since MongoDB 6.1, and its off switches were removed. commitIntervalMs trades a wider loss window against fewer syncs; syncPeriodSecs trades more journal to keep and replay against less frequent checkpoints.

open as a page

A MongoDB server restarted with blockCompressor set to zstd in mongod.conf shows no drop in disk usage: why, and how do you estimate and realise the saving?

level: seniorimportance: should knowfreq 24%

basics

~20 s

blockCompressor only sets the default for collections created afterwards; existing collections keep their compressor. Estimate by copying a sample into a snappy and a zstd collection, then rewrite the data to collect the saving. Indexes will not shrink.

open as a page

What compressor policy would you set for a fleet of MongoDB deployments, and how would you decide where to depart from snappy?

level: principalimportance: should knowfreq 18%

basics

~20 s

Set one default for new collections by workload class, usually snappy or zstd, based on measured disk savings against CPU headroom. Avoid mass conversions unless the saving pays for the rewrite, and leave journal and index defaults alone.

open as a page