A C++ producer sets a std::atomic<bool> flag without locking the mutex, then calls notify_one; why can the waiting consumer still block forever?
answer
- check, then block: a window
- the mutex is what closes it
- atomic removes the race, not the window
- set the flag under the waiter's mutex
basics
~20 sThe consumer can read the flag as false and then block; a store and notify_one landing between those two steps find no blocked thread, so the notification is lost. Storing the flag under the waiter's mutex closes that window.
solid answer
~40 s`wait(lock, pred)` checks the predicate and then blocks, and its unlock-and-block is atomic only with respect to the mutex. A producer that changes the flag under that mutex cannot run between the check and the block. A producer that skips the mutex can: the consumer reads `false`, the producer stores `true` and calls `notify_one` while nobody is blocked yet, and the consumer then blocks with no further notification coming. `std::atomic` removes the data race, so ThreadSanitizer stays quiet, but it does nothing for that interleaving. The fix is to take the same mutex around the store — even for an atomic — and notify inside or after that scope. Notifying while still holding the lock is correct; notifying after unlocking only spares the woken thread a brief block on the mutex.
code
cpp · 33 lines#include <atomic>
#include <chrono>
#include <condition_variable>
#include <iostream>
#include <mutex>
#include <thread>
using namespace std::chrono_literals;
std::mutex m;
std::condition_variable cv;
std::atomic<bool> ready{false};
int main(int argc, char**) {
const bool lock_in_producer = argc > 1;
std::thread producer([&] {
std::this_thread::sleep_for(50ms);
if (lock_in_producer) { std::lock_guard<std::mutex> g(m); ready = true; }
else { ready = true; } // store without the mutex
cv.notify_one();
});
auto t0 = std::chrono::steady_clock::now();
std::unique_lock<std::mutex> lk(m);
while (!ready) { // what wait(lk, pred) does
std::this_thread::sleep_for(200ms); // widen the check-then-block window
cv.wait_for(lk, 2s);
}
auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::steady_clock::now() - t0);
std::cout << (ms > 1s ? "notification lost, waited " : "woken after ")
<< ms.count() << " ms\n";
lk.unlock();
producer.join();
}go deeper
Remember the rule: whatever the predicate reads is changed only while holding the same mutex, then notify.
Trace the four-step interleaving that loses the notification and show why the mutex around the store makes it impossible.
Diagnose rare hangs in start-up and shutdown handshakes, know that ThreadSanitizer is silent here, and choose notify inside or outside the lock for the object's lifetime.
Make the mutex-around-the-state rule part of review checklists and prefer one shared, tested queue or latch over ad-hoc flags with condition variables.
## The window the mutex closes A waiter using `std::condition_variable` runs, in effect, `while (!pred()) wait(lock);` with the mutex held. Between the moment `pred()` returns `false` and the moment `wait` has unlocked and blocked, the waiter is holding the mutex. `wait` performs "atomically calls `lock.unlock()` and blocks", so there is no instant at which the mutex is free and the waiter is not yet blocked. Any producer that must **take that mutex** to change the state therefore runs either entirely before the check (and the check sees the new value) or entirely after the waiter is blocked (and its notify finds it). That is the whole correctness argument for condition variables, and it rests on the producer taking the mutex. ## Why `std::atomic` does not help Making the flag a `std::atomic<bool>` removes the **data race** — concurrent unsynchronized access to a non-atomic object is undefined behaviour, and the atomic fixes that. But the lost wake-up is not a race on the flag; it is an interleaving that is perfectly well defined: 1. The consumer locks the mutex and evaluates the predicate: `ready.load()` returns `false`. 2. The producer, not taking the mutex, stores `ready = true`. 3. The producer calls `notify_one()`. No thread is blocked on the condition variable, so the standard's effect, "if any threads are blocked waiting for `*this`, unblocks one of those threads", does nothing. 4. The consumer enters `wait`, unlocks and blocks. Nothing will notify it again. With `wait(lock, pred)` and no timeout, the consumer sleeps until a later notification happens to arrive, which in a one-shot handshake is never. Because no access is unsynchronized, **ThreadSanitizer reports nothing**; the bug only shows up as a rare hang. ## The fix Change the flag under the same mutex the waiter holds, even though it is atomic: ```cpp { std::lock_guard<std::mutex> g(m); ready = true; // atomic or plain bool: either works now } cv.notify_one(); ``` Once the store needs the mutex, step 2 cannot run while the consumer is between its check and its block. At that point the flag no longer needs to be atomic at all, unless other code reads it without the lock. ## Notify inside or outside the lock | Where `notify_one` runs | Correct? | Effect | |---|---|---| | Inside the locked scope | yes | the woken thread may block briefly on the mutex until the notifier unlocks | | Just after unlocking | yes | the woken thread can usually take the mutex at once | | With no lock ever taken around the state change | **no** | the lost wake-up described above | Two refinements matter in practice: - Notifying **after** unlocking is the usual default for throughput, but when the waiter may destroy the condition variable as soon as it sees the flag — a one-shot object owned by the waiter — notifying **inside** the lock keeps the condition variable alive until the notify has returned. - `notify_all_at_thread_exit(cv, std::move(lk))` exists for the case where the signal must come after a thread's `thread_local` objects are destroyed; it holds the lock until the thread exits. ## The same bug with a plain `bool` If the flag is a plain `bool` written without the mutex, the program has a second, worse defect: the write and the predicate's read are a **data race**, which is undefined behaviour in C++. The compiler may then keep the value in a register or reorder the accesses, and anything can follow. ThreadSanitizer does report that version, and the usual "fix" is to make the flag `std::atomic<bool>`, which silences the report but leaves the lost wake-up in place. The real fix is the same for both versions: write the flag under the mutex. Seeing an atomic flag next to a condition variable is therefore a cue to check where it is written. ## How it shows up and how to find it - A hang that appears once in thousands of runs, typically in start-up or shutdown handshakes that signal only once. - A worker blocked in `wait` while a debugger shows the flag already `true`. - ThreadSanitizer is silent, because there is no data race. - A timed wait (`wait_for`) in the waiter turns the hang into a delay of exactly the timeout, which is a useful diagnostic: the waiter wakes on the deadline and finds the condition already true. - Code review is the best detector: every write to state read by a condition-variable predicate must be under that condition variable's mutex.
- If the producer must lock the mutex anyway, does the flag still need to be std::atomic?Not for the condition-variable handshake. Once every write happens under the mutex and the predicate reads it under the same mutex, a plain `bool` is race-free. Keep it atomic only if some other code reads or writes it without the lock, for example a fast-path check before taking the mutex.
- When is notifying while still holding the mutex the better choice?When the waiter may destroy the condition variable as soon as it sees the condition, for instance a one-shot object owned by the waiting thread. If the notifier unlocked first, the waiter could wake (even spuriously), return and destroy it before `notify_one` runs. Notifying under the lock keeps the waiter from returning until the notifier has released the mutex.
- Why does ThreadSanitizer not report this bug?ThreadSanitizer reports data races: conflicting accesses not ordered by synchronization. With an atomic flag every access is synchronized, so there is no race to report. The defect is a missing mutual exclusion between the waiter's check-then-block and the producer's store-then-notify, a logic error no race detector models.
- An atomic flag makes it safe to skip the mutex when signalling
- notify_one called before anyone waits is delivered to the next waiter
- Calling notify_one while holding the mutex loses the notification
- ThreadSanitizer would catch a lost wake-up
- notify_all instead of notify_one fixes a lost wake-up