CI intermittently fails on assertions written as verify(handler, timeout(50)).handle(event), which always pass on a developer laptop. How do you diagnose that, and when would you replace the timed verification with a CountDownLatch or a polling library such as Awaitility?
answer
- late vs never vs contamination
- timeout is an upper bound: make it generous
- latch = happens-before for counts/absence
- Awaitility = waiting on state, not mocks
- same-thread executor removes the wait
basics
~20 sA 50 ms budget is a guess about machine speed; loaded CI with fewer cores and GC pauses blows through it. Because timeout returns on success, raising it to a couple of seconds costs nothing and removes the flake. Replace it with a latch or Awaitility when you need a real happens-before edge or must poll state rather than an interaction.
solid answer
~60 sFirst confirm the failure mode: a Mockito 'Wanted but not invoked' after the budget means the interaction never arrived in time — either it was late or it never happened. Check whether the worker swallowed an exception (executors capture throwables in the Future), whether the executor pool is saturated, and whether the test leaks threads from previous tests. If the work genuinely completes but late, the timeout is simply mis-sized. Timeouts are upper bounds, not sleeps, so 2000 ms is as fast as 50 ms on the happy path and only slows real failures. Tight timeouts buy nothing and cost stability; that is the single most common fix. Switch tooling when the shape is wrong rather than the number: use a CountDownLatch (counted down from a stubbed answer or a callback) when you want a deterministic barrier before asserting counts or absence; use Awaitility when the thing to wait for is observable state (a repository row, a queue depth, a metric) rather than a mock interaction, and you want polling with a readable failure message. Best of all, inject a same-thread executor so the flow is synchronous and no waiting exists.
code
java · 9 linesCountDownLatch done = new CountDownLatch(1);
doAnswer(inv -> { done.countDown(); return null; })
.when(handler).handle(any());
service.submit(event);
assertTrue(done.await(5, TimeUnit.SECONDS), "handler never invoked");
verify(handler).handle(event);
verifyNoMoreInteractions(handler);go deeper
Say that the timeout is too small for a slow CI machine and that raising it is safe because timeout waits at most that long.
Separate 'late' from 'never happened', mention swallowed executor exceptions, and know that a latch gives a real barrier.
Diagnose systematically (late / never / contamination), argue the cost asymmetry for sizing, and pick latch vs Awaitility vs synchronous executor per situation.
Set suite-wide policy: no sleeps, generous named timeout constants, injectable executors as the default seam, no blanket retries, flaky tests quarantined and root-caused.
## Diagnose before you tune An intermittent timed-verification failure has three distinct causes and they need different fixes. 1. Late but correct. The work happens, just slower than the budget on a shared runner with fewer cores, noisy neighbours and GC pauses. Signature: raising the timeout makes it green and the duration on success stays small. 2. Never happens. The worker threw and the executor swallowed it, a queue was full and the task was rejected, or a conditional branch was not taken. Signature: raising the timeout does not help and the test now fails slower. Surface the cause by asserting on the Future, adding an uncaught-exception handler, or using a rejection-logging executor. 3. Cross-test contamination. A previous test left a thread pool or listener alive that consumes the event, or a shared mock was not reset. Signature: fails only in a particular ordering, passes in isolation. Fix lifecycle, not timing. ## Why tight timeouts are a pure loss Mockito's timeout() polls and returns on the first successful check, so the budget is an upper bound. A passing test with timeout(5000) takes exactly as long as the same test with timeout(50). The only thing a small number buys is a faster failure; the thing it costs is every flake caused by a slow runner. The rule that follows: make timeouts generous (1-5 s) and instead keep the number of waiting tests small. This is the opposite of after(ms), where the number is paid every run. ## When a latch is the right tool A CountDownLatch converts a guess into a happens-before edge. Count it down from a doAnswer on the mock, or from a completion callback in production code, then await it with a generous timeout and assert afterwards. This is required when you need exact counts or a negative assertion, because those are only meaningful once the system has provably settled — a timeout-based verify cannot tell you the pipeline is finished, only that one call has appeared. The cost is coupling the test to a specific completion point. ## When Awaitility fits better Mockito's timed verification can only wait for an interaction with a mock. When the observable outcome is state — a row committed, a cache entry, a gauge value, a file on disk — Awaitility (or any polling helper) expresses 'await until this condition holds, with this interval and this at-most' and produces a condition-oriented failure message. It also composes with several conditions and supports ignoring transient exceptions while polling. Use it at integration boundaries; do not layer it on top of mocks where a plain timeout already works. ## The best fix is usually structural Most async unit tests exist only because the class under test constructs or hides its execution strategy. Inject an Executor and pass Runnable::run (or a direct executor) in tests, or a synchronous task executor in a Spring context, and the asynchrony — and the flakiness — disappears. Keep genuinely async tests for the boundary where the asynchrony itself is the behaviour under test, and accept generous timeouts there. ## Operating rules that keep it green Give every wait a generous, named constant rather than scattered magic numbers. Never use a fixed sleep. Ensure each test shuts down or awaits its own executors so leaked threads cannot bleed into the next test. Quarantine and fix flaky tests rather than retrying them, because a retried async test hides the exception-swallowing case, which is a real bug.
- The team's fix is to retry failing tests up to three times in CI. What is wrong with that here?Retrying hides the 'never happens' class of failure, where a swallowed worker exception or a rejected task is a genuine product bug that happens to be intermittent. It also lets tight timeouts survive, so the suite stays fragile and every future timing regression is masked. Fix sizing and add a completion barrier instead, and quarantine what still flakes.
- How do you make an async unit test deterministic without changing its assertions?Inject the Executor rather than creating it inside the class, and pass Runnable::run in the test so submitted work runs on the calling thread before submit returns. The production code path is unchanged, but the test becomes synchronous, so plain verify works and no timeout is needed. In Spring, the equivalent is a SyncTaskExecutor for the @Async bean.
saying these in an interview costs you the question
- Shrinking timeouts to make the suite faster, unaware they are upper bounds
- Replacing timeout with Thread.sleep and a plain verify
- Assuming every timeout failure means 'too slow' and never checking for a swallowed exception
- Adding CI retries as the fix for async flakiness
- Using Awaitility to poll a mock's invocation count when timeout() already does that