Adding Keras's TensorBoard callback doubled epoch time — which settings cause that?

## Callbacks are on the critical path A callback is not an observer running elsewhere — its hooks execute synchronously inside the training loop, between steps. Anything a hook does is added directly to wall-clock training time. The TensorBoard callback is the usual culprit because it is the one people configure for maximum visibility and then forget about, but the reasoning generalises to any callback you write. The multiplier that matters is *cadence*. An epoch hook runs a few dozen or a few hundred times per run; a batch hook runs once per step, which on a large dataset is tens of thousands of times per epoch. The same body of work is negligible in one and dominant in the other. ## update_freq `update_freq` controls how often losses and metrics are written. `"epoch"` (the default) writes once per epoch — a handful of scalars, effectively free. `"batch"`, or an integer like `10`, writes every step or every N steps. Each write serialises scalars and touches the event file. On a small model where a step is a couple of milliseconds, the write can cost as much as the computation, which is exactly how an epoch doubles. Per-batch curves are genuinely useful when you are debugging a divergence in the first few hundred steps; they are waste for the rest of a run. If you want them, use a small integer rather than every batch, and turn the setting back to `"epoch"` when the investigation ends. ## histogram_freq and write_images `histogram_freq=0`, the default, means no weight histograms. Setting it to 1 makes the callback, at every epoch end, read every weight and bias tensor in the model, bin the values, and write the histograms. The cost scales with parameter count, and there is a second cost that shows up later: event files grow by orders of magnitude, which makes the TensorBoard UI slow to load and the artifacts expensive to keep. In Keras 3, computing histograms also requires materialising weight values on the host, so on an accelerator there is a transfer involved. `write_images=True` additionally renders weight tensors as images on the same schedule — more work, much more file. Both are diagnostic tools for "are my weights exploding or dead", not defaults for routine runs. `write_graph=True` is a one-off cost at the start rather than a per-epoch one, but it can still bloat the event file for a large model. ## profile_batch `profile_batch` runs the tracing profiler over a batch or a range of batches. Profiling is intentionally heavy — that is the point — and it is meant for a short run of a few hundred steps whose only purpose is to produce a trace. Leaving it enabled on a full training job pays the tracing cost and produces trace data nobody reads. In Keras 3 this argument is only meaningful under the TensorFlow backend; with the JAX or PyTorch backend there is no TensorFlow profiler to invoke, which is worth remembering when a snippet copied from a Keras 2 tutorial appears to do nothing. ## Diagnosing it The method is dull and reliable. Time an epoch with the callback list empty, then with the callbacks one at a time. Because callbacks are independent and synchronous, the difference attributes cleanly. If the gap appears only with `update_freq` below epoch level, it is I/O per step; if it appears at epoch boundaries only, it is histograms. The same test catches hand-written callbacks with expensive `on_train_batch_end` bodies — computing a metric over a full batch, appending to a list that grows without bound, writing a row to disk per step, or pulling weights to host memory. The fix is the same shape: accumulate something cheap per batch, and do the expensive part once per epoch. ## What logging should actually cost A sensible default set is `TensorBoard(log_dir=..., update_freq="epoch", histogram_freq=0)` plus `CSVLogger(filename, append=True)`, which writes one row per epoch and is essentially free — and, unlike a TensorBoard event file, is trivially readable when the run is on a machine you cannot point a browser at. Turn on histograms deliberately, for a specific question, and turn them off again. If per-step visibility is a standing requirement rather than a debugging trip, the honest fix is to reduce the *volume* per write and the frequency, not to accept a permanent tax on every epoch.