skip to content

How does batch_size affect Sentence Transformers' encode() when embedding a few million passages, and how do you choose a value?

level: middleimportance: should knowfreq 12%

answer

  1. inputs per forward pass
  2. default is 32
  3. padding up to the longest in a batch
  4. length-sorted, then restored to input order
  5. benchmark on a real sample

basics

~20 s

batch_size (default 32) is how many inputs share one forward pass. Larger batches raise GPU throughput until memory runs out; encode() length-sorts inputs so batches pad little, then restores your order. Benchmark a few sizes on a sample.

solid answer

~50 s

`encode()` splits the inputs into batches of `batch_size` — default 32 — and runs one forward pass per batch. On a GPU, bigger batches amortize per-call overhead and keep the device busy, so 32 is often too small for a bulk job, up to the point where the batch no longer fits in memory. A batch costs roughly its size times its longest input, because shorter inputs are padded to it; `encode()` already sorts inputs by length before batching and returns results in your original order, so you need not sort yourself. The library's advice is to benchmark a few sizes on real data. Keep the numpy default for a bulk job: each batch moves to CPU as it finishes, whereas `convert_to_tensor=True` keeps every vector on the GPU. For several GPUs, pass a list of devices to `device=`.

go deeper

for a junior

Know the default of 32, that bigger batches are faster on a GPU until memory runs out, and that results come back in your input order.

for a middle

Explain padding to the longest input, encode's internal length sorting, and why the numpy default frees GPU memory batch by batch while convert_to_tensor does not.

for a senior

Plan a multi-million passage job: benchmark sizes on real data, expect early out-of-memory on the longest batches, and split across devices with a device list or a pool.

for a principal

Size the embedding job as capacity work: GPU hours per corpus refresh, how often refreshes happen, and whether faster batching or fewer re-embeddings saves more.

## What `batch_size` controls `SentenceTransformer.encode()` — and `encode_query()` / `encode_document()`, which pass it through — processes inputs in **batches**: `batch_size` texts are tokenized together, padded to a common length, and run through the model in one **forward pass**. The default is `32`, and a value of zero or less raises `ValueError`. Batching matters because a GPU does far more work per call than one sentence needs. Small batches leave the device idle between calls; large ones keep it saturated. On a CPU the effect is weaker, because the hardware has less parallelism to fill. ## Length sorting inside `encode()` Within a batch, every input is padded to the longest one, so a batch that mixes a 10-token title with a 500-token passage pays for 500 tokens twice. To limit that waste, `encode()`: 1. estimates each input's length (character count for strings); 2. sorts inputs from longest to shortest, so each batch holds similarly sized texts; 3. runs the batches in that order; 4. puts the embeddings back into the **original input order** before returning. When the model runs with flash attention and inputs can be concatenated without padding, it interleaves long and short inputs instead, to keep the token count per batch even. Either way, you get results aligned with what you passed in. The longest possible input is capped by the model's `max_seq_length`, beyond which text is truncated; choosing that limit is a model-selection concern. One consequence: the **first** batches hold the longest texts and are the most memory-hungry. An out-of-memory error, if it comes, usually comes early. ## Choosing a value 1. Take a few thousand inputs that are representative of the real corpus, long ones included. 2. Time `encode()` on them at several sizes — for example 32, 64, 128, 256. 3. Pick the largest size before throughput stops improving or memory runs out, and leave headroom for the longest inputs. 4. Re-check after changing the model, the precision it runs at, or the hardware; the library's own docs say the best value depends on hardware, model size, precision and input length. `batch_size` changes speed and memory, not results: the same inputs produce the same embeddings whatever the batch size, apart from floating-point noise. ## Memory: where the output lives | Output setting | Where finished batches go | Bulk-job fit | |---|---|---| | default (`convert_to_numpy=True`) | copied to CPU as each batch finishes, returned as one numpy array | good: GPU memory holds only the current batch | | `convert_to_tensor=True` | kept on the model's device and stacked into one tensor at the end | risky: every vector of the corpus stays on the GPU | For millions of passages, the stored vectors alone can reach gigabytes, so a GPU-resident result plus the final stack can exhaust memory near the end of a run that started fine. Keep the numpy default, or encode in slices and write each slice to disk. ## Beyond one device - **Several devices:** pass a list such as `device=["cuda:0", "cuda:1"]` to `encode()`; it starts one process per device, splits the inputs into chunks (`chunk_size`), and merges the results in order. - **Reusable pool:** `model.start_multi_process_pool(...)` creates worker processes you pass as `pool=` to repeated `encode()` calls, and `stop_multi_process_pool(pool)` shuts them down. - **Progress:** `show_progress_bar=True` prints a bar over batches; by default it appears only when logging is at INFO or DEBUG level. Starting processes has a fixed cost, so multi-process encoding pays off on large inputs, not on a few hundred sentences. ## Reading a benchmark honestly - **Warm up first.** The first call pays for moving the model to the device and for kernel set-up; time a second call, not the first. - **Use real lengths.** A benchmark on short sentences overstates the batch size that survives a corpus of long passages. - **Measure inputs per second end to end,** including the copy back to CPU, since that is what a bulk job pays. - **Watch peak memory,** not average: the longest batches come first and set the ceiling. ## What not to tune with `batch_size` - It does not reduce how often you encode: a corpus re-encoded per query stays slow at any batch size. - It does not change embedding size or precision; those are separate options. - It is unrelated to `util.semantic_search`'s chunk sizes, which bound the scoring step, not encoding.

  • Why does an out-of-memory error during encode() tend to appear in the first batches?
    `encode()` sorts inputs from longest to shortest before batching, so the first batches pad to the longest texts and need the most activation memory. If those fit, later batches usually do too. Lower `batch_size`, or cap the input length, until the first batches pass.
  • Does changing batch_size change the embeddings?
    No, beyond floating-point noise: each input is padded (or packed) and masked, so its embedding does not depend on its batch neighbours, and `encode()` returns results in the original order. `batch_size` trades speed against memory only.
  • How do you spread one encode() job across two GPUs?
    Pass a list of devices, for example `device=["cuda:0", "cuda:1"]`. `encode()` starts a process per device, sends each a chunk of inputs (`chunk_size` controls how many), and merges the results in input order. For repeated calls, create a pool with `start_multi_process_pool` and pass it as `pool=`.

  • batch_size changes the embedding values, so tune it for quality
  • You must sort inputs by length yourself before calling encode()
  • A larger batch_size always speeds encoding, with no memory limit
  • encode() returns embeddings in length-sorted order, not input order
  • convert_to_tensor=True is the memory-safe choice for huge corpora