In TGI, when is --quantize bitsandbytes the wrong way to shrink a model?

## What the flag selects `--quantize` tells the TGI shards which weight-quantization path to use when loading the model. The accepted values name distinct paths: `awq`, `gptq` and `marlin` for pre-quantized checkpoints in those formats, `fp8` for 8-bit floating point on hardware that supports it, and `bitsandbytes`, `bitsandbytes-nf4`, `bitsandbytes-fp4` for load-time quantization of an ordinary checkpoint. The important split is not the bit width. It is **when** the quantization happened: - **Offline, ahead of time** — someone produced a quantized checkpoint, typically with a calibration pass, and published it. TGI loads weights that are already in the target format and runs kernels written for that format. - **At load, on the fly** — TGI takes fp16 weights and converts them as it loads. Nothing was published; nothing was calibrated against your data. When a checkpoint's own configuration already declares how it was quantized, TGI can pick that up without you naming it, so in practice `--quantize` is most often either an override or the switch that turns on the on-the-fly path. ## Why bitsandbytes is the wrong production default Its appeal is obvious: point it at any fp16 model on the Hub and it fits in less memory immediately. No offline job, no second artifact, no format compatibility to check. For a laptop-adjacent dev GPU or a quick experiment that is exactly right. In production it is usually the wrong trade, because the kernels are substantially slower than the fp16 path — TGI's own guidance flags it as the easy but slow method. You are therefore paying latency and throughput for memory. On a rented GPU those are the same currency: if the on-the-fly path halves your tokens per second, the memory you freed did not make the deployment cheaper, it made it more expensive per token. The 4-bit variants (`bitsandbytes-nf4`, `bitsandbytes-fp4`) push memory savings further and do not reverse this reasoning. The alternative costs one offline step and pays it back on every request: use a checkpoint quantized ahead of time in a format with fast kernels, and serve it directly. That is the entire argument, and it is the answer an interviewer is listening for when they ask "we're out of memory, can we just quantize it?" ## What quantization does not fix The most common follow-on mistake is expecting `--quantize` to solve a concurrency problem. It shrinks **weights**. The other large consumer of device memory when serving is the per-request cache, which scales with how many requests you admit and how long their slots are — and weight quantization does not touch it. So: - If you cannot load the model at all, quantization is the right lever. - If the model loads and you then hit memory pressure as concurrency rises, the levers are your configured token limits and how many concurrent requests you admit, not the weight format. A deployment that quantizes to 4-bit and then keeps a full-length token slot has freed memory it immediately hands back to the cache. ## Validate, don't assume Quantization changes the arithmetic the model performs. Sometimes the effect is invisible; sometimes it lands squarely on the behaviour you care about — long-form coherence, structured output that must parse, a low-resource language, precise instruction following. The failure is not a crash, it is a quality regression that only your task evaluation will see. So treat a quantization change as a model change, with the same gate: run the same evaluation set against the fp16 baseline and the quantized server, on the same prompts, and compare on your task metric. "It still writes fluent English" is not evidence. This is also why the pre-quantized route has a second advantage — a published checkpoint has usually been evaluated by its author, whereas an on-the-fly conversion has been evaluated by nobody. ## Hardware compatibility is part of the choice Quantization formats are not universally fast. A format is only worth choosing if the GPUs you actually run on have optimized kernels for it — `fp8` in particular depends on hardware support, and formats like `marlin` exist precisely because a faster kernel for an existing weight format was worth a separate path. A checkpoint that is fast on your newest nodes and falls back to something slow on the older half of the fleet turns into a latency mystery that only reproduces on some pods. Before standardising on a format, check it against every GPU SKU in the pool, and pin the deployment to SKUs where it is fast rather than letting the scheduler discover the slow ones for you. ## A working decision order 1. Do the memory arithmetic first — weights plus the cache you need plus overhead against the card you have. Maybe you do not need to quantize at all. 2. If you do, prefer a **pre-quantized checkpoint** in a format your GPUs run fast, served with the matching `--quantize` value or read from the checkpoint's own config. 3. Use `bitsandbytes` only for development, a one-off experiment, or a model for which no quantized checkpoint exists and producing one is not worth it yet. 4. Evaluate before and after on your own task, not on general fluency. 5. Re-check the memory picture at your real concurrency, since weight savings are not cache savings.