How do you choose between Cohere's float and binary embeddings at 100M scale?

## Frame the question correctly Junior framing: "which type is better?" Senior framing: "what does each cost and what does each lose?" Principal framing: "what does this commit us to, and what will it cost to change our mind?" All three matter, but the last one is what the level is being probed for. ## The infrastructure arithmetic For 100M vectors at 1024 dimensions: - **float** — 4 KB each, roughly **410 GB**. - **int8** — 1 KB each, roughly **102 GB**. - **binary** — 128 bytes each, roughly **13 GB**. ANN indexes generally want their vectors resident in memory. 410 GB means sharding across a fleet of large-memory nodes, with the operational tax that implies: shard routing, rebalancing, replica cost multiplied per shard, longer index builds, slower recovery. 13 GB fits comfortably on a single machine with room for the graph structure on top, and replicas become cheap enough to over-provision for availability. That is not a 30× storage saving; it is a different system architecture, with different failure modes and a different on-call burden. Search throughput moves the same way. Distance computation over millions of candidates is memory-bandwidth-bound, so packed binary vectors scan far faster per candidate than float arithmetic. Compression buys latency headroom as well as capacity. ## Why "or" is the wrong conjunction `embedding_types` accepts a list, and embed pricing counts input tokens — you pay to read the text, not per representation returned. So `["float", "binary"]` in the same call yields both forms of every vector for the same token bill as either one alone. That converts the decision from an irreversible commitment into a storage question. Keep binary in the hot ANN index; keep float (or int8) in cheap storage — object storage, a columnar store, a disk-backed keyed lookup. Later, if evaluation says binary-only recall is inadequate, or if a new store gains a better quantisation mode, you already hold the high-fidelity vectors and never re-read the corpus. The alternative — embedding float-only now and adding binary in six months — means re-billing 100M documents' worth of input tokens and re-running an ingest that takes days. That asymmetry is the entire argument for capturing both up front. ## The two-stage serving pattern The standard design once you hold both: 1. **Search the binary index wide.** Retrieve substantially more candidates than you need — hundreds where you want ten — because compression perturbs ranking near the boundary. 2. **Rescore the candidates with float.** Fetch the float vectors for those few hundred ids and score them precisely. You pay compressed cost for the expensive step (scanning 100M vectors) and full fidelity for the cheap one (scoring a few hundred). Recall recovers close to the float-only baseline while the index stays small. The knob is the candidate count: too small and rescoring cannot recover what the first pass missed; too large and you lose the latency benefit. Tune it empirically. This is also where a reranking model would sit in a full search stack — but note that rescoring with float vectors and reranking with a cross-encoder are different stages solving different problems, and neither substitutes for the other. ## What actually decides it: your evaluation No published recall figure transfers to your corpus. Build a labelled eval set from real queries against real documents — a few hundred query/relevant-document pairs is enough to see differences — and measure recall@k and end-task metrics for: float-only, int8-only, binary-only, and binary-plus-rescore at a couple of candidate depths. Then read the numbers against your latency and cost budgets. Corpora with many near-duplicate documents suffer more from compression, because the discarded precision was exactly what separated the near-duplicates. Corpora of diverse topics suffer much less. Budget for re-running that evaluation whenever the model, the chunker or the corpus mix changes materially. ## The real lock-in Storage type is cheap to change if you kept both forms. What is genuinely expensive: - **Changing embedding model.** Every vector must be regenerated. That is the migration to plan for, and the reason to keep raw source text alongside the vectors forever. - **Changing input type.** Same cost, same reason. - **Losing provenance.** An index whose model id, input type and embedding types were never recorded cannot be safely extended or migrated — you cannot tell whether new vectors are compatible with old ones. So the principal-level deliverable is not a chosen type. It is: capture both representations in the one billed pass, record the generation metadata, keep the source text, define an eval set, and design the index so swapping the served representation is a config change rather than a project. ## What a strong answer sounds like Do the storage arithmetic out loud, refuse the false binary choice by pointing at the multi-type request and token-based billing, describe the two-stage rescore, insist on measuring recall on your own data, and close on re-embedding as the cost that actually constrains future choices.