How do shared text-image embeddings let a description retrieve untagged photos?

## The problem it solves A museum has 200,000 scanned photographs and almost no metadata. Cataloguing them by hand is a multi-year project. Yet visitors want to search by describing what they remember: a tram in the snow, a woman reading on a bench. Keyword search over filenames is useless; there is no text to match. A shared text-image embedding space attacks this by making the description and the photograph comparable *directly*, with no intermediate text about the image at all. ## How the shared space is built The architecture is a **dual encoder**, sometimes called a two-tower model: - an image encoder that maps a picture to a vector, - a text encoder that maps a string to a vector of the same dimensionality, - both projecting into one common space. Training is contrastive over pairs. Take a large batch of image-caption pairs; for each image, its own caption is the positive and every other caption in the batch is a negative. The loss maximizes similarity for the true pair and minimizes it for the rest, in both directions. Run that over hundreds of millions of pairs harvested from the web and the two towers converge on a shared coordinate system: the region of the space meaning "dog on a beach" is the same region whether you arrived there from pixels or from words. Nothing about this is magic, and it is worth being precise about what was learned — the model learned to match *photographs* with *the kind of caption people write on the internet*. That is the distribution it will be good at. ## Why the pipeline is boringly familiar Once the space is shared, cross-modal retrieval is not a special system: 1. Embed every photograph once, at ingest, and store the vectors in an index. 2. Embed the incoming text query with the *paired* text tower. 3. Nearest-neighbour search, same as any text retrieval. The same trick runs in reverse (find captions or documents matching an image) and image-to-image (find visually similar photographs) with no change to the index. This is the practical appeal: one representation, several search modes. ## The hard constraint: one model pair, one space The single most important operational rule. **Vectors are only comparable when they come from encoders that were trained jointly.** A text vector from a general-purpose sentence encoder and an image vector from a vision model are in unrelated coordinate systems; the cosine between them computes to a number and that number means nothing. Matching dimensionality does not imply compatibility. This has consequences beyond the obvious. If you already run a text corpus indexed with a strong text-only embedding model, you cannot simply add image vectors to that index — you need the joint model's text tower for anything that must be comparable with images, and that tower is usually weaker at pure text-to-text semantics than a dedicated text encoder, because it was optimized for caption-style matching. Many systems therefore keep two indexes and merge results, rather than pretending one space serves both purposes. ## Where it degrades Be candid about the limits — an interviewer is listening for whether you have actually deployed one: - **Phrasing sensitivity.** Short, concrete, caption-like queries beat long analytical sentences, because captions are what training saw. - **Fine-grained distinctions.** Distinguishing two similar architectural styles or two similar bird species is much weaker than distinguishing a beach from a kitchen. - **Counting and spatial relations.** "Three people to the left of the door" is a known weak spot for contrastive dual encoders. - **Text inside images.** Reading a shop sign or a document scan is not what the model was trained for; a dedicated OCR step is the right tool. - **Domain shift.** A century-old monochrome archive looks nothing like the web photography the model was trained on, so quality on that collection is an empirical question, not an assumption. - **Abstract or curatorial concepts.** "Melancholy", "post-war austerity", or an accession category will not be reliably encoded. ## Choosing on modality The practical lesson generalizes: **modality is a selection criterion that comes before quality scores.** If your artefacts are images, audio or video, the first question is not which model ranks highest but which models can represent that modality at all, and whether they place it in a space shared with the query modality you actually have. A text-only model, however strong, is simply not a candidate. ## Validating before you commit Have a curator write 50 to 100 realistic queries against the actual archive and mark the correct photographs. Measure recall at 10 and 50 on that set. It is a few hours of work and it converts "CLIP-style search should work" into a number you can defend — including, sometimes, the finding that a zero-shot joint model is good enough for exploratory browsing but not for authoritative catalogue search, in which case its right role is to accelerate human cataloguing rather than to replace it.