Vision-Language Models (VLMs): how machines read images

A Vision-Language Model is any model that takes images and text as input and produces meaningful output that mixes the two. Two families dominate:

• Embedding VLMs (CLIP, SigLIP, EVA): produce a vector for an image and a vector for a piece of text in the same space. Two things that match score high.
• Generative VLMs (GPT-4V, Claude 4 visual, Gemini, LLaVA): take image+text in, generate free-form text out.

Different tools for different jobs. Embeddings are cheap and great at search/classification. Generative models are flexible but slow and pricey.

OpenAI's CLIP was trained on 400M (image, caption) pairs scraped from the web. The clever bit: it learned not by predicting captions but by pulling matching pairs together and pushing mismatched pairs apart in a shared embedding space.

For each batch of $N$ pairs you get an $N \times N$ similarity matrix. The diagonal should be high (each image matches its own caption); off-diagonal should be low. The loss is symmetric cross-entropy on both rows and columns. That's it — no labels, no human annotation, just "these two go together, those two don't". Astonishingly, this produced a model that could classify images by writing a prompt instead of training a classifier.

Two encoders, one shared space, contrastive matching.

Because CLIP embeds images and text in the same space, you can classify images without training a classifier. Define each class by a natural-language prompt, embed those prompts, then pick whichever prompt is closest to your image embedding.

import open_clip, torch
model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32', pretrained='laion2b_s34b_b79k')
tokenizer = open_clip.get_tokenizer('ViT-B-32')

labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
img = preprocess(Image.open("input.jpg")).unsqueeze(0)
with torch.no_grad():
    img_emb = model.encode_image(img)
    txt_emb = model.encode_text(tokenizer(labels))
    probs = (100 * img_emb @ txt_emb.T).softmax(dim=-1)
print(labels[probs.argmax()])

No training data, no fine-tuning. The prompts are the classes.

Models like GPT-4V, Claude 4 visual, and LLaVA take a different route: they stick a vision encoder onto a pretrained LLM and train the connection so the LLM can "see".

A common recipe (LLaVA-style):

1. Take a vision encoder (often CLIP's image tower).
2. Pass its features through a small projection MLP.
3. Insert the projected features as if they were token embeddings in front of the LLM.
4. Fine-tune the projection (and sometimes the LLM) on image-text pairs.

The result: a language model that can answer questions about images, describe scenes, transcribe charts, OCR sloppy handwriting. The vision tower contributes perception; the LLM contributes reasoning and language.

Embedding VLMs are ~1000x cheaper per query and run on a single GPU. Reach for a generative VLM only when you actually need natural-language output.

VLMs are not infallible. Known weaknesses:

• Counting: "how many people are in this picture" routinely wrong past ~5.
• Spatial reasoning: "what's to the left of the lamp" is shaky, especially in cluttered scenes.
• Fine-grained discrimination: telling apart breeds of a dog, or two similar product SKUs.
• Reading small text: most do OCR well at large sizes but fall apart on tiny print.
• Bias from training data: web-scale image scrapes reflect the web's biases.

For production, treat VLMs as a useful but imperfect signal. Combine with traditional CV (OCR engines, object detectors) when accuracy matters.

Closed (GPT-4V, Claude 4 visual, Gemini): best quality on hard reasoning, easiest to use, pay per call. Good default.

Open (LLaVA, Qwen-VL, InternVL, OpenCLIP for embeddings): self-hostable, predictable cost, customisable. Quality has caught up dramatically for common tasks. Use them when (a) data can't leave your infra, (b) you need to fine-tune on a niche domain, or (c) you're running at high volume where per-call pricing hurts.

For embeddings specifically, open beats closed handily: OpenCLIP variants are essentially free to run and competitive with closed embeddings. For generative tasks, closed still has an edge on the hardest cases, but the gap is closing every few months.