OpenCLIP: the open-source CLIP that everyone actually uses

OpenCLIP is an open-source re-implementation of OpenAI's CLIP, trained mostly on the LAION image-text datasets. It ships the training code, the model checkpoints, the eval suite, and dozens of model variants ranging from tiny ViT-B/32 to enormous ViT-G/14.

Why it matters: OpenAI's original CLIP weights are openly available but the training pipeline is not. OpenCLIP is the version you can actually reproduce, retrain on your own data, audit for bias, and deploy without an API call. It has become the de-facto vision-language embedding model in research and production.

OpenCLIP has hundreds of checkpoints. A pragmatic mental model:

• Default starter: ViT-B-32 pretrained on laion2b_s34b_b79k. Fast, runs on CPU at a pinch, great enough for prototypes.
• Quality bump: ViT-L-14 on laion2b_s32b_b82k. ~3x slower, materially better on hard cases.
• State of the art: ViT-bigG-14, EVA02-L-14, or one of the SigLIP variants. Expensive to host but tops most public leaderboards.
• Multilingual: xlm-roberta-base-ViT-B-32 — text encoder speaks ~100 languages, image encoder is the same.

Bigger models are not always better. For dense retrieval over millions of images, latency dominates and ViT-B/32 wins on cost-per-result.

The API is small. Two calls — encode_image and encode_text — produce vectors in the same space. Cosine similarity tells you how well they match.

import open_clip, torch
from PIL import Image

model, _, preprocess = open_clip.create_model_and_transforms(
    'ViT-B-32', pretrained='laion2b_s34b_b79k'
)
tokenizer = open_clip.get_tokenizer('ViT-B-32')
model.eval()

image = preprocess(Image.open('cat.jpg')).unsqueeze(0)
text = tokenizer(['a cat', 'a dog', 'a sailboat'])

with torch.no_grad():
    img_emb = model.encode_image(image)
    txt_emb = model.encode_text(text)
    img_emb /= img_emb.norm(dim=-1, keepdim=True)
    txt_emb /= txt_emb.norm(dim=-1, keepdim=True)
    sim = (100.0 * img_emb @ txt_emb.T).softmax(dim=-1)

print(sim[0].tolist())  # [0.97, 0.02, 0.01]

Normalise embeddings before comparing — the model returns unnormalised vectors and cosine similarity assumes unit length.

Most production uses of OpenCLIP fall into three buckets:

1. Image search: embed your library once, embed user queries on the fly, return nearest neighbours. Pair with FAISS or pgvector and you have a CLIP-powered visual search engine in 100 lines.
2. Zero-shot classification: embed your class labels as text prompts, compare to image embeddings, take argmax. No training needed.
3. Dataset curation: filter noisy web-scraped data by embedding similarity. "Keep only images that match a photo of a cat with >0.25 similarity." Most training datasets today are filtered this way.

Embed once at index time, embed once per query, do nearest-neighbour.

Out-of-the-box OpenCLIP does well on "web-distribution" content — pets, food, landscapes, common consumer products. It can be weak on:

• Medical imaging
• Satellite or aerial
• Industrial/defect inspection
• Specific fashion brands or SKUs

For these, fine-tune. Two approaches:

• Linear probe: freeze the encoders, train a single classification head on top. Cheap, no large GPU needed, works when you have labelled categories.
• Contrastive fine-tuning: continue training the whole model on (image, text) pairs from your domain. More expensive, more powerful — closes the domain gap rather than working around it.

The OpenCLIP repo includes training code; community recipes like DataComp and open_lpips document the standard settings.

Decision rule: if you're doing >100K embeddings/day, host OpenCLIP. If you're prototyping or running low-volume, use a closed API. The break-even is roughly where one GPU's monthly cost equals your API spend.

OpenCLIP inherits two big issues from its training data:

1. LAION bias: the LAION datasets were scraped from Common Crawl. They overrepresent western content, English text, photogenic subjects, and have well-documented racial/gender biases in association tasks.
2. Compositional weakness: like CLIP, OpenCLIP confuses attribute binding — "red cube, blue sphere" sometimes matches an image with a blue cube and red sphere. The model encodes a bag of concepts more reliably than their relationships.

Use OpenCLIP for retrieval and similarity. Avoid using it for fairness-sensitive classification without auditing the specific checkpoint on your population.