CLIP (Contrastive Language-Image Pre-training) is a model designed to learn shared representations between images and text by training on large datasets of paired image-text examples. It uses a dual-encoder architecture: one encoder processes images (e.g., Vision Transformer or ResNet), and another processes text (e.g., a Transformer-based text encoder). Both encoders map their inputs into a shared embedding space where semantically similar images and texts are positioned close together. The core idea is contrastive learning: during training, the model learns to maximize the similarity between embeddings of correct image-text pairs while minimizing similarity for incorrect pairs. For example, an image of a dog and the text “a brown dog” would be pulled closer in the embedding space, while the same image would be pushed away from unrelated text like “a blue car.”
The training process involves feeding batches of image-text pairs into the two encoders. Each image is converted into an embedding vector by the image encoder, and each text is converted into a separate vector by the text encoder. A contrastive loss function (typically a form of normalized softmax) is then applied. For each image in a batch, the model calculates its similarity score with all text embeddings in the batch, and vice versa. The loss encourages the correct pairs (e.g., image 1 with text 1) to have high similarity scores and incorrect pairs (e.g., image 1 with text 2) to have low scores. A key detail is the use of a temperature parameter, which scales the similarity scores to control how “sharp” the softmax distribution is. CLIP was trained on 400 million publicly available image-text pairs, enabling it to generalize across a wide range of visual and textual concepts without task-specific fine-tuning. The encoders are trained from scratch, which avoids biases from pre-trained models and ensures alignment between the two modalities.
CLIP’s embeddings enable applications like zero-shot image classification and cross-modal retrieval. For example, to classify an image of a cat, you could generate text embeddings for labels like “a photo of a cat” or “a photo of a dog” and compare their cosine similarity with the image embedding. The label with the highest similarity becomes the prediction. Developers can also use CLIP for tasks like image search by encoding a query text (e.g., “a sunset over mountains”) and finding images whose embeddings are nearest to the query. However, CLIP has limitations: its performance depends on the diversity of its training data, so niche or culturally specific concepts may not be well-represented. Additionally, the embeddings might not capture fine-grained details (e.g., distinguishing between similar bird species) unless the text prompts are precise. Despite these caveats, CLIP provides a flexible foundation for multimodal systems by unifying image and text understanding into a single framework.