Skip to main content

CLIP: Learning Transferable Visual Models From Natural Language Supervision

CLIP (Contrastive Language-Image Pre-training) is a new approach to learning visual representations proposed by researchers at OpenAI in 2021. Unlike traditional computer vision models which are trained on large labeled image datasets, CLIP learns directly from natural language supervision.

The Core Idea

The key insight behind CLIP is that we can connect images and text captions without generating the captions. By training the model to predict which caption goes with an image, it learns a rich visual representation of the world.


As illustrated above, CLIP consists of two encoders - an image encoder and a text encoder. The image encoder takes in an image and outputs a visual representation vector. The text encoder takes in a caption and outputs a text representation vector.

During training, these representations are optimized to be closer for matching image-text pairs, and farther apart for non-matching pairs. This is known as a contrastive loss objective.


Benefits of CLIP

There are several advantages to this approach:


  • Transferable visual representations - The image encoder learns generally useful visual concepts like objects, scenes, actions, etc. This representation can be transferred to other vision tasks through fine-tuning.
  • Scale - CLIP is trained on a dataset of 400 million image-text pairs scraped from the internet. This provides a rich supervision signal.
  • Zero-shot transfer - Remarkably, CLIP can perform zero-shot classification without seeing any examples from a dataset. It simply relies on comparing image embeddings to text prompts.
  • Robustness - CLIP shows robustness to image variations like blur, brightness changes etc. This indicates it learns high-level semantic representations.



CLIP In Action


A few examples of what CLIP can do:


  • Classify an image of a dog as a "dog" without seeing any dog images during training.
  • Correctly match the caption "A happy couple on a gondola" to the corresponding image.
  • Be fine-tuned to get 97.9% accuracy on CIFAR-10 in just 10 minutes of training.

CLIP provides simple yet powerful off-the-shelf visual representations for a variety of computer vision tasks. Its limitations include a lack of spatial understanding and bias in the training data. Nonetheless, it's an impressive demonstration of learning visual concepts directly from natural language.



Comments

Post a Comment

Popular Posts

TX-CNN: DETECTING TUBERCULOSIS IN CHEST X-RAY IMAGES USING CONVOLUTIONAL NEURAL NETWORK

In this paper , the authors propose a method to classify tuberculosis from chest X-ray images using Convolutional Neural Networks (CNN). They achieve a classification accuracy of 85.68%. They attribute the effectiveness of their approach to shuffle sampling with cross-validation while training the network. Methodology Convolutional Neural Network This has been the ultimate tool for researchers and engineers for computer vision tasks. It has been widely used for many general purpose image and video related tasks. There are many great resources to learn about them. I will link a few of them at the end of this post. In this paper, the authors study the famous AlexNet and GoogLeNet architectures in classifying tuberculosis images. A CNN model usually consists of convolutional layers, pooling layers and fully connected layers. Each layer is connected to the previous layers via kernels or filters. A CNN model learns parameters of the kernel to represent global and local features ...
Post a Comment
Read more

BLIP: Bootstrapping Language-Image Pretraining for Unified Vision-Language Understanding

BLIP is a new vision-language model proposed by Microsoft Research Asia in 2022. It introduces a bootstrapping method to learn from noisy image-text pairs scraped from the web. The BLIP Framework BLIP consists of three key components: MED  - A multimodal encoder-decoder model that can encode images, text, and generate image-grounded text. Captioner  - Fine-tuned on COCO to generate captions for web images. Filter  - Fine-tuned on COCO to filter noisy image-text pairs. The pretraining process follows these steps: Collect noisy image-text pairs from the web. Pretrain MED on this data. Finetune captioner and filter on the COCO dataset. Use captioner to generate new captions for web images. Filter noisy pairs using the filter model. Repeat the process by pretraining on a cleaned dataset. This bootstrapping allows BLIP to learn from web-scale noisy data in a self-supervised manner. Innovations in BLIP Some interesting aspects of BLIP: Combines encoder-decoder capability in one...
Post a Comment
Read more