Skip to main content

CLIP: Learning Transferable Visual Models From Natural Language Supervision

CLIP (Contrastive Language-Image Pre-training) is a new approach to learning visual representations proposed by researchers at OpenAI in 2021. Unlike traditional computer vision models which are trained on large labeled image datasets, CLIP learns directly from natural language supervision.

The Core Idea

The key insight behind CLIP is that we can connect images and text captions without generating the captions. By training the model to predict which caption goes with an image, it learns a rich visual representation of the world.


As illustrated above, CLIP consists of two encoders - an image encoder and a text encoder. The image encoder takes in an image and outputs a visual representation vector. The text encoder takes in a caption and outputs a text representation vector.

During training, these representations are optimized to be closer for matching image-text pairs, and farther apart for non-matching pairs. This is known as a contrastive loss objective.


Benefits of CLIP

There are several advantages to this approach:


  • Transferable visual representations - The image encoder learns generally useful visual concepts like objects, scenes, actions, etc. This representation can be transferred to other vision tasks through fine-tuning.
  • Scale - CLIP is trained on a dataset of 400 million image-text pairs scraped from the internet. This provides a rich supervision signal.
  • Zero-shot transfer - Remarkably, CLIP can perform zero-shot classification without seeing any examples from a dataset. It simply relies on comparing image embeddings to text prompts.
  • Robustness - CLIP shows robustness to image variations like blur, brightness changes etc. This indicates it learns high-level semantic representations.



CLIP In Action


A few examples of what CLIP can do:


  • Classify an image of a dog as a "dog" without seeing any dog images during training.
  • Correctly match the caption "A happy couple on a gondola" to the corresponding image.
  • Be fine-tuned to get 97.9% accuracy on CIFAR-10 in just 10 minutes of training.

CLIP provides simple yet powerful off-the-shelf visual representations for a variety of computer vision tasks. Its limitations include a lack of spatial understanding and bias in the training data. Nonetheless, it's an impressive demonstration of learning visual concepts directly from natural language.



Comments

Post a Comment

Popular Posts

Chest X-Ray Analysis of Tuberculosis by Deep Learning with Segmentation and Augmentation

In this paper , the authors explore the efficiency of lung segmentation, lossless and lossy data augmentation in  computer-aided diagnosis (CADx) of tuberculosis using deep convolutional neural networks applied to a small and not well-balanced Chest X-ray (CXR) dataset. Dataset Shenzhen Hospital (SH) dataset of CXR images was acquired from Shenzhen No. 3 People's Hospital in Shenzhen, China. It contains normal and abnormal CXR images with marks of tuberculosis. Methodology Based on previous literature, attempts to perform training for such small CXR datasets without any pre-processing failed to see good results. So the authors attempted segmenting the lung images before being inputted to the model. This gave demonstrated a more successful training and an increase in prediction accuracy. To perform lung segmentation, i.e. to cut the left and right lung fields from the lung parts in standard CXRs, manually prepared masks were used. The dataset was split into 8:1:1...
Post a Comment
Read more

Ocean: Object-aware Anchor-free Tracking

The paper titled " Ocean: Object Aware Anchor Free Tracking " presents a novel approach to visual object tracking that is poised to outperform existing anchor-based approaches. The authors propose a unique anchor-free framework named Ocean, designed to address certain challenges in the current field of visual tracking. Introduction Visual object tracking is a crucial part of computer vision technology. The widely utilized anchor-based trackers have their limitations, which this paper attempts to address. The authors present the innovative Ocean framework, designed to transform the visual tracking field by improving adaptability and performance. The Problem with Anchor-Based Trackers Despite their wide usage, anchor-based trackers suffer from some notable drawbacks. They struggle with tracking objects experiencing drastic scale changes or those having high aspect ratios. The anchors, with their fixed scale and fixed ratios, can limit the flexibility of the trackers, making the...
Post a Comment
Read more