NOVIC: Unconstrained Open Vocabulary Image Classification: Zero-Shot Transfer from Text to Image via CLIP Inversion

You can try out NOVIC yourself on your own images using the Hugging Face Spaces Live Demo! Alternatively, you can easily get started inferencing NOVIC yourself on a local GPU, using the pre-built NOVIC Docker image and the associated compact instructions.

NOVIC is an innovative uNconstrained Open Vocabulary Image Classifier that uses an autoregressive transformer to generatively output classification labels as language. Leveraging the extensive knowledge of CLIP models, NOVIC harnesses the embedding space to enable zero-shot transfer from pure text to images. Traditional CLIP models, despite their ability for open vocabulary classification, require an exhaustive prompt of potential class labels, restricting their application to images of known content or context. To address this, NOVIC uses an object decoder model that is trained on a large-scale 92M-target dataset of templated object noun sets and LLM-generated captions to always output the object noun in question. This effectively inverts the CLIP text encoder and allows textual object labels to be generated directly from image-derived embedding vectors, without requiring any a priori knowledge of the potential content of an image. NOVIC has been tested on a mix of manually and web-curated datasets, as well as standard image classification benchmarks, and achieves fine-grained prompt-free prediction scores of up to 87.5%, a strong result considering the model must work for any conceivable image and without any contextual clues.

At the heart of the NOVIC architecture is the object decoder, which effectively inverts the CLIP text encoder, and learns to map CLIP embeddings to object noun classification labels in the form of tokenized text. During training, a synthetic text-only dataset is used to train the object decoder to map the CLIP text embeddings corresponding to templated/generated captions to the underlying target object nouns. During inference, zero-shot transfer is used to map CLIP image embeddings (as opposed to text embeddings) to predicted object nouns. The ability of the object decoder to generalize from text embeddings to image embeddings is non-trivial, as there is a huge modality gap between the two types of embeddings (for all CLIP models), with the embeddings in fact occupying two completely disjoint areas of the embedding space, with much gap in-between.

Despite being an image classification model, NOVIC is trained exclusively on synthetic textual data in the form of matching caption-object pairs. These are primarily generated using a multi-object prompt templating strategy based on the Object Noun Dictionary. The remaining caption-object pairs were one-time generated using an LLM based on the same dictionary of nouns. The complete data required for training is available on Papers With Code as the NOVIC Caption-Object Data.

In order to properly evaluate a totally novel generative image classification model like NOVIC, three new open vocabulary datasets were constructed and annotated. These datasets are generic, i.e. not specific to NOVIC, and are referred to as the OVIC Datasets. Refer to Papers With Code and the WACV Paper for more information. The three new evaluation datasets are as follows: