Classification Done Right for Vision-Language Pre-Training

W1. Other popular few-shot and zero-shot classification benchmarks.

A1. Thanks for the advice. We have added more evaluation benchmarks, including 10-shot classification on ImageNet-1k, pets and cars, and zero-shot classification on 8 more datasets.

10-shot classification
We follow the setting of Cappa[1]. For each dataset and model, we run 3 times, and report the mean results and variance in the following table. Our method surpasses CLIP on IN-1K and Pets by clear margins with improvements of 1.6 and 2.2 points, while being comparable with CLIP on Cars (92.6 v.s 92.7).

zero-shot classification
Following LiT[2], we train a text encoder on Datacomp-1B by keeping the image encoder locked. In this way, we are able to perform zero-shot classification. We tested the model on the following 10 datasets. Superclass beats openCLIP on seven of the datasets. It is worth noting that SuperClass uses a ViT large model with patch size 16, while the baseline method adopts a patch size of 14. This makes the superiority of SuperClass over openCLIP even clearer.

[1] Tschannen, Michael, et al. "Image captioners are scalable vision learners too." NeurIPS 2024. [2] Zhai, Xiaohua, et al. "Lit: Zero-shot transfer with locked-image text tuning." CVPR. 2022.

W2. The application in text-to-image generation systems.

A2. Thanks for the advice. Text-to-image generation is indeed a particularly important application of vision-language models. However, due to the limited time available for rebuttal, we were unable to attempt and train this task. We will explore this issue further in the future.

W3. The effect of synthetic captions

A3. Thanks for the nice advice. We use the code provided by LaCLIP[1] https://github.com/LijieFan/LaCLIP for investigation. We compare SuperClass against CLIP with ViT-B/16 following the setting of LaCLIP. The numbers of CLIP and LaCLIP are directly borrowed from the paper. As shown by the results in the following table, Superclass can also benefit from rewritten captions, and the improvement is even greater than that of CLIP (+1.1 vs. +1.6 in zero-shot and +1.2 vs. 1.9 in Linear probing). The possible reason is that the rewritten captions transform the sentence structure but keep the major objects and subjects intact, indirectly enhancing the weight of those visual-related words.

[1] Fan, Lijie, et al. "Improving clip training with language rewrites." NeurIPS 2024.

Q1. The effect of removing stop-words

In Table 7, removing stop-words leads to a decrease in classification accuracy, for example, the linear probing accuracy decreases from 76.0 to 75.7 (-0.3), and the zero-shot accuracy reduces from 61.7 to 61.0 (-0.7). Therefore, we conclude that stopwords could help the vision encoder. This might be explained by the fact that stop-words also carry some useful visual information. For example, 'she', 'her', and 'him' can indicate a person's gender; 'on', 'off', 'above', 'below', 'up', and 'down' can indicate operational status or position; '@' is likely to indicate an email address.

Stop words and punctuation are listed below. ['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]

{'{', '+', '(', '$', '}', '!', '%', '\\', '<', ';', '|', ']', '"', "'", ',', '&', '=', ')', '_', '^', '~', '#', '@', '.', '[', '*', '?', ':', '/', '>', '-'}

Q2. Online IDF

A2. To implement online IDF, we set up two global variables to track the number of seen samples (N) and the occurrence count of each subword. As training proceeds, these values will gradually approach the true word frequency of the entire dataset. The advantage of online IDF is its ease of use, allowing the code to be directly transferred to new datasets. We also conducted experiments to compare recognition accuracy using online IDF and offline IDF. The zero-shot accuracy with online IDF drops by about 0.3, and the linear probing accuracy drops by about 0.1.

Q3. The analysis of classification results

A3. We evaluated the performance of CLIP and Superclass on the 1000 classes in ImageNet. Among them, Superclass performs better in 502 classes, while CLIP outperforms in 270 classes, and they perform equally well in the remaining classes. We also extracted the top 10 classes with the largest performance differences and displayed them in the table below. We do not find any obvious patterns, but it is possible that Superclass and CLIP can complement each other. This is an interesting direction for future research.

At its core, this paper introduces a new way to pre-train open-world Vision-Language foundational models, it's simple and elegant and seems to have maintained the zero-shot capabilities that CLIP offers.

But I believe that using such a model might not be suited for generation, where one may want a text encoder to infer the relationships between words, i.e. "A dog over a cat" vs "A cat over a dog".

Additionally, I believe that such a technique can also be used to clean the pertaining data of VLMs, a problem that LAION has faced in the past [1].

[1] https://cyber.fsi.stanford.edu/news/investigation-finds-ai-image-generation-models-trained-child-abuse

Thanks for the comprehensive answers to my questions, I will increase the score for soundness in my review.

W1: The robustness of the proposed method to different model types remains unclear.

A1. Thank you for the valuable advice. To evaluate the robustness of our proposed method across different model types, we selected two representative convolution-based networks: ResNet50 and ConvNext-Tiny. We compare SuperClass against CLIP for ImageNet zero-shot (LiT) and linear probing classification, as shown in the table below. All experiments were conducted with a batch size of 16k and 1.28B seen samples.

We observe that SuperClass surpasses CLIP in all settings by a clear margin, ranging from 1.64 to 2.91. These results demonstrate that the superiority of SuperClass over CLIP is robust across different model architectures.

W2. The robustness of softmax loss on a broader range of vision-language tasks

A2. Thank you for the advice. We further compare the sofrmax loss against different losses on various vision-language tasks. We selected three model sizes (ViT-S/16, ViT-B/16 and ViT-L/16) and trained them on Datacomp-1b with 512 million seen samples. The detailed results are presented in the following table. We observe that softmax is the best-performing loss function across different vision-language tasks and model sizes.

W3. The scaling properties of loss function

A3. Due to the limited time for rebuttal, we only ablated the effect of different losses with different model sizes. As the model size increases, softmax consistently achieves the best accuracy while also demonstrating equal or better scalability compared to other losses.

W1: Comparisons with other related SOTA works

A1. Thanks for the advice. We have added more SOTA methods for comparison, including self-supervised learning methods (MoCov3, Dinov1&v2, MAE, BEiT, CAE) and weakly-supervised methods (CatLIP, Cappa). Additionally, we have evaluated more downstream tasks and datasets, such as semantic segmentation on ADE20K and instance segmentation on COCO. Please refer to our response to Reviewer1(aefc) and Reviewer4(bBvT) for more results.

Linear probing on ImageNet-1K

Instance segmentation and semantic segmentation

Results of instance segmentation are obtained by using Mask R-CNN on COCO with an input resolution of 1024×1024. Semantic segmentation results are obtained by using UperNet on ADE20K with an input resolution of 512×512.

VLM downstream tasks

W2: The proposed approach might not be capable of doing multi-label classification or zero-shot classification ...how it can comprehensively show effectiveness over CLIP and other vision-language models.

A2. Our paper focuses on the issue of vision encoder pretraining, specifically whether the pre-trained model can perform better in vision-language models. Therefore, the native zero-shot capability may not be the primary property we consider. However, we can use Lock Image Tuning(LiT)[69] to equip our pre-trained vision backbone with zero-shot classification and retrieval capabilities. After LiT, we also could do zero-shot segmentation using MaskCLIP[a]. The experimental results show that SuperClass could achieve much better performance on the Pascal context and COCO stuff dataset.

[a] Extract Free Dense Labels from CLIP (ECCV 2022)

W3: using the subwords as labels does not correspond to any visual concept

A3. Firstly, it is important to note that a single object class can be mapped to one or multiple subwords. Here is a detailed explanation of how our method works in both scenarios:

1. Single Subword Mapping:

• When a class is mapped to a single subword, the process is similar to traditional classification tasks. The model learns to associate the subword with the corresponding visual patterns.
• This is akin to standard classification where each class is represented by a unique label, and the model learns the association between the label (subword) and the visual features.

2. Multiple Subword Mapping:

• When a class is mapped to multiple subwords, our optimization objective is to maximize the co-occurrence probability of subwords that belong to the same class.
• This means the model learns to associate multiple subwords with the corresponding visual patterns, effectively capturing the relationship between subwords and the visual concept they represent.

W4: The results for the zero-shot and linear probing in Tab1 and Tab 2 of the main paper are different for the same model.

A4. Because the models are trained with different #seen samples. In Table 1 and Table 2, we use the ViT-L/16 as the backbone. The models in Table 1 are trained with 12.8B seen samples. In Table 2, we study the effect of different seen samples. The models are trained with 128M, 512M, and 1.28B seen samples, respectively.

Q4 Compared with these "bag-of-word classification" pre-trained methods

A4. We have included the comparison with other classification-based methods, like CatLIP in the subsection Word-level tokenizer vs. Subword-level tokenizer. The word-level tokenizer is used in CatLIP [46], which carefully selected approximately 40,000 "gold labels" from the datacomp-1B dataset. Aside from the tokenizer being different, all models are trained under the same settings. The results of Table 4 show that with the increasing size of the model, the subword-level tokenizer gradually outperforms the word-level tokenizer, whether in classification tasks or vision & language tasks. We also provide the results of finetuning on ImageNet-1k in the below Table. The superclass could achieve better performance than CatLIP.

Q5 GPU usage

A5. This is a typo. What we meant to say is all experiments were carried out on 80G A100 GPUs. We will fix it in the revised version.

Q1 Zero-shot capacity.

A1. We thank the reviewer for triggering the discussion on zero-shot abilities of CLIP and our model. Honestly, our model does not come with trivial zero-shot image-text retrieval usage. However, we can enable this behavior by levering LiT [a], which learns a text encoder by keeping the image encoder fixed. As shown by the following table, we test the zero-shot ability of our model on 10 datasets. Superclass beats openCLIP on seven of the datasets. It is worth noting that SuperClass uses a ViT large model with patch size 16, while the baseline method adopts a patch size of 14. This makes the superiority of SuperClass over openCLIP even clearer.

Moreover, we'd like to emphasize that there is another increasingly more important application of the CLIP model. Specifically, CLIP is predominantly the default vision encoder in existing vision-language models (e.g., LLaVA, BLIP). We show that when combined with a large language model, SuperClass is able to substantially improve performance over CLIP. Please refer to Table 2 and Table 3 on the paper for details.

[a] Zhai, Xiaohua, et al. "Lit: Zero-shot transfer with locked-image text tuning." CVPR. 2022.

Q2 The baseline should be other weakly-supervised methods or self-supervised methods like MAE, etc

A2. Thanks for the suggestion. We have added more SOTA methods for comparison, including self-supervised methods (MoCov3, Dinov1&v2, MAE, BEiT, CAE) and weakly-supervised methods (CLIP, Cappa). Additionally, we have evaluated more downstream tasks and datasets, such as semantic segmentation on ADE20K and instance segmentation on COCO. Finally, Following the LLaVA setup, we combine frozen CLIP models, self-supervised models, and SuperClass models with the pre-trained Vicuna-V1.5-7B and perform downstream tasks. The experimental results demonstrate that the proposed method could achieve better than self-supervised ViT pre-training methods, like Dinov2, and weakly-supervised methods, like CLIP.

Linear probing on ImageNet-1K

Instance segmentation and semantic segmentation

Results of instance segmentation are obtained by using Mask R-CNN on COCO with an input resolution of 1024×1024. Semantic segmentation results are obtained by using UperNet on ADE20K with an input resolution of 512×512.

VLM downstream tasks

Q3 Some arguments about previous so-called "bag-of-word classification"

A3. Thanks for pointing it out. We will rewrite the description here. CatLIP [46] is a concurrent work. We have already cited and compared it in the paper.

Thanks for your reviews and feedback.

Q1: So far, we only see zero-shot performance on classification tasks. It seems that CLIP training style shows broader zero-shot ability in different downstream tasks.

A1: We have evaluated zero-shot retrieval on COCO dataset and zero-shot segmentation on Pascal context and COCO stuff (see more details in the response to Reviewer W9bX). The experimental results show that Superclass could achieve competitive performance compared to CLIP.

Besides, we also present results of linear probing and fine-tuning on ImageNet-1K. Compared to CLIP, SuperClass outperforms openCLIP on seven out of ten zero-shot classification datasets. Our method achieves a 1.1% higher accuracy in IN-1K linear probing (85.0 vs 83.9). To further demonstrate the transfer ability of our method, we conducted semantic segmentation on ADE20K and instance segmentation on COCO. Moreover, we provide experimental results on various vision & language tasks after integrating with large language models. The results show that our method outperforms CLIP.

We hope that by showcasing our competitive performance in zero-shot retrieval, zero-shot segmentation, linear probing, fine-tuning, semantic segmentation, instance segmentation, and various vision & language tasks, we can address the reviewer’s concern of "only see zero-shot performance on classification tasks".

Q2: From the comparison, I do not see the superiority of the proposed pre-trained method compared to self-supervised learning, especially since the proposed method is actually weakly-supervised.

A2: Compared to the current SOTA self-supervised model DINOv2, our method achieves a 0.5% higher accuracy in IN-1K linear probing (85.0 vs 84.5). Although SuperClass has seen more samples, our method is very simple and straightforward. DINOv2 adopts a dual-tower structure and adds a bunch of bells and whistles as shown in Table 1.

Furthermore, we would like to remind Reviewer aefc the very important comparison on vision and language tasks. Our method outperforms DINOv2 in 7 out of 9 tasks, and the overall score is significantly higher than that of DINOv2 (Dinov2 54.66 vs. Ours 58.94). Please note that the Dinov2 used here is distilled from the ViT-giant model. These improvements are significant.

We thank the reviewer bBvT for the comment, "At its core, this paper introduces a new way to pre-train open-world Vision-Language foundational models," and Reviewer W9bX for noting, "this is a new pretraining style for vision backbones rather than a variant of CLIP," our proposed new pretraining method is distinctly different from previous pretraining approaches. The differences from other most closely related classification-based pretraining methods are also evident: our method does not require manually curated "golden labels" and directly uses tokenized raw text as supervision. In the paper subsection "Word-level tokenizer vs. Subword-level tokenizer," we demonstrate through experiments that our method has better performance and scalability. As Reviewer 3a7k mentioned, "Compared to previous classification-based pretraining methods, the proposed approach demonstrates greater practical applicability."

Overall, we believe we have provided a thorough response to Reviewer aefc's comments and hope that Reviewer aefc will review our feedback in light of the perspectives that align their rating with other reviewers.

Dear authors:

Thank you for you response.

• Q1

---

Could you please tell me where the zero-shot 46.9 COCO Text-to-Image result comes from? I check the paper of CLIP[1], the zero-shot image retrieval result is 37.8.

• Q2

---

• CLIP is born in 26 Feb 2021. Many other VLM pre-trained under language supervision emerged, e.g., EVA-CLIP[2]
• Self-supervised pre-training and weaky-supervised methods are also hot. According to the provided information, the proposed method uses 10x data (12.8B), and it is close to DINOv2(1.92B). If we talk about "performance and scalability", DINOv2 ViT-g/14@448 achieves 86.7% linear probing results on ImageNet-1K.

Overall, the baselines are far from strong. The reviewer does not know why we need such a method when we have many other powerful and versatile pre-trained VLMs/self-supervised large models/weakly supervised large models.

I still retain my score at this time.

Best,

Reviewer aefc

[1] https://arxiv.org/pdf/2103.00020

[2] https://arxiv.org/abs/2303.15389

Thank the Reviewer aefc for the feedback.

We have addressed all Reviewer aefc's questions in our previous responses, including but not limited to

• performance and scalability

  Line #212 subsection "Data Scaling results Line" and #221 subsection "Model scaling results" in the paper

  The results of Zero-shot classification, VLM downstream tasks ... in https://openreview.net/forum?id=Hd2EOwKItm&noteId=JxjyAuZo5M (Rebuttal by Authors)

  Few-shot classification in A1 https://openreview.net/forum?id=Hd2EOwKItm&noteId=1hHvBn6szv (Response to Reviewer bBvT)

• comparisons and differences with CatLIP[46]

  A3 in https://openreview.net/forum?id=Hd2EOwKItm&noteId=JxjyAuZo5M (Rebuttal by Authors) "CatLIP [46] is a concurrent work. We have already cited and compared it in the paper. "

  Line #69 and Line #233 subsection "Word-level tokenizer vs. Subword-level tokenizer" in the paper

  Fine-tuning results in A4 https://openreview.net/forum?id=Hd2EOwKItm&noteId=OOhKhswjjl (Rebuttal by Authors Part 2)

  https://openreview.net/forum?id=Hd2EOwKItm&noteId=KeV1R5CtDE (Rebuttal by Authors Part 3) "The differences from other...classification-based pretraining methods are also evident...manually curated "golden labels"..."

• Performance comparisons with CLIP and Dinov2

  The results of Zero-shot classification, VLM downstream tasks ... in https://openreview.net/forum?id=Hd2EOwKItm&noteId=JxjyAuZo5M (Rebuttal by Authors)

  The results of 10-shot classification in A1 of https://openreview.net/forum?id=Hd2EOwKItm&noteId=aVM6gFdSHR (Response to Reviewer bBvT)

  ...

• Different data from CLIP paper

  https://openreview.net/forum?id=Hd2EOwKItm&noteId=ogRsx4SgwE (Rebuttal by Authors Part 4) "Regarding Q1, we utilized the open-sourced checkpoint [1] from paper [2]...This serves as a stronger baseline compared to the original OpenAI CLIP."

We do not intend to engage in repetitive responses and hope to strengthen and solidify our work based on the reviewers' valuable feedback.

We further thank all the reviewers.

We thank reviewer aefc for their timely response.

Regarding Q1, we utilized the open-sourced checkpoint [1] from paper [2], which employs ViT-Large as its backbone and is trained with Datacomp-1B and 12.8 billion seen samples. This serves as a stronger baseline compared to the original OpenAI CLIP.

Reviewer aefc commented "The reviewer does not know why we need such a method when we have many other ... pre-trained ... large models" (VLMs/self-supervised/weakly supervised).

It seems the reviewer suggests that foundational modeling research might be redundant when the absolute performance are weaker compared to a "large model" pretrained in a systematic way. The reviewer appears to prioritize "higher numbers" even when comparing in an apple-to-orange setting. While we acknowledge that a big model with higher numbers is often advantageous from a downstream users' perspective, we believe that a scientific foundational model researcher would in favor of careful ablations rather than chasing higher numbers with biggest models.

For instance, the reviewer argues that SuperClass cannot demonstrate better "performance and scalability" than DINO v2, citing that the DINO v2 ViT-g 448 model, pretrained on a customized dataset at 448 resolution - superior to our largest ViT-L 224 model at 224 resolution.

We appreciate the reviewer's observation that our ViT-L 224 resolution model is less powerful compared to a ViT-g 448 resolution model. It appears that the reviewer has overlooked various systematic differences - unintentionally or for some other unknown reason - comparing scientific papers in an apple-to-orange manner.

The reviewer aefc further suggested "Overall, the baselines are far from strong. "

The baselines we compared here are CLIP trained with DataComp [2] and DINOv2 [3], which are important milestone papers accepted at NeurIPS 2023 and TMLR 2024, respectively. For some unknown reason, the reviewer believes CLIP, DataComp and DINOv2 are far from strong. We kindly respect the reviewer's false claim.

We would like to wish the reviewer the best, and politely disagree with the reviewer's comments and rating.

We further thank other reviewers for their professional, scientific, objective reviews.

[1] https://huggingface.co/laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K

[2] DATACOMP: In search of the next generation of multimodal dataset, NeurIPS 2023.

[3] DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024.

Thank the authors for the response.

1. "performance and scalability"

---

Well, the authors say that your methods have better "performance and scalability" in https://openreview.net/forum?id=Hd2EOwKItm&noteId=KeV1R5CtDE A2. So,

• The reviewer provides examples that previous methods have better "performance and scalability".
• The proposed method use 10x data, but is close to DINOv2 with the same parameters.
• The reviewer does not even mention similar work CatCLIP[1], which also trains CLIP in a classification way. It also demonstrates better "performance and scalability".

2. It seems the reviewer suggests that foundational modeling research might be redundant when the absolute performance are weaker

---

The authors are trying to mislead readers.

• The paper does not propose something novel, CatCLIP[1] already does the classification pre-training job.
• The performance/scalability of the paper is no better than recent literature [2,3].

The proposed method does not demonstrate its necessity when there are many other strong self-supervised/weakly-supervised/language supervised (may belong to the weakly-supervised class) methods, so it is reasonable that the reviewer thinks this work may not be necassary for the community.

3. Different data from CLIP paper

The authors do not answer this question. https://openreview.net/forum?id=Hd2EOwKItm&noteId=vqnfjpW7aT Q1.

Could you please tell me where the zero-shot 46.9 COCO Text-to-Image result comes from? I check the paper of CLIP[1], the zero-shot image retrieval result is 37.8.

4.

Finally, as an independent reviewer, the reviewer thinks he can have his own opinion and rating about the paper.

Best,

Reviewer aefc

[1] https://arxiv.org/pdf/2404.15653

[2] https://arxiv.org/abs/2303.15389

[3] DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024.