Q1&W1. More evaluations on benchmarks for VL compositionality.

A: Thank you for the valuable suggestion. Following your advice, we have conducted extensive quantitative evaluations on several established benchmarks for vision-language compositionality, including SugarCrepe [T1], ARO [T2], and VL-Checklist [T3].

Furthermore, to strengthen our validation, we have included results on Winoground [T4], a standard benchmark for visio-linguistic compositional reasoning, and the recently proposed ColorSwap [T5] benchmark (introduced after 2023), which specifically targets color-object compositional understanding through word order variation.

To ensure a fair comparison, both the CLIP and SuperCLIP models used in this evaluation were trained under identical settings: a batch size of 16k, 512M seen samples, and ViT-L/16 as the backbone. The experimental results are as follows:

SugarCrepe

ARO

VL-Checklist

Winoground

ColorSwap

These experiments demonstrate that SuperCLIP consistently achieves superior performance compared to CLIP under the same training settings on nearly all benchmarks, validating its enhanced compositional capabilities. We will include the corresponding results and analysis in the revised manuscript.

[T1] SugarCrepe: A benchmark for faithful vision-language compositionality evaluation. NIPS’23

[T2] When and why vision-language models behave like bags-of-words, and what to do about it? ICLR’23.

[T3] Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations.

[T4] Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. CVPR’22.

[T5] ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation. ACL'24

Q2. Typos.

A: Thank you for your attentive reading and for pointing out this detail. We have corrected the typo as you suggested.

Q3. Ablation on Loss Weighting.

A: Thank you for your insightful suggestion. We have conducted an ablation study on loss weights λ to investigate their impact:

L_Total = L_CLIP + λ × L_Class.

To expedite the experiments without affecting the main conclusions, we used a ViT-B/16 model trained on 256M seen samples (the default in the paper is 512M). All other settings remained consistent with those described in the paper.

(Experiments with λ = 0.8 and 1.2 is still ongoing due to time and resource constraints.)

As λ increases from 0.4 to 1.0, model performance consistently improves across all tasks. When λ is increased beyond 1.0, Text-Retrieval (I2T) tasks continue to improve, while Image-Retrieval (T2I) and classification performance begin to plateau. This demonstrates the effectiveness of introducing the classification loss, and we recommend using a weight of 1.0 or higher for L_Class in practice.

Dear Reviewer,

Thank you for your valuable comments and questions. We have done our best to address them in our responses. Please kindly let us know if there are any remaining concerns or points that need further clarification.

Sincerely,

SuperCLIP Authors

Q1 More Comparisons with Multi-Class Token Based or Other Alternatives

A: Thank you for the thoughtful suggestion.

(1) More Comparisons with Fine-Grained Alignment Baselines:

As discussed in the Introduction, many existing methods addressing CLIP’s fine-grained alignment either rely on extra labeled data or introduce significant complexity. In contrast, our method requires no additional supervision, is simple, and easily scalable to CLIP-style pretraining.

That said, we acknowledge that some recent approaches—such as DetailCLIP [T1], A-CLIP [T2], and MaskCLIP [T3]—also focus on improving fine-grained alignment during pretraining only, without using extra data. Although their designs are more complex and computationally costly, comparison with them is still meaningful. To that end, we have added experiments following DetailCLIP’s setup (15M samples, batch size 4K, 25 epochs) and conducted zero-shot evaluation on standard benchmarks. The experimental results are as follows:

As shown in the results, SupeCLIP achieves the best performance across all benchmarks, with particularly clear gains on retrieval tasks, while remaining significantly simpler. In contrast, DetailCLIP—though effective—relies on heavier mechanisms such as token filtering, self-distillation, and reconstruction losses. This demonstrates that SupeCLIP offers a simple yet highly effective solution for enhancing fine-grained visual-text alignment in CLIP-style models.

(2) On Multi-Class Token Approaches:

While methods like the Multi-Class Token Transformer [T5] are effective in segmentation tasks, applying them to CLIP-style pretraining is nontrivial. Such methods require predefined class labels per image—unavailable in CLIP’s web-scale, caption-only training setup. Extending such methods to CLIP would require introducing ~50K unique class tokens—one per subword—and feeding them alongside the image into the ViT, resulting in prohibitive computational cost.

Our approach instead reuses existing caption tokens via a simple classification head, avoiding external labels or overhead—making it a more scalable and compatible solution for CLIP.

[T1] DetailCLIP: Detail-Oriented CLIP for Fine-Grained Tasks. ICLR’25

[T2] Attentive Mask CLIP. ICCV’23.

[T3] MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining. CVPR’23

[T4] SLIP: Self-supervision meets Language-Image Pre-training. ECCV’22

[T5] Multi-class Token Transformer for Weakly Supervised Semantic Segmentation. CVPR’22

Q2 Addressing Rich Text Utilization in Long Caption Scenarios

A: The performance drop when switching from short to long captions generated by LLMs/MLLMs is a common phenomenon. Prior works such as Long-CLIP [T1], VeCLIP [T2], and What-If [T3] have all reported similar trends. This degradation is likely due to the lack of data diversity in model-generated data (long captions) [T1, T2, T3].

Furthermore, our experimental results indicate that SuperCLIP is better at leveraging long-caption data than the standard CLIP model. As shown in Table 5, while both CLIP and SuperCLIP experience performance drops with long captions, SuperCLIP-B/L consistently mitigates this degradation. Specifically, SuperCLIP-B/L outperforms CLIP-B/L by 1.1/2.9 points with short captions and extends this lead to a larger margin of 3.6/3.8 points with long captions, demonstrating its superior ability to handle longer, more descriptive text.

Moreover, our Dual setting directly targets the issue raised by your question: it uses short captions for the contrastive branch and long captions for the classification branch, effectively combining their strengths. This strategy yields a 2.3/2.7-point improvement over the best-performing CLIP-B/L baseline trained on a carefully balanced caption mix.

In summary, while contrastive learning alone struggles with rich textual signals, our method explicitly addresses and alleviates this limitation. We will make this clarification more explicit in the revised manuscript.

[T1] Long-CLIP: Unlocking the Long-Text Capability of CLIP. ECCV’24

[T2] VeCLIP: Improving CLIP Training via Visual-enriched Captions. ECCV’24

[T3] What If We Recaption Billions of Web Images with LLaMA-3? ICML’25

Q3 Investigation of Limited Classification Improvement

A: Thank you for this insightful comment.

Your comment touches on an important and subtle point in CLIP-style training with long captions. Our proposed Dual setting is essentially an exploratory attempt to integrate long captions into CLIP-style training. While our formulation is new, the impact of long captions on CLIP-like models has been well studied in recent work such as Long-CLIP [T1], VeCLIP [T2], and What-If [T3]. These studies show that while integrating long captions with short captions significantly benefits retrieval tasks—due to richer semantic details—it often has a negative effect on zero-shot classification. This is because classification relies on concise, prompt-like inputs, and long captions can dilute class-specific signals.

Our Dual setting directly addresses this issue: it retains short captions for the contrastive branch to preserve strong classification performance, while the classification branch leverages long captions to enhance retrieval. As a result, although the classification gains are modest (44.5→45.1; 48.6→49.5), we observe substantial improvements in retrieval (SuperCLIP-B/L gain 2.45/2.4 points in Recall@1).

In summary, compared to other methods [T1,T2,T3] that incorporate long captions, which tend to degrade zero-shot classification performance, our approach maximizes retrieval performance without hurting classification — and even improves it.

We appreciate your suggestion and will incorporate a clearer discussion of this design rationale in our revised manuscript.

[T1] Long-CLIP: Unlocking the Long-Text Capability of CLIP. ECCV’24

[T2] VeCLIP: Improving CLIP Training via Visual-enriched Captions. ECCV’24

[T3] What If We Recaption Billions of Web Images with LLaMA-3? ICML’25

Q4 Applicability of SuperCLIP to Other Vision-Language Models

A: Thank you for your insightful and constructive comment.

First, although SuperCLIP was originally designed for CLIP, our results (Table 6) show consistent gains on recent variants like SigLIP and FLIP in both classification and retrieval tasks. This confirms its effectiveness across modern CLIP-style models.

Second, while evaluating SuperCLIP beyond the original CLIP is valuable, it’s important to note that CLIP-style models still underpin most modern VLM systems. Recent models like LLaVA [T1], MiniGPT-4 [T2], and SmolVLM [T3] all adopt CLIP-based encoders such as EVA-CLIP [T4], SigLIP [T5], and FLIP [T6], highlighting the continued importance of improving CLIP-style training approaches like SuperCLIP.

To assess SuperCLIP beyond contrastive VLMs, we integrate both CLIP and SuperCLIP (ViT-B/16, 512M samples) into the LLaVA-1.5 framework with Vicuna-7B [T7], allowing a fair comparison in the MLLM setting:

These experiments confirm SuperCLIP’s superior performance compared to standard CLIP encoders across multiple downstream datasets, particularly in visual question answering and fine-grained recognition tasks. These results demonstrate SuperCLIP’s general applicability and effectiveness in broader VLM architectures, beyond traditional contrastive models.

Third, while these results are promising, exploring SuperCLIP’s applicability to VLMs beyond the CLIP family (e.g., non-contrastive or generative VLMs) is beyond the scope of this paper. We believe this is a valuable direction for future work and plan to investigate it in more depth.

[T1] LLaVA: Large Language and Vision Assistant. NIPS’23

[T2] MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. ICLR’24

[T3] SmolVLM: Redefining small and efficient multimodal models.

[T4] EVA-CLIP: Improved Training Techniques for CLIP at Scale.

[T5] Sigmoid Loss for Language Image Pre-Training. ICCV’23

[T6] Scaling Language-Image Pre-training via Masking. CVPR’23

[T7] Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality.

Q1 Ablation on Loss Weighting.

A: Thank you for the suggestion. To examine how performance varies with different weights for L_Class, we introduce a scalar λ in the total loss formulation:

L_Total = L_CLIP + λ × L_Class.

To expedite the experiments without affecting the main conclusions, we used a ViT-B/16 model trained on 256M seen samples (the default in the paper is 512M). All other settings remained consistent with those described in the paper.

(Experiments with λ = 0.8 and 1.2 is still ongoing due to time and resource constraints.)

As λ increases from 0.4 to 1.0, model performance consistently improves across all tasks. When λ is increased beyond 1.0, Text-Retrieval (I2T) tasks continue to improve, while Image-Retrieval (T2I) and classification performance begin to plateau. This demonstrates the effectiveness of introducing the classification loss, and we recommend using a weight of 1.0 or higher for L_Class in practice.

Q2 One-Sided Supervision

A: Thank you for the suggestion. To investigate this, we have conducted a focused analysis of the text encoder in isolation using the Massive Text Embedding Benchmark (MTEB) [T1]. This evaluation allows us to more fully understand whether the one-way supervision mechanism in SuperCLIP influences the quality of textual representations.

As shown in the results below, we compare CLIP and SuperCLIP across 8 representative text embedding tasks, including semantic similarity (STSBenchmark, BIOSSES), text classification (AmazonCounter, Emotion), clustering (TwentyNews, Reddit), and retrieval (SciFact, Touche2020):

From these results, we observe that the SuperCLIP text encoder consistently achieves slightly better performance than the original CLIP across almost all tasks. This indicates that, although our classification supervision is applied only to the vision encoder, the text encoder does not degrade—in fact, it appears to benefit indirectly from the more effective supervision introduced by the classification loss, which enhances the overall training dynamics of the joint model.

However, we acknowledge that the improvement is modest and not the primary objective of SuperCLIP. In future work, we will further explore effective supervision flowing from images (as labels) to the text encoder to enable a more comprehensive, bidirectional co-optimization framework.

[T1] MTEB: Massive Text Embedding Benchmark

Q3 Role of IDF Weighting

A: Thank you for the suggestion. We have conducted an ablation comparing IDF-weighted labels with unweighted K-hot labels. To speed up experiments without impacting conclusions, we used a ViT-B/16 model trained on 256M samples (vs. 512M in the paper), keeping all other settings unchanged.

Above results show that IDF weighting improves performance across tasks, supporting its effectiveness. We will add these results in the revised version to highlight the benefit of IDF weighting.

Dear Reviewer,

Thank you for your valuable comments and questions. We have done our best to address them in our responses. Please kindly let us know if there are any remaining concerns or points that need further clarification.

Sincerely,

SuperCLIP Authors

We appreciate your thoughtful observation. This seemingly counterintuitive result can be explained as follows:

1. Synergistic Effects of Alignment Objectives

Both L_{CLIP} and L_{Class} serve to improve image-text alignment, albeit through different mechanisms. While L_{CLIP} enforces global image-text correspondence, L_{Class} provides a complementary fine-grained supervisory signal by aligning image representations with caption-derived N-hot vectors. Increasing the weight of L_{Class} does not simply reduce the effect of L_{CLIP}; rather, it introduces additional discriminative information that can further refine the joint embedding space. As a result, retrieval performance may improve due to richer and more robust visual-semantic alignments.

2. Enhanced Dual-Encoder Representations

Retrieval tasks depend on the quality of both visual and textual encoders. While L_{Class} has a clear benefit for visual representations, our additional experiments (see our response to Q2: secondary effects on the text encoder's representations) show that it also improves text encoder performance. Specifically, the text encoder trained with SuperCLIP consistently outperforms the vanilla CLIP model across various text benchmarks, including semantic similarity, text classification, clustering, and retrieval. These improvements in the text encoder directly contribute to better image-text retrieval performance.

In summary, increasing the weight of L_{Class} not only avoids harming retrieval performance but can enhance it, thanks to the complementary and mutually reinforcing effects on representation learning for both modalities. We hope this response addresses your question. Thank you again for your insightful feedback.

Q1 Post-training strategies to reduce computational cost on pretrained CLIP.

A: Thank you for the insightful comment. Due to time constraints, we have not yet fully explored post-training strategies on a pretrained CLIP model.

However, based on our previous experiments, we observed that adding the classification loss not only enhances visual representation but also accelerates convergence. This suggests that our method could potentially adapt a pretrained CLIP model to new tasks or data domains more quickly during a post-training phase. We plan to investigate this further in our future work.

Q2 Inference procedure and the role of the linear head in SuperCLIP.

A: Thank you for the question. Inference in SuperCLIP follows the standard CLIP procedure: predictions are made via cosine similarity between image and text embeddings. The added linear classification head is only used during training to enhance supervision for the vision encoder. It is not used at inference time, and its outputs are not combined with CLIP predictions. We will clarify this in the revised paper.

Q3 Equivalence between pretrained B-512M and CLIP ViT-B/16.

A: Thank you for your question. We are not entirely sure which version of CLIP ViT-B/16 you are referring to, so we will address both possibilities.

1. Compared to the open-source CLIP ViT-B/16 from OpenCLIP: We use the same model architecture and training dataset. The primary difference in accuracy stems from the number of seen samples. OpenCLIP achieves 73.5% on ImageNet-1K with 13B seen samples, whereas our model achieves 60.5% with 512M seen samples.

2. Compared to the original OpenAI CLIP ViT-B/16: While we use the same architecture, the performance difference is mainly due to different training datasets and the number of seen samples. The original OpenAI model achieves 68.6% on ImageNet-1K using the WIT-400M dataset with an undisclosed number of seen samples. Our model achieves 60.5% using the DataComp-1B dataset with 512M seen samples. Additionally, minor differences in training details, such as learning rate and batch size, may also contribute to the performance gap.

In our experiments, we ensured fair comparisons by using the same training settings, including dataset and data volume, for both our proposed SuperCLIP and the baseline CLIP model.

Q4 Limitations of using token frequency to estimate lexical information.

A: Thank you for your question.

In SuperCLIP, We adopt IDF-based token frequency as a proxy to estimate lexical information primarily to downweight common but less informative tokens. While simple, this method effectively highlights semantically meaningful words and helps guide visual attention. Our additional ablation study confirms that this strategy improves performance. To speed up experiments without impacting conclusions, we used a ViT-B/16 model trained on 256M samples (vs. 512M in the paper), keeping all other settings unchanged.

While we currently adopt token frequency as a proxy for lexical information, we plan to explore additional dimensions such as lexical diversity and semantic complexity in future work.