CLIP-Enhance: Improving CLIP Zero-Shot Classification via von Mises-Fisher Clustering

This paper proposes an approach that learns teacher-student von Mises-Fisher (vMF) mixture models on test data to improve zero-shot image classification performance of CLIP. It first reformulates the zero-shot classification into the vMF clustering problem parameterized by vMF mixture weights and concentration parameters. Then it constructs a teacher-student pipeline and performs knowledge distillation by encouraging the teacher predicted probability (based on full view of image) to be close to the student predicted probabilities (based on augmented local views).

The authors propose CLIP-Enhance to tackle the misalignment between multi-modal embeddings and the long-tailed distribution of CLIP's training dataset. They reinterpret CLIP's zero-shot classification as a clustering problem on a hypersphere using a von Mises-Fisher mixture model and use a knowledge distillation loss for optimization. Empirical results demonstrate that CLIP-Enhance improves CLIP's zero-shot classification performance across various benchmarks.

This paper proposed a new approach, named CLIP-Enhance, to improve the zero-shot ability of CLIP. CLIP-Enhance re-interprets the CLIP zero-shot classification as a clustering problem on a hypersphere using a von Mises-Fisher mixture model. Besides, CLIP-Enhance is optimized to improve the alignment of text and image vectors. Experiments on multiple benchmarks show the effectiveness of the proposed method.

The paper introduces CLIP-Enhance, a method to improve CLIP's zero-shot classification by addressing two key issues: misalignment between text and image embeddings and the long-tailed distribution of the training data. CLIP-Enhance reinterprets zero-shot classification as a clustering problem on a hypersphere using a von Mises-Fisher (vMF) mixture model. This model optimizes both the alignment and concentration of embeddings through self-supervised learning, inspired by the DINO framework. The empirical findings reveal a remarkable enhancement in the performance of zero-sample classification across multiple benchmarks, manifesting superior management of data distribution disparities and robustness under limited data circumstances.

This paper proposes a vMF-based teacher-student knowledge distillation tuning framework to enhance the CLIP zero-shot performance. Technically, it proposes two strategies: (1) closed-form approximation of normalization term of vMF distribution to model variance statistics of different categories; (2) knowledge distillation with confident view selection for stabler fine-tuning. The experimental results demonstrate the superior performance of CLIP-Enhance compared with some baselines on multiple downstream zero-shot classification benchmarks.

This paper propose a method named CLIP-Enhance to improve CLIP zero-shot classification. CLIP-Enhance addresses two challenges: (1) the misalignment between image and text embeddings, and (2) the long-tailed distribution of CLIP's training data. The authors re-fomulate zero-shot classification as a von Mises-Fisher clustering problem. Then an self-supervised learning method is proposed to optimize both alignment and concentration. Experiment results show that the proposed method can improve zero-shot classification accuracy.

Updated Zero-Shot Baseline Comparison

In response to feedback from multiple reviewers, we have expanded our zero-shot baseline comparison to include additional fine-grained classification datasets. Table 1 in our paper has been updated to reflect these results, covering the following datasets:

• OxfordIIITPets: A dataset with classes from cat and dog families.
• Stanford Cars: A dataset for classifying various car models.
• Caltech101: A dataset with 101 object categories.
• UCF101: A dataset focused on classifying human actions.

Analyzing Kappa initialization

we provide results on the impact of different choices for the initial concentration parameter κ0. Results from κ0 = 500 and κ0 = 5000 are provided in Table 6 along with the results of the value we used everywhere else in the paper, namely κ0 = 3000. Building on the discussion in § 4.3 regarding the choice of κ0, we further illustrate in Figure 3 how increasing κ0 reduces the contribution of the normalization term to the logit score. This observation provides insight into the observed performance degradation: as the model increasingly relies on the normalization component rather than the image data for prediction its performance declines, around the κ0 = 2000 the contribution reaches close to zero and we observe that model accuracy drop near the values obtained by our model without Vmf formulation as discussed in section § 5.2. Additionally, we observed that larger values of κ0 result in numerical instability during training in our setup, leading to a significant drop in performance.

CLIP alignment Redefined

CLIP's training objective aims at a relative alignment of captions embeddings with embeddings of their corresponding image embeddings [1]. More precisely, for a random batch from CLIP's training dataset, the symmetric cross entropy is minimized when 1): every caption embedding is more are aligned (larger cosine similarity) with its corresponding image embedding than with any other image embedding 2) every image embedding is more are aligned (larger cosine similarity) with its corresponding caption embedding (positive pair) than with any other caption embedding (negative pairs). Moreover, since “a temperature parameter which controls the range of the logits in the softmax is directly optimized during training” -- which would appear to take approximately the value $\exp(\tau)=100$ [2], which is the maximal value allowed during training to prevent instability [1] -- a small difference between the cosine similarity of a positive pair compared to negative pairs leads to cross entropy loss which is roughly zero. For instance, for the training batch size ($b=32,768$), a positive cosine similarity of $0.5= \cos(60^\circ)$ for negative cosine similarities of $0.225\approx \cos(77^\circ)$ obtains a cross entropy loss (when logits are scaled by $100$) which is smaller than the smallest representable number (with Pytorch, Float32).

Hence, CLIP's training does not aim at aligning text embeddings and their corresponding images in the sense that the angle between them is small (cosine almost $1$). On the contrary, in CLIP's multimodal embedding space, the angle between a text representation and its corresponding image representation can be as large as $60^\circ$ (cosine $0.5$) as long as negative pairs are less aligned, namely with an angle larger than $77^\circ$. Therefore, we conclude that the quality of CLIP's multimodal embeddings really resides in their RELATIVE alignment, not in the actual alignment of positive pairs, an observation that is also supported by [3].

Furthermore, we consider that this relative alignment highly depends on the training distribution. When leveraging these representations for a downstream classification task, we aim at a relative alignment of text-based class reference embeddings with respect to the image distribution. A specific balanced classification task like CIFAR10 is expected to differ greatly from the long tail distribution of CLIP's training dataset, resulting in a poor relative alignment. Our approach aims at adapting in an unsupervised manner the initial text-based class reference embeddings to improve their relative alignment on the image dataset. By endowing each class reference with a concentration parameter and formulating the classification problem as a von Mises--Fisher mixture model, our approach achieves this goal as reflected by the improved accuracy.

[1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

[2] CLIP's official git repository. API. url:https://github.com/openai/CLIP.

[3] Liang, Victor Weixin, et al. "Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning." Advances in Neural Information Processing Systems 35 (2022): 17612-17625.

We have updated the paper to better explain the questions raised by the reviewers

Dear Reviewers,

Thank you for your efforts in reviewing this paper. We highly encourage you to participate in interactive discussions with the authors before November 26, fostering a more dynamic exchange of ideas rather than a one-sided rebuttal.

Please feel free to share your thoughts and engage with the authors at your earliest convenience.

Thank you for your collaboration.

Best regards, ICLR 2025 Area Chair

This paper explores clustering in unsupervised (zero-shot) classification using CLIP and introduces a vMF-based teacher-student knowledge distillation framework. While the clustering approach shows potential, the work suffers from notable experimental shortcomings, as highlighted by the reviewers. These include incomplete results (e.g., NA in Table 1), missing key baselines, and limited dataset coverage. Additionally, reviewers flagged the term "zero-shot classification" as potentially inappropriate, suggesting "unsupervised classification" as more accurate. Following the author-reviewer discussion, all reviewers recommended rejection. The Area Chair concurred with their assessment and also recommended rejection.

Reject