Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models

We thank Reviewer u8xU for the thoughtful and constructive feedback on our work. The following is our response to the comments and questions in the review.

---

Preserving the original CLIP space in the alignment with DINOv2

We would like to clarify that our objective function for the alignment task is designed to minimize the changes to the CLIP embedding while aligning it with DINOv2. We agree that we have to make some modifications to the CLIP model to have a higher alignment with the DINOv2. Meanwhile, to minimize the changes to CLIP in alignment, our proposal is to match only the kernel similarity matrices of the CLIP and DINOv2 embeddings, which only focus on the relative positioning of the points. In other words, the proposed objective function does not require the absolute position of CLIP embedded point to be identical to that of DINOv2, but the goal is to only obtain similar similarity scores between the two embeddings. In this sense, our alignment method attempts to mitigate the changes to the CLIP model as much as possible.

---

Relationship with L2SP

Our regularization minimizes the $L_2$ difference between visual embeddings of the fine-tuned and original encoders, directly preserving visual-text alignment (as demonstrated in Proposition 3.2). We note that this is different from L2SP that penalizes the $L_2$-norm of parameter changes. While L2SP effectively prevents overfitting to the target domain in transfer learning, our direct feature-based penalty aims to maintain compatibility with the text encoder while enhancing fine-grained visual capabilities. We will include the discussion in the revised paper.

---

Clarification on training loss

The choice of the polynomial kernel function is inspired by the literature of generative model evaluation and the KID metric [Bińkowski et al., 2018], which is an established metric to measure the similarity between two sets of representations. In the standard KID evaluation, the kernel function is commonly chosen to be polynomial degree 3. This choice has been commonly adopted by other well-known studies in the literature, e.g. [Stein et al., 2023], [Kang et al., 2023]. We have also drawn the kernel loss in Fig. 4 of the Appendix, which remains at a reasonable magnitude throughout training.

Furthermore, we would like to clarify that the choice of poly-3 kernel function is only to have a reasonable similarity score between already embedded samples, and the structure of the kernel similarity matrix is supposed to be governed by the choice of embedding (DINOv2 in our work), not by different kernel functions. Considering this, we think our alignment method is flexible and allows the application of different embedding models. For example, we have already attempted to align with MLCD embedding in Appendix C.3.

---

Additional baseline on imagenet fine-tuning

We thank the reviewer for pointing this out. We agree that comparison with a vanilla ImageNet-fine tuned CLIP (without kernel-based alignment) will shed light on how important the role of kernel-based alignment is in the improvement. To address this comment, we tested the following baseline: fine-tuning the CLIP visual encoder with contrastive loss, where the paired text for each image is “This is a photo of {class name}”. While this approach improves ImageNet zero-shot classification, it deteriorates the generalization of the model to other datasets.

If the reviewer has any specific fine-tuning method in mind, we would be happy to run and report the results during discussion.

---

Ablation on the dataset size

Thank you for the suggestion. We have conducted ablation studies using 25% and 50% of the imagenet for fine-tuning. The results show that our alignment fine-tuning consistently improves the scores even with 25% of the samples. We will include the results and analysis in the revised paper.

---

Regarding projection heads

We would like to clarify that we do not use any projection heads when computing the kernel; it is computed directly on the representations produced by the models. Meanwhile, in Table 1, we also show experiments on aligning two models with different representation dimensions (e.g., CLIP-B and DINOv2-L), where we observe similar improvements.

---

Reference

• Bińkowski et al. "Demystifying MMD GANs." ICLR (2018).
• Stein et al. "Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models." NeurIPS (2023).
• Kang et al. "StudioGAN: a taxonomy and benchmark of GANs for image synthesis." TPAMI (2023).

We thank Reviewer 3gYu for the thoughtful and constructive feedback on our work. The following is our response to the comments and questions in the review.

---

Significance of the improvements

We would like to highlight that our alignment fine-tuning achieves consistent improvements on several kinds of vision-centric benchmarks. We improve the performance of CLIP by 1.28%, 1.34%, 5.51%, and 2.96% on zero-shot classification, image-text retrieval, linear probing, and MMVP-VLM benchmark, respectively. For the MLLM benchmarks, we focus on consistent gains across multiple benchmarks and aim to highlight that our kernel-based approach provides a principled and efficient way to enhance model performance, even under resource-constrained settings. Meanwhile, following the suggestions of Reviewer ue7L, we have conducted experiments with standard two-stage training of LLaVA, which shows even greater improvements (under the response for Reviewer ue7L: The explanation for additional PEFT). Overall, we think our proposed alignment fine-tuning provides considerable improvements with very low demand for data and computational resources.

---

Comparison with AM-RADIO models

We thank the reviewer for recommending the related work. AM-RADIO [Ranzinger et al., 2024] distills knowledge from multiple teachers into a student model, which achieves superior performance on multiple downstream tasks. However, the original AM-RADIO model is trained with DataComp-1B, which is composed of 13B samples. This training process incurs a computational cost equivalent to the pre-training of CLIP, making it resource-intensive. In contrast, our proposed kernel-based alignment focuses on achieving strong accuracy and generalizability through fine-tuning on relatively small datasets, such as ImageNet-1k, for only a few epochs. To provide a fair comparison under our settings, we trained AM-RADIO on ImageNet-1k using CLIP and DINOv2 as teacher models (both with ViT-L-14 as backbones) and another ViT-L-14 as the student model. We evaluated its zero-shot classification performance and observed that AM-RADIO does not generalize well to out-of-distribution datasets under this setup. In contrast, our alignment method demonstrates both strong performance and superior generalizability, even with limited data and computational resources. We will include the results and discussion in the revised version of the paper.

---

Reference

• Ranzinger et al. "Am-radio: Agglomerative vision foundation model reduce all domains into one." CVPR (2024).

We thank Reviewer ue7L for the thoughtful and constructive feedback on our work. The following is our response to the comments and questions in the review.

---

Comparison with DINOv2 encoder

We would like to clarify that our numerical evaluation of the fine-tuned CLIP using alignment with DINOv2 performs experiments of the following two types of image-centric models:

1. Text-image cross-modality tasks such as zero-shot classification and image-text retrieval: We note that DINOv2 is not applicable to these tasks, since they require a text encoder that is not existing in DINOv2. This is why we did not report the performance of DINOv2.
2. Image-only tasks: In these experiments, our goal is to highlight that aligning CLIP with image-modality expert DINOv2 enhances CLIP's visual performance. Per the reviewer's suggestion, we include DINOv2's performance on linear probing and probing bench. Results show DINOv2 significantly outperforms CLIP on purely perceptual tasks (e.g., spatial reasoning, locality probing), with alignment successfully narrowing this gap. For tasks requiring both visual and multimodal abilities (e.g., caption recognition), the aligned model leverages strengths from both models, outperforming each individual model.

In both settings, we aim to show that aligning with modality experts improves multi-modal models, that is why we aligned CLIP with DINOv2 rather than the opposite, and focused on the comparison with the original CLIP. We will include the additional comparison with DINOv2 in the revised text.

---

The explanation for additional PEFT

Thank you for pointing this out. Since we have fine-tuned the visual encoder, we also attempted to finetune the downstream component so that we can achieve even a stronger improvement. Because our alignment method ensures the visual encoder does not deviate significantly from the original model, fine-tuning the downstream component can be efficiently performed using PEFT. Following the reviewer’s suggestion, we further experimented with standard two-stage training, and compared the CLIP encoder w/wo alignment. In our experiments, we observed that this led to even greater performance and confirmed that the alignment improves the MLLMs performance.

Due to the time limit, we could not finish the experiment with DINOv2. However, [Karamcheti et al., 2024] has shown that DINOv2’s MLLM performance does not match that of CLIP, which may due to its image-only pre-training protocol.

---

Explanations on “projection” and “feature” methods

The term “projection” in Table 1 refers to the case with a trained projection head to project the DINOv2 features into the CLIP feature space. Specifically, we freeze the CLIP and DINOv2 encoders, and train a projection head $P(\cdot)$, to minimize the ${\mathbb{E}}[\Vert P(g(I_i)) - f(I_i) \Vert^2]$. Then we use $P(g(I_i))$ as the visual representation to conduct zero-shot classification. The term “feature” refers to aligning the CLIP and DINOv2 representations directly, subject to a linear transformation. Formally, we fine-tune the CLIP encoder to minimize the following objective:

$\min_{\theta, \mathbf{R}} {\mathbb{E}}[w\Vert f_\theta(I_i) - \mathbf{R}g(I_i)\Vert_2^2 + \Vert f_\theta(I_i) - f_{\theta_0}(I_i)\Vert_2^2]$

We will make it more clear in the revised text.

---

Effects of the regularization term

Regarding the reviewer's suggestion, we would like to note that there is an ablation study of the regularization term in Fig. 3(b). Removing the regularization term results in a significant drop of the zero-shot classification performance, which shows its importance for maintaining the visual-text alignment. For a more detailed evaluation, we have run a new experiment where we plugged the encoder fine-tuned without regularization into LLaVA without fine-tuning the LLM. The results below also indicate the performance drop. We will include the additional results in the revised paper.

---

Reference

• Karamcheti et al. "Prismatic vlms: Investigating the design space of visually-conditioned language models." ICML (2024).