CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling

W1: The paper claims that compared to the standard CLIP which often encodes inputs in a very coarse-grained manner, CLIP-MoE is able to encode richer information. I am curious about CLIP-MoE’s performance in fine-grained image classification tasks, such as on the Stanford Cars, FGVC-Aircraft, Flowers102 and so on.

W2: From the experimental results on LLaVA-1.5, replacing the CLIP in the LLaVA model with CLIP-MoE does not significantly improve performance across the five benchmarks; in fact, there are even some cases of performance decline. Additionally, the test results of the MLLM on benchmarks like MME and POPE fluctuate considerably. Are there other benchmark results, or perhaps reasoning cases for MLLM, that can further demonstrate CLIP-MoE’s ability to encode richer information in MLLM?

A1&A2: Due to server migration, we will try to provide more evaluation on fine-grained image classification and MLLM before the discussion period ends.

W3: Why did you choose 4 experts? Is there a particular reason for this choice, and is there any observed relationship between the number of experts and CLIP-MoE’s performance?

A3: We chose 4 experts as it is the upper limit that allows training on 8×A100 GPUs while maintaining a reasonable batch size (8×100). Future work will explore configurations with more experts and evaluate their relationship to performance.

W1: Firstly, there is a mislabeling in the paper. In Section 5.5, Table 1 presents the results of the text-to-image retrieval experiment on the COCO dataset. The CLIP-MoE model trained on the Recap-DC dataset achieved an @10 value of 79.4, which is lower than the Upcycling's 79.9. However, the value of 79.4 is erroneously bolded, which may lead to confusion. It is recommended that this be corrected to avoid misinterpretation of the results.

A1: We sincerely apologize for the incorrect bolding in the table. Thank you for bringing this to our attention. We will ensure this error is corrected in the final version to avoid any confusion.

W2: Secondly, it might be beneficial to include an analysis of the number of experts utilized in the method. Currently, the CLIP-MoE employs 4 experts to balance performance and computational cost. However, conducting experiments to demonstrate how varying the number of experts—either increasing or decreasing—affects both performance and computational efficiency could provide more compelling evidence for the chosen configuration.

A2: Thank you for your insightful suggestion. We selected 4 experts as it represents the upper limit that allows training on 8×A100 GPUs while maintaining a reasonable batch size (8×100). Exploring configurations with more or fewer experts and analyzing their relationship to performance and computational efficiency is a promising direction for future work.

W3: Thirdly, while Multistage Contrastive Learning is described in the paper as enabling experts to learn different aspects of features—such as color, texture, and shape—this explanation seems to lack supporting experimental results. To strengthen this claim, it would be beneficial to provide concrete evidence, such as demonstrating that models trained at different stages exhibit distinct capabilities on datasets characterized by these specific features. Alternatively, offering a more nuanced explanation of the effectiveness of Multistage Contrastive Learning would help clarify its role and prevent potential misunderstandings.

A3: MCL aims to generate experts with distinct expertise through iterative clustering and training. At a high level, the clustering results of each stage reflect the feature distributions learned so far, while the MCL loss encourages the model to learn new distributions distinct from the existing ones. We do not impose assumptions on the clustering outcomes, and as noted in Figure 1, terms like "color," "shape," and "texture" are metaphorical representations of abstract features. In practice, clustering results may not always align with human-intuitive categories. However, as shown in the original MCL paper[1], experiments on synthetic datasets demonstrate that clustering can yield interpretable groupings, validating MCL's effectiveness. To further support that MCL can produce experts with different focus, we will provide the evaluation of the obtained experts on the MMVP[2] benchmark later, which contains ten manually defined attributes.

W4: Lastly, I think the explanation in the ablation study section to be unclear. This section mentions training a CLIP-MoE model with two experts, while Table 4 compares a fully trained CLIP-MoE with a model that has not undergone the first and second stages of MCL training. First, is the latter the CLIP-MoE model with two experts mentioned in the paper? Additionally, based on my understanding, since the fully trained CLIP-MoE in the paper includes 4 experts, it should undergo 4 stages of MCL training. The results related to the CLIP-MoE in Table 4 are also obtained after 4 stages of MCL training. Why is a model that has not gone through the first and second stages of training used for comparison?

A4: The ablation study aims to evaluate how much of the performance gain stems from our proposed MCL initialization. CLIP-MoE (w/o S1, S2) integrates the original dense CLIP checkpoint with FFN fine-tuned on ShareGPT4V (MCL-Stage0). In contrast, CLIP-MoE incorporates FFNs from the original dense CLIP checkpoint, MCL-Stage0, MCL-Stage1, and MCL-Stage2. This demonstrates the incremental benefits of MCL stages, which further validates the MCL initialization in DMU. We acknowledge that the explanation in the current version of the paper could be clearer, and we will revise this section in the final version to improve clarity and address these concerns.

[1]Zhang, J., Lan, X., Qu, X., Cheng, Y., Feng, M., & Hooi, B. (2025). Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning. In European Conference on Computer Vision (pp. 35-52). Springer, Cham.

[2]Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., & Xie, S. (2024). Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9568-9578).

Thank you for your thoughtful feedback and suggestions throughout the review process. I wanted to kindly draw your attention to the additional experiments we’ve included in the latest revision, which address the points raised and further strengthen our findings. Your insights on these additions would be highly valuable to us.

W1: The motivation behind this work—that CLIP often encodes inputs in a very coarse-grained manner—is neither novel nor particularly compelling. Additionally, the paper lacks an in-depth analysis of why this issue arises and does not clearly explain how the proposed method addresses it. The introduction could benefit from further refinement, as it currently seems to have chosen a weak motivation simply to justify applying MoE to CLIP.

A1: Thank you for your valuable comments. We believe that the coarse-grained encoding in CLIP can be attributed to feature suppression in contrastive learning. Feature suppression occurs when, due to competing features, the model prioritizes certain features while ignoring others. Although the reasons for this phenomenon are complex and covered in prior works[1,2,3], our primary focus is not on feature. Thus, we provided limited detail on this aspect in the introduction. Our proposed CLIP-MoE is designed not only to mitigate feature suppression but also to introduce several additional advantages. Through our Diversified Multiplet Upcycling (DMU) approach, CLIP-MoE flexibly updates pre-trained CLIP models with high-quality data, maintains the original model's performance, improves performance on tasks related to new data, and avoids the need for training from scratch. We acknowledge that the introduction could be further refined to include more background and clearly articulate our motivation.

W2: The rationale for using Multistage Contrastive Learning (MCL) is not fully convincing. The paper suggests that clustering can simulate grouping raw data by attributes, such as color or shape. However, as clustering is conducted entirely within CLIP’s feature space, this outcome is not necessarily assured. The authors could strengthen their argument by visualizing each expert’s focus on distinct details, using methods such as t-SNE. Additionally, the routing analysis and case study presented in the experiments are too general to effectively illustrate MCL’s advantages. Besides, a more targeted comparison that highlights each expert's independent downstream task performance would provide clearer insights into the effectiveness of MCL.

A2: MCL aims to generate experts with distinct expertise through iterative clustering and training. At a high level, the clustering results of each stage reflect the feature distributions learned so far, while the MCL loss encourages the model to learn new distributions distinct from the existing ones. We do not impose assumptions on the clustering outcomes, and as noted in Figure 1, terms like "color," "shape," and "texture" are metaphorical representations of abstract features. In practice, clustering results may not always align with human-intuitive categories. However, as shown in the original MCL paper[1], experiments on synthetic datasets demonstrate that clustering can yield interpretable groupings, validating MCL's effectiveness. To further support that MCL can produce experts with different focus, we will provide the evaluation of the obtained experts on the MMVP benchmark[4] later, which contains ten manually defined attributes.

W3: The zero-shot tasks for CLIP lack sufficient comparisons to other state-of-the-art methods. This work mainly compares against Long-CLIP, which was primarily designed for sequence extension rather than enhancing CLIP’s zero-shot performance in downstream tasks.

A3: Our primary claim is that DMU can convert a dense CLIP checkpoint into a sparse MoE architecture with performance improvements. Thus, our comparisons mainly focus on the base dense checkpoint. To further validate our method, we plan to extend our experiments to additional dense checkpoints in future work. We chose Long-CLIP as a baseline for two reasons: (1) it also enhances a pre-trained CLIP checkpoint without training from scratch, similar to our approach; and (2) it allows for fair comparisons by using the same training data. Many state-of-the-art CLIP models rely on superior data, making direct comparison challenging. However, these models can serve as new base checkpoints for CLIP-MoE in future work. Additionally, we include CLIP-ViT-L-16-HTxt-Recap as a performance reference in Section 5.6.

W4: The design of the ablation study is somewhat unclear. The sole comparison (w/o S1, S2) introduces too many variables, such as the number of experts, whether the experts were fine-tuned, and whether fine-tuning occurred on the original dataset or the clustered data (assuming clustering aims to mine hard negatives). The authors should design more distinct ablation studies to clearly demonstrate the contribution of each component.

A4: The ablation study aims to evaluate how much of the performance gain stems from our proposed MCL initialization. CLIP-MoE (w/o S1, S2) integrates the original dense CLIP checkpoint with FFN fine-tuned on ShareGPT4V (MCL-Stage0). In contrast, CLIP-MoE incorporates FFNs from the original dense CLIP checkpoint, MCL-Stage0, MCL-Stage1, and MCL-Stage2. This demonstrates the incremental benefits of MCL stages, which further validates the MCL initialization in DMU.

Q1: In the MCL process, is each FFN initialized from the previous stage’s FFN or from the pre-trained CLIP FFN?

AQ1: Each FFN is initialized from the pre-trained CLIP FFN (dense checkpoint). This follows the methodology outlined in MCL[1]. Initializing from the pre-trained checkpoint allows each stage to learn new and distinct features without being constrained by previously learned properties in previous stages. Retaining prior properties could hinder the model's ability to acquire new information.

Q2: Why does upcycling appear less effective than direct fine-tuning in Table 1, yet w/o S1 S2 in Table 4, it outperforms both methods significantly, such as on Flickr I2T?

AQ2: As mentioned above, CLIP-MoE (w/o S1, S2) integrates the original CLIP dense checkpoint with FFNs fine-tuned on ShareGPT4V (MCL-Stage0). This differs from both upcycling and direct fine-tuning. Upcycling is primarily designed for larger-scale continuous training settings. In our experiments, the dataset may not have been sufficient to fully support upcycling’s ability to learn enough diversified experts, which could explain why it does not significantly outperform direct fine-tuning in this context.

[1]Zhang, J., Lan, X., Qu, X., Cheng, Y., Feng, M., & Hooi, B. (2025). Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning. In European Conference on Computer Vision (pp. 35-52). Springer, Cham.

[2]Bleeker, M., Yates, A., & de Rijke, M. (2022). Reducing Predictive Feature Suppression in Resource-Constrained Contrastive Image-Caption Retrieval. arXiv preprint arXiv:2204.13382.

[3]Robinson, J., Sun, L., Yu, K., Batmanghelich, K., Jegelka, S., & Sra, S. (2021). Can contrastive learning avoid shortcut solutions?. Advances in neural information processing systems, 34, 4974-4986.

[4]Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., & Xie, S. (2024). Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9568-9578).

W1: The zero-shot image classification results should be more diverse. Only including ImageNet, ImageNet-O, ImageNet-V2, CIFAR-10, and CIFAR-100 is not sufficient. Please refer to the CLIP benchmark [1]. I believe that including ImageNet, ImageNetV2, ImageNet-A, ImageNet-R, ImageNet-Sketch, and ObjectNet datasets is essential for a more comprehensive evaluation.

A1: Our comparison on zero-shot image classification follows the protocols in Recap-1B[1] and Long-CLIP[2]. Furthermore, as MLLMs are gaining prominence, recent works[3,4] suggest that testing within the MLLM framework may provide a more comprehensive evaluation. We appreciate your valuable feedback and will include evaluations from the CLIP benchmark in the final version of the paper. We will try to provide more evaluation on zero-shot image classification before the discussion period ends.

W2: I’m interested in the performance on the MM-Vet benchmark.

A2: Thank you for raising this point. Due to server migration, we will try to provide an evaluation of the MM-Vet benchmark before the discussion period ends. A full evaluation will be included in the final version of the paper.

[1]Li, X., Tu, H., Hui, M., Wang, Z., Zhao, B., Xiao, J., ... & Xie, C. (2024). What If We Recaption Billions of Web Images with LLaMA-3?. arXiv preprint arXiv:2406.08478.

[2]Zhang, B., Zhang, P., Dong, X., Zang, Y., & Wang, J. (2025). Long-clip: Unlocking the long-text capability of clip. In European Conference on Computer Vision (pp. 310-325). Springer, Cham.

[3]Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., & Xie, S. (2024). Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9568-9578).

[4]Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., ... & Xie, S. (2024). Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860.