Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling

Thank you for your thorough review

W1. Weaker models are more specialized Thank you for pointing us to these works, which we agree are closely related to ours. We will add all of them to the related work section. Regarding more precise comparisons, we would like to point out the following:

• [2] focuses on model averaging to approximate an ensemble method with a single one for out-of-distribution generalization but does not study CLIP backbones or adaptive ensembling like ours.
• [3] investigates instance-level performance differences in pre-trained language models, which is tangentially related but distinct from our focus on vision-language models and CLIP-specific complementarity.

Due to the above, the rest of our response concerns [1], which we believe is more closely related to our proposals. We would like to bring attention to the following distinctions.

• Training Setup: [1] uses timm models trained on ImageNet with arbitrary settings, meaning that methods are not necessary using the same loss function or regularizer. The selection of loss functions impacts the inductive biases of the model, which could potentially help the models to have more diversity. In contrast, we use a Foundational model, CLIP backbones, trained on the same dataset in a contrastive learning approach using the same loss function. In this way, we are sure that the diversity of predictions comes from the backbone. We believe that the results provided by [1] are not sufficient to conclude that the differences in classification performance are due to either utilizing a weaker architecture or the selection of a loss function.
• Complementarity Analysis: First, we would like to clarify that our analysis is about diversity in the correct predictions of different backbones. Based on these results, then propose a mechanism to exploit their diversity through complementarity. While [1] conducts pairwise teacher-student analysis, in our paper, we analyze nine backbones simultaneously. Although we agree that the pairwise analysis by [1] is informative, we think it does not provide evidence to show if a given backbone is better for a specific type of data. In contrast, our analysis does show that some backbones are more resilient to certain types of image variations than others, Figure 7.
• Methodology: The knowledge distillation approach in [1] requires access to a large training set (e.g. ImageNet), a data augmentation procedure, and partitioning of data into sets for continual learning. These partitions are built from a complementarity analysis, and the idea is to perform knowledge transfer to mitigate catastrophic forgetting. For example, [1] performs knowledge transfer for the set of data where one backbone is better than the other and retention on the set where they would like to maintain performance. Additionally, this approach requires retraining the model in order to distil the knowledge. In contrast, our method does not require data partitioning nor backbone retraining: it preserves the model's original knowledge, and it is shown to be effective in few-shot scenarios. Reaching up to 120% (Figure 2) relative improvement.
• Dataset Breadth: [1] examines four datasets (ImageNet, CUB, StanfordCars, Caltech256), whereas we evaluate diversity and complementarity across 21 datasets. Additionally, we specifically show the diversity of zero-shot CLIP backbones is higher for certain datasets.
• Performance improvement: [1] obtains improvements of nearly 5% for the ConvNeXt network after the knowledge distillation process. In contast, our approach can directly leverage the backone prediction diversity in order to improve the performance of our best Convolutional network ResNet50x64 by a similar 5%.
• Contemporaneity: We believe [1] is contemporary to our work.

Thank you for your clarifications. Most of my concerns have been addressed. Following is a clarification on the prior work mentioned in W1

• While [3] might not be exactly the same, they tackle the same underlying problem that different models have different instance level predictions regardless of the model size. Hence, even if this work is specifically for vision-language models, the writing should reflect that prior work in this area of instance level analysis been done. In particular, statements like "This non-obvious insight is the key motivation for the ensemble method" are misleading, but can be revised to include reference to other works that have done the same for other domains, but not for vision language models.
• [2] might not be working with vision language models, but they also tack a similar problem - models that are finetuned on different subsets of data (i.e., have different specializations) can be combined to perform well on ood data for any of the domains, i.e., helps with generalization.
• I understand that there are differences in the way [1] and this work handles analysis of complementary data. This work is fairly recent - it was published in ICLR2024 - but I believe it is old enough for it to be discussed (contemporary work according to ICLR2025 guidlines is within 4 months prior to submission deadline). I would not expect a comparison with the work and believe the method is different in terms of how the ensembles handles complementary data. However, the limitations section needs a revision to include a refernece to [1], since the fusion of backbones during training has been done. This particular point is not a major weakness, and a citation and comparison of differences should be sufficient.

I don't mind increasing my score to 6 given that the paper is revised to include the new results and references to prior works.

Follow up questions

In Appendix G, lines 1153-1154 mention that the degradation stops when 8 training samples are used , but the degradation appears to be present when number of samples used is 32. This seems counter intuitive. Can the authors comment on the reason for this?

Thank you for your insightful feedback. We greatly appreciate your recognition of the importance of systematically evaluating foundational models like CLIP. As you pointed out, with the increasing availability of these models, understanding their strengths, weaknesses, and complementarity is critical for advancing the field. We agree that exploring novel aggregation, ensembling, and merging methods is vital to fully harness the potential of these foundation models, and we are excited to contribute to this ongoing exploration with our work. Your feedback further strengthens the motivation behind our study. Moreover, We deeply appreciate your recognition of the systematic nature of our approach and the value of our proposed NLC. We also appreciate your comments on the large-scale experiments and evaluations.

W1, Q1 Thank you for your observation. We agree that the large performance improvement seen in Figure 8 could be influenced by the injection of the supervised signal. However, our method outperforms other parametric ensembles such as SL. This demonstrates that the benefits of our approach go beyond just the supervised signal, and it remains an improvement over standard ensemble techniques.

Moreover, The decrease in performance improvement from 9.1% to 1.6% between Figure 8 and Figure 9 can be attributed to the fact that linear probing reduces the diversity in predictions across backbones. By relying on a fixed linear head, the model tends to converge towards a more uniform prediction across backbones, limiting the space for improvement. As a result, the gains observed with the NLC method are smaller in this setup. We will clarify this point in the manuscript to highlight the role of reduced diversity in linear probing. We would like to refer the reviewer to Tables B1 and D1, where we see linear classifiers reaching performance and thus leaving relatively smaller room for improvement compared to zero-shot backbones. For example, in the case of Eurosat, the best backbone reaches 97.1%, and the Oracle reaches 99.6%.

W2 Injection of the supervision: Thank you for your question! Yes, the details about the injection of the supervision signal using one labelled example per class can indeed be found in Appendix G and Table G.1. We will make sure to clarify this process further in the main text to ensure the methodology is fully understandable.

W3 Minor things:

• Page 3, Line 134: You are correct—the first encoder is indeed the language encoder (φ_l), and the second is the vision encoder (φ_v). We will clarify this distinction in the manuscript.

• Figure 4: We appreciate your comment. We will ensure that the dashed lines are properly labelled or annotated to clarify the setup they correspond to in the figure.

Q2 MoE Missed Thank you for your question. The MoE (Mixture of Experts) results are not shown in Figure 8 because the focus there is on comparing the proposed NLC approach with other ensemble methods, and MoE was not included in that specific experiment. Regarding your assumption, the linear heads used in the experiments shown in Figure 9 are indeed part of the broader context of how MoE-like structures might work within our framework. We will clarify this in the manuscript to avoid any confusion and ensure the setup is fully understood.

Q3 MLP with one hidden layer Thank you for pointing this out we will fix this in the final version. We meant a MLP with one hidden layer.

Thank you for your thorough review.

• Q1.1 How the model is trained? As explained in Lines 301, 366, and 401, we use the training set of each dataset to train NLC. In Section 4.2, where we train a linear classifier on top of the logits, we use 90% of the training set for the linear classifier and reserve 10% for training the NLC. For the Few-shot adapter method, we follow the Tip-Adapter setting and use the same validation split L455.

• W2, Q1.2 and Q1.3 Does it generalize well to novel classes? and Novel compositions of concepts? Thank you for your suggestion. NLC is trained using the image representation of the target dataset. Then, it combines the logits such that creates a new one that is tailored to the target dataset. Thus, using NLC that has been trained on a dataset A in a dataset B is possible as long as the classes are fixed. Due to the limited time during the rebuttal process, we are unfortunately unable to provide results now, but are commited to doing so for the camera-ready version.

• Q 2.1 Diversity on Vanilla Supervised Learning We evaluated Oracle performance on MAE backbones (ViT-Base-16, ViT-Large-16, and ViT-Huge-14) using the official implementation and shared fine-tuned weights on ImageNet-1k.

  Our results indicate a 3% improvement in performance with the Oracle compared to the best individual backbone, reinforcing our argument of complementarity across backbones.

• Q2.2 Is diversity always preferred when selecting models in a per-sample way, no matter whether using CLIP or not? We believe diversity is key to exploiting the strengths of different models. Figure 2, illustrates the correlation between diversity and performance improvement. When diversity is low—meaning all models predict the same label—there’s limited room for improvement. Therefore, diversity is always preferred, but the challenge lies in how to exploit it effectively. In the context of CLIP, leveraging diverse backbones helps capture different aspects of the data, but for other models, careful management of diversity is crucial to avoid inefficiencies.

• Q3.1 Does the user need to load all CLIP models to the device? No, the user does not need to load all CLIP models to the device. In the proposed method, only the selected backbones for a given input need to be loaded. This is done dynamically, allowing for efficient use of resources. The Neural Logit Controller (NLC) can adjust and combine the outputs of different backbones as needed, without requiring all models to be loaded simultaneously on the GPU.

Q3.2 Comparison of Memory and time cost with other ensembles

We thank the reviewer for bringing up this point, which we believe is a valuable insight to add to our paper. In terms of computation and time cost SL and MoE is similar to ours, NLC, in the sense that they do not require to run all the backbones simultaneously. Altenatively, one can also run the backbones sequentially and store the feature representations and logits in memory. Thus, all of these models add the timecost of the forward-pass of N backbones. NLC is an MLP with one hidden layer that receives the concatenation of the features and outputs a scalar per backbone to the ensemble. In contrast, SL is a vector of the size of the number of backbones in the ensemble. Therefore, the memory requirements for SL are smaller compared to NLC. Conversely, MoE is marginally more expensive than NLC in terms of memory consumption. Please see the Table below.

Thank you for your thorough review.

W1 More Evaluation Please check section E and K in the supplemental material, where we present the complementarity of CLIP backbones on the image-to-text and text-to-image retrieval tasks. Table K.3 shows patterns of complementarity across backbones comparable to those in the classification setting. On Flick30k, the upper bound shows possible improvements of 20 percentage points on both text and image retrieval. On MSCOCO, >25pp. We selected Flick30k and MSCOCO because they are standard benchmarks for image-to-text and text-to-image retrieval, covering diverse and challenging scenarios.

Note that we conducted extensive experiments across various tasks, including image classification, few-shot adaptation, out-of-distribution evaluation, and image-to-text and text-to-image retrieval, utilizing a total of 27 datasets.

W2 Ensemble on logit Thank you for the observation. While ensemble methods using late fusion are not new, our approach introduces adaptive fusion through instance-specific calibration, setting it apart from static methods. This addresses the limitations of traditional ensembles. Thus, we believe the Neural Logit Controller (NLC) introduces a significant advancement over traditional ensembling methods. Specifically:

• Adaptive Fusion: Unlike standard ensemble methods (e.g., averaging or voting), NLC dynamically learns to weight the contributions of different backbones using a one-layer MLP. This allows for instance-specific calibration, rather than static fusion strategies. We show that is needed to exploit the diversity in backbones predictions.

• Efficient Use of Diversity: NLC explicitly exploits diversity in backbone predictions to enhance performance while maintaining computational efficiency when combined with Cascade. It achieves higher accuracy with fewer GFLOPs compared to naive ensembles.

• Minimal Supervision: NLC achieves substantial improvements using as little as one labeled example per class for tuning. This low-resource requirement differentiates it from other adaptive techniques.

These innovations showcase that while our method builds on established ensemble techniques, it addresses their limitations and offers practical advancements for real-world deployment. To the best of our knowledge, our method is compared against a comprehensive selection of alternative resembling methods, so we would appreciate it if the reviewer could point out more specifically in which sense our work offers limited novelty when "Compared with the previous ensemble".

W3. Test-Time:

Thank you for pointing this out. While all ensemble methods do increase test time and introduce additional steps in the pipeline, their performance gains often justify these complexities, particularly in applications where accuracy and robustness are critical, e.g., healthcare (PCAM) or autonomous systems (Eurosat). Additionally, many real-world pipelines already involve post-processing or model refinement, making such methods more feasible in practice. That said, the trade-offs depend on the specific use case.

W4. More Training Shots Analysis: We think the Reviewer may be referring to the results in Table 1, where it is possible to see that the performance of NLC via k-shots on ImageNet-1k grows more slowly compared to our baselines. Regarding this observation, we would like to point out that we do not observe this is the case for our other studied datasets, as shown in Table F.3, F.4, F.5, F.6, G.1 and C.1. Moreover, the Table C.1 shows that when using the full training set to tune the NLC, we observed an average performance improvement of 9.3%, compared against a 2.9% when using 16 shots, suggesting that overall the NLC generalizes better with more data. However, the amount of improvement will depend of on prediction diversity across backbones, as shown in Figure 2.