Robust Fine-tuning of Zero-shot Models via Variance Reduction

Q1: Weaknesses on computation costs.

A1: For efficiently performing k-NN search, we use Faiss library [11], which can perform billion-scale similarity search with GPUs. For our ImageNet experiments, the inference speed of kNN search for a single image is averaged 3.2 ms. For our ImageNet experiments, the storage size for the ZSF set features is 289 MB. While this does require additional storage compared to some traditional methods, we believe it is a manageable overhead given the benefits in robustness and performance that our VRF framework provides.

Q2: Limited implication of the theoretical result.

A2: In non-ensemble-based fine-tuning methods, the model can be seen as using fixed coefficients, essentially $g_{ft}(x) = 1$. Since this is not the optimal strategy for balancing ID and OOD performance, the generalization error for non-ensemble-based fine-tuning is expected to be relatively larger compared to our VRF method, which optimizes these coefficients.

Q3: Lack of possible comparison with more advanced fine-tuning.

A3: Since our VRF framework is orthogonal to the fine-tuned models, we use the [4][5] as the fine-tuned models and report results below.

Q4: Could the proposed method also be applied to the weight-space ensemble?

A4: Yes, our method can be applied to weight-space ensemble models. To do this efficiently, we first generate a set of weight-ensembled models with varying coefficients $\alpha=[0.1,0.2,..,0.9]$. For each sample, we calculate the weight function and select the cached WSE model with the closest $\alpha$. The results for CIFAR10 -> STL10 validate that our VRF approach is effective for weight-space ensemble methods as well.

Q5: How sensitive is the proposed method against varying numbers of training samples that determine the quality of the zero-shot failure set?

A5: Our proposed method is robust to variations in the number of training samples in the zero-shot failure set. To validate this, we randomly downsampled the zero-shot failure set by 10%, 20%, and 50%. The results showed only minor fluctuations of ID and OOD performances. This demonstrates that our method maintains its effectiveness even with a reduced zero-shot failure set.

Q1: Applying VRF to other robust fine-tuning methods.

A1: To further demonstrate the versatility and effectiveness of our VRF method, we applied it to another robust fine-tuning method, FTYP, and conducted experiments on ImageNet and its variants. As expected, our VRF framework further improves OOD performance without sacrificing ID accuracy, reinforcing its applicability across different fine-tuning strategies. We will include these additional results in the revised version.

Q2: What happens if the value of is increased further? Would it still outperform OSE and WSE baselines?

A2: For the hyperparameter $a$, we chose a range less than 1.6 based on the nature of the ZSF set of ImageNet, where nearly 99% of validation sample distances were less than 1.6. Exploring larger values of $a$ (i.e., $a>1,6$) is not necessary because it results in almost all predictions being assigned to the fine-tuned models, reducing the contribution of the zero-shot models. Since the features are L2-normalized, the distance d(x) is bounded between [0, 2], we further tested values $a = 1.7, 1.8, 1.9$ and observed that OOD performance decreased as expected, with the fine-tuned model dominating the ensemble contribution.

For the hyperparameter $b$, as shown in Figure 6(a), we noticed that increasing $b$ beyond 1 leads to a decrease in ID performance. Therefore, we limited our exploration to $b \leq 1$ to maintain a balance between ID and OOD performance.

Q3: Could the proposed VRF method also extend to "Linear classifier" models?

A3: Yes, using linear classifiers can indeed reduce the inference costs, as the image encoder only needs to be processed once, effectively halving the costs compared to VRF's standard approach. To explore this, we trained a linear classifier based on CLIP-ViT/16 models and compared the performance with WSE and our VRF. The results confirmed that our VRF framework effectively addresses the ID-OOD trade-offs, achieving higher OOD performance without compromising ID accuracy. This demonstrates that VRF can extend to linear classifier models, providing a versatile and effective solution for robust fine-tuning.

Q4: In Figures 3, does "Ensembling with varying coefficient " refer to OSE? It would be helpful to also provide the curve for WSE.

A4: Yes, the "Ensembling with varying coefficient" refers to OSE. We have also evaluated the ID and OOD performance with varying $\alpha$ for WSE. The curves are plotted in the attached PDF.

Q5: Table 7 shows the results of exploring values for OSE. It would be beneficial to provide the same results for WSE as well.

A5: Thanks for your suggestions. We provide the WSE results with varying $\alpha$ values as below and we will add them in the revised version.

Q6: It would be advantageous to visualize a scatter plot across a broad range of a and b values.

A6: We appreciate the suggestions. We have visualized the trade-off curves in the attached PDF.

I have updated my rating to 5 based on the authors' assurance that the issues will be addressed. Wishing you the best of luck.

Q1: VRF requires identifying and saving the zs classification error samples for subsequent use, which presents certain limitations.

A1: For our ImageNet experiments, the storage size for the ZSF set features is 289 MB. While this does require additional storage compared to some traditional methods, we believe it is a manageable overhead given the benefits in robustness and performance that our VRF framework provides.

Q2: The description of Step 2 is unclear: is it calculating the distance to the k-th nearest sample in , or is it clustering the representations in first and then calculating the distance to the -th cluster center?

A2: We apologize for the unclear description. In Step 2, we calculate the distance to the k-th nearest sample in the ZSF set. We do not perform any clustering of the representations. We will clarify this in the revised version of the paper.

Q3: Is the representation v of x calculated by ft or zs?

A3: We use the representation from the fine-tuned model, as mentioned in Line 120: "we collect its feature representation from the fine-tuned model."’

Q4: For each sample, recalculating and sorting the distance to incurs additional inference costs. How much extra computational cost does this introduce?

A4: To address the efficiency of k-NN search, we leverage the Faiss library [11], which is optimized for large-scale similarity searches using GPUs. In our ImageNet experiments, the inference speed of the k-NN search for a single image is approximately 3.2 milliseconds, demonstrating that our approach can be executed efficiently even with large datasets.

Q5: Since line 278 indicates that different values of have a minor impact on the model, why not use the nearest sample to calculate the distance? What is the reason behind choosing the -th sample for distance calculation?

A5: We avoid using the nearest sample to reduce the impact of label noise in the training set. If the nearest sample is mislabeled, the distance calculation could be unreliable. By selecting the $k$-th nearest sample, we mitigate this risk, as the likelihood of all $k$ nearest samples being mislabeled is low.

Q1: There should be a discussion on the theoretical and practical computational complexity regarding space and time.

A1: To address the efficiency of k-NN search, we leverage the Faiss library [11], which is optimized for large-scale similarity searches using GPUs. In our ImageNet experiments, the inference speed of the k-NN search for a single image is approximately 3.2 milliseconds, demonstrating that our approach can be executed efficiently even with large datasets.

For our ImageNet experiments, the storage size for the ZSF set features is 289 MB. While this does require additional storage compared to some traditional methods, we believe it is a manageable overhead given the benefits in robustness and performance that our VRF framework provides.

We will include the discussion of the computational complexities in the revised version.

Q2: The proposed distance calculation using the k-th nearest neighbor is borrowed from the previous work on OOD detection [24], and there is a lack of explanation for the specific choice of this metric. It seems that other metrics like the averaged k nearest neighbor distance discussed in [24] may also work in VRF.

A2: We appreciate the reviewer’s observation and would like to clarify our rationale for choosing k-NN distance as the metric for density estimation in our VRF framework:

• Training-Free and Easy Implementation: The k-NN method is training-free, making it simple to implement without requiring additional trained models. This aligns with our goal of maintaining a lightweight and efficient approach.
• Non-Parametric Nature: k-NN is a non-parametric method, meaning it does not make any assumptions about the underlying distribution of the data. This flexibility contrasts with methods like those in [16, 22], which assume the feature space follows a Gaussian distribution, making k-NN a more robust choice across different datasets.
• Strong ID and OOD Performance: We selected k-NN based on its demonstrated ability to effectively capture the relationship between distance measures and performance metrics (Acc_ft/Acc_zs) in both ID and OOD settings. This consistent performance across different scenarios reinforces our confidence in k-NN as a reliable metric.

While we acknowledge that the averaged k-NN distance could also be a viable alternative, our experiments show that it yields similar results (82.3 ID and 61.8 OOD on ImageNet benchmarks) to the k-NN distance. However, the averaging step introduces additional computational complexity, albeit slight. To keep our method succinct and efficient, we chose to use k-NN distance. We will include the discussion in the revised version.

Q3: It is unclear whether the “best ID and OOD accuracy” refers to the comparison with other existing methods or only considers the performance of the proposed method under different hyperparameter settings. Judging from Figure 3 (a.1), VRF does not always achieve the “best ID and OOD accuracy” compared with the peak ID/OOD accuracy of the existing ensembling method.

A3: The statement “best ID and OOD accuracy” refers to the comparison with other existing methods. Our framework is designed to be orthogonal to existing ensembling techniques, meaning it can be applied in conjunction with them to enhance performance. Traditional ensembling methods often face trade-offs where optimizing for ID performance can degrade OOD performance, and vice versa. However, our VRF framework is intended to mitigate these trade-offs by simultaneously optimizing for both ID and OOD accuracy.

In Figure 3 (a.1), the best OOD performance of conventional ensembling is 98.5%, while VRF achieves 98.4%. Although VRF is marginally lower by 0.1%, it is important to note that this difference is within a very small margin, particularly as both approaches are nearing the upper performance limit (close to 100%). Thus, we consider VRF's performance to be effectively on par with the best OOD accuracy, while also offering strong ID performance without requiring a trade-off.

Q4: To what extent is VRF less efficient than existing ensembling methods?

A4: The key distinction between VRF and conventional ensembling methods lies in their objectives. While traditional ensembling methods primarily focus on enhancing in-distribution (ID) performance, our VRF framework is designed to simultaneously improve both ID and out-of-distribution (OOD) performance. This broader objective naturally introduces additional computational overhead. However, as mentioned earlier, we utilize the Faiss library for efficient k-NN search, significantly reducing the time complexity. For example, in our ImageNet experiments, the k-NN search for a single image takes approximately 3.2 milliseconds, which we consider manageable given the overall performance benefits.

Q5: why is k-th nearest neighbor distance selected.

A5: We avoid using the nearest sample to reduce the impact of label noise in the training set. If the nearest sample is mislabeled, the distance calculation could be unreliable. By selecting the $k$-th nearest sample, we mitigate this risk, as the likelihood of all $k$ nearest samples being mislabeled is low.

Q6: What is the meaning of solving the ID-OOD trade-offs?

A6: Solving the ID-OOD trade-offs means developing models that can maintain high accuracy on in-distribution (ID) data while also performing well on out-of-distribution (OOD) data. Typically, improving OOD performance can reduce ID accuracy, but addressing these trade-offs aims to balance both, ensuring models are reliable in both familiar and new environments.

Q1: The main drawback of the proposed method is the reliance on multiple hyperparameters.

A1: We understand and acknowledge the reviewer’s concern regarding the complexity of tuning multiple hyperparameters. However, we would like to provide further clarification and context to address this issue:

• No Need to Optimize p: First, we emphasize that the hyperparameter $p$ does not require optimization in our approach. We adopt the default value from [24], and as demonstrated in Figure 5(c), the performance is not sensitive to changes in $p$. To further validate this, we conducted experiments on CIFAR and Entity-30 datasets, where we varied $p$ from 0.0002% to 50%. The ID and OOD performance fluctuated by less than 0.1% and 0.3%, respectively, confirming that $p$ does not significantly impact the overall performance.
• Simplified Search for a and b: Regarding the hyperparameters $a$ and $b$, the optimization process is straightforward and can be managed through grid search on the ID validation set. Additionally, we observed that the performance tends to peak at consistent values of $b$ even as $a$ varies. This pattern significantly reduces the search space for $b$, making the tuning process more efficient and less complex than it might initially appear.

In conclusion, while our method involves multiple hyperparameters, we have shown that the tuning process is manageable and not as complex as it may seem. The robustness of $p$ and the structured tuning of $a$ and $b$ make our approach both practical and efficient across different datasets.

Q2: Need to report the hyperparameters found in experiments with different datasets and architectures to analyze their variability and sensitivity.

A2: We appreciate the reviewer’s insightful suggestion. We have reported the searched hyperparameters across different datasets and architectures below and will include them in the revised version of the paper.

Our findings indicate that for the same dataset, the hyperparameters remain consistent across different architectures, suggesting that our method's performance is robust to architectural changes. However, the hyperparameters do vary between datasets, which is expected due to differences in data distribution. For example, the k-NN distance ranges from 0.4 to 1.6 for ImageNet and from 0.4 to 1.2 for Entity30, reflecting the underlying data characteristics. Correspondingly, the searched values for $a$ are 1.5 for ImageNet and 1.1 for Entity30. This variability in $a$ highlights the importance of adapting hyperparameters to specific datasets to achieve optimal performance.

ViT/32

ViT/16

Q3: The performance when finding hyperparameters for each test set as an oracle result.

We have conducted additional experiments where we optimized the hyperparameters for each test set of the ImageNet benckmarks. The results are summarized in the following table.

Q4: iWILDCam and FMoW results.

Q4: We have conducted experiments (E2E-FT model as the fine-tuned models) on fmow and iwildcam and report the performance with the hyper-parameters below. We will add the results of iWILDCam and FMoW in the revised version.

Q5: Comparison with other robust fine-tuning methods.

A5: Our VRF framework is designed to be orthogonal and complementary to existing fine-tuned models. To demonstrate this, we conducted additional experiments using FLYP as the fine-tuned model within our VRF framework. The results show that our VRF framework enhances the performance of OSE by improving the distribution shift performance by 1.1% while maintaining the in-distribution (ID) performance.