Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models

We appreciate the reviewer's time and valuable insights. It is encouraging that the reviewer found our DRM approach is supported by "rigorous mathematical proofs" and it “effectively enhances the robustness of the model”. We appreciate that the reviewer has thoroughly engaged with our work and acknowledged its merits.

Regarding the concerns of the reviewer, we provide detailed clarifications below.

1. Accuracy of the estimate of $p_c(y|x)$

There is certainly still a gap between our estimate and the real $p_c(y|x)$. However, we would like to highlight that the estimate is already substantially better than one-hot labels, as demonstrated by the comparison between (S) and (c3) in Table 7: OOD performance of 49.2 vs. 45.1. Moreover, our approach requires minimal human input, making it highly scalable. In principle, with more human input, we should be able to further improve the estimate, and this would be an interesting future direction.

2. “... why not directly fine-tune cd instead of using cd to assist in fine-tuning df?”

Please see B. Why not directly fine-tune a concept-description classifier? in the global response. There, we show experiment results and discuss why solely fine-tuning the concept-description classifier is not better than our DRM fine-tuning.

3. “... why is the performance of cd inferior to df when comparing row 2 and row 3 in Table 3?”

Please note that only the ID performance of cd is inferior to df. This is expected since cd is not (while df is) trained with ERM which aims to maximize ID performance.

4. “In Table 7, if the models trained with df and cd (rows d1 and d2) are directly used for inference with (13), how will the performance be?”

This is an interesting question. Upon further experiments, we find that: 1. Using a mixture model to make predictions for (d1), which is trained with default prompts (df), results in slightly improved OOD performance, though it still significantly lags behind that of the full DRM approach; 2. Conversely, using a mixture model for predictions in (d2), which is trained with concept descriptions (cd), actually leads to a decrease in OOD performance.

Recall that, (d1) refers to the ERM-trained model that uses a concentration of text embeddings from default prompts as the classification head, while (d2) represents the ERM-trained model utilizing a concentration of text embeddings from concept descriptions as the classification head. For both (d1) and (d2), the mixture hyperparameter $\beta$ is determined based on achieving the best ID validation performance.

We appreciate the reviewer's time and valuable insights. It is encouraging that the reviewer found our DRM approach "novel", the use of LLMs to obtain core-feature descriptions is "innovative", and the method is "well-supported by theoretical foundations and empirical results". We appreciate that the reviewer has thoroughly engaged with our work and acknowledged its merits.

Regarding the concerns of the reviewer, we provide detailed clarifications below.

1. Impact of hyperparameter $\lambda$ in DRM

$\lambda$ is a balancing hyperparameter for the two risks in DRM. While it may seem that DRM is sensitive to $\lambda$, we would like to draw your attention to the fact that the range of $\lambda$ with decent performance is still wide. As shown in Table 6 (which we copy below), for $\lambda$ between 2 and 5, DRM maintains high-level OOD performance (above 48.1, compared to FLYP’s 42.1) on iWildCam. In other words, the performance drop (from 49.2 to 48.1) is relatively small compared to the gain.

Copy of Table 6

Following your great suggestion, we have conducted the same experiments on ImageNet with CLIP ViT-B/16 to further study the impact of $\lambda$:

Similar to the results on iWildCam, DRM achieves great performance across a wide range of $\lambda$ between 2 and 6. All in all, these results suggest that DRM is fairly insensitive to the choice of $\lambda$ as it is easy to find appropriate values of $\lambda$ for DRM to significantly outperform the state of the art.

2. “The results in Table 6 show no trade-off between ID and OOD performance, which is counterintuitive ...”

Indeed, this was also counterintuitive to us at first, but then a closer inspection revealed a rather intuitive explanation. The ID performance is determined by two factors: (i) how well the model fits the training data, and (ii) how severe the overfitting (if any) is. The results in Table 6 show no trade-off because for relatively small $\lambda$, (i) the model can still fit the training data reasonably well, and (ii) the WRM objective can in fact reduce overfitting, as shown by the training/validation performance of DRM on iWildCam below:

This explains why ID performance improves as $\lambda$ increases (for $\lambda \leq 3$), and consequently why no trade-off is observed in Table 6.

3. Writing

Thank you for your great suggestions.

• We will make the explanation of the DRM formulation relaxation process clearer by showing the key intermediate steps from Eq. (3) to line 159.
• We will specify the meaning of the bars in Figure 1: the 1st bar – the predicted probability of skis being present in the image; and the 2nd bar - no skis being present in the image.
• To avoid potential misunderstanding, we will add the affinities w.r.t. other images for Figure 2. We provide several such examples in Figure 2 of the attached PDF in the global response.

4. Quality of concept descriptions and its impact on the results

Please see A. Quality and Robustness of LLM-Generated Concept Descriptions in the global response. There, we have included the results of a quantitative study on the reliability of concept descriptions, an analysis of the stochastic variability in LLM generations, and an investigation into the robustness of DRM with respect to concept descriptions generated by various LLMs.

In summary, the results demonstrate that GPT-4's concept descriptions effectively capture the core features, with the resulting affinities proving stable and unaffected by randomness in the LLM generation process, and DRM is robust to the concept descriptions generated by different LLMs.

5. Combining DRM with LP-FT or FLYP

Yes, our DRM can indeed be integrated with other methods such as LP-FT and FLYP. In fact, we have combined DRM with FLYP in our experiments, as detailed in lines 276-278 of our paper. Specifically, we utilized FLYP for the ERM component of Eq. (12). It is crucial to highlight that even without FLYP, where DRM is applied with standard full fine-tuning (row 2), our method still outperforms FLYP (row 1), which itself has been shown to surpass standard full fine-tuning in the ERM setting [1]. This performance advantage is demonstrated in the results of fine-tuning CLIP ViT-L/14 on iWildCam:

As the table above demonstrates, combining DRM with LP-FT (row 3) enhances performance over just DRM with standard full fine-tuning (row 2). Furthermore, integrating DRM with FLYP (row 4), as we have explored in our paper, yields even more significant improvements.

In the next version of our paper, we will further clarify that our primary experimental configurations involve DRM combined with FLYP, and will include the above discussions you kindly suggested.

References:

[1] Finetune like you pretrain: Improved finetuning of zero-shot vision models. CVPR, 2023.

We are grateful for the reviewer's time and insightful feedback. It is encouraging to know that the reviewer found our DRM “novel and interesting” and “achieves state-of-the-art results on various benchmarks”. We appreciate that the reviewer has thoroughly engaged with our work and acknowledged its merits.

Regarding the concerns of the reviewer, we provide detailed clarifications below.

1. Impact of hyperparameter $\lambda$ in DRM

$\lambda$ is a balancing hyperparameter for the two risks in DRM. We have discussed its impact on the performance of DRM in Appendix D.3 of our paper. Here, we copy Table 6 from Appendix D.3 below. Notably, for $\lambda$ within the wide range from 2 to 5, DRM maintains high-level OOD performance (above 48.1, compared to FLYP’s 42.1) on iWildCam.

Copy of Table 6

To further study the impact of $\lambda$, we have also conducted the same experiments on ImageNet with CLIP ViT-B/16:

Similar to the results on iWildCam, DRM achieves great performance across a wide range of $\lambda$ between 2 and 6. All in all, these results suggest that DRM is fairly insensitive to the choice of $\lambda$ as it is easy to find appropriate values of $\lambda$ for DRM to significantly outperform the state of the art.

2. Computation time and cost

In our experiments, we implemented the ERM part of DRM with FLYP [1]. The additional computational cost of DRM, compared to FLYP, primarily arises from the preparation of soft labels for the targets of the second risk in DRM and the minimization of this second risk. In short, the training and inference cost for DRM increased by about 20% from FLYP. This additional cost is insignificant compared to the attained performance gain. Below, we detail the computational costs and timing for fine-tuning CLIP ViT-L/14 models on ImageNet.

We use two Nvidia H800 GPUs with 80GB VRAM (a modified version of the Nvidia H100 with a reduced chip-to-chip data transfer rate) and 24 CPU cores from Intel Xeon Scalable processors. The computation time reported below is based on the setting that training batch size=256 and inference batch size=1024. There are 1,281,167 training images in ImageNet, and thus there are 5005 training batches.

This detailed analysis of the computation cost and time will be included in the next version of this paper.

References:

[1] Finetune like you pretrain: Improved finetuning of zero-shot vision models. CVPR, 2023.

We are grateful for the reviewer's time and insightful feedback. Regarding the concerns and questions the reviewer raised, we provide detailed clarifications below. If not specified otherwise, all experiments below are conducted with CLIP ViT-B/16.

1. Reliability of the core features obtained with concept descriptions

Please see A. Quality and Robustness of LLM-Generated Concept Descriptions in the global response.

2. “At Line 217, the paper mentions that the affinity between $x$ and any other class $y'$ should ideally be zero ... this might ignore the fact that the ground-truth class $y$ exhibits varying degrees of similarity with different $y'$ ...”

This is not what we stated in the paper. At lines 217-218, please note that “$x$ is completely void of core visual features for $y'$” and only in such case the affinities should ideally be 0.

It is generally true that, as you mentioned, “the ground-truth class $y$ exhibits varying degrees of similarity with different $y’$”. Our method does not disregard the learning of such inter-class similarities; on the contrary, our method specifically captures the similarities with Eq. (11) where the probabilities of non-ground-truth classes are weighted by their affinities w.r.t. the image.

To illustrate this point, for each class in ImageNet, we identify the classes with the highest average probabilities in the soft labels generated by Eq. (11). Below are some examples:

• horizontal bar: horizontal bar, balance beam, parallel bars, pole, swing;
• whiskey jug: whiskey jug, water jug, pitcher, beer bottle, beaker;
• goldfish: goldfish, rock beauty, tench, barracouta, anemone fish.

We can see that the top-1 class is consistently the class itself as intended (in fact, this holds for all 1k ImageNet classes), and the other top classes are indeed visually related to the top-1 class.

3. Comparison with knowledge distillation methods

Yes, the term is related to knowledge distillation, or more precisely, self-distillation. There is a self-distillation method, L2-SP, in Table 1 of our paper. Here, we compare our method with two more state-of-the-art self-distillation methods, CaRot [1] and MIRO [2]. Their ID/OOD performances are shown in the table below.

We will further discuss related methods in the next version of the paper, including CaRot, MIRO, and DELTA [3] which you kindly suggested. Please note that DELTA has been surpassed by MIRO [2] on various benchmarks so we only included MIRO here.

4. “... could we simply fine-tune the CLIP model utilizing these concept descriptions? ...”

Please see B. Why not directly fine-tune a concept-description classifier? in the global response.

5. “What would be the ID and OOD accuracy for the concept description classifiers? ... would they be more robust to distribution changes, even without fine-tuning?”

As shown below, they are slightly better than the default-prompt classifiers, but much worse than the fine-tuned DRM classifiers. In short, concept descriptions are helpful, but fine-tuning is still important, especially when the domain gap is large, e.g., in the cases of iWildCam and FMoW.

6. “How does the min-max normalization affect the training on long-tailed datasets ...?”

This is a great question. We concur that using min-max normalization on long-tailed classes can lead to less stable normalization. That said, its impact on overall performance should be small.

The reason is two-fold. First, even if there are only a few samples for a class, most samples of the class would still have higher affinities with the class compared to samples from other classes. This means that $\gamma(x, y)$ would be small for $y \neq y_x$, and thus any impact on the estimation of $p_c(y|x)$ (via Eq. (11)) would likely also be small. Second, for the estimation of $p_c(y_x|x)$, although with fewer samples the difference in $p_c(y_x|x)$ between $x$ of the same class $y_x$ would likely be magnified, their relative ordering would still be intact, and thus the classifier would still learn to preserve the core features of the class.

Surprisingly, we find that DRM can even enhance the learning of long-tailed classes on iWildCam. This is reflected by the table below where we can see the training F1 score increases while the training accuracy decreases as $\lambda$ increases.

In particular, for classes with fewer than 50 training examples, we find that FLYP achieves an accuracy of 59.61% on the ID validation set, whereas DRM achieves a significantly higher accuracy of 69.38%.

7. About limitations

Thank you for suggesting the use of specific examples to illustrate DRM limitations due to LLMs' restricted domain knowledge. In our next version, we will showcase GPT-4's inaccuracies in medical imaging fields like ocular disease and breast histology.

References:

[1] Towards calibrated robust fine-tuning of vision-language models. arXiv, 2023.
[2] Domain generalization by mutual-information regularization with pre-trained models. ECCV, 2022.
[3] DELTA: Deep Learning Transfer using Feature Map with Attention for Convolutional Networks, ICLR 2019.