fine-tuning with very large dropout

We highly recommend that Reviewer HNY5 read our general response to gain a better understanding of this work.

---

Question 1: lack of novelty, because this paper doesn’t introduce substantial new concepts or techniques.

To start with, substantial new concepts do not necessarily lead to novelty.

Please allow me to ask two questions to justify the novelty of this work:

• Before this paper, did the community know a simple very-large dropout on the penultimate layer outperformed ensembling, weight averaging, and other .xxx methods on ood?

• Before this paper, would you even try to fine-tune with dropout rate >= 0.9 and explore the possibility of very-large dropout approach on ood generalization?

Answering these two questions will justify the novelty of this work. To the best of our knowledge, the answer to these two questions is likely “NO”

---

Question 2: reviewer HNY5 feels the experimental setup is minimal. And thus ask for more datasets and baselines.

Please check our general response 1 and 2.

The experimental setup follows DomainBed, which is a widely used setup. Thus, it is not a “minimal” setup, but a “widely used” setup.

DomainBed is not one dataset, but a set of datasets. Following [5], this paper uses 5 different datasets.

This work compares very-large dropout with ensemble and weight averaging baselines. The comparison between weight averaging and other baselines has been conducted in [9].

As we stated in lines 202-208: "Using these same datasets, Gulrajani & Lopez-Paz (2020) argue that plain Empirical Risk Minimization (ERM) equals and often betters the o.o.d. performance of purposefully designed methods, such as CORAL (Sun & Saenko, 2016), VREX (Krueger et al., 2021), and IRM (Arjovsky et al., 2019). More recently, Arpit et al. (2022), Cha et al. (2021), Ramé et al. (2022b), and Ramé et al. (2022a) find that ensemble and weight averaging methods consistently outperform the o.o.d. performance of ERM. We now show that fine-tuning with very large dropout outperforms the o.o.d. performance of these state-of-the-art methods."

We can, of course, reference the numbers of these methods. But that only make the paper wordy.

---

Question 3: Lack of theoretical discussion.

Please refer to our general response 1 for more information.

---

[5] Rame, Alexandre, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. "Diverse weight averaging for out-of-distribution generalization." Advances in Neural Information Processing Systems 35 (2022): 10821-10836.

[9] Cha, Junbum, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. "Swad: Domain generalization by seeking flat minima." Advances in Neural Information Processing Systems 34 (2021): 22405-22418.

Dear reviewer HNY5

Thank you for taking the time to review our paper and provide valuable feedback.

We have carefully considered your comments and have provided detailed responses to each of your points. We hope that our answers adequately address your concerns and provide a clearer understanding of our work.

If you have any further questions or concerns, please do not hesitate to leave a comment.

Thank you, Reviewer HNY5, for your thoughtful feedback. We greatly value the opportunity to enhance understanding between authors and reviewers, beyond just paper-review scores. To facilitate this understanding, we posed several questions in our initial response, seeking clarity on certain points. Unfortunately, we were unable to find clear answers in your response.

We would like to reiterate our initial questions and sincerely hope to receive clear answers from you, which will enable us to improve our paper effectively.

---

Novelty Concerns: You raised a question about the novelty of our work. We posed two questions in response:

• "Before this paper, did the community know that a simple very-large dropout on the penultimate layer outperformed ensembling, weight averaging, and other methods on OOD?"
• "Before this paper, would you have considered fine-tuning with a dropout rate >= 0.9 and exploring the possibility of a very-large dropout approach on OOD generalization?"

To the best of our knowledge, the answer is "No." Could you kindly provide an answer from your perspective?

---

Experimental Setup: You commented that our experimental setup is "minimal". We clarified that the DomainBed benchmark is widely used, and our work compares two architectures (ResNet, Transformer), three pre-trained models, five OOD tasks, and thousands of fine-tuning configurations. The superior performance of our proposed very-large dropout approach is evident. Reference [9] has compared other baselines (e.g., CORAL, VREX, IRM) with our baseline (ensemble, weight-averaging), demonstrating the superior performance of our baseline. We can also reference the CORAL, VREX, IRM baselines in [9].

We believe our experiment is sufficiently comprehensive to support our claims. We found Reviewer HNY5's suggestion to "explore a more comprehensive experimental design" in the response. We truly want to ensure the experimental design in this work is comprehensive.

Could you kindly point out which parts of our current experimental design make it less comprehensive / minimal / fail-to-support-our-claim?

---

Theoretical Discussion: Reviewer HNY5 further pointed out that this work lacks theoretical discussion.

In our response, we clearly stated that this is an experimental work. It's worth noting that ICLR accepts experimental works, which aligns with the nature of our submission.

We appreciate your guidance and look forward to your insights, which will be invaluable in refining our work.

We appreciate Reviewer Lz8E's positive feedback on the superior performance of our proposed large dropout approach.

---

Question 1: Could you add additional theoretical results or visual analysis?

Please refer to our general responses 1 and 2 for more information.

---

Question 2: Compare the proposed method with other existing methods in the DomainBed benchmark.

We have already done so. Please see lines 202-208, which state: "Using these same datasets, Gulrajani & Lopez-Paz (2020) argue that plain Empirical Risk Minimization (ERM) equals and often betters the o.o.d. performance of purposefully designed methods, such as CORAL (Sun & Saenko, 2016), VREX (Krueger et al., 2021), and IRM (Arjovsky et al., 2019). More recently, Arpit et al. (2022), Cha et al. (2021), Ramé et al. (2022b), and Ramé et al. (2022a) find that ensemble and weight averaging methods consistently outperform the o.o.d. performance of ERM. We now show that fine-tuning with very large dropout outperforms the o.o.d. performance of these state-of-the-art methods."

We can reference the numbers of these methods because the comparison between these methods and weight-averaging & ensemble has been conducted in related works [9].

---

Question 3: Analyze the reason why the IID performance of the very-large dropout approach doesn’t always outperform weight averaging and ensemble (in appendix figure 7).

Please refer to our general response 3 for more information.

---

Question 4: Why not adopt the large dropout in the deeper layers?

Please refer to our general response 4 for more information.

---

Question 5: If the number of ensembling members is 1296, it is too large.

The number of ensemble members is actually 2 (learning rate) x 3 (weight decay) = 6, not 1296. Therefore, it is not too large.

---

[9] Cha, Junbum, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. "Swad: Domain generalization by seeking flat minima." Advances in Neural Information Processing Systems 34 (2021): 22405-22418.

Dear reviewer Lz8E

Thank you for taking the time to review our paper and provide valuable feedback.

We have carefully considered your comments and have provided detailed responses to each of your points. We hope that our answers adequately address your concerns and provide a clearer understanding of our work.

If you have any further questions or concerns, please do not hesitate to leave a comment.

Thanks for reviewer Lz8E's quick reply.

After our response, reviewer Lz8E's primary concern is that this empirical work doesn't contain a "formal theory". We are happy to see that our responses addressed reviewer Lz8E's other question 2-5.

Please note that this is an empirical work on the early-stage of a counter-intuitive problem. Because it is the early-stage of a "unfamilar" problem, establishing a consistent & helpful "formal theory" requires a lot of empirical "observations" as support. This work serves as one of these essential empirical observations.

We encourage Reviewer RZvb to read the general response to gain a comprehensive understanding of this work.

---

Question 1: In the limit of large sampling, this work conflicts with the expected theory. Thus practitioners need an extensive grid search to employ this solution.

The assumption of "large sampling" is not applicable to our work. This work focuses on "limited samples" in the OOD fine-tuning area. Consequently, theories based on large sampling assumptions do not apply here.

Regarding the claim of an extensive grid search, it is unfounded. Our very-large dropout approach involves only one hyper-parameter: the dropout rate. Figure 2 demonstrates that the relationship between dropout rate and OOD accuracy is simple and smooth, indicating that our approach is not sensitive to the choice of dropout rate. Therefore, there is no extensive grid search involved.

---

Question 2: Reviewer RZvb points out that “there is a lot of engineered choices related to learning rate scaling at layer's scale, dropout etc.”

This is incorrect. “Learning rate scaling at layer’s scale” is merely a method for comparison, not a hyper-parameter of our proposed very-large dropout approach.

In fact, our work aims to minimize engineering choices, such as learning rate and weight decay. The bar plot in Figure 1 illustrates various hyper-parameter configurations, comparing the worst hyper-parameters of very-large dropout with the best hyper-parameters of other methods. Therefore, it is incorrect to write that "a lot of engineered choices” are involved.

---

Question 3: Some statements are overly (and maybe, unnecessarily) strong. For example, "It is impossible today to pretend that the practice of machine learning is compatible with the idea that training and testing data follow the same distribution": there are many cases in which inference happens in distribution.

The claim “there are many cases in which inference happens in distribution.” does not conflict with the statement "It is impossible today to pretend that the practice of machine learning is compatible with the idea that training and testing data follow the same distribution".

There are carefully designed cases of in-distribution. In the meanwhile, there are more cases of out-of-distribution in the wild (e.g. each in-distribution case can be easily transformed into multiple ood cases by changing spurious features).

Reviewer RZvb is talking about the existence of in-distribution. Our sentence states that not all cases and not even a majority of cases are in-distribution.

---

Question 4: How does it work on other tasks and other architectures?

Please refer to our general responses 1 and 2.

---

Question 5: How are the learning hyper-parameters found? Has a validation set been used?

This work utilizes the DomainBed suite [7] and weight-averaging as a baseline [8], adhering to the setups in these references. As stated in lines 282-283, "Following Gulrajani & Lopez-Paz (2020), we prevent overfitting by early-stopping at the best i.i.d. performance." We use an IID validation set as per the setups in [7].

---

[7] Gulrajani, Ishaan, and David Lopez-Paz. "In search of lost domain generalization." arXiv preprint arXiv:2007.01434 (2020).

[8] Rame, Alexandre, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. "Diverse weight averaging for out-of-distribution generalization." Advances in Neural Information Processing Systems 35 (2022): 10821-10836.

Dear reviewer RZvb

Thank you for taking the time to review our paper and provide valuable feedback.

We have carefully considered your comments and have provided detailed responses to each of your points. We hope that our answers adequately address your concerns and provide a clearer understanding of our work.

If you have any further questions or concerns, please do not hesitate to leave a comment.

We highly recommend that reviewer hP51 read our general response, as most of their concerns are addressed there.

---

Question 1: The choice of baseline methods in this paper is narrow.

Please refer to our general responses 1 and 2 for a detailed explanation.

---

Question 2: Reviewer hP51 concerns the proposed very-large dropout method is sensitive to dropout rate hyper-parameter = 0.9.

No, this paper provides the Ablation of dropout rate in Figure 2. First, the curve of dropout rate vs OOD acc is simple & smooth, which makes it easy to search hyper-parameters. Second, both dropout rates 0.9 and 0.95 provide good OOD performance.

---

Question 3: Why does the proposed method help OOD generalization? Why apply dropout on the penultimate layer, instead of everywhere?

Please read our general responses 1 and 4 for an explanation.

---

Question 4: Why does IID performance drop in certain datasets?

Please read our general response 3 for an explanation.

---

Question 5: Reviewer hP51 points out that Raghu et al., 2019 is a case about MAML and likely does not extend to all fine-tuning settings.

Thanks for your suggestion. There are also other works [4] that demonstrate a close phenomenon as Raghu et al., 2019 figure 2, but in a more general setting. We will cite [4] in the final version.

---

Question 6: Reviewer hP51 believes that Evci et al., 2022 “doesn’t seem to make any claims about the usefulness of intermediate features being specific to models using skip connections”. Thus

Question whether this work should use Evci et al., 2022 to motivate the choice of models. The claim that “skip connections in residual networks expose the inner block features” is supported by Equation 1 in our paper. Evci et al., 2022 experimentally demonstrate the usefulness of inner block features on networks with skip connections (resnet, transformer), and thus support the claim above in the experimental view. We use Evci et al., 2022 to motivate the choice of ResNet and Transformer because they use these two choices and support our claim.

---

Question 7: Has it been demonstrated in prior work that the linear connectivity property disappears when the dataset sizes get close (i.e., is there a citation for this claim)?

The size of the pretraining dataset (in this work) is large compared with the fine-tuning dataset size. As the fine-tuning dataset gets larger, one needs more fine-tuning iterations (line 183) or other aggressive hyper-parameters (such as large learning rate) to pursue better performance. Many weight-averaging works, e.g., [5] (section E.1.4) and [6], show that aggressive fine-tuning hyper-parameters breaks the linear connectivity property. We will add citations for this claim in the final version.

---

Question 8: “A barely matches B” -> “A matches B”. “setup is a commonly used” —> drop “a”.

Thanks for pointing them out. We will fix them in the final version.

---

[4] Zhang, Chiyuan, Samy Bengio, and Yoram Singer. "Are all layers created equal?." Journal of Machine Learning Research 23, no. 67 (2022): 1-28.

[5] Rame, Alexandre, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. "Diverse weight averaging for out-of-distribution generalization." Advances in Neural Information Processing Systems 35 (2022): 10821-10836.

[6] Wortsman, Mitchell, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong et al. "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time." In International conference on machine learning, pp. 23965-23998. PMLR, 2022.

Dear reviewer hP51

Thank you for taking the time to review our paper and provide valuable feedback.

We have carefully considered your comments and have provided detailed responses to each of your points. We hope that our answers adequately address your concerns and provide a clearer understanding of our work.

If you have any further questions or concerns, please do not hesitate to leave a comment.

Thank you, Reviewer hP51, for your thoughtful feedback. We appreciate the opportunity to address your concerns and provide further clarification.

---

Regarding the benchmark discussion, DomainBed [1], WILD [2], and NICO++ [3] were all released between 2020 and 2022. They share a similar dataset structure (multiple training environments), and benchmark a comparable set of methods (e.g., IRM, ERM, VREX, GroupDRO, etc). Therefore, it is challenging to label one as out-dated while considering the others current.

More importantly, these benchmarks are designed to evaluate whether a machine learning algorithm can learn interesting properties, rather than focusing solely on absolute performance, such as that achieved by "massively trained models." Training directly on test data would yield the best performance, but it would not be interesting.

Thus, the statement, "One must also consider the current era we live in, where massively trained models can likely solve most of the DomainBed flavor of tasks," does not imply that evaluating a machine learning algorithm on DomainBed is less convincing. DomainBed benchmark evaluates the ood generalization problems that are still challenging to solve.

---

In response to Reviewer hP51's inquiry about the sensitivity of dropout parameters, we present two arguments:

• "As shown in Figure 2, the curve of dropout rate versus OOD accuracy is simple and smooth, facilitating easy dropout hyper-parameter search."
• "Both dropout rates of 0.9 and 0.95 yield good OOD performance."

Reviewer hP51's observation that "the performances at 0.9 and 0.95 are fairly close for 3 of the 4 datasets" supports our claim that the very-large dropout approach in this work is not sensitive to dropout hyper-parameters. More importantly, the former argument shows that dropout rate is not a sensitive hyperparameter in this work.

---

Reviewer hP51 further notes that "and more significant for the challenging TerraIncognita dataset." We argue that it is not inherently "challenging"; rather, TerraIncognita contains more training examples. Please refer to, lines 318-338 and the DomainNet dataset experiment in Figure 3 for further clarification.

---

Regarding the analogy of OOD generalization to the IID train/test setup, Reviewer hP51 mentions that it is "not convincing, and is in fact a limitation with the narrative in this area of work." We would like to clarify that this analogy is general and not exclusive to our work.

To elucidate, consider a simple question in IID: "If we model the training data with a Dirac delta function, does the model provide any insight into IID test data? answer: No."

Similarly, in OOD: "If we model training data from distribution A, does the model offer any insight into test data from arbitrarily distribution B? answer: No "

We can have a "Yes" answer when we eliminate "arbitrarily" in OOD (building connections between distirbution A and B) and "Dirac delta function" in IID (building connections between IID train data and IID test data).

[1] Gulrajani, Ishaan, and David Lopez-Paz. "In search of lost domain generalization." arXiv preprint arXiv:2007.01434 (2020).

[2] Koh, Pang Wei, et al. "Wilds: A benchmark of in-the-wild distribution shifts." International conference on machine learning. PMLR, 2021.

[3] https://arxiv.org/abs/2204.08040

I thank the authors for their thorough response. Having read it, and considered the points being raised by the other reviewers, I am inclined to keep my rating. In particular,

• while it is true that purely empirical results are within the scope of ICLR, a paper that presents itself as purely empirical without very clear reasoning would need to prove itself much more thoroughly. DomainBed is arguably at this point a somewhat out-dated benchmark, with more relevant ones such as NICO++ or WILDS available, where one can more convincingly illustrate the empirical merit of an idea. One must also consider the current era we live in, where massively trained models can likely solve most of the DomainBed flavour of tasks.
• No, this paper provides the Ablation of dropout rate in Figure 2. First, the curve of dropout rate vs OOD acc is simple & smooth, which makes it easy to search hyper-parameters. Second, both dropout rates 0.9 and 0.95 provide good OOD performance. There are only 4 points of evaluation in Fig 2. While the performances at 0.9 and 0.95 are fairly close for 3 of the 4 datasets, and more significant for the challenging TerraIncognita dataset, the overall improvements over the baselines are not so high that one can ignore the differences.
• The analogy of OOD generalization to the IID train/test setup is not convincing, and is in fact a limitation with the narrative in this area of work. One cannot know how OOD a deployment is going to be, and that the model will never be evaluated in an IID setting. In fact, it feels a much more likely scenario that models are generally going to be evaluated in IID settings, but may face OOD settings at varying rates depending on circumstances.

We encourage reviewer azwu to read our general response first.

---

Question 1: Why does the proposed approach work? And thus suggest evaluating the proposed approach on other tasks and architectures?

Please refer to our general responses 1 and 2 for a detailed explanation.

---

Question 2: Concerns about the dropping IID performance in certain datasets.

Please see our general response 3 for an explanation.

---

Question 3: Suggest using "structured dropout methods" to avoid the dropping IID performance in certain datasets.

We appreciate reviewer azwu's suggestion. Please check our general response 3. In short, IId performance in the OOD generalization area is analogous to training dataset performance in traditional IID machine learning. Thus IID performance is not the primary concern in the OOD generalization area.

---

Question 4: Suggest increasing the ensemble baseline (OOD) performance by training a model with one joint classifier and multiple separate backbones.

We thank reviewer azwu for the suggestion. However, this approach has been shown to lead to poor OOD performance. As demonstrated in related work [2], training a model with one joint classifier and multiple separate backbones can harm rich representation and result in suboptimal OOD performance.

---

Question 5: Why should evenly distributed information across dropped features lead to "weakly relevant" features? Is further theoretical support available?

To satisfy the learning objectives, a model needs to identify relevant features. Under the sparsity bias (either from SGD optimizer or L1/L2 regularization), a few "strongly relevant" features can be discovered by minimizing the learning objectives. With the existence of “strong relevant” features, other features ( “weakly relevant”) are not incrementally helpful in the training distribution.

“However, features that are not strongly relevant might nevertheless (a) be incrementally useful when the data follows a different distributions of interest, or (b) be useful under the training distribution when added to certain subsets of the other existing features instead of all of them (“weakly relevant”).”(line 049-052)

A very-large dropout masks a large fraction of features, making it possible for the model to learn from these weakly relevant features.

[2] Zhang, J., Lopez-Paz, D., & Bottou, L. (2022, June). Rich feature construction for the optimization-generalization dilemma. In International Conference on Machine Learning (pp. 26397-26411). PMLR.

Dear reviewer azwu

Thank you for taking the time to review our paper and provide valuable feedback.

We have carefully considered your comments and have provided detailed responses to each of your points. We hope that our answers adequately address your concerns and provide a clearer understanding of our work.

If you have any further questions or concerns, please do not hesitate to leave a comment.