Unraveling the Key Components of OOD Generalization via Diversification

We thank the reviewer for their time reviewing and providing useful feedback, we kindly answer raised questions as follows.

What are the unique advantages of “diversification” methods to solve OOD problems？Does it contribute much to the OOD community to study the components of “diversification”?

We kindly refer to the ‘The focused scope of our work’ section in the general response for a detailed answer. We will be sure to add more discussion in the next revision.

More complicated diversification loss, approaches, need to be included. In addition to the methods used in the paper, does it need to consider more models to support the conclusions of the paper?

At the time of writing, to the best of our knowledge, there were no other diversification losses or approaches that are fundamentally different from D-BAT and DivDis. The two methods differ not only on diversification losses but also on the optimization strategies, thus representing a spectrum of possible dynamics in this learning framework well.

Throughout the paper, we convey crucial messages that provide insights to guide the future design of diversification methods and the usage of current ones, and each message is supported by theoretical proofs (proposition 1, proposition 2), and experimental verifications as below:

• For proposition 1, we further justify the performance of diversification methods is highly dependent on spurious ratio by firstly verifying the solution of synthetic 2D illustrative example and secondly showing the conclusion holds in practice with datasets in different scales (M/C, M/F, and CeleA-CC).
• For proposition 2, we show with 10 models (spanning different training strategies) that diversification alone cannot lead to OOD generalization the inductive bias brought by the used learning algorithm is important.

For the reasons above, we believe our conclusions are well-supported. However, we would be very glad to provide further evidence, but was wondering which kind of models or approaches the reviewer would be interested to see?

We thank the reviewer for their time reviewing and providing useful feedback, we kindly answer raised questions as follows.

The focused methods of our work

We would like to clarify that our analysis shows unique axes that make diversification methods work, and is actually acknowledging the significance (rather than criticizing) and providing further investigation and evidence to support these methods to stand the test of time. We kindly refer to the ‘The focused scope of our work’ section in the general response for a detailed answer.

The significance of the no-free-lunch results

We kindly refer to the ‘The significance of the no-free-lunch results’ section in the general response to all reviewers.

Concerns on the datasets and problem setup we considered / Can you add some more background for how OOD concerns can arise in the industry?

We kindly answer the two questions together. As shown in [1, 2], when multiple predictive features are present in the training set, neural networks tend to learn the simple (but can be spurious) feature rather than the true causal features because of their ''simplicity'' inductive bias. The case where multiple predictive features co-exist but only the true feature is generalizable is called spurious correlation. This can cause real-world and industrial failure cases, for example:

• In a chest X-ray dataset [3], many images of patients with pneumothorax include a thin drain used for treating the disease. An empirical risk minimization (ERM) model trained on such a dataset can erroneously identify such drains as a predictive feature of the disease, while it's actually a spurious feature. A good deployable model for the model would require sensitivity to physiological signals while being invariant to environmental or operational signals that can change between deployment contexts. (please see https://wilds.stanford.edu/datasets/ for more real-world examples)

which is especially crucial when the reliability of the model is expected to be high, such as in medical/financial/self-driving fields. This has motivated the community to study spurious correlation and the problem has become a main branch of OOD generalization problem and is long-standing, well-defined, realistic but challenging. Following the many existing methods (e.g., [5, 7, 8]), both D-BAT & DivDis and our work use the well-established and real-world baselines and benchmarks (CelebA, WaterBirds, Camelyon17, etc). Going one step further, D-BAT and DivDis also try to handle a more difficult case, where the spurious feature/hypothesis always correlates with the true feature/hypothesis, and this is important because:

• Sometimes it's hard to acquire minor group samples due to temporal or spatial reasons (e.g., a particular species can only be observed in a specific time/location).
• Therefore, the model has to work on this harder case in order to reliably work on simpler cases.
• As an example, the medical images of a specific disease tissue are solely available in a hospital/research lab, and the medical model fails to recognize the disease because it learns to only look at the hospital token rather than the tissue.

For our purpose, using MNIST-CIFAR and Waterbirds-CC unlocks more controlled experiments (e.g., a more fine-grained analysis under various spurious ratios) to simulate the different scenarios that could arise in the industry, which makes it possible to understand the conditions required for the diversification methods to work, which, in turn, allows for developing more reliable and powerful methods for real-world applications.

We thank the reviewer for their time reviewing and providing useful feedback, we kindly answer raised questions as follows.

Given that the methods use 2-3 different hypotheses, they obviously are relying on the training procedure's inductive bias (otherwise how can 2-3 samples explore the full space of "good" hypotheses?)

Indeed, it is expected that one needs to outline many hypotheses to considerably explore the hypothesis space (as shown in proposition 2). However, D-BAT and DivDis were shown to work in practice with only two hypotheses, making us curious about the components (we study inductive bias and data distribution property) that they rely on to confine the hypothesis space enough to achieve this efficiency. We investigate this and show that the spurious ratio, architecture, and pretraining strategy are important components of the success of these methods. Furthermore, we show the influence of each component (Sec 4 and Sec 5) and their interaction (Sec 5.3).

What if we do model selection over multiple architectures and multiple diversity algorithms? Is there risk that the results we get on a cross-validation set may not generalize to the test set?

As shown in Figure 3, diversification methods do not work well without pretraining, so the model selection boils down to choosing a diversification loss and a pre-trained model (which includes pretraining strategy and architecture).

As mentioned above (also shown in Figure 3), the pretraining performance on ImageNet is surprisingly not a good proxy for selecting the pretraining strategies. However, a validation set can be used as one of the ways of model selection, and the risk of results' generalization to the test set depends:

• When the validation set is i.i.d to the test set, we can safely do the model selection. Through this work, we have provided important axes (diversification loss, pretraining methods, and their combination) to identify the best model.
• In terms of cross-validation, i.e., assuming the validation set is i.i.d to the training set and not to the test set, it is not possible to do the model selection. To the best of our knowledge, existing methods in the field do not consider this case and there is generally no solution for this setting.
• Finally, when the validation set is i.i.d to the unlabeled data, which may be non-i.i.d to the test set, it depends on how many spurious features are present in the dataset.
  • When there is only 1 spurious feature, knowing the spurious ratio of the test set can help in simulating the corresponding validation set, and the model selection can be done across diversification loss, architectures, and pretraining strategies.
  • When there is more than 1 spurious feature, knowing such summary statistics is still good, but there is a risk that the results may not generalize to the test set. (see further discussion in the next answer)

What are the summary statistics of the test set that the above procedure would need to know? For example, if the spurious ratio of the (unseen) test set is known, can that be used to simulate a pseudo-test set and then do model selection over it?

As discussed above, the number of spurious hypotheses and the spurious ratio of each of those hypotheses are good summary statistics to know. In most cases, the number of spurious hypotheses is equal to 1 and there will therefore be only one spurious ratio.

However, as shown in Sec 5.3, when there are more than 1 (2 in this case) spurious hypotheses ($h_1$ & $h_2$) present in the training set, things can be a bit different. Having the spurious ratio for each spurious hypothesis may not help, as different types of the model may latch on a different spurious hypotheses because of their inductive bias, and it’s known to be hard to measure such inductive bias apriori. Thus, the given spurious ratio may not be valid to simulate a pseudo test-set which is valid for all models.

Although there is risk in some cases where no good model selection strategy is available, we would like to highlight and clarify that, the efforts done in this work are focused on the diversification stage (first stage of the two-stage framework shown in Sec 3.2). In cases where there is no reliable way of model selection, which can be known when given the above summary statistics, an ultimate way of model selection can be considering all the axes (diversification loss, pretraining, architecture) above and conducting the second stage (the disambiguation stage) with the hypotheses produced by spanning those axes. Therefore, in the second stage, the best diversification loss & pretraining strategy can still be easily selected with a small amount of additional human supervision, e.g., labeled data points or human observation on the Grad-CAM [1] feature map. This means the diversification method is still useful but may have to pass more burden to the second stage in order to reliably find the hypothesis that is close to the true one.

We thank the reviewer for their time reviewing our paper and providing useful feedback, we kindly answer the raised questions as follows.

Regarding Weakness 1: the spurious ratio

Thanks for pointing out the confusion. Indeed, this can be confusing and we will carefully consider a better name (e.g., agreement ratio as suggested) in the next revision. The new definition can be:

Definition 1 (Agreement Ratio) Given a hypothesis h, the agreement ratio $r_D^h$, with respect to a distribution $D$ and its true labeling function $h^∗$ is defined as the proportion of data points where $h^∗$ and $h$ agree, i.e., have the same prediction: $r_D^h = \mathbb{E}_{x \sim D}[h^*(x) = h(x)]$.

We also would like to comment that, though $h^*$ has $r_D^{h^*}=1$, this relation is not symmetric, i.e., a hypothesis $h$ with $r_D^{h}=1$ does not mean $h=h^*$. This is because the quantity is defined on a distribution $D$. For example, on $D_t$ where complete spurious correlation exists, there exists a $h$ such that $r_{D_t}^{h}=1$, but on unlabeled data $D_u$, the same $h$ may have $r_{D_u}^{h} \in [0,1]$. In the scope of our work, we mostly examine this quantity on $D_u$, and we assume the test set $D_{ood}$ is a distribution where no spurious correlation exists, i.e., $r_{D_ood}^{h} \approx 0.5$ (so that the test set can reflect how well the diversification methods find the true hypothesis $h^*$). We hope this could address your confusion.

Regarding Weakness 2: Figure 2 and definition of hyperplanes

As mentioned in the description of Sec 4.1, $x_1$ is the horizontal axis and in this case, the reviewer is correct that $h^*(x)=\mathcal{I}\{x_1 > 0\}$ would be 1 for the points on the right. This is indeed a typo, and we thank the reviewer for noticing it. We will correct it. However it should be noted that this doesn't make any difference in the conclusions made in Proposition 1, and in the behavior illustrated in Figure 2, given that we're in the symmetric binary classification case and the classes are balanced.

For defining the hyperplanes, we wanted to match what D-BAT does (in Figure 17) and therefore used the radian of the classification plane w.r.t horizontal axis $x_1$ as mentioned in Sec 4.1, instead of the normal. We hope this allows for a better readability.

Regarding the no-free-lunch findings

We kindly refer to the ‘The significance of the no-free-lunch results’ section in the general response for a detailed answer.

The results for DivDis in Table 1, section 5.2 for increasing K are not convincing because alpha needs to increase as the square of the number of hypotheses in the set (i.e., O(K^2)).

We assume the reviewer means alpha needs to decrease (rather than increase) when increasing K in order for the regularization not to become too strong. The reviewer's concerns are correct given Equation 2, as the diversification losses are summed, and the value is not normalized w.r.t the number of hypotheses K. In practice, however, we always average the diversification losses instead of summing them up, which results in having $\alpha_K = \frac{\alpha}{K(K-1)}$. The $\alpha$ in the numerator is what we keep fixed in practice, which corresponds to scaling the loss by a factor of $~1/K^2$ as the reviewer rightfully suggested. We thank the reviewer for noticing this confusion. We will update the loss function to reflect the normalization term.

We hope we have addressed your questions, and we of course remain at your disposal for any further questions you may have.

Thank you for replying to my questions and concerns. Here are my responses:

1. Regarding the spurious ratio, I had understood that r_D^(h) depends on D and that r_D^(h) can be 1 even when h != h*. But thank you for verifying I was aware of this.

2. On figure 2 and the definition of hyperplanes, thank you for confirming my assumption and dispelling my confusion. I had stared at that figure for a long time :-)

3. Regarding alpha and table 1, that you for clarifying that there was a hidden correction for the number of hypotheses, "K". I am now convinced for the results in table 1.

4. Finally, on the NFL, you state that "your results do do not directly follow from the original NFL, similar to the results that researchers in the meta-learning field [2,3] or other fields [4,5] obtained, which can also be seen as expected in hindsight."

But actually I think they do follow from NFL. Imagine that a diversification method existed called "DivBest" was in fact sufficient for OOD generalization. Then one could have a free lunch in the supervised setting by: i) Collecting a large training set of labeled examples. ii) Discarding half or more of the labels to form an unlabeled data set. iii) Running DivBest on the labeled + unlabeled data to learn a model \hat{M} with DivBest's OOD generalization guarantees.

You have addressed 3/4 of my concerns so I will revise my overall score upward.

The larger concern that "results follow from the no-free-lunch theorem" still seems open to me. Or, if the results do not follow from NFL, the paper in its current form does not make it clear why. If the authors have insights here, I'd be appreciate if they could share them.

The significance of the no-free-lunch results

We would like to clarify the significance of our results on the insufficiency of diversification alone for OOD generalization, i.e., the no-free-lunch result.

1. Indeed, this result might be expected given the no-free-lunch theorem for the standard supervised setting [1]. However, it does not directly follow from the original NFL, similar to the results that researchers in the meta-learning field [2,3] or other fields [4,5] obtained, which can also be seen as expected in hindsight.
2. We would like to note that we present this result not as a mere critique of the diversification methods but as a meaningful step in studying them from the theoretical perspective. It suggests that there is no significant benefit in diversification-only from the theoretical perspective.
3. However, despite neural networks being universal approximators [6], and NFL suggesting that each task requires a specific inductive bias, neural networks are successful in practice. Similarly, our theoretical result does not need to translate into poor generalization in practice. We, therefore, follow by a more holistic experimental analysis of these diversification methods and study how they perform in practice. We show that while being successful in some scenarios demonstrated in the original papers, the diversification mechanism does not bring benefits on top of standard ERM training in more pathological situations (e.g., as shown in Tab. 1, and Sec. 4).

Based on the no-free-lunch suggested in our theoretical results, our findings provide overlooked factors to be aware of in order for diversification methods to achieve good results, and we clarify that in practice the actionable point for applying existing diversification methods is to jointly consider the axes we identified (data spurious ratio, model choices, etc) and refer to the disambiguation stage (as mentioned in Sec 3.2) for the best model.

[1] Wolpert et al., No Free Lunch Theorems for Optimization, IEEE Transactions on Evolutionary Computation, 1997
[2] J. Baxter, A Model of Inductive Bias Learning, Journal of artificial intelligence research, 2000
[3] J. Baxter, Theoretical Models of Learning to Learn, in Learning to Learn, 1998
[4] Bendidi et al., No Free Lunch in Self Supervised Representation Learning, 2023
[5] Hanneke et al., A No-Free-Lunch Theorem for Multi-Task Learning, The Annals of Statistics, 2022
[6] Hornik et al., Multilayer Feedforward Networks are Universal Approximators, Neural Networks, 1989