Scalable Ensemble Diversification for OOD Generalization and Detection

We thank the reviewer for the thorough review and positive comments about the method/results. The questions/comments are very useful for improving the paper. We added a number of new results (in the attached PDF) and made numerous clarifications to the paper (summarized below).

W1: Overlap of label spaces.

We will clarify upfront in the paper that this work considers two types of shifts - semantic and covariate [a - j]. Openimages-O and iNaturalist represent datasets with semantic shifts and their labels sets are disjoint with ImageNet1k label set. ImageNet-C dataset is a representative example of covariate shift for OOD detection. Its labels coincide with labels of ImageNet-1K, while the "style" of the images does not. Note that this dataset was also used in the latest version of the OpenOOD [a] referenced by the reviewer.

W2: Additional ablations.

We performed additional ablations as requested (different number of layers for the DeiT-3b feature extractor). See Table 5 in the attached PDF. Happy to add others if deemed necessary.

W3: Performance on small-scale datasets?

The whole point of this paper is to scale up the work presented in the original A2D and DivDis papers to a more realistic and useful ~ImageNet level.

Nonetheless, we performed additional experiments on the (small) Waterbirds dataset (see Table C in the authors rebuttal), since both A2D and DivDis also provided results on it. We report the worst group (test) accuracy for ensembles of size 4. We did not use the stochastic sum for our method to factor out its influence. A2D and DivDis use the validation set for disagreement. While DivDis discovers a better single model, the ensemble is clearly better with the proposed SED method.

W4: Additional comparisons

Thanks for the suggestion. We performed additional comparisons (see Table A in the author rebuttal). The proposed SED with model-soup aggregation performs better than the compared methods [m,n] across all OOD datasets.

W5: Demonstration of speedup. We performed an additional evaluation (Table 7 in the attached PDF) that shows the benefit of controlling the "subset size" in the stochastic sum ($\mathcal{I}$) on the speed of training an ensemble. For example, to train an ensemble of size 5, the time required for 1 epoch grows from 53s to 585s without stochastic sum ($\mathcal{I}=2$). We could not train an ensemble of 50 models without stochastic sum with our resources, but it already requires 7244s for ($\mathcal{I}=10$) vs 2189s for ($\mathcal{I}=2$).

Q1: Where is $\alpha_B$ used?

We do not use it directly in our method. It was provided in text to highlight the fact that weights $\alpha_n$ are small when ensemble is undertrained because the sum of all weights in batch equals $\alpha_B$ and is inversely proportional to average classification loss on this batch.

Limitations. Thanks for the suggestion. We added an evaluation of the computational time of the proposed method vs. others (see the attached PDF).

Thanks for the suggestion. We are keen to include additional comparisons in the final version to strengthen the paper. Suggestions of specific methods are welcome.

We provided results for additional OOD detection methods in the authors' rebuttal (see Table B: Ensemble Entropy, Average Entropy, Mutual Information, Average Energy, and the A2D disagreement as an uncertainty score [k,p]).

The comparison with BMA/deep ensembles (already in the paper) is probably the most important since this is considered state-of-the-art in recent studies [s, v, w].

Summary of results: the proposed PDS proves superior to all compared methods on all covariate and semantic shift datasets, i.e. C-1, C-5, iNaturalist, OpenImages.

[k] Pagliardini et al., Agree to disagree: Diversity through disagreement for better transferability, ICLR 2022.

[p] Xia and Bouganis, On the Usefulness of Deep Ensemble Diversity for Out-of-Distribution Detection, ICLR 2022.

[s] Ovadia et al., Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift, NeurIPS 2019.

[v] Mukhoti et al. Deep deterministic uncertainty: A new simple baseline, CVPR 2023.

[w] Dusenberry et al. Efficient and scalable bayesian neural nets with rank-1 factors. International conference on machine learning, PMLR 2020.

We thank the reviewer for the thorough review and positive comments about the intuitiveness of the method, the extensiveness of the evaluation, and the empirical results. The questions/comments are very useful for improving the paper. We added a number of new results (in the attached PDF) and made numerous clarifications to the paper (summarized below).

W1: Tradeoff between ID and OOD performance.

We supplemented the results tables with the ID performance were missing (see Table 1 and 4 in the attached PDF). The tradeoff in ID vs. OOD performance is inherent to the very nature of the problem and has been discussed extensively in the literature on domain generalization [u] (e.g. a model discarding spurious features because they do not generalize OOD necessarily becomes less performant on domains where these features are informative). The advantage of the "diversification" approach is that one member of the ensemble with particularly high ID performance can be selected, if this is the objective/selection criterion.

W2: Ablation study on stochastic sum.

We performed additional experiments on the subset size $\mathcal{I}$ of the stochastic sum, see Table 7 in the attached PDF. They evaluate the speed up from different $\mathcal{I}$ vs. performance. As we can see by comparing SED with $I=2$ and $I=5$ which have 42.6 vs 37.6 drop on IN-A and 48.1 vs 44.9 drop on IN-R speed-up helps OOD generalization. However, results for OOD detection are not so straightforward - having growth of AUROC on covariate shift datasets, e.g. 0.896 vs 0.903 on C-5 while dropping from 0.977 to 0.970 on iNat for semantic shift datasets.

W3: Comparison of PDS against other baselines.

We performed additional comparisons, see Table B in the authors rebuttal (ensemble entropy, average entropy, mutual information, and average energy [p]). BMA/deep ensemble is state-of-the-art as reported in recent studies [s]. Happy to include other methods if deemed necessary by the reviewer.

W4: Deep ensembles remain competitive in many settings, and the best values for C-1 and C-5 OOD generalization are still achieved using ensembles.

Deep ensembles are indeed competitive, especially with diverse sets of hyperparameters across its members. However, our SED-A2D shows better OOD generalization on 15/23 of the cases (i.e. 65%) in Table 1.

Q1: Application to DivDis.

Our preliminary experiments with DivDis showed that it was unable to create a diverse ensemble of >2 models. Intrinsic limitations of DivDis were indeed reported in prior work (see Appendix Section F.7 of A2D).

Q2: It's interesting that diversifying on ID samples improves OOD generalization compared to diversifying on ImageNet-R (Table 2). What was the setup for these experiments? Did the ensembles see fewer "OOD" data points in the latter?

SED makes the soft selection of "OOD" datapoints via $\alpha$ term in Equation 6. It is difficult to directly compare the number of used "OOD" datapoints for SED and when IN-R is used.

One possible explanation for why diversifying on IN-R did not help is that many samples in IN-R are still within the distribution of the ID dataset (IN-train). Therefore, enforcing disagreement on them may severely hinder the training of the main task for the ensemble, leading to a decrease in OOD accuracy. It is better to let the adaptive loss $\alpha$ decide which samples are safe for disagreeing - i.e. the ones which already have high CE loss.

Q3: In Table 1, I'm surprised that SED-A2D has each model giving a different prediction for C-1, but OOD detection AUROC using PDS is quite low. Does this imply that SED-A2D also has very high disagreement on ID data?

Yes, SED-A2D also has quite high disagreement on ID data. We show #unique and PDS on ID and OOD datasets, ID accuracy and OOD detection scores for the models in Table 1, 4 in the attached PDF.

It is not a problem because what matters is the relative values of PDS between ID and OOD samples. The PDS values (in parentheses) are respectively 4.16 and 3.98 for the covariate shift detector and 3.53 and 1.54 for the semantic shift detector (see Table 1 in the attached PDF). This enables OOD detection at a state-of-the-art level.

Thank you for providing so many additional experiments and ablations during the rebuttal process! Based on these new results, I am willing to increase my score to a 5.

We thank the reviewer for the thorough review and positive comments about the idea/clarity. The questions/comments are very useful for improving the paper. We added a number of new results (in the attached PDF) and made numerous clarifications to the paper (summarized below).

W1: Ablations; why the method works.

• Compare two methods that only differ on the OOD data used. See the additional experiment in Table 7 in the attached PDF. The ablation shows SED-A2D without the stochastic sum. Now we can compare the line of SED with $I=5$ in Table 7 in the attached PDF and Naive A2D from Table 2, 3 in the manuscript as they both do not include stochastic sum. Namely, in OOD generalization they become on par with each other with 44.9 vs 44.3 on IN-R dataset and 37.6 vs 37.8 on IN-A dataset, while in OOD detection SED is still superior in covariate shift case achieving 0.903 vs 0.850 AUROC on C-5 and on par in semantic shift case achieving 0.941 vs 0.939 on OI. As expected, average time spent on training for one epoch significantly decreases for ensembles of various sizes, e.g. from 585s to 53s.
• Why does stochastic sum improve performance? Stochastic sum is similar to bagging [t]: each model is trained on a different subset of the training data at each epoch. That imposes further diversity in addition to the disagreement loss. This is why it is unsurprising that the stochastic sum improves OOD generalization.
• Loss terms with contradicting objectives. The $\alpha_n$ (eq. 4) indeed balances the opposing effects of these two terms, depending on the "OODness" of the sample. Its value is proportional to the sample-wise classification loss. For an OOD sample, $\alpha_n$ is far greater (e.g. in the order of ~15 vs 0.01), which dwarfs the effect of the cross-entropy loss.

W2: Additional insights.

Thanks for these suggestions! We added a discussion of these points to the paper, which makes the analysis of the results much more informative.

• Why does oracle selection perform worse compared to simple average in Table 2? (despite using privileged information) Results with "oracle selection" refer to the selection of the best model per test dataset, not per test sample. It is possible that the best individual member is still worse than an ensemble or averaged weight of the members on a test dataset.

• Why was the A2D loss chosen instead of other losses e.g. DivDis? Our preliminary experiments with DivDis showed that it was unable to create a diverse ensemble of >2 models. Intrinsinc limitations of DivDis were indeed reported in prior work (see Appendix Section F.7 of [k]).

• Tables 2,4: why more ensemble components make SED-A2D worse? This is a mere artefact of the experimental conditions, which are now made clearer in the paper. Computational constraints dictated us to fix the number of epochs to a smaller value, such that each individual model is undertrained and suboptimal as seen in the table.

• Why is Eq. 6 preferable to an OOD dataset from model errors? The proposed approach is more straightforward and efficient, since there is no need to train an initial model to determine OOD samples. We also performed additional experiments showing on-par or better performance by SED (see Table 6 in the attached PDF). Namely, in OOD generalization SED approach (called joint in the table) tied with 2-staged one on IN-A while being slightly worse on IN-R - 48.1 vs 48.5. However, in OOD detection SED approach leads to superior performance across all OOD datasets having 0.896 vs 0.845 AUROC on covariate shift dataset C-5 and 0.941 vs 0.911 on semantic shift dataset OI.

• Can we introduce OOD samples via augmentations? This is an interesting suggestion. We performed a preliminary evaluation of this idea (see "synth" in Table 8 in the attached PDF). We performed ensemble diversification with small random crops (size sampled within 8-100%) on 30k samples of the ImageNet training set (same size as IN-R). That resulted in achieveing better OOD detection performance across the board in combination with any regularizer - having 0.647 vs 0.600 AUROC on C-1 and 0.974 vs 0.971 AUROC on iNat.

W3: What is the model batch size for the stochastic sum? How does performance for generalization/detection change with and without this speedup?

The size selected for each batch is $\mathcal{I}=2$ (L237). We report additional experiments in Table 7 in the attached PDF. They evaluate the speed up from different $\mathcal{I}$ vs. performance. As we can see by comparing SED with $I=2$ and $I=5$ which have 42.6 vs 37.6 drop on IN-A and 48.1 vs 44.9 drop on IN-R speed-up helps OOD generalization. However, results for OOD detection are not so straightforward - having growth of AUROC on covariate shift datasets, e.g. 0.896 vs 0.903 on C-5 while dropping from 0.977 to 0.970 on iNat for semantic shift datasets.

Q1. Other datasets than IN-R for the OOD samples.

We performed additional experiments with ImageNet-A ("Natural Adversarial Examples"). See Table 8 in the attached PDF. They show that disagreement on IN-A and IN-R have almost identical results on both OOD generalization and OOD detection tasks when using A2D regularizer. With the biggest gap in accuracy on IN-R for Div regularizer at 45.2 for disagreeing on IN-A vs 41.8 for disagreeing on IN-R. Similar drop can be observed on IN-A: 37.8 vs 36.3.

Q2: Instead of computing the detection scores, how does the proposed diversity measure correlate with correct predictions?

We provide an additional analysis in Figure-Table 3 in the attached PDF. It visualizes the ensemble accuracy vs. PDS for IN-Val, C-1 and C-5. Based on the figure, the accuracy seems to correlate negatively with PDS.

Q3: In Table 1, what is the IN-Val #unique? We added this and additional details to the table (see the attached PDF, Table 1).

Thanks to the reviewer for the thorough review and encouraging comments. The questions and suggestions are very helpful to improve the paper, and we propose several improvements (details below) that should make the final version much clearer.

We also added a number of new results (in the attached PDF) that were requested by other reviewers and clearly strengthen the paper.

W1: Terminology ("OOD samples" in an ID dataset)

Thanks for pointing this out, the choice of words was indeed confusing. The paper now simply uses "hard (ID) samples".

W1b: Theoretical questions.

Indeed these are important questions. A formal theoretical treatment is out of the scope of this paper but prior work has given support to the idea of diversification ([l]], [k], [r] on underspecification). We propose to summarize these points in the paper as summarized below.

• Why would diversifying models' predictions on hard samples help?
• Is such prediction diversification always conducive to OOD generalization?

The difficulty of OOD generalisation is rooted in the task being underspecified, i.e. multiple hypotheses are consistent with the training data. Diversification methods allow discovering a set of such hypotheses. In the proposed method, the assumption is that enforcing diversity in prediction space on hard samples can drive this set of hypotheses to contain one with better generalization properties. In particular, the chosen "hard samples" are assumed to be those where the model could make several valid predictions while still generalizing correctly on other "easier" samples.

The validity of these assumptions depends on a complex interaction between the data and the inductive biases of the model. Its formal study is an important direction for future investigations.

W2: Connection between SED and PDS.

The comments by the reviewer are very relevant. We propose to clarify these points with an additional discussion in the paper summarized below.

Does PDS measure epistemic uncertainty? Epistemic uncertainty measures how "abnormal" a data point is w.r.t. the training distribution. We propose to measure this degree of abnormality a given sample through the diversity of model predictions on it. Let us consider the following example [borrowed from the reviewer]. The predictions from an ensembles of two members on two samples $v$ and $w$:

• $p_1(v)=[0,1]$ and $p_2(v)=[1,0]$
• $p_1(w)=[1/2,1/2]$ and $p_2(w)=[1/2,1/2]$

Even though the naive ensemble returns the same prediction $(p_1+p_2)/2=[1/2,1/2]$, we would evaluate $v$ as more likely to be OOD w.r.t. the training distribution, as the predictions by ensemble members are more diverse. BMA would fail to capture this effect unlike PDS.

W3: Baselines.

Thanks for the suggestion. We have added a number of additional comparisons to the paper.

• OOD generalization: see Table A in the authors rebuttal. The proposed SED with model-soup aggregation performs better than the compared methods [m,n] across all OOD datasets. We could not obtain the code for [o] (which pursues the different goal of feature-space diversification).
• OOD detection: see Table B in the authors rebuttal. BMA/deep ensemble is state-of-the-art in several recent studies [s]. We also added comparisons with standard baselines: Ensemble Entropy, Average Entropy, Mutual Information, Average Energy [p].

We're happy to include other methods if deemed necessary by the reviewer.

W4: Definition of #unique.

Thanks for pointing this out. We clarified the definition in the paper.

• Does the value 5 means that all 5 models disagree with each other on every C-1 example? Yes.
• This suggests that for many examples, 4 out of 5 models are probably wrong. Why is this a good sign? If the models are to be ensembled, this extreme diversity could hinder generalisation. If the goal is to perform OOD detection (PDS score), the extreme diversification on OOD samples is beneficial as the degree of diversification is informative about whether a sample is OOD or not.

Thank you for the clarification. We show below how the provided example does not invalidate PDS as a metric of epistemic uncertainty. We also clarify our understanding of epistemic uncertainty and the soundness of ensembles in measuring it. The final version of the paper should be clearly improved by including this discussion.

Definition of epistemic uncertainty

Epistemic uncertainty is defined by a lack of training data in certain input regions [a,b].

Ensembles and epistemic uncertainty

The lack of supervision in OOD regions means that multiple models exist with similar training risk. They agree on training/ID data but disagree in their predictions on OOD examples [c,d,e]. This is the reason why the rate of agreement on OOD data is a valid measure of epistemic uncertainty.

Example given by the reviewer

Regardless of how much entropy each ensemble member exhibits, what eventually matters for the epistemic uncertainty is the degree of agreement among the members. It makes sense that both $v$ and $w$ have the identically low epistemic uncertainty as long as $p_1(v)=p_2(v)$ and $p_1(w)=p_2(w)$. The overall entropy of the predictions matters more for predictive uncertainty (concerned with the correctness of predictions) and aleatoric uncertainty (data uncertainty stemming from the entropy of the ground-truth $p(y|x))$. Epistemic uncertainty is thus related to the ensemble agreement rather than entropies [c,d,e].

We prepared a concrete demonstration of case suggested by the reviewer [1/2, 1/2], [1/2, 1/2] to be added to the appendix. We use a combined dataset of ImageNet-Val (ID) and OpenImages-O (OOD). We compute the predictive distribution of our SED ensemble on this data. We retain examples with a highly-similar distribution across ensemble members (i.e. high agreement) then rank them by decreasing entropy of the predictive distribution. As we expected/argued, the top examples (high entropy) do correspond to ID samples (from ImageNet-Val). Entropy thus cannot be used to reliably separate ID from OOD samples.

Soundness of PDS

PDS is precisely measuring the agreement among ensemble members. Unlike BMA, PDS disregards the overall entropy and focuses purely on the agreement. Previous studies [c,d,e] and our empirical results (Table 3 in the paper) support that PDS is a more suitable measure to determine the OODness of a sample than BMA and entropy.

Thanks to the reviewer for recognizing the key values of the paper - clear motivation, clear writing, and experimental results supporting the effectiveness of the method. The questions are also very useful to further improve the paper as detailed below.

We also added a number of new results (in the attached PDF) that were requested by other reviewers and clearly strengthen the paper.

W1: Additional comparisons

Thanks for the suggestion. We performed additional comparisons (see Table A in the authors rebuttal). The proposed SED with model-soup aggregation performs better than the compared methods [m,n] across all OOD datasets, namely, the respective accuracies gaps are: 38.1 vs 36.7 on IN_A, 45.2 vs 44.4 on IN_R, 77.3 vs 76.0 on C-1 and 40.6 vs 39.1 on C-5.

W2: Other/non-frozen feature extractor.

Indeed the method is applicable to other feature extractors. We performed additional experiments (Table 5 in the attached PDF). These results while almost identical for OOD generalizatoin show improvements on OOD detection with ResNet-18 (the main paper used DeiT-3b). Namely, AUROC grows from 0.670 to 0.686 on C-1 dataset with covariate shift and from 0.802 to 0.812 on OI dataset with semantic shift.

We are also preparing additional results with non-frozen feature extractor. They should be ready during the discussion phase and we will be added to the final version of the paper

W3: Applicability to other OOD tasks?

Yes indeed. We did not want to overclaim the applicability of the method but we propose to add the following discussion to the paper. We currently focus on OOD generalization and OOD detection, yet other relevant settings include:

• Domain adaptation: after diversifying the ensemble on the source domain, the ensemble member most suited to the target domain could be selected using labeled samples from the target domain.
• Unsupervised domain adaptation/test-time training: similarly, the ensemble member most suited the target domain could be selected unsing an unsupervised objective (e.g. [q]) from the UDA/TTT literature.
• Continual learning: diversifying the ensemble on the inital domain could provide multiple hypotheses to eliminate/adapt as new domains come in. Alternatively, further diversification could be triggered by training novel members that disagree on new domains to facilitate generalization to these new domains.

We thank the reviewer for the response to the rebuttal. We assume that there are some remaining weaknesses of the paper that do not allow to increase the score. Could you please share them as well to improve the final manuscript?