Diversity-Aware Agnostic Ensemble of Sharpness Minimizers

• I think that Theorem 1 does not help understanding the paper, and I am not convinced by its significance, as explained below.

  • First, the ensembling nature of $\theta$ is actually very superficial in the proof; it's simply dealt by considering that ensembling performs in average better that its members, thanks to the convexity of the loss. And thus the long computations (from page 13 to the middle of page 16) can actually be made for a single model. Actually, is this novel or is it from an existing paper? Then authors state "For the first time, we connect the sharpness-aware minimization with ensemble learning"; but actually your proof is not specific to ensemble, as you actually simply use the convexity of the loss to upper bound the ensemble by its members. This also questions the tightness of the bound.
  • The underlying assumptions are only stated along the proof, and not before.
  • Should'nt it be Q (instead of P) in the Equation from Page 16 just after "For the loss on the ensemble classifier, we use the assumption"?
  • The last line of the proof is not explicited. The "Finally, we obtain" is not clear; could you please clarify which equations are you combining to obtain the final equation?
  • Also, The C universal constant is not defined.
  • Also, please keep the equations numbered when possible, it facilitates reviewing.
  • More specifically, authors state that "The trade-off parameter $\gamma$ signifies the levels of sharpness-aware enforcement for the ensemble model alone and its base learners themselve". This coefficient gamma should be better introduced, to clarify how it appears in the proof (I suspect this is related to the unclear "Finally, we obtain"). Therefore it is not clear how you change it in Figure 1 and Appendix D.3. I speculate that for $\gamma=1$, you actually compute the upward gradient on the ensemble rather than each member, which would reduce performance because you would give the same upward gradient for each member and thus reducing diversity.

• The explanations to explain why applying diversity in the upward are hand-wavy and not compelling. The arguments (based on Taylor expansion) between the congruence (do you mean (cosine) similarity?) between the gradients and the diversity loss are not clear.

• The training paradigm would be complex to scale, as it requires simultaneous training of multiple deep networks.

• The experiments are only on medium scale datasets, and the performances are far from state-of-the-art, despite using very costly ensembling solutions. In case of computational limitation, I would suggest fine-tuning scenarios on larger datasets rather than training from scratch.

• A missing and important baseline is one that includes diversity regualrization in the downward of the SAM rather than the upward.

• Appendix D.5. should be included as an important section of the main paper, and needs to be further analyzed. Also note that in terms of disagreement (D), your strategy actually remains very far from standard SGD in Table 14.

• "To the best of our knowledge, we are the first to explore the connection between ensemble diversity and loss sharpness". No actually, "Diverse Weight Averaging for Out-of-Distribution Generalization" previously showed that SAM flattens loss landscape but actually reduces diversity across different models.

• Minors

  • The anonymous github is expired.
  • Authors state "The reason is that we aim to diversify the base learners without interfering the their performance on predicting ground-truth labels. " This was first used in "Improving adversarial robustness via promoting ensemble diversity", please cite accordingly.
  • Tables do not fit in pages in Appendix

It would have been good if the paper compared the trained models not only on the ensemble performance, but in terms of the individual models.

1. How do the individual models compare in terms of performance? Do the average SAM and average DASH models perform similarly, with the only difference being diversity among the models?
2. How different are within-ensemble models? You could measure pairwise disagreement or even your diversity objective. Are DASH models empirically more diverse?

(minor) the bright green links are somewhat jarring. (minor) Figures 1-3 use space inefficiently.

• (Unclear motivation and underdeveloped problem statement) This paper aims to integrate two previously established and effective methods and its novelty appears to be notably constrained.

• (Figure 2 and confusing claim) I am confused if Figure 2 reflects any real empirical or theoretical insights to support the claim that the diversity is reduced because different models use the same mini-batch $B$ in the normalized gradients. A question arises: deep ensemble trains the models independently, in that case, how could the two independent models use the same ascent gradient computed from the same mini-batch data?

• (Lack of theoretical proof) The theoretical work only shows that combining SAM and deep ensembling can provide a reduced generalization bound. The main contribution of using the KL divergence term is not supported by any proof.

• (The deep ensemble baseline accuracy is unusually low) The SGD deep ensemble baseline (first row of Table 1) uses 3 ResNet18 to obtain 93.7% test accuracy on CIFAR10, and 75.4% on CIFAR100. However, in Table 1 of [1], a single ResNet18 with SGD can obtain 95.41% on CIFAR10 and 78.17% on CIFAR100. Therefore, the validity of the reported baseline results needs to be clarified.

• (Unfair baseline comparison) The paper mentions they use SGD optimizer for baseline methods (DST, EDST, Snapshot). However, their proposed method DASH uses SAM as the default optimizer. The evaluation can be more fair if all of the baseline ensembles can be combined with SAM and the hyperparameter $\rho$ can be carefully tuned. Then we can know that DASH is an effective ensemble method compared to the previous ensembling method instead of observing the improvement from better optimizer SAM.

• (Lack of baseline comparison) The proposed DASH method is only compared with a few efficient ensemble methods (like Snapshot, FGE, and EDST) without the need to train multiple runs. However, the proposed DASH indeed needs to be trained multiple times. The evaluation can be more solid if compared more with methods that also need multiple training runs(e.g. [1]).

• (Limited evaluation of small models and small datasets). The author claims they only studied small models because their method inherits the "principles of classic ensembles (bagging or boosting)". I don't think it is a valid claim since a lot of NN ensembling work also grounds their method on the classic ensemble principles and works well in medium and large models (e.g. WideResNet). Also, relative to larger datasets ImageNet, the WRN, and ResNet50 can be considered weak learners.

• (Missing baseline hyperparameter settings) For results of Table 1, 2, the hyperparameter ($\rho$) range of SAM is not provided, it is possible the SAM baseline is not carefully tuned to show the improvement of the DASH method.

• (minor) The provided code link is expired.

[1] Du et al. "Efficient sharpness-aware minimization for improved training of neural networks."

[2] Wenzel et al "Hyperparameter ensembles for robustness and uncertainty quantification."

1. The relation between the upper bound presented in Theorem 1 and the proposed DASH algorithm appears weak. The insight provided by Theorem 1 is that the generalization error can be upper-bounded by a weighted sum of the SAM losses for individual ensemble members and the SAM loss for the entire ensemble model. However, it appears that the diversification term proposed by the DASH algorithm is unrelated to this upper bound.

2. There is a lack of experimental evidence to support the occurrence of phenomena similar to what is depicted in conceptual Figures 2 and 3 when applying sharpness-aware minimization in ensemble learning. Besides, it appears improbable that ensemble members, beginning with different initializations, would converge to the same basin (Fort et al., 2019).

3. The proposed algorithm simultaneously trains $m$ parameters, which has limitations in terms of scalability (i.e., we should load $m$ model copies and corresponding computational graphs to memory). Furthermore, while the training method where multiple model copies repel each other resembles the POVI approaches involving repulsion (e.g., D’Angelo and Fortuin, 2021), this connection remains not addressed in the paper.

4. There are no standard deviations present in the experimental results.

Fort et al., 2019, Deep Ensembles: A Loss Landscape Perspective.
D’Angelo and Fortuin, 2021, Repulsive Deep Ensembles are Bayesian.