Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks

Thanks for reviewing our paper and finding our paper "well structured and easy to follow" and stating "the claims are clearly formulated and well supported empirically". Also, the authors thank you for your recognition that the contributions are novel and significant when compared to the previous work [24].

########

Q1: The Investigation of 6 Special Noise Directions.

A1: A great question! Our paper's major observation is that the sign consistency between DNN parameters and noise determines the symmetry. Through your question, the authors guess that the level of asymmetry may be also related to the property of perturbed parameters, e.g., the total number and distribution of the parameters.

To answer Q1, the special noise $\epsilon_5$ and $\epsilon_6$ does not show asymmetry because it only affects the positive parameters.

If the parameters are all positive, they may be relatively stable when adding or subtracting some specific values. Otherwise, if the distribution of parameters is more disordered (e.g., containing various scopes of positive and negative values), the parameters may be more sensitive to sign-consistent perturbation. We investigate the negative-sign noise directions and find similar results.

This guess may also provide explanations for BN layers, i.e., Q2.

########

Q2: The Finding Holds for ImageNet and Various Parameter Groups.

A2: First, we use Gaussian noise. Second, compared with other groups of parameters, BN parameters (especially the BN.weight) have a simpler parameter distribution. According to the first question's guess (e.g., A1), BN parameters tend to be less sensitive to sign-consistent noise.

This observation does not contradict the findings in [24].

First, they find that adding random noise directions on BN parameters is more asymmetric, which is due to the possible larger sign consistency.

Second, adding totally sign-consistent noise to BN parameters indeed leads to asymmetry valleys (e.g., Fig.6 and Fig.7), but the asymmetry tendency is less obvious.

Third, applying totally sign-consistent noise to other parameters (e.g., a complex parameter distribution) leads to obvious asymmetry valleys.

As an initial guess, the significance level of asymmetry has the following rank: A > B > C > D.

A denotes a complex parameter distribution perturbed by sign-consistent noise. (100% sign consistency)

B denotes a simpler parameter distribution perturbed by sign-consistent noise. (100% sign consistency, but the parameter distribution is simpler)

C denotes a simpler parameter distribution perturbed by random noise. (a larger probability of > 50% sign consistency)

D denotes a complex parameter distribution perturbed by random noise. (about 50% sign consistency)

The previous work [24] found C > D (i.e., directions on BN parameters are more asymmetric), while our paper found A > D and B > C (i.e., sign-consistent noise leads to asymmetry).

Your question helps us guess that A > B, which derives the entire rank.

In summary, your great question brings us a novel guess about the influence of parameter distribution on valley symmetry. This guess should be checked further. Thanks for your insights.

########

Q3: Line 186.

A3: We are sorry that this is a mistake. The distribution is centered around 0.5, while the ones under the initialization of U(0, 1) are centered around 0.2.

########

Q4: Section 4.2.1.

A4: We only apply noise sampled from {0, 1} to BN parameters, which conforms to the experimental setting in [24].

We also have some experimental studies that apply noise to the whole parameters when considering the influence of BN, and please see the details in Appendix A.4. Specifically, we compare the valley symmetry of DNNs with or without BN layers.

########

Q5: Section 5: Did the authors train a plain linear classifier or a neural network in the sklearn.digits experiment?

A5: As we stated in Section 5 (line 260), the demo code is provided in Appendix D.3. The demo code is listed in Python code 2 (Page 23).

The code trains a plain linear classifier as a simple illustration.

########

Q6: Section 6.2

A6: Thanks for your correction! The design of this loss follows the idea of Negative Log Likelihood, where we miss a minus. We will correct this formula in the future version.

We rerun the experimental studies five times and list the std. of accuracies for a portion part in Table. 1. The experimental results are as follows. The accuracy doesn't fluctuate very much.

All results will be added in the future version.

FedAvg does not have additional hyper-parameters. FedProx, MOON, and FedDyn have a regularization coefficient like $\gamma$, and we search for them in the scope of {0.1, 0.01, 0.001}. The best results are reported. The hyper-parameters of FedPAN are set according to the original paper.

########

Q7: A valuable related work.

A7: Thanks for your recommendation! This is indeed related to our work.

Thanks for reviewing our paper and finding our observation reasonable.

########

Q1: The asymmetry is not unexpected and has been studied in [24].

A1: Surely, our work is majorly motivated by the proposal of asymmetry valley in [24], and we have declared this relation in lines 28-29 and 67-70. We have also declared that "the asymmetric valley is initially proposed by [24], while it does not propose the inherent principles behind the phenomenon" in Appendix A.4. Appendix A.4 also in-detail presents our motivation for changing the sign of noise to explain the asymmetry valley, which has not been studied in previous works. Compared with [24], we have the following additional contributions:

(1) we provide both empirical explanations and theoretical insights for the asymmetry valley;

(2) we provide a novel concept named sign consistency ratio;

(3) we provide detailed analysis for the DNNs with BN layers and show the initialization matters;

(4) we also provide specific applications to model fusion, including model soups and federated learning.

Additionally, the reviewer QEpC has recognized that "to my knowledge, the question of minima asymmetry was not systematically investigated after its first appearance in [24], so I consider the contributions of this study novel and significant".

########

Q2: The sign consistency is intuitive and does not emerge as a significant contribution.

A2: We don't agree with the reviewer's view. When reviewers think that this explanation is intuitive and reasonable, they recognize the comprehensibility and acceptability of our work.

However, they do not know the process of discovering this explanation required a lot of experimental studies, such as the process in Appendix A.4 and the extensive visualization results in the paper. These experiments include validation results on multiple data sets, various DNNs, different initialization methods, and even different DNN blocks on ImageNet.

The authors advocate that summarizing a reasonable explanation that is easy to understand and accepted by the community through abundant experimental results in various scenarios is a significant contribution to the research area.

########

Q3: Lack of theoretical analysis.

A3: The lack of strict theoretical proof is indeed our weakness, but we provide some theoretical insights to support our findings.

The reviewer MUA2 thinks that "the theoretical insights are convincing".

The reviewer QEpC also declares that "although theoretical support is mostly intuitive, with no rigorous proofs (the authors mention that in the Limitations section, which is commendable), it paves the way for future work that will address this limitation".

We will try our best to provide formal theoretical results in future work.

########

Q4: The application to local-search based optimization methods, such as SGD and SAM.

A4: Surely, the traditional works focus on searching for a wider minima that has better generalization performance, which has been stated in lines 76-79. However, these previous works majorly assume the width of minima is well correlated with the generalization performance, which is not inclusive until now.

Hence, we provide a novel application area (i.e., model fusion) and apply our findings to federated learning. Additionally, our application to federated learning is well-motivated, which is also recognized by the reviewer MUA2 and the reviewer QEpC.

Thanks for reviewing our paper and finding our paper's strengths, e.g., "the paper is well-structured and clear", "the theoretical insights are convincing", and "the proposed method is simple and effective".

########

Q1: The different architectures in Section 6.1.

A1: As stated in lines 263-265, the pre-trained model could make the traditionally encountered barrier disappear when linearly interpolating models [55, 64]. The previous works have already shown that pre-training is important for the success of model fusion [55, 64]. Hence, our focus in Section 6.1 is not to support this conclusion further. Instead, we would like to show the correlation between sign consistency and the model interpolation performance. Using different DNN architectures could support the correlation convincingly.

According to the reviewer's suggestion, we also replace the pre-trained resnet18 with a random initialized one, and we find the plots tend to be like the left part of Fig. 11 (i.e., becoming similar to the random initialized VGG16BN). We will add these plots for better illustration in the later version. These three groups of plots could better verify the correlation found in Section 6.1.

########

Q2: The valley flatness of BS 32 and BS 2024 in Fig. 8 and Fig. 9.

A2: To answer this question, we must declare two points of view.

(1) The valley width is closely related to the method of visualization.

This point of view has been stated multiple times in our paper, e.g., the lines 60-69, the lines 95-99, and the lines 198-200. Indeed, large batch training may lead to poor generalization compared to small batch training. However, whether the large batch training leads to sharp minima is still inconclusive.

First, the minima width in the visualization depends on both the parameter scale of $\theta_f$ and $\epsilon$ (e.g., previous works [37, 42] found) and the direction of $\epsilon$ (e.g., our paper found). For example, the plot in Fig. 8 adds $\epsilon=\theta_{f_2} - \theta_{f_1}$ to the BS 32 solution $\theta_{f_1}$, while Fig. 9 adds the norm-scaled $\epsilon \sim G(0, 1)$ to the BS 32 solution $\theta_{f_1}$. Different noises have different parameter scales and directions, which makes the visualized shape inconsistent.

Second, this phenomenon could also be found in the previous work [42], i.e., the Fig.2 vs. Fig.3 in [42]. In other words, the inconsistency of valley width in Fig.8 and Fig.9 is rational.

(2) Our work does not focus on the valley width but on the valley symmetry.

We have also stated this point of view in lines 27-30, 69-70, 94-95, 106-107, 198-200, and 205-207, etc. The inconsistency of valley width has been explained in [42]. We find that the valley symmetry between Fig.8 and Fig. 9 also differs a lot, which is because of the sign consistency ratio as shown in the upper-right of Fig. 8.

To conclude, perturbed by norm-scaled noise, BS 32 indeed lies on a flatter minima (Fig.3 in [42] and Fig.9 in our paper). However, perturbed by the same noise $\epsilon=\theta_{f_2} - \theta_{f_1}$, BS 32 lies on a sharper minima (Fig.2 in [42] and Fig.8 in our paper). This implies that the valley width is closely related to the visualization method, which further makes the correlation between valley width and generalization performance not so clear. Our work does not focus on this debate and aims to study the reasons behind valley asymmetry.

########

Q3: The conclusion from Fig. 8 also conflicts with the statement in related work that “large batch size training may lead to sharp minima with poor generalization.”

A3: Indeed, large batch training may lead to poor generalization compared to small batch training. However, whether the large batch training leads to sharp minima is still inconclusive, which is stated in lines 26-27. Additionally, the "conflict" has already been explained in [42] by proposing a proper filter-norm scaled visualization method.