The Implicit Bias of Gradient Descent toward Collaboration between Layers: A Dynamic Analysis of Multilayer Perceptions

Thank you for your valuable feedback and careful review. The rebuttals are as follows:

About the relation between co-correlation and adversarial robustness

The relationship between co-correlation and adversarial robustness is bridged by the Dirichlet energy, as depicted in Theorem 4.5 and Eq. (15), shown as

While co-correlation $\varrho_{\boldsymbol{\phi}, \boldsymbol{\varphi}}$ is not directly related to adversarial robustness, it is the most significant factor affecting the Dirichlet energy and the Dirichlet energy is directly related to adversarial robustness as shown in Thm. 4.1. Specifically, let $\phi \circ \varphi$ be a 2-layer neural network where $\varphi$ represents the first layer and $\phi$ represents the second layer. First, as demonstrated in Eq. (15), the Dirichlet energy of the network $\mathfrak{S}(\phi \circ \varphi)$ is dominated by the co-correlation $\varrho$, since the terms $\Big(1 + \frac{var_{\boldsymbol{\phi}, \boldsymbol{\varphi}}}{\mu^2_{\boldsymbol{\phi}, \boldsymbol{\varphi}}}\Big)^{\frac{1}{2}}$ are negligible as shown in Fig. 4. Seondly, our work concerns more about how the Dirichlet energy of layer 1 and layer 2, i.e., $\mathfrak{S}(\boldsymbol{\phi})$ and $\mathfrak{S}(\boldsymbol{\varphi})$, can be passed to overall Dirichlet energy of neural networks. And this interaction between layers can be quantified by co-correlation $\varrho$.

It is important to note that this paper primarily aims to understand the interplay between layers concerning adversarial robustness, i.e. $\varrho$, rather than adversarial robustness itself. Therefore, the direct relationship between co-correlation and adversarial robustness is not our primary focus.

About the value and the rate of convergence

As indicated in Thm. 5.3 and Thm. 5.5, Eq. (17) and Eq. (20), the exact value of $C(t)$ depends on several assumptions about the inputs and the state of the weight matrix $W(t)$. It is possible to provide a more precise evaluation of $C(t)$ based on specific assumptions about the inputs. However, doing so requires particular assumptions, such as Gaussian distribution of the inputs and weight matrix.

Because our work primarily aims to answer the research question of whether there exists collaboration between layers to improve adversarial robustness, we focused on qualitative analysis of dynamics change rather than quantitative analysis of the convergence speed. This is an interesting problem, but it is impossible to address all issues within a limited article. Future studies may explore this idea.

About co-correlation of MLP is smaller than linear net

As is shown on page 5, the co-correlation represents the feature alignment of layers in neural networks. Introducing an activation function, at least ReLU, as an 'isolation' layer between $W_1$ and $W_2$, is suggested to reduce co-correlation. This concept is experimentally verified in Fig. 1 (c) and (d). While we did not formally prove this idea, and different activation functions other than ReLU may even enhance co-correlation, our conclusion holds as long as the derivative of the activation function is bounded, as detailed in Assumption 5.4.

The effect of Dirichlet energy for each mapping

Yes, as shown in Eq. (15), the overall Dirichlet energy is also affected by that of individual mappings. However, since our primary focus is on the interplay between layers, this aspect is beyond the scope of our paper. Even if the Dirichlet energy for each mapping, $\mathfrak{S}(\boldsymbol{\phi})$ and $\mathfrak{S}(\boldsymbol{\varphi})$, are large, co-correlation still represents the interplay between $\phi$ and $\varphi$. Additionally, since other statistics are negligible shown in Fig. 4, whether the high value of Dirichlet energy for each mapping can be transferred to $\mathfrak{S}(\boldsymbol{\phi} \circ \boldsymbol{\varphi})$ still highly depends on $\varrho$.

Given the limited time available to respond and the lack of feedback on whether I adequately addressed your question, I kindly request that you let me know if my response was satisfactory. This would help me identify areas where I can improve my work. Thank you!

Additionally, I noticed that your question is somewhat vague, and many aspects fall beyond the scope of our research. If possible, providing more specific questions would be greatly appreciated.

Thank you very much for your valuable feedback and careful review. Your detailed examination of the paper and identification of its flaws are greatly appreciated. My rebuttals are as follows:

The reason why study co-correlation:

The motivation for studying co-correlation actually comes from the research question:

Can different layers of a neural network collaborate against adversarial examples?

This is an important point, especially in an era dominated by complex neural networks. For example, in the main block of a vanilla Vision Transformer, we have Multi-head Self Attention (MSA) followed by an MLP. People may expect these components to have specific functionalities and imagine the forward propagation working like independent robots on a conveyor belt. However, even for a simple MLP, there is hardly any research on whether different layers interact with each other, let alone on the effects on adversarial robustness. Our work aims to fill this gap to some extent.

How co-correlation related to adversarial robustness:

The relationship between co-correlation and adversarial robustness is bridged by the Dirichlet energy, as depicted in Theorem 4.5 and Eq. (15). While co-correlation $\varrho_{\boldsymbol{\phi}, \boldsymbol{\varphi}}$ is not directly related to adversarial robustness, it is the most significant factor affecting the Dirichlet energy as shown in Thm. 4.1. Specifically, let $\phi \circ \varphi$ be a 2-layer neural network where $\varphi$ represents the first layer and $\phi$ represents the second layer. Since the terms $\Big(1 + \frac{var_{\boldsymbol{\phi}, \boldsymbol{\varphi}}}{\mu^2_{\boldsymbol{\phi}, \boldsymbol{\varphi}}}\Big)^{\frac{1}{2}}$ is negligible, the co-correlation $\varrho$ dominate how the Dirichlet energy of each layer can be passed to that of neural network.

About L163:

The use of Dirichlet energy to measure adversarial robustness was first considered in the paper by Dohmatob et al. [8], as mentioned in line 137. It was originally proposed to replace the Lipschitz constant as a better measure of adversarial robustness, as shown on page 14, Appendix A, in [8]. In our work, we first demonstrate that Dirichlet energy can measure the gap between generalization and adversarial risk, implying that Dirichlet energy is a proper measurement for adversarial robustness since higher Dirichlet energy indicates a larger gap.

With simple modifications, we can also calculate the Dirichlet energy for a single layer or blocks within neural networks, as defined in Eq. (7) in Def. 3.2. We use this concept because of the decomposition of the Dirichlet energy for neural networks, as shown in Eq. (15). If we regard the neural networks as compound functions, i.e., $\theta \circ \phi$, the decomposition of the Dirichlet energy for this compound function requires the computation of Dirichlet energy for $\phi$ and $\theta$, each of which can be an individual layer inside the neural network. This explains the first sentence that we can evaluate the adversarial robustness of individual layers. I apologize for the misuse of “adversarial robustness,” which will be corrected in a later version.

For the second sentence, as is shown in Eq. (15), since the term $\Big(1 + \frac{var_{\boldsymbol{\phi}, \boldsymbol{\varphi}}}{\mu^2_{\boldsymbol{\phi}, \boldsymbol{\varphi}}}\Big)^{\frac{1}{2}}$ is negligible shown in Fig. 4, leave co-correlation $\varrho$ dominating whether Dirichlet energy for each mapping can be transferred to overall Dirichlet energy. Hence, we say that we can assess the collaboration between layers via $\varrho$.

About L117

The random and fixed second layer is used for simplification of our analysis, following the previous work by Du et al. [9], which was published in ICLR. Since we use similar approaches—directly analyzing the gradient descent—and to make the results easier to understand, we adopted the same setting. We will emphasize this in the paper of a later version.

About the claim in L160

Each neural network in Fig. 1 (a) and (b) was trained to converge under identical conditions, except for width and weight initialization. Therefore they demonstrate the relationship between Dirichlet energy and adversarial robustness, though not causally. However, as verification of Thm 4.1, it is adequate to show the correctness of the proof. And more experiments will be included in a later version.

About L192-L194

L192-L194 discusses the interpretation of co-correlation in a linear case without activation functions to simplify understanding. Since the activation functions will make the transformation at layers become non-linear, we can replace the weight matrix with the Jacobian. Let $\theta$ and $\phi$ be the weight matrices with activation functions, then Eq. (11) becomes

The explanation at L190-L194 is still applicable for the Jacobians of both layers. The difference is that the Jacobian varies with different inputs, potentially attect co-correlation, as empirically shown for ReLU in Fig. 1 (c) and (d).

Extra experiments

We conduct the experiments with ResNet50 and Wide-ResNet50 on CIFAR10. We divided the ResNets in two ways and used the Adam optimizer with a learning rate of 0.003 to track the dynamics of co-correlations. The experiment results are available at the link: Dynamic of co-correlation of the divide as $A_1$, $A_2$ and Dynamic of co-correlation of the divide as $B_1$, $B_2$. As shown in both figures, the co-correlation increases over the training epochs, although the upward trend for the separation of $A_1$ and $A_2$ is not strictly monotonic. The figure shown in the anonymous link: Illustration of the divide for ResNet50 shows the way we divide ResNet50.

We will check all lines and make them more clear in a later version.

Due to the limited time available to respond, and since I have not received any feedback on whether I answered your question, I kindly request that you let me know if my response was satisfactory. This would help me understand where I can improve my work. Thank you!

Thank you very much for your valuable feedback and careful review, you indeed checked the details of the paper and pinpointed the flaws. It is highly valued feedback. My rebuttals are as follows:

More experiments

Experiments on ResNet50 and Wide-Resnet50

We conducted additional experiments on larger and deeper networks, specifically ResNet50 and Wide-ResNet50 on CIFAR10. These experiments will be included in the paper in a later version. In the figures shown in the anonymous links: Illustration of the divide for ResNet50, we divided the ResNets in two ways and used the Adam optimizer with a learning rate of 0.003 to track the dynamics of co-correlations. The experiment results are available at Dynamic of co-correlation of the divide as $A_1$, $A_2$ and Dynamic of co-correlation of the divide as $B_1$, $B_2$. As shown in both figures, the co-correlation increases over the training epochs, although the upward trend for the separation of $A_1$ and $A_2$ is not strictly monotonic.

About varying depth and width

Your suggestion to conduct experiments on networks with varying depth and width, but the same number of total parameters, is quite interesting. However, in this paper, we focus on the theoretical analysis of whether there is collaboration between layers against adversarial examples. How different architectures of neural networks impact the co-correlation, especially with varying depth and width, will be considered in future work.

Negligible of other terms on different weight initialization

Yes, Fig. 4 in the appendix demonstrates that the other terms are negligible. Additional experiments with different initializations are available at Relative std and Linear correlation. We consider the MLP with a width of 512 with varying weight initializations. Similar to Fig. 4 in the appendix, the first and second figure show the $\frac{var^{\frac{1}{2}}}{\mu}$ and $\rho$ in Eq. (15). Empirically, it shows that $\Big(1 + \frac{var_{\boldsymbol{\phi}, \boldsymbol{\varphi}}}{\mu^2_{\boldsymbol{\phi}, \boldsymbol{\varphi}}}\Big)^{\frac{1}{2}} \rho_{\boldsymbol{\phi}, \boldsymbol{\varphi}}$ is quite close to 1, making it negligible in the Eq (15).

About Eq.(19) and training stage

According to Eq. (19), the first term of Eq. (19) is

The exact value of this term is complex; however, the sign of this term depends on $\sum_{\tau=1}^{t}\widetilde{\boldsymbol{x}}(\tau)^T\widetilde{\boldsymbol{x}}(t)$, since other terms are positive, as explained in Thm 5.3. According to Eq. (17), $\widetilde{\boldsymbol{x}}$ is a weighted average of all inputs, where the weight is the difference between the ground truth $y_i$ and the predicted likelihood $sig(u_i(t))$.

At the initial stage with a small learning rate, $\widetilde{\boldsymbol{x}}(\tau_1)$ is similar to $\widetilde{\boldsymbol{x}}(\tau_2)$ for $\tau_1, \tau_2 \in[t]$. More specifically, let $t$ be 5 epochs. Since the model hasn't learned much, the predicted likelihood $sig(u_i(t))$ of $\tau_1 = 1$ and $\tau_2 = 4$ are all similar to each other; therefore, $\widetilde{\boldsymbol{x}}(\tau_1)$ and $\widetilde{\boldsymbol{x}}(\tau_2)$ are similar in terms of cosine similarity.

However, in the later stages, as the model learns more from the data, most of the predictions become correct, and $y_i - \sigma(u_i)$ will approach zero, causing $\sum_{\tau=1}^{t}\widetilde{\boldsymbol{x}}(\tau)^T\widetilde{\boldsymbol{x}}(t)$ to converge. Continued training will also enlarge $\Vert W(t) \Vert_2^2$ and make $\Big(1 - \big(\boldsymbol{v}(t)^T\boldsymbol{a}\big)^2\Big)$ approach zero. Considering all the impacts, the co-correlation will increase very fast at the beginning, then the rate of increase will become smaller, and in the final stage, it will approach zero, as shown in Fig. 2 (c) and (d). In Fig. 2 (c) and (d), all the co-correlation of networks flattened at the end.

About weight regularization and co-correlation on Property 2

As shown in Eq. (19), the co-correlation is inversely related to the $L_2$-norm of the weight matrix. This $L_2$-norm is the operator norm, which is different from the norm used in weight regularization, such as L2 regularization, which is the Frobenius norm (F-norm). This implies that merely controlling the F-norm of the weight matrix may not have a direct impact on $\Vert W \Vert_2$. However, since all norms are equivalent in finite-dimensional vector spaces, L2 regularization will somehow restrict its increase during training. Additionally, since the value of $C(t)$ also depends on $\Big(1 - \big(\boldsymbol{v}(t)^T\boldsymbol{a}\big)^2\Big)$, where $\boldsymbol{v}$ is the maximal singular vector of $W(t)$, it is difficult to determine whether $C(t)$ will become larger because of weight regularization.

About L267

Yes, if the learning rate is too large, whether at the initial stage or at any stage, $\widetilde{x}^{\star}$ may fluctuate too much, but it may also prevent the model from converging or result in poor performance. In such cases, it is possible that $C(t)$ could become negative. However, in most machine learning settings, it is required that the learning rate not be too large, such as using 3e-4 with Adam, to ensure convergence and good performance. Therefore, I believe the assumption that the learning rate is not too large is realistic.

Due to the limited time available to respond, and since I have not received any feedback on whether I answered your question, I kindly request that you let me know if my response was satisfactory. This would help me understand where I can improve my work. Thank you！

Thank you for your valuable feedback and careful review. We appreciate the opportunity to clarify and expand on our work. Our rebuttals are as follows:

Theoretical Implications

Although it may not be immediately apparent in the paper, the implications of our work are significant. From a theoretical perspective on implicit bias research, our approach is distinct in assuming only L-2 norm-bounded inputs, Gaussian weight initialization, and bounded derivatives of the activation functions. This differs from most works on implicit bias, such as those by Lyu et al. [23], Frei et al. [12], Kunin et al. [20], and Frei et al. [13], where proofs are typically provided for 2-layer neural networks and rely on strong assumptions on inputs, making it difficult to generalize to deep neural networks. The extension to deeper neural networks in these works may require the same strong assumptions on features between layers, which is unrealistic. However, due to our assumptions on inputs needing only to be $L_2$-norm-bounded, our work can be more easily extended to deep networks. This extension was not included in the paper due to page limitations. However, we have provided experiments with ResNet50 and Wide-ResNet50 on CIFAR10 in this rebuttal and the later version of this paper.

In the figures shown in the anonymous links: Illustration of the divide for ResNet50, we divided the ResNets in two ways and used the Adam optimizer with a learning rate of 0.003 to track the dynamics of co-correlations. The experiment results are available at Dynamic of co-correlation of the divide as $A_1$, $A_2$ and Dynamic of co-correlation of the divide as $B_1$, $B_2$. As is shown in both figures, the co-correlation increases over the training epochs, although the upward trend for the separation of $A_1$ and $A_2$ is not strictly monotonic.

Additionally, our work offers a possible explanation for the observation that wider neural networks are more resilient to adversarial attacks. This complements recent findings by Frei et al. [13], who suggest that implicit bias leads to non-robust solutions. Our work suggests that these non-robust solutions can be mitigated by increasing the network width.

Practical Implications

Practically, our findings imply that designing an isolation layer to decouple the correlation between layers could enhance adversarial robustness. However, due to page limitations, we concentrate on the theoretic analysis of whether there is a collaboration between layers against adversarial examples. This aspect of how to reduce the co-correlation will be explored in future works.

We greatly appreciate your careful review. All typographical and grammatical errors will be corrected in a subsequent version. And all the suggestions will be considered. Thank you for your understanding and feedback.

Due to the limited time available to respond, and since I have not received any feedback on whether I answered your question, I kindly request that you let me know if my response was satisfactory. This would help me understand where I can improve my work. Thank you!