When Are Bias-Free ReLU Networks Effectively Linear Networks?

Thank you for your detailed review. We'd like to clarify our contributions and how they differ from prior work.

• Importance

  We very much agree with you that some prior works have studied cases where the ReLU networks behave like one or several linear networks. However, the connections between the ReLU and linear networks have not been explicitly highlighted and systematically summarized. We feel that these connections lend a valuable and attractive understanding of ReLU networks. To this end, our paper focused on specifying and explaining these connections as clearly as we can, aiming to make a number of previous results in the ReLU network subfield more intuitive and accessible to a broader audience. We're glad to see that you and other reviewers found our presentation very clear.

  Additionally, as noted by all three other reviewers, the impact of removing bias is an interesting topic because bias-free networks have been studied often in prior theoretical works. Discussing their limitations sheds new light on the conclusions from these prior studies.

• Soundness

  You are correct in pointing out that we did not pursue the direction of deriving bounds for initialization scale and learning rate. However, we believe that the rigor of our claims is at an appropriate level, in the sense that we did not overstate our results. For instance, the equality in Assumption 5 is indeed exactly conserved throughout training if it is true at initialization, which we prove in Section C.2.

  We do recognize that giving bounds on error terms is important. However, we believe that providing closed-form solutions under well-stated assumptions is also important and valuable. Moreover, while our assumptions and conclusions differ from those of Lyu et al, some are actually stronger: 1) we study square and logistic loss (Lyu et al focused on logistic loss); 2) we give closed-form time-course solutions to certain two-layer ReLU networks, which has not been written out; 3) we relaxed the assumption on the target from being linearly separable to being odd. We thus see our contributions as being complementary to theirs.

  To quote the final paragraph in Lyu et al: "A critical assumption for our convergence analysis is the linear separability of data. We left it as a future work to study simplicity bias and global margin maximization without assuming linear separability." Our work addresses part of this open question: to study simplicity bias without assuming linear separability.

• Large Initialization, Large Learning Rate

  Thank you for bringing this interesting question into discussion. We do have empirical evidence that some of our results still hold with a moderately large learning rate and large initialization. In our revised pdf, we have added Section C.6 in Appendix and some signposts in the main text to make it clear that: while our analytical derivations used vanishing initialization and infinitesimal learning rate, we have empirical evidence that some of our results extend to large initialization and large learning rates.

  In the added Figure 8, we use a learning rate of $0.6$, which is 150 times larger than the learning rate of $0.004$ used in Figure 2. Similar to Figure 2, the loss curves in Figure 8 with different leaky ReLU slopes collapse to one curve after rescaling time and the differences between weight matrices are small. If the learning rate is further increased, the loss and weights curves oscillate and the equivalence breaks; but we typically wouldn't let our networks train in this oscillating regime.

• Rephrasing

  We have deleted "as special cases" in line 48 and deleted "which has never been done for nonlinear networks outside the lazy learning regime" in line 58.

  You are correct that the rank-one structure is approximate for non-vanishingly small initialization. We have revised line 400 to clarify this point. Thanks for your suggestion.

• Thank you for sharing the references. We have included them where you suggested.

We look forward to hearing whether our revision and response address the reviewer's concern. Please let us know if you have further questions.

Thank you very much for your response. We really appreciate the time and effort you invested in helping us improve our manuscript for publication. We have updated our pdf according to your suggestions. Please let us know if our revision is appropriate and if you have any further feedback.

• Balancing Narrative, Specifying Assumption

  We do like how you frame our contributions not as an advancement of what is wanted, but as a warning of what is to be avoided, which is often under-represented in literature but just as important. We actually intended to convey the same message in the opening paragraph of our introduction: "This paper seeks to illuminate the implications of bias removal in ReLU networks, and so provide insight for theorists on when bias removal is desirable." We're sorry that we didn't get this point across effectively. We have revised lines 20, 53, 60 to adopt a more balanced narrative.

  We have revised line 48 and added a clarification of our initialization assumption in Remark 6. We hope that now Remarks 4 and 6 together clarify the relationship between our setup and that of Lyu et al -- our assumption on datasets is weaker and our assumption on initialization is stronger.

• Large Initialization & Large Learning Rate

  Thank you for this follow-up question. We have added Figure 6d to demonstrate that the loss and weights curves with a large initialization and a large learning rate are qualitatively similar to the curves with a large initialization and a small learning rate. As in the case with a small learning rate, the loss curves in Figure 6d with different leaky ReLU slopes collapse to one curve after rescaling time.

Thank you for your interest in our work and for your thoughtful feedback. We'd like to respond below.

• Significance of Contribution

  We acknowledge that prior works, including the references you mentioned, have studied cases where the ReLU networks behave like one or several linear networks. However, the connections between the ReLU and linear networks have not been explicitly highlighted and systematically summarized. We feel that these connections lend a valuable and attractive understanding of ReLU networks. To this end, our paper focused on specifying and explaining these connections as clearly as we can, aiming to make a number of previous results in the ReLU network subfield more intuitive and accessible to a broader audience.

  Additionally, as you and other reviewers have noted, the impact of removing bias is an interesting question due to the large body of existing works on networks with bias removal simplification. Discussing their limitations sheds new light on the conclusions from these prior studies. For instance, if we do not extend Lyu et al's results from linear separable functions to odd functions, the disadvantages of two-layer bias-free ReLU networks may not be apparent. For linearly separable tasks, two-layer bias-free ReLU networks converge to the max-margin linear classifier, which is presumably a good solution. However, with our extension to odd functions, it becomes clear that: while convergence to a max-margin linear map is advantageous when the target task is linear, it can be a disadvantage when the target is nonlinear.

• Comment on Assumption

  Thank you for bringing this issue to discussion. The primary purpose of Section 4.2 (orthogonal or XOR data) is to identify two common cases where a two-layer ReLU network behaves like multiple independent linear networks. Our goal was not to improve upon the analytical analysis of these cases, which have been studied in existing literature, as you correctly noted. Rather, we aim to illustrate and explain their connections to linear networks.

  As for our assumptions in Section 4.1 (symmetric data), while our assumptions and conclusions differ from those of Lyu et al, some are actually stronger: 1) we study square and logistic loss (Lyu et al focused on logistic loss); 2) we give closed-form time-course solutions (Corollary 8) to certain two-layer ReLU networks, which was not given in Lyu et al or other prior works; 3) we relaxed the assumption on the target from being linearly separable to being odd. We thus see our contributions as being complementary to theirs.

• Non-negative Weights in Deep ReLU Networks

  Line 438 says the second-layer weights $\boldsymbol W_2$ are non-negative based on the empirical observation in Figure 4b. We plot $\boldsymbol W_2$ in color in Figure 4b and only see gray and white elements, representing positive and zero numbers. Analytically showing $\boldsymbol W_2$ is approximately non-negative is an intriguing future direction. We have revised the sentence in line 438 to clarify that the non-negativity of $\boldsymbol W_2$ is a statement based on empirical observation.

• Thank you for the useful suggestion. We added "Section F Depth Separation" in our revised pdf to include a plot of the depth separation function and some discussions.

  We added the missed reference (Glasgow, 2024). Thanks.

  We corrected the typo "summed". Nice catch!

We hope our response and revision address the reviewer's questions and welcome any further suggestions.

Thank you for your feedback and constructive suggestions. We're glad you found our insights novel and important. We'd like to respond to your questions as follows.

• Comparison of Expressivity Results

  Your understanding of our expressivity result is correct. Regarding Basri et al, their Theorem 2 & 4 showed that in the harmonic expansion of two-layer bias-free ReLU networks with input $x$ uniformly sampled on a sphere, the coefficients corresponding to odd frequencies greater than one are zero. (Functions with odd frequency one are linear functions; functions with odd frequency greater than one are nonlinear odd functions.) They didn't study the probability of $f(x)\neq h(x)$ conditioned on $x$ uniformly sampled on a sphere. Thus, their statement on the expressivity is of the same strength as ours while they used an input assumption that we didn't use.

  We have updated our pdf to specify the Theorem numbers when citing Basri et al. Hope this helps clarify. Please let us know if we misunderstood your suggestion.

• Comments on Limited Expressivity

  We have reworded the beginning of Section 3 to clarify that the limitation of positively homogeneous functions is a known fact. Further, we'd like to add some comments about this topic below.

  Though we agree it is somewhat evident that bias-free ReLU networks have limited expressivity, the practical usage of bias-free ReLU layers is actually not that limited. As we have cited, a best paper in ICLR2024 [1] showed bias-free convolutional ReLU networks are state-of-the-art models in image denoising and that the removal of bias specifically helps generalization. Meta's Llama [2], one of the most influential open-source large language models, doesn't seem to have bias terms. Hence, we believe that giving a more accurate description of the expressivity of bias-free ReLU networks is useful.

  [1] Kadkhodaie et al. Generalization in diffusion models arises from geometry-adaptive harmonic representation. ICLR 2024.

  [2] https://github.com/meta-llama/llama/blob/main/llama/model.py

• Clarification of Assumption 5

  Thank you for this useful suggestion! We have clarified in our revised pdf that Assumption 5 is made only at initialization, and we proved (instead of assumed) that Assumption 5 will remain true throughout training in the proof of Theorem 7. What $\boldsymbol r$ is depends on the random initialization. We have reworded the assumption to clarify that there exists some $\boldsymbol r$ such that $\boldsymbol W_1= \boldsymbol W^\top_2 \boldsymbol r^\top$.

• Clarification of Conjecture in Section 5

  We're sorry that the conjecture was unclear. We have now re-written the last paragraph of Section 5. We hope it is clearer now and welcome any further suggestions for improvement.

• Finite Data

  The exact assumption we used is that the empirical distribution of input $x$ satisfies $p(x)=p(-x)$. For infinite data, it incorporates common distributions such as any zero-mean normal distribution. For finite data, it means that if $x$ is present in the dataset, $-x$ is also present, which was exactly the assumption used in Lyu et al [3]. We have revised Remark 3 to explicitly describe what our assumption means for infinite and finite data.

  [3] Lyu et al. Gradient descent on two-layer nets: Margin maximization and simplicity bias. NeurIPS 2021.

• Gradient Descent versus Gradient Flow

  We empirically find that our results still hold with a moderately large learning rate. In our revised pdf, we have added Section C.6 in Appendix and some signposts the main text to make it clear that: while our analytical derivations used infinitesimal learning rate, we have empirical evidence that some of our results extend to large learning rates.

  In the added Figure 8, we use a learning rate of $0.6$, which is 150 times larger than the learning rate of $0.004$ used in Figure 2. Similar to Figure 2, the loss curves in Figure 8 with different leaky ReLU slopes collapse to one curve after rescaling time and the differences between weight matrices are small. If the learning rate is further increased, the loss and weights curves oscillate and the equivalence breaks; but we typically wouldn't let our networks train in this oscillating regime.

Thank you for your thoughtful comments and feedback. We are glad to know that you found our contributions important and interesting, and that our presentation is very clear. We'd like to respond to your questions below.

• Dynamics for Different Datasets, Quantifying Non-Robustness

  This is an intriguing direction. We added Figure 5b in our revised pdf to show the plateau duration in the loss curves for perturbed symmetric datasets scales approximately linearly with $1/\Delta y$. Our intuition is that the gradient update during the plateau has a $\Delta y$ component and the time will thus have a $1/\Delta y$ scaling factor. The simulations indeed match our intuition. This result is empirical at the moment, but we are working to provide an analytical analysis and a more general metric for quantifying how asymmetric a dataset is.

• Reemphasize Assumption

  We have now explicitly written out our symmetric condition in the sentence you quoted. We have also edited summarizing sentences in the Introduction to make sure that we write out the symmetric condition and/or use a clickable hyperlink to the condition (Condition 3).

• Clarification of Assumption 5

  We have revised to clarify that Assumption 5 is made only at initialization. Thank you for your useful suggestions!

• Challenges of Analytical Derivation for Deep ReLU Networks

  This is indeed an interesting and challenging problem. We made an attempt to derive the late phase dynamics in Appendix E, showing that weights that formed a low-rank structure as Equation 15 will maintain the structure. We left the early phase dynamics to future work.

  Technically, studying the dynamics of deep ReLU networks is challenging because the nested nonlinearities introduce nested derivatives in the gradient descent differential equations. Most tools for analyzing the dynamics of two-layer ReLU networks do not apply to deep ReLU networks. For two-layer ReLU networks trained on symmetric datasets from small initialization, we use the tool of approximating the early phase dynamics with a linear differential equation (Equation 27), whose closed-form solution is available. For depth-$L$ ReLU networks, we can't reduce the early phase dynamics to a solvable differential equation with the same tool: the gradient update of a weight involves the multiplication of some nonlinear functions of $(L-1)$ weights, which is generally intractable. Thus, unsurprisingly, the literature on the learning dynamics of ReLU (and perhaps also other nonlinear) networks in the rich regime is quite sparse.

  We will include our discussion and review relevant literature on the dynamics of deep ReLU networks in final revision.

We hope our response and revision are beneficial and welcome any furthur suggestions.