Comment: "...view the adaptations as individual low-rank matrix factorization problems?"

Response: For each weight matrix, suppose there existed a “target” weight matrix $\\mathbf{M}$ that serves as the optimal weight matrix for the fine-tuned model. Then, one way of finding the low-rank adaptors to fit $\\mathbf{M}$ can be viewed as a low-rank matrix factorization problem: \underset{\\mathbf{A},\ \mathbf{B}}{\\mathrm{min}} \, \\| \\mathbf{W}\_0 + \\mathbf{AB}^\\top - \\mathbf{M}\\|^2_{\\mathsf{F}}, where $\mathbf{W}_0$ is the frozen weight matrix. This is what we envisioned when mentioning viewing the adaptations as individual problems.

Comment: "It would be better to have a quantitative understanding of the 'low-complexity learning'"

Response: Fitting low frequency images with convolutional neural networks can be considered a low complexity task because of spectral bias [1] of such networks. Smooth/low frequency images are much easier to fit for convolutional neural networks due to spectral bias where the networks incrementally learns each Fourier frequency (low to high). Hence, a measure of complexity of each image can be the Fourier bandwidth of the image. Furthermore, while fitting low frequency images or simple datasets like MNIST, intrinsic update dimension in the parameter space of networks tend to be much smaller [2, 3]. However, the exact relation between the intrinsic dimension of parameter update and its sharpness is unknown and calls for further in-depth study. We revised the manuscript stating that we highlight the difference between the low complexity learning in CNNs vs linear networks as in our setting.

Thank you for pointing out the typos as well -- they are fixed in the revised manuscript.

References:

[1] Nasim Rahaman and Aristide Baratin and Devansh Arpit and Felix Draxler and Min Lin and Fred A. Hamprecht and Yoshua Bengio and Aaron Courville. "On the Spectral Bias of Neural Networks." https://arxiv.org/abs/1806.08734

[2] Chunyuan Li and Heerad Farkhoor and Rosanne Liu and Jason Yosinski. "Measuring the Intrinsic Dimension of Objective Landscapes". https://arxiv.org/abs/1804.08838

[3] Phillip Pope and Chen Zhu and Ahmed Abdelkader and Micah Goldblum and Tom Goldstein. "The Intrinsic Dimension of Images and Its Impact on Learning". https://arxiv.org/abs/2104.08894

Comment: "Lemma 2 does not tell us that the singular values become balanced"

Response: Thank you for highlighting this important aspect. Indeed, even if a sequence $\\{ a(t) \\}\_t$ is lower bounded by $0$ and obeys $a(t+1)<a(t)$, the sequence is not guaranteed to converge to $0$. However, if there exists a $0<c<1$ such that $a(t+1)<c\cdot a(t)$, then the sequence is guaranteed to converge to $0$ as $t \rightarrow \infty$ (see Lemma 9). To this end, we made an attempt to modify Lemma 2 to show strict balancing such that $b(t+1) < c\cdot b(t)$. To do this, we analyze 2 cases: (i) $\\pi(t) < \\sigma_*$, and (ii) $\\pi(t)> \\sigma_*$, and showed that there exists a constant $0<c<1$ such that it holds, where $\\pi(t)$ is the product of singular values at time $t$ and $\\sigma_*$ being the target singular value (comparable to the machine precision scale). However, when $\\pi(t)> \\sigma_*$, we can only prove for $c=1$, i.e, $b(t+1)< b(t)$. Hence, the balancing gap cannot be proven to go to exactly zero but our experiments suggest that is converges to something very small.

For the final manuscript, we can make these points even more rigorous by performing an error analysis -- we will prove our results with an error term $e(\alpha)$, where $\alpha$ is the initialization scale, which arises when balancing only approximately holds. Then, we can re-state our results including the eigenvalue calculation (requires matrix perturbation theory) in the order of $e(\alpha)$. However, due the time limitations, we leave it for the final manuscript but will update the manuscript if completed during the discussion period. Finally, we note that we also added in the balanced initialization in Equation 3, and hence the conclusions made in our paper do not change.

Comment: “Large lr may lead to fast decreasing of the training loss because of its large stepsize not because of the catapults.”; “notable for ranks r=4 and r=8, there are catapults…”

Response: We agree with this point – our experiments at the current stage do not clearly show that the increase in Pearson correlation is because of simply using a large learning rate or if it is because of catapults. We made this conclusion because if you zoom in to the very early iterations in Figure 21 of the revised manuscript, there exists large “jumps” in the training loss for ranks r=4, 8, whereas there are much smaller jumps in ranks r=1, 2. We jumped to the conclusion that this hinted that potential catapults lead to faster optimization. Since we cannot make a stronger claim other than these experiments, we moved this section to the Appendix as suggested by Reviewer n4c2.

Comment: Self-stabilization in DLNs

Response: We have modified the statement in our revised manuscript to better reflect this point. To summarize, recall that in DLNs, sharpness oscillates periodically about the limit $2/ \\eta$ and this comes from the property that in each eigenvector direction, it satisfies the stable oscillation condition. This is in contrast to the self-stabilization effect explored in Damian et al. where sharpness decreases below a specified threshold and is a result of a closed loop sharpness reduction feedback. The difference between sustained oscillations and damped oscillations/catapults are classically studied in control system by locating the poles of transfer function of a closed loop system. We believe a similar study can be made to study these two cases but it is currently beyond the scope of our work.

Comment: "what is $\\rho_i(t)$ exactly?"

Response: We have changed the notation in the revised manuscript to remove the dependence on $t$. $\\rho_i$ are the two values that the first singular value takes when it exhibits two-period oscillation.

Comment: "to show oscillations in two or more subspace... I don’t think this is a trivial result from Thm 1."

Response: We apologize for the confusion -- this is a direct result from Theorem 2 and not Theorem 1. Theorem 2 states that whenever $\\eta$ is set such that it exceeds the stability condition $\eta>2/\\lambda_i$ in each eigenvector direction $\Delta_{i}$, then it stably oscillates about the eigenvector direction $\Delta_{i}$. Hence, by ensuring that $\eta$ lies in $\lambda_{3}<\frac{2}{\eta}< \lambda_{2} < \lambda_{1}$, oscillations only occur along $\Delta_{1}$ and $\Delta_{2}$ and not along $\Delta_{3}$. We have revised our theorem statement to better reflect this as well. We also included a figure in 2-dimension to visualize the stable subspace oscillation in Figure 10.

Thank you for your review. Overall, we fixed typos and added in the references as suggested. Below, we provide a few concrete responses regarding weaknesses and concerns.

Comment: "main concern is that Lemma 2 does not ensure convergence to strict balancing"

Response: Thank you for highlighting this important aspect. Indeed, even if a sequence $\\{ a(t) \\}\_t$ is lower bounded by $0$ and obeys $a(t+1)<a(t)$, the sequence is not guaranteed to converge to $0$. However, if there exists a $0<c<1$ such that $a(t+1)<c\cdot a(t)$, then the sequence is guaranteed to converge to $0$ as $t \rightarrow \infty$ (see Lemma 9). To this end, we made an attempt to modify Lemma 2 to show strict balancing such that $b(t+1) < c\cdot b(t)$. To do this, we analyze 2 cases: (i) $\\pi(t) < \\sigma_*$, and (ii) $\\pi(t)> \\sigma_*$, and showed that there exists a constant $0<c<1$ such that it holds, where $\\pi(t)$ is the product of singular values at time $t$ and $\\sigma_*$ being the target singular value. However, when $\\pi(t)> \\sigma_*$, we can only prove for $c=1$, i.e, $b(t+1)< b(t)$. Hence, the balancing gap cannot be proven to go to exactly zero but our experiments suggest that is converges to something very small (comparable to the machine precision scale).

For the final manuscript, we can make these points even more rigorous by performing an error analysis -- we aim to prove our results with an error term $e(\alpha)$, where $\alpha$ is the initialization scale, which arises when balancing only approximately holds. Then, we can re-state our results including the eigenvalue calculation with an error order of $e(\alpha)$. However, due the time limitations, we leave it for the final manuscript but will update the manuscript if completed during the discussion period. Finally, we note that we also added in the balanced initialization in Equation 3, and hence the conclusions made in our paper do not change.

Comment: "the model fails to capture (non-monotonic) decreasing training loss"

Response: We point out that the linear model also decreases the loss non-monotonically where oscillations occur while incremental rank learning takes place (e.g., see Figure 2). Here, oscillations occur after the first and second saddle jumps take place. However, when the iterates reach the global minimum, the model cannot decrease the training loss further since there are no spurious local minimas and sustained oscillations take place near the global minimum. We discussed this point of difference and included a loss landscape figure comparing linear scalar deep networks with highly non-convex landscapes with local minimas like the Holder table (see Figure 8, 10, 11). Furthermore, DLNs exhibit oscillatory behaviors beyond the stability regime instead of divergence, which Theorem 1 shows. Since, the optimization community has mostly focused on optimization in the stable regime, we believe our work lays the foundation to study deep models that operate beyond stability, which neural networks typically operate on.

Comment: "The claims on LoRA dynamics in Section 4.2 lack sufficient evidence."

Response: As stated in the global response, we have moved this section to the Appendix, and changed the language such that it warrants further investigation to avoid making strong claims. We hypothesized that the oscillations at lower learning rates occur due to the stochasticity in the updates by using minibatches, and that there exists actual catapults due to the large learning rate. This is because in the very early stages of training, the catapults for larger learning rates happen at a much larger magnitude (see new caption in Figure 21) However, since we cannot precisely delineate these points, we decided to remove it as a contribution and remove it from the main text.

Comment: "When initialization is outside the singular vector invariant set, how do loss and sharpness behave..."

Response: For concreteness, we added Section B.4 in the Appendix to answer this question thoroughly. To summarize, even when initialization is outside the SVS set, the loss still oscillates within a 2-period orbit and the weights indeed converge to period-2 orbit (within the correct learning rate regime). Near the global minimum, the loss cannot decrease non-monotonically since the oscillations are contained in a single loss basin and there are no spurious local minima. This implies that our theory still holds, but the evolution of the singular vectors would need to be accounted for if they do not converge to the SVS set. Though, we also consider two more initializations outside the SVS set in Section B.4: orthogonal and random initialization (see Figure 15). These initialization do not belong to the SVS set but even at EOS, they converge to the SVS set.

As for the balancing, we consider an initialization larger than the one required in Theorem 1 in Section B.4, where strict balancing fails to hold. Here, although oscillations start before strict balancing holds, the loss and singular values of weights exhibits 2-period oscillations. We show this in Figure 14.

I appreciate the authors for their efforts in revising the paper and conducting additional experiments. The revised version has significantly improved the clarity of the contributions, and I find that most of my initial concerns have been adequately addressed. Thank you for the detailed and thoughtful rebuttal.

That said, I still have a remaining concern regarding Lemma 2. While the revised lemma provides a clearer statement ensuring the balancing gap converges to zero under the condition $\pi(t) < \sigma_*$ for all time steps $t$, I find this condition potentially restrictive.

• Initialization condition: Could the authors clarify whether there exists a practical initialization condition—particularly when strict balancing does not hold initially—that ensures $\pi(t) < \sigma_*$ for all $t$? It would be helpful to understand if this condition can emerge naturally under typical initialization schemes or during the training dynamics.

• Case when $\pi(t) > \sigma_*$: Additionally, I am curious about the implications when $\pi(t) > \sigma_*$ at some time step $t$. Specifically, is the result $b(t+1) < c \cdot b(t)$ with $0 < c < 1$ fundamentally unattainable in such scenarios, thereby necessarily leading to $c = 1$?

Addressing these questions would further strengthen the theoretical contribution of the paper and provide greater clarity on the assumptions underlying the convergence result. I am inclined to increase my score if the authors can address these points.

Thank you for your timely response. Firstly, we would like to clarify when the two cases $\pi(t) < \\sigma\_\star$ and $\\pi(t) > \sigma_\star$ actually occur.

Consider the case in which the learning rate is stable (i.e., oscillations do not occur) and the global minima $\pi(t) = \\sigma\_\star$ is stably achieved. In this case, $\pi(0) = 0$ (due to zero-$\alpha$ initialization) and grows until it hits $\sigma_\star$ stably. Mathematically, this implies that the iterates $\pi(t)$ always stay in the regime of $\pi(t) < \\sigma\_\star$, until it reaches $\pi(t) = \\sigma\_\star$.

Now, let us consider the EOS regime. If the learning rate is unstable, it starts from $\pi(0) =0$ (due to zero-$\alpha$ initialization), where $\pi(t) < \\sigma\_\star$ (Case 1), but overshoots the global minima $\pi(t) = \\sigma\_\star$ to enter Case 2 ($\pi(t) > \\sigma\_\star$). Then due to EOS, it oscillates between the regions Case 1 ($\pi(t) < \\sigma\_\star$) and Case 2 ($\pi(t) > \\sigma\_\star$). In this attached figure, we delineate these two regions for clarity. Notice that this is directly implied by Theorem 1, as if you look at $\rho_1$ and $\rho_2$ carefully, the oscillations imply that we alternate between the case $\pi(t) < \\sigma\_\star$ and $\\pi(t) > \sigma_\star$.

With this in place, we answer your questions, highlighting how our proof explains these two phases:

Initialization condition (Case 1 $\pi(t) < \\sigma\_\star$): Consider the initialization in the manuscript, where $\\mathbf{W}\_L = \\mathbf{0}$ and the rest are initialized to $\\mathbf{W}\_l = \alpha \mathbf{I}$. Notice that strict balancing does not hold initially here. In the stable regime, as long as $\alpha$ is sufficiently small, we will have $\pi(t) < \sigma_\star$ until it hits the global minimum $\pi(t) = \sigma_\star$.

For the case we analyze, EOS ensures oscillation about the global minima $\pi(t) = \\sigma\_\star$. So for all initializations, assuming that $\alpha$ satisfies the conditions in our paper, we will have $\pi(t) < \sigma_\star$ until it starts to alternate between the conditions in Case 1 and Case 2. In the attached Figure, notice that the balancing gap does indeed decrease when $\pi(t) < \sigma_\star$, which is when there exists a $0< c < 1$ for which $(1-g(t))<c$ (since $g(t)<1$ in figure). Here $b(t+1)=b(t)(1-g(t))$, $g(t)$ being the quantity we defined in line-1750 in manuscript.

Oscillation between Case 1 ($\pi(t) < \\sigma\_\star$) and Case 2 ($\pi(t) > \\sigma\_\star$): Once it overshoots ($t>500$ in figure) $\pi(t) = \\sigma\_\star$, it hits Case 2 ($\pi(t) > \\sigma\_\star$) and oscillates between Case 1 and Case 2 (see Figure). For Case 2 strictly, we could only prove that $(1-g(t))<c$ for $c=1$, but note that in Figure, $g(t)<1$ ensuring there exists a $0< c < 1$ even when there is oscillation between these two cases. Although our proof in Case 2 can only ensure $c=1$, the iterates mostly undergoes through Case 1 and then alternates Case 1 and Case 2. The strict balancing effect has undergone in Case 1 (in both the time regions) and is sufficient to reduce the balancing gap to a very small value. This is reflected perfectly in the figure, where we see balancing gap decreases monotonically and $g(t)<1$ for all $t$.

We will add a discussion of this in appendix.

Comment: Experiments: LoRA initialization does not match practice.

Response: As suggested by another reviewer, we moved the section LoRA to the Appendix, as we do not state it as a contribution but rather pose it as a future work. Generally, we use the same setup as described by [7] where $\alpha$-scaled random initialization was used along with adapting the attention and MLP weights. We did not clarify that we used random weights in the original manuscript, which we have now included in appendix. The Pearson correlation corresponds to measuring the correlation of the predicted labels to the true labels in the test set, and can be viewed as testing the performance of the fine-tuned model. If the Pearson correlation is higher than another, then we can conclude that one fine-tuned model is better than the other.

Comment: A natural experiment to try: In Line 188, step size is suggested to be chosen based on the depth

Response: We performed this experiment in Section B.4, we do indeed verify this claim (see Figure 16).

Comment: In the introduction, 3 observations are described from [2, 3], but it is not clear to me how or which ones the paper addresses. It is also not clear to me how sharpening and catapults from the two papers are related to each other.

Response: This confusion probably arose since [3] examines catapults in the top eigenvector space of the NTK matrix, whereas in our paper we study the catapult in the top eigenvector space of the Hessian. The NTK and the Hessian behaves almost similarly near the end of training since the residual goes to zero for MSE loss, which the authors in [3] address in Appendix A.1. In our paper, we study sharpening (or progressive sharpening) in DLNs for the max eigenvalue of the Hessian like in Cohen et al. and then prove the oscillatory phenomenon in the top eigenvector subspace of the Hessian based on the learning rate. The latter observation validates experiments in [3] since they observe catapults in the top eigenvector space of the NTK (or equivalently the Hessian). We make this clear in the introduction.

Comment: Definition 2 is introduced abruptly without need or context.

Response: Our revised manuscript removes this abruptness and addresses the importance of strict balancing before defining it.

Comment: "Theorem 1. Why is this called 1-d subspace oscillation?"

Response: In Theorem 1, the oscillations in subspace is a rank-1 outer-product. Hence, we changed the definition to a rank-1 oscillation instead of 1-d subspace oscillation. Initially, we stated it as 1-D oscillations since oscillations took place along the top eigenvector of the flattened Hessian of the DLN loss.

Comment: "Figure 2. There are 2 kinks in the EOS regime in the left plot but only 1 is visible in the right plot. Do the authors know why?"

Response: Thanks for pointing this out. This is an artifact of plotting in the logarithm scale. For DLNs with small initialization, the sharpness is negative in the earlier learning stages, since this region is a saddle point with a negative curvature. Hence, the logarithm (along with some irregular quantization) made the negative sharpness look like a "kink". We updated the figure.

Comment: Why do smooth, low-frequency images correspond to low task complexity? For the CIFAR dataset, number of samples is used as a proxy task complexity, why? Both datasets are image datasets.

Response: Smooth/low frequency images are much easier to fit for convolutional neural networks due to spectral bias
[4], where the networks incrementally learns each Fourier Frequency (low to high). Similarly, fitting simple datasets such as MNIST or even datasets with less images are low complexity tasks since the intrinsic update dimension in parameter space is much smaller [5, 6]. However, the exact relation between the intrinsic dimension of parameter update and it's sharpness is unknown, it is a common observation that sharpness do not rise much when the network easily/quickly fits the underlying function (appendix in Cohen et al). Although currently the relation between complexity of the tasks in linear and non-linear network are far from perfect, the top singular value of the target matrix $M$ is a proxy for complexity. We updated the manuscript to reflect this.

We thank the reviewer for their detailed feedback on our work that helped improve our manuscript. Below, we respond to each of the concerns in more detail.

Comment: I'm not sure the results shed light on dynamics in deep nonlinear networks.

Response: Thank you for highlighting this point. We have revised our manuscript to mainly focus on DLNs, highlighting the similarities and differences between linear and non-linear networks. We removed the claim that DLNs can explain the phenomena that happen in practical non-linear networks at EOS and added a 2D toy experiment that highlights a significant difference between these two models (see Figure 8).

To also better reflect our contributions in DLNs along with its (potential) connections to nonlinear networks, we have revised our manuscript to focus more on DLNs. Primarily, we changed the tone of the paper to focus on DLNs, while still explaining how our results might contribute to some of the unexplained phenomena in deep nonlinear networks. Moreover, to have a fairer comparison between the two networks, we added a figure in Section 4.2 that highlights the differences of the behaviors in EOS. We do this by visualizing the loss landscape using a 2-D depth-4 network and a Holder table function. We chose the Holder table loss function as a proxy test for deep non-linear network which is known to have numerous local minimas. Whereas deep linear networks contains only saddle points and global minimas (no spurious local minimas).

Regarding our connections on the similarities between linear and nonlinear networks, we referenced [1] to highlight particular cases in which behaviors at EOS are similar. For example, our experiments with bias-free ReLU networks in Figure 17 imply that there are basins that support sustained oscillations over some local neighborhood before catapulting happens.

Comment: $\alpha$-scaled initialization scheme is not used in practice.

Response: While the $\alpha$-scaled initialization may not be used for nonlinear networks, it is a very common setting to study for DLNs, as it corresponds to the “rich” training regime. The rich regime promotes "incremental learning", where training performs greedily to fit the target singular values [2]. In fact, Figure 4 of [2] shows that test accuracy improves with a smaller alpha for image classification datasets. To the best of our knowledge, a small initialization scale is used in most (if not all) works that study DLNs. Furthermore, to show that the convergence to the singular vector stationary set is not an impractical assumption, we added additional experiments in the Appendix to show that many other initializations converge to such a set. This includes identity (which is from the original manuscript), orthogonal, and random initialization (see Figure 15).

Comment: the assumptions on singular values' "strict balanced state" and initialization's "stationary set condition" are strong

Response: Although the strict balance state may seem strong, we only consider it for unbalanced initializations to cover a wide range of initialization schemes. However, for balanced initialization, the strict balanced state is not an assumption, which we added to the revised manuscript. Moreover, all of our experiments show that these two conditions are generally met, provided that they satisfy the conditions in our theorem statements (e.g. the initialization scale needs to satisfy the threshold). We provided additional experiments for these assumptions, which are available in Appendix B.4 (see Figure 15 and Figure 14). Furthermore, we would like to point out that almost all the literature on deep matrix factorization assumes that these two conditions hold exactly. In our work, we go one step further to show increasingly balancedness.

Finally, we note that these assumptions are mainly used such that we can focus on analyzing the periodicity that arises in the singular values – they are tools to make the analysis a little bit “simpler”.

[1] Yedi Zhang, Andrew Saxe, Peter E. Latham. "When Are Bias-Free ReLU Networks Like Linear Networks?". https://arxiv.org/abs/2406.12615

[2] Blake Woodworth and Suriya Gunasekar and Jason D. Lee and Edward Moroshko and Pedro Savarese and Itay Golan and Daniel Soudry and Nathan Srebro. "Kernel and Rich Regimes in Overparametrized Models". https://arxiv.org/abs/2002.09277

Comment: Line 71. What is "sharpening"? First time this term is mentioned.

Response: We have defined "progressive sharpening" at the start of Section 3 initially. Now, we included it where we defined EOS so the reader can understand what "sharpening" is referred to as.

Comment: I think more text is needed to explain how the analysis is done for r-dimensional oscillations. A figure would really help the reader.

Response: We added figure which highlights the statement of this theorem in the Appendix (Figure 10). For example the X-axis and Y-axis here refers to the two perpendicular lines in which oscillation occurs. In higher dimensions, these orthogonal directions are denoted by $\Delta_{i}$, (which are the eigenvector directions of the Flattened Hessian) which passes through the global minima (we referred to as $x$ in paper). Hence, this line is denoted as $y=x+t\Delta_{i}$.

References:

[2] Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, Ameet Talwalkar "Gradient descent on neural networks typically occurs at the edge of stability". arXiv preprint arXiv:2103.00065.

[3] Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, Mikhail Belkin. "Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning". arXiv preprint arXiv:2306.04815.

[4] Nasim Rahaman and Aristide Baratin and Devansh Arpit and Felix Draxler and Min Lin and Fred A. Hamprecht and Yoshua Bengio and Aaron Courville. "On the Spectral Bias of Neural Networks." https://arxiv.org/abs/1806.08734

[5] Chunyuan Li and Heerad Farkhoor and Rosanne Liu and Jason Yosinski. "Measuring the Intrinsic Dimension of Objective Landscapes". https://arxiv.org/abs/1804.08838

[6] Phillip Pope and Chen Zhu and Ahmed Abdelkader and Micah Goldblum and Tom Goldstein. "The Intrinsic Dimension of Images and Its Impact on Learning". https://arxiv.org/abs/2104.08894

[7] Yaras, Can, et al. "Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation." arXiv preprint arXiv:2406.04112 (2024).

We sincerely thank the reviewer for their thoughtful analysis of our work and for providing constructive feedback, which has greatly helped us improve the manuscript. We would like to clarify that while the revisions may appear substantial, the major changes are primarily related to the connections with deep nonlinear networks and the removal of the LoRA section.

If there are any additional clarifications or questions that could help address your concerns further, we would be happy to provide them. We kindly request you to consider these improvements and reassess the score for our submission.

We appreciate the reviewers insightful comments. We have also fixed the typos. Below, we provide concrete responses regarding the concerns:

Comment: "the claim in the abstract seems overstated and exaggerated."

Response: As stated in the global response, we have updated our manuscript by toning down their similarities and highlighting the similarities and differences between the two networks. For example, in Figure 8, we plot the landscape of a two variable depth-4 linear scalar network and GD dynamics at EOS. Along with it, we also plot GD optimization trajectory for a Holder table landscape (which is highly non-convex with local minimas) for increasing learning rates Figure 11. Here, the sharpness closely follows the limit $2/\\eta$ supporting the EOS observations made in Cohen et al. This helps visualize the differences (along with similarities) between the two networks. Overall, we had adapted our paper to focus more on DLNs.

Comment: "It is not clear if all statements work only based on the initialization (eqn (3)) or for any initialization in the singular vector stationary set."

Response: The theoretical results hold for any initialization that converges to the SVS set in Proposition 1. We choose the initializations in Equation (3) for concreteness, as we prove in Propositions 2 and 3 that these initializations belong to the SVS set. Hence, our analysis can be extended to many different forms of initializations, as long as they can be shown to satisfy Proposition 1.

Comment: "Why is the period of the fixed orbit is always 2? It seems that it comes from the two-step GD update. How about other periodicities?".

Response: This is an excellent question. In Figure 1 of the revised manuscript, we provide a bifurcation plot showing that other periodicities also exist for a properly chosen learning rate. This shows a period-doubling route to chaos (and then to divergence). Our paper focuses on the period-2 regime, as it still shows a sufficient amount of information to understand what happens in the EOS regime for DLNs. We leave the other cases (i.e., analyzing what learning rates are required for more periodicities) for future work.

Comment: "on page 19-20 seem to have typos".

Response: Thanks for pointing it out. We have fixed it.

Thank you for pointing out the typos—those will be fixed in the final manuscript. We also appreciate your efforts in overseeing the discussion during the rebuttal phase.

Regarding the balancedness, while your argument is correct, the singular vector stationary set assumption does make the higher order terms match and so the preservation does hold for the balanced initialization case. For the unbalanced case, please note that we never assumed that the conserved flow in GF (i.e., the balancedness of the weights) holds for finite step sizes in GD. We apologize for any confusion in the manuscript that may have led to this conclusion. In fact, as you pointed out, with finite step sizes, the layerwise balancing breaks; however, at EoS, this balancing gap converges to zero. This is the central argument of our proof detailed in Lemma 2, and has been discussed extensively with Reviewer nnRE and n4c2.

To clarify this point, we plan to add a figure in the camera-ready version that clearly delineates the differences between GF, GD with step sizes below the EoS regime, and GD in the EoS regime. Our theorem and lemma are consistent with our empirical observations: balancedness breaks for finite step-size GD, and at EoS, the balanced quantity monotonically converges to zero.