Thank you very much for your thorough review of our work, your careful assessment, and the positive evaluation. It was humbling to read that "loss landscape have been tremendously studied, this works uncovers a novel structure therein." We will address your questions, concerns, and suggestions in detail below.

Sharpening of channels:

Indeed, the channels are very sharp in some orthogonal directions, and these directions become progressively sharper along the channel. This is not visible in Fig. 4c because the Hessian is evaluated at the saddle-line for a negative eigenvalue, i.e., it shows the direction which "leads away" most strongly from the saddle-line. This direction simply does not correspond to the direction of maximum curvature inside the channel in the high-dimensional parameter space. To give a more formal argument, we extended our scaling analysis to the eigenvalues of the Hessian which shows that the maximum eigenvalue diverges as $\lambda_\mathrm{max} \sim 1/\epsilon^2$.

Discrete optimizers:

This is a very helpful comment! We can directly combine the above scaling of the sharpness with the main result of the 'edge of stability': the maximum eigenvalue scales inversely with the learning rate $\eta$, i.e., $\lambda_\mathrm{max} \sim 1/\eta$. This results in a simple relation between the learning rate and the maximum depth: $\epsilon_\mathrm{min} \sim \sqrt{\eta}$. We empirically confirm in the setup of Figure 6 that gradient descent converges to the level in the channel in which $\lambda_\mathrm{max} = 2 / {\eta}$, and that the maximum eigenvalue diverges with $1/\epsilon^2$.

Vicinity of local minima:

We would like to clarify that it is not necessary to find local minima of smaller sub-networks in order to find channels. We sincerely apologize because this misunderstanding clearly arose from a crucial omission on our side: The experiments in Fig. 5, including the quantification of channels in 5c, all start from random initial conditions (Glorot normal). Thus, channels are found in significant number starting from random initial conditions; it is not necessary to first identify minima of sub-networks. We will add this important point by appending "; importantly, we start in all cases from random initial conditions (Glorot normal initialization)" to line 135. We believe this also answers your question (ii), but please let us know in case we misinterpreted this question.

Interpolating regime:

We agree that it seems unlikely to find a channel in the interpolating regime, certainly deep in the interpolating regime with large manifolds of zero-loss solutions. We will change the first sentence of the discussion (line 205) to: "Neural network loss landscapes are non-convex, but the empirical success in training shows that their structure is benign also outside the deeply over-parameterized regime."

Generalization to multiple neurons:

The generalization of similar structures to multiple neurons is an interesting question. Empirically, we find channels composed of multiple neurons in various settings (see Fig 5g), which effectively makes them multi-dimensional. From a theoretical perspective we show in appendix C.5.2 how three neurons could together implement a second derivative using two divergences with different speed ($1/\epsilon$ and $1/\epsilon^2$) along two particular directions in parameter space. Empirically, we verified that a few neurons could fit to zero loss a target that is a linear combination of the activation function and its first and second order derivative.

Relation to Lim, Michalek, and Qi (2022):

Thank you very much for pointing out this interesting result, which we were unaware of. First, Lim, Michalek, and Qi explicitly construct an example of a minimum at infinity with multiple (six) data points for their Theorem 1, hence we will change line 67 to "but there are examples, where minima at infinity can arise with two or more datapoints [Josz23,Lim22]". However, their construction is different from a channel to infinity because only a single readout weight diverges, in contrast to the joint divergence of two readout weights that characterize a channel. Furthermore, Lim, Michalek, and Qi show for a single-hidden layer network that any sequence of weights converging to a global minimum at infinity is unbounded. This is consistent with the example in Fig. 6 where the upper channel (channel 2) leads to the global minimum, hence we will add "; we note that this is consistent with Proposition 1 in [Lim22]" to line 197. Importantly, channels do not only occur in combination with a global minimum at infinity: the lower channel (channel 1) in Fig. 6 provides an explicit example. Thus, channels are consistent with Proposition 1 of Lim, Michalek, and Qi but are a more generic phenomenon because they are not tied to a global minimum at infinity. Moreover, in contrast to Lim et al, our example of Fig. 6 shows that solutions at infinity can arise also with infinite data.

Minor suggestions:

Thank you very much for bringing unclear notations and wordings to our attention! (i) We will fix the notation of mineigval direction following your suggestion. (ii) Throughout section 4 we consider the limit $\epsilon \to 0$ at fixed $a$ (convergence of $a$ is shown in Fig.\ 6a, middle panel). This was indeed not clearly defined in the text and we will fix the definition of the limit to "$\epsilon\to0$ with fixed $\mathbf{w}$, $\mathbf{\Delta}$, $a$, $c$". (iii) Precisely, with 'attractive' we mean a point where gradient flow can converge, e.g., on the plateau saddle. We will add "attracting (all eigenvalues of the Hessian are non-negative)" to line 103. (iv) Yes, very good suggestion; we will add ", and $h(\mathbf{\theta}_0; \mathcal{D})$ denotes the leading order correction term" to the end of the sentence in line 177.

Figures:

The impression that the figures are too crowded was also echoed by the other reviewers, in particular Figs. 2 and 3 seemed to obstruct the flow of the manuscript. To address this we will combine them into one figure and move the top right panels in Fig. 2 as well as panels a, b, e in Fig. 3 to the supplement. Please do let us know if you have further suggestions to improve the readability!

Conclusion:

Thank you very much again for meticulously checking and deeply engaging with our work. We sincerely appreciate your constructive criticism as well as your helpful questions and suggestions, and we hope our rebuttal was able to comfort your opinion of our work.

Thank you very much again for your careful review, and also for actively engaging in the follow up discussion.

Vicinity of local minima

Each channel runs parallel to a $\gamma$-line. However, this $\gamma$-line could be quite far away in parameter space. Empirically, we observe that perturbations from (non-zero loss) $\gamma$-lines can lead to channels, but currently we do not know, if each channel has a higher loss $\gamma$-line nearby.

It is easy to construct examples, where the global minimum is in a channel, even in large scale settings. Take, for example, an arbitrarily large network as a target network and replace two units with the gated-linear unit expression on the right-hand-side of Eq. 4: the global minimum of any student network will be at the end of a channel for infinite data (and probably at least some way into the channel for finite data), even if the student is wider than the teacher. Whether global minima are in channels also for typical machine learning datasets is an open question.

Even in cases where typical initializations lead to a zero-loss solution, local minima, including channels, could exist at higher losses. We agree with your intuition that they could be found with perturbations of $\gamma$-lines that are constructed from local minima of "higher-order subnetworks".

Catapult

This is an interesting question. In the experiments with infinite data and finite learning rate, we did observe an effect that could be related: for the higher-loss channel in Fig. 6 (at $\alpha\approx-1$) the dynamics jump into the lower-loss channel (at $\alpha \approx 2.5$). However, this occurred only with large learning rates and only from high-loss to low-loss. Otherwise, the dynamics stop at the point where the maximum curvature in the channels reaches the edge of stability. Thus, there does not seem to be a generic catapult mechanism leading out of a channel. We hypothesize, however, that a jump would occur with a sudden increase of the learning rate at the point where gradient descent with a low learning rate gets stuck in the channel.

Practical Guidelines

These are very interesting questions, and we will add some of them as an outlook to the extended discussion in the revised manuscript (please let us know if you disagree with this). However, we need more empirical results with setups used in practice to be able to provide reliable guidelines. For example, we can find channels in bigger networks (VGG trained on CIFAR10; see our answers on "significance for practical scenarios" to reviewers Po89 and YsiZ) but we did not yet investigate under which circumstances going down the channels is beneficial for generalization.

Thank you very much for your careful reading of the manuscript, your insightful detailed, and helpful suggestions. We are pleased to read that you found our work to be "very creative and original" and that it "could prove very insightful". We address your concerns and suggestions in detail below.

Investigating pre-trained networks:

This is an interesting point, but it is difficult to address it adequately. We expect channels to infinity to be present in the loss landscape of large neural networks. Indeed, preliminary experiments applying the jump procedure (Appendix C.1) to a VGG pretrained on CIFAR10 leads into a channel. However, the problem is to identify under which circumstances the training dynamics lead into a channel --- addressing this requires significant additional work. Furthermore, the channels may not be trivial to identify, because standard training with SGD and regularization may lead only to the very beginning of it.

Edge of stability:

This is an excellent suggestion. Extending our scaling analysis (lines 180-181; Fig. 6 top row) we can show that the maximum eigenvalue of the Hessian $\lambda_\mathrm{max}$ diverges with $1/\epsilon^2$ in a channel. Thus, the sharpness does not saturate and, following the "edge of stability", any optimizer with finite learning rate will eventually get stuck. Using the scaling of the sharpness, we can quantify where the optimizer gets stuck for a given learning rate $\eta$: according to the edge of stability, the maximum eigenvalue scales inversely with the learning rate, $\lambda_\mathrm{max}\sim1/\eta$, leading to $\epsilon_\mathrm{min} \sim \sqrt{\eta}$. We empirically verified this on the setup of Fig. 6: we find indeed that GD trajectories get stuck at $\lambda_\mathrm{max}=2/\eta$. Thus, this setup presents a stylized example of the edge of stability phenomenon, where the geometrical properties of the landscape are defined by the channel.

Weight decay:

Indeed, weight decay would prevent the dynamics from continuing along the channel. Nonetheless, the computation would be a finite-difference approximation of the directional derivative that gives rise to the Gated Linear Unit, which provides a clear understanding of the underlying computation. We will add the following brief discussion to the paragraph in lines 223-227 to expand the brief statement from line 198: "Adding regularization has a similar effect: from a certain point the regularization outweighs the decrease in loss such that the dynamics get stuck inside the channel. Nonetheless the network implements an approximation of the gated linear unit."

Figures 2 & 3:

This is a very helpful suggestion. We will follow your advise and combine parts of Figs. 2 and 3 into a single figure. To this end, we will move the top right panels of Fig. 2 and panels a, b, e from Fig. 3 into the appendix. Fig. 3 c, d will be combined with the remainder of Fig. 2, thereby summarizing the main results on saddle-lines in a single figure.

Conclusion

Thank you very much again for the work that you put into this review and the helpful comments. We are particularly grateful for your suggestion of the link to the "edge of stability", which we believe provides an interesting link to the literature and improves the significance of our results. If you agree, we would appreciate if you would revise your scores to reflect this.

We thank you very much for your detailed reading of our manuscript, your insightful questions and suggestions, and the balanced evaluation. We truly appreciate your effort put into reviewing our work and we are pleased to read that you found it "interesting" and "very well presented". We address your questions and concerns in detail below. In particular, we would like to clarify two important points regarding the activation function and the effect of biases that might have biased your rating.

Effect of adding biases:

Adding biases does not have a negative impact on the number of channels, all experiments in Fig. 5b are with biases. In contrast, adding biases strongly affects the presence of plateau-saddles (local minima on $\gamma$-lines) -- they indeed seem to "go away" in more realistic scenarios, as expected from inspection of the structure of the Hessian matrix (see lines 113-114 and supplemental Fig. A1 and our response to reviewer jaHr on this point). Thus, simply adding biases does not suggest that the channels go away in more realistic scenarios, in contrast, plateau-saddles disappear.

Unusual activation function:

This comment seems to be based on a misunderstanding. We used a range of different activation functions throughout the manuscript: Figs. 2, 3 are indeed based on $\mathrm{softplus}(x)+\mathrm{sigmoid}(4x)$ but Figs. 4, 5 are based on $\mathrm{softplus}(x)$ (see appendix B3; a preliminary analysis for $\mathrm{erf}$ is consistent with the $\mathrm{softplus}$ results), and Fig. 6 is based on $\mathrm{erf}(x)$ (see line 183). Thus, our results show that channels to infinity occur with different activation functions.

Significance for practical scenarios:

We expect channels to infinity to be present in the loss landscape of large neural networks. Indeed, preliminary experiments applying the jump procedure (Appendix C.1) to a VGG pretrained on CIFAR10 leads into a channel. However, the tricky question is under which circumstances the training dynamics lead into a channel --- addressing this requires significant additional work. Furthermore, the channels may not be trivial to identify, because standard training with SGD and regularization may lead only to the very beginning of it.

Generalization:

Train and test loss are almost identical in our setups because the data is relatively low-dimensional and densely sampled. In particular, the distribution of the test loss for channels and finite minima is similarly overlapping as the distribution of the training loss.

Momentum:

The reason why SGD and ADAM get stuck is their finite step size in combination with the increasing sharpness (maximum eigenvalue of the Hessian) of the channels. As shown in the literature on the "edge of stability" [Cohen, et al.; Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability], the maximum sharpness is inversely related to the step size $\eta$. Note that this effect occurs with and without momentum (see their Eq. 1). Extending our scaling analysis (lines 180-181; Fig. 6 top row) we can show that the sharpness diverges with $1/\epsilon^2$ in the channels. Furthermore, SGD training in the setup of Figure 6 converges exactly at a point in the channel corresponding to the edge of stability ($\lambda_\mathrm{max} = 2/\eta$). Regarding your proposal of an optimizer operating in "a rotated parameter space", we believe this is very similar in spirit to the jump procedure that we used to proceed when the ODE solver got stuck (lines 169-172; appendix C.1).

ReLU activation:

We avoided the ReLU activation due to its non-smoothness (the homogeneity can be accounted for, see next point). ReLU loss landscapes present non-differentiable cusps at points in parameter space defining the boundary between a ReLU neuron being activated or dead for a specific datapoint. We noticed, especially nearby saddles where trajectories can get very close to each other, that ReLU network trajectories would get stuck at cusp-like points near the $\gamma$-line. However, given that softplus is a smooth approximation of ReLU, we speculate that ReLU landscapes can still present channel structures.

Identifying channels:

This is an excellent point and we thank you very much for raising it. Indeed the threshold on weight similarity provides only a necessary condition. Thus, we will scale the numbers reported in Fig. 5b down by including only trajectories that have > 90% fraction of the updates parallel to the saddle line. This reduces the number of confidently identified channels to 80-90% the reported number (depending on the dataset), which is still a significant fraction of runs. On top of this, we improved the channel identification criteria in two important ways:

1. We now account for the approximate homogeneity of softplus (in the limit of large inputs ReLU and softplus coincide) which allows to rescale input- vs. output-weights to correctly identify putative channels with high norm input weights. Specifically, channels of high input weight norm softplus neurons can have weights grow both parallel to the saddle line and parallel to the direction of the input weight vector (becoming effectively a two dimensional space).
2. The originally selected threshold for the rosenbrock data was including solutions that were not channels (see dashed line in Fig B8). We selected better thresholds for each dataset and saw a drastic reduction of runs that had updates not parallel to the saddle line (or the extra manifold described above). The down-scaling by 80-90% already includes these two corrections.

Association between channels and $\gamma$-lines:

Quite often, a small perturbation from a $\gamma$-line leads to a channel. However, it is an open question, whether the attractive region of all channels contains points that are arbitrarily close to a $\gamma$-line. In some preliminary work, we tried with mixed success to find $\gamma$-lines nearby the region of attraction of given channels. A direct association between $\gamma$-lines and channels is that they are asymptotically parallel. To see this we can rewrite the reparameterization introduced in section 4 as $\mathbf{\theta} = (\mathbf{w_1}, \ldots, \mathbf{w}, \mathbf{w}, a_1, \ldots, c/2, c/2) + (\mathbf{0}, \ldots, \epsilon\mathbf{\Delta}, -\epsilon\mathbf{\Delta}, 0, \ldots, a/2\epsilon, -a/2\epsilon)$ where the update given by the second term is asymptotically equal to $\frac{a}{2\epsilon}(\mathbf{0}, \ldots, 0, 1, -1)$, which is parallel to a $\gamma$-line.

Star-shaped connectivity:

We would like emphasize that the connection to star-shaped connectivity was clearly indicated as a speculation (line 228). That being said, we agree that the connection is a bit premature since channels could only explain aspects of star-shaped connectivity---clearly, it could not explain the structure observed by Annesi et al. for a spherical perceptron (but note that they consider geodesics on the sphere, not in euclidean space)---and, re-reading our formulation from this point of view, we agree that the link to star-shaped connectivity is premature. We will remove this speculation from the discussion.

Conclusion

We thank you again for the careful reading and the insightful questions and comments. We hope our rebuttal convinced you that the channels can be of practical significance; in particular we hope we were able to clarify that the channels occur for commonly used activation functions and that they also occur with biases. If this is the case please consider raising your overall score to reflect this.

Thank you for carefully reading our paper, the fair and positive evaluation, and your questions and suggestions. We are pleased to read that our work "seems to be a valuable contribution to the study of loss landscapes". We address your concerns in detail below. If our answers will increase your confidence in your judgment we would appreciate if you could raise your score to reflect this.

Significance of the connection to Gated Linear Units:

When we saw the channels for the first time, and realized that they occur in a significant number of cases, we naturally wondered about their functional role. The connection to Gated Linear Units answers this question: the channels implement a new function that is the derivative of the activation function multiplied by a linear projection of the input. When naively looking at the architecture of an MLP, fitting such functions does not seem possible. As a concrete example, a network with a finite number of softplus neurons can perfectly fit targets composed of Gated Linear Units. Fig. 6B (left panel) provides another stylized example.

Regularization:

Indeed, regularization would prevent the dynamics to reach the "end" of the channel. However, even if the dynamics "get stuck" in the channel, the network would still employ a finite-difference approximation of the Gated Linear Unit, which provides a clear understanding of the underlying computation (see lines 226-227). In other words, the landscape would contain a channel whose "end" is at finite values due to regularization. We will expand the discussion of regularization (as also suggested by Reviewer YsiZ) to make sure it is clear that the idealized behavior (convergence to GLUs) can only occur in the absence of regularization. Concretely, we will add this brief discussion to the end of the paragraph in lines 223-227: "Adding regularization has a similar effect: from a certain point the regularization outweighs the decrease in loss such that the dynamics get stuck inside the channel. Nonetheless the network implements an approximation of the gated linear unit."

Number of runs vs. number of unique minima in Fig. 2:

We ran $50 \cdot 2^r$ simulations per network, where $r$ is the hidden layer size. We experimentally verified that conducting more simulations for the smaller networks does not lead to the discovery of new minima. Furthermore, we have empirically noticed that the number of unique minima follows roughly $2^r$, and we estimated that running 50 times that number of simulations would allow good coverage of the landscape. There may be minima whose basins of attraction are too small or unreachable from standard Glorot initializations; those would not be quantified in our figure.

Distance between channels and $\gamma$-line:

The distance between the channel and the $\gamma$-line does not necessarily remain small because the other parameters can move significantly; this is quantified in supplemental Fig. B5.

Effect of biases on plateau-saddles:

The stability of the $\gamma$-line is dependent on two sub-matrices that can be derived from the Hessian of the loss (Fukumizu & Amari, 2000); in Appendix A.1 we briefly describe their structure. For a plateau-saddle, these matrices must be positive or negative semi-definite (eigenvalues must be all positive or all negative), or zero. In general it is very difficult to make predictions on the signs of the eigenvalues because they depend on the whole training dataset. For the case of multi-dimensional output weights, Petzka & Sminchisescu 2021 (section 5.5) simply note that constraints on all eigenvalues are too strong to be satisfied by any practical example with noticeable probability. We speculate that the same accumulation of constraints can explain our results for adding biases to the network. Our empirical evidence, in combination with Petzka & Sminchisescu 2021, shows that symmetry-induced $\gamma$-lines are not a source of local minima in loss landscapes. Note that this only applies to plateau-saddles, not the channels to infinity.

Fig. 3a:

For the final version we will combine Figs. 2 and 3 to improve the flow of the manuscript (see also the comment by Reviewer YsiZ), which includes dropping 3a, because the main intuition that the plateau-saddle looses stability is also conveyed in 3c.

Thank you very much again for your constructive feedback!