Uncovering Critical Sets of Deep Neural Networks via Sample-Independent Critical Lifting

We thank the reviewer for providing a detailed review and giving valuable suggestions on improving our paper. Below we first give a comment on motivation and importance of the paper, then address the reviewer’s questions.

$**Motivation and importance of paper:**$ Understanding the global convergence and training dynamics of neural networks remains a fundamental challenge. A key obstacle is the prevalence of non-global critical points and manifolds, which hinder efficient training and convergence to global minima. Although recent work has identified high-dimensional critical manifolds—embedded from narrower networks' critical points—the geometry of these sets and their dependence on training data are still poorly characterized. Without this understanding, it is difficult to analyze the distribution of local minima, saddles, and strict saddles, or to estimate escape probabilities, convergence rates, and acceleration strategies near them.

Our work takes a significant step toward characterizing critical sets geometrically in two ways. First, building on existing work, we demonstrate the prevalence of low-complexity critical points lifted from narrower networks, which exhibit favorable generalization properties. The tendency of training dynamics to stagnate at these low-complexity critical points—combined with early stopping—may help networks generalize well regardless of sample noise. Second, we show that saddles exist among sample-dependent lifted critical points, thus establishing a foundation for further studying escape dynamics from these saddles. We particularly emphasize that for one hidden layer networks, all sample-dependent lifted critical points are saddles, thus narrowing the potential presence of local minima to the sample-independent subset.

$**Importance of the analytic assumption in Assumption 3.1:**$ The assumption that the activation function is analytic is mainly used to 1.establish the linear independence of neurons (Lemma A.1.1) and 2. guarantee that the level sets of the function has zero measure (Lemma A.1.3). These are used in, e.g., Proposition 4.2.1 to show that any critical point $\theta_{\mathrm{wide}}$ of the form (2) is a saddle, and in Theorem 4.2.1 to construct a sample-dependent lifted critical point $\theta_{\mathrm{wide}}$ of the form (2). A twice-continuously differentiable activation satisfying 1. and 2. also works.

$**Definition of critical embedding operators:**$ In this paper we mentioned three critical embedding operators, namely the null embedding operator, splitting embedding operator and general compatible embedding operator. Intuitively speaking,

(a) The null embedding operator adds neurons of zero input weight. In [1] the authors define null embedding operator for neural networks with bias terms. For unbiased neural networks, we need $\sigma(0)=0$ and to set the output weight to be zero.

(b) The splitting embedding operator copies one neuron and “splits” the output weight; for example, a parameter (1/6a, w, 1/3a, w, 1/2a, w) is obtained from (a, w) via splitting embedding operator.

(c) A general compatible embedding operator generalizes the previous two operators by taking into account, e.g., the composition of them and the permutation of indices in different layers. We will add more formal definitions of these critical embedding operators in the paper.

[1] Y. Zhang, Y. Li, Z. Zhang, et al., “Embedding Principle: a hierarchical structure of loss landscape of deep neural networks”, arXiv: 2111.15527, 2021.

$**Why would sample-dependent lifted critical points be more important than sample-independent ones:**$ The main contribution of this paper is $**three-fold:**$ introducing a critical lifting operator (contribution (a)), discovering sample-independent lifted critical points which do not arise from previously studied embedding operators (contribution (b)), identifying sample-dependent lifted critical points and show that saddles exist among them (contribution (c)). None of them is prioritized: each advances a different understanding of deep learning.

$**Essentially, the condition for the sample-dependent critical points is that the sample size needs to be larger than the dimension:**$ Yes, it is the basic idea of the condition for sample-dependent lifted critical points to exist. More rigorously, it is related to the rank and kernel of the network’s Jacobin matrix (which follows from linear independence of neurons), as well as the structure of loss function (which yields the existence of samples). Moreover, we give lower bounds on sample size for sample-dependent lifted critical points to exist, further clarifying the interplay between critical points and samples.

We thank the reviewer for providing a detailed review and giving valuable suggestions on improving our paper. We will correct the typos in the revision of this paper. Below we briefly discuss how our work is related to a broader field in machine learning:

Understanding the global convergence and training dynamics of neural networks remains a fundamental challenge. A key obstacle is the prevalence of non-global critical points and manifolds, which hinder efficient training and convergence to global minima. Although recent work has identified high-dimensional critical manifolds—embedded from narrower networks' critical points—the geometry of these sets and their dependence on training data are still poorly characterized. Without this understanding, it is difficult to analyze the distribution of local minima, saddles, and strict saddles, or to estimate escape probabilities, convergence rates, and acceleration strategies near them.

Our work takes a significant step toward characterizing critical sets geometrically in two ways. First, building on existing work, we demonstrate the prevalence of low-complexity critical points lifted from narrower networks, which exhibit favorable generalization properties. The tendency of training dynamics to stagnate at these low-complexity critical points—combined with early stopping—may help networks generalize well regardless of sample noise. Second, we show that saddles exist among sample-dependent lifted critical points, thus establishing a foundation for further studying escape dynamics from these saddles. We particularly emphasize that for one hidden layer networks, all sample-dependent lifted critical points are saddles, thus narrowing the potential presence of local minima to the sample-independent subset.

We thank the reviewer for providing a detailed review and giving valuable suggestions on improving our paper. Below we first give a comment on motivation and importance of the paper, then address the reviewer’s questions.

$**Motivation and importance of paper:**$ Understanding the global convergence and training dynamics of neural networks remains a fundamental challenge. A key obstacle is the prevalence of non-global critical points and manifolds, which hinder efficient training and convergence to global minima. Although recent work has identified high-dimensional critical manifolds—embedded from narrower networks' critical points—the geometry of these sets and their dependence on training data are still poorly characterized. Without this understanding, it is difficult to analyze the distribution of local minima, saddles, and strict saddles, or to estimate escape probabilities, convergence rates, and acceleration strategies near them.

Our work takes a significant step toward characterizing critical sets geometrically in two ways. First, building on existing work, we demonstrate the prevalence of low-complexity critical points lifted from narrower networks, which exhibit favorable generalization properties. The tendency of training dynamics to stagnate at these low-complexity critical points—combined with early stopping—may help networks generalize well regardless of sample noise. Second, we show that saddles exist among sample-dependent lifted critical points, thus establishing a foundation for further studying escape dynamics from these saddles. We particularly emphasize that for one hidden layer networks, all sample-dependent lifted critical points are saddles, thus narrowing the potential presence of local minima to the sample-independent subset.

$**Why is the index of$w$and$w'$up to$k_3$:**$ $k_3$ is both for enumerating the entries of $a_1$ and the $w_{k_3}^{(3)}$'s. The dimension of $a_1$ is $m_3$ or $m_3+1$ depending on the narrower or wider network.

$**For simplicity assume the activation is even or odd. Which result does this assumption hold for exactly:**$ The result ''By linear independence of neurons …then $\theta_{\mathrm{wide}}$ is a sample-independent lifted critical point''.

$**The argument of Remark 4.5 does not make sense at all:**$ We apologize for the typo. Here it should be “Therefore, up to permutation of the entries, a $*sample-dependent*$ lifted critical point from $\theta_{\mathrm{narr}}$ takes the form (2)”.

$**Definition of critical embedding operators:**$ In this paper we mentioned three critical embedding operators, namely the null embedding operator, splitting embedding operator and general compatible embedding operator. Intuitively speaking,

(a) The null embedding operator adds neurons of zero input weight. In [1] the authors define null embedding operator for neural networks with bias terms. For unbiased neural networks, we need $\sigma(0)=0$ and to set the output weight to be zero.

(b) The splitting embedding operator copies one neuron and “splits” the output weight among these copies; for example, a parameter (1/6a, w, 1/3a, w, 1/2a, w) is obtained from (a, w) via splitting embedding operator.

(c) A general compatible embedding operator generalizes the previous two operators by taking into account, e.g., the composition of them and the permutation of indices in different layers. We will add more formal definitions of these critical embedding operators in the paper.

[1] Y. Zhang, Y. Li, Z. Zhang, et al., “Embedding Principle: a hierarchical structure of loss landscape of deep neural networks”, arXiv:2111.15527, 2021.

$**Line 153 and 154 very vague:**$ Please refer to the intuitive definition of critical embedding operators. Hopefully this can make the statement clearer.

$**The notion of ‘saddle’ is not defined:**$ A saddle of a real-valued, differentiable function is a critical point which is neither a local minimum nor a local maximum.

$**The work is not inclusive:**$ Under mild assumptions on activation, we are able to discover all sample-independent lifted critical points for one hidden layer neural networks. For example when $\sigma$ is odd, we have a linear independence of neurons and their derivatives: {\sigma(w_k x), \sigma(w_k x)x_1, ..., \sigma(w_k x)x_d: 1 <= k <= m} are linearly independent for any $m \in \mathbb{N}$ and non-zero $w_k$'s such that $w_k \pm w_j \ne 0$ for distinct $k,j$ (here $x_t$ denotes the t-th entry of input $x$). This result implies that a sample-independent lifted critical point \theta' = (a_k', w_k') from \theta = (a_k, w_k) iff.

(a) w_k' \in {\pm w_k}_{k=1}^m for any $k$ such that $a_k' \ne 0$.

(b) w_k' \in {\pm w_k}_{k=1}^m \cup {0} for any $k$ such that $a_k' = 0$.

(c) Given $k' \in \{1, ..., m'\}$, if $w_{k'}' \in \{\pm w_k\}$ for some $k \in \{1, ..., m\}$, then for any $x \in \mathbb{R}^d$ we have $\sum_{j:w_j' = \pm w_{k'}'} a_j' \sigma(w_{k'}^{'\mathrm{T}} x) = \sum_{j: w_j = \pm w_k} a_j \sigma(w_k^{\mathrm{T}} x)$.

In short, the null embedding, splitting embedding and flipping the signs of $w_k$'s produce all sample-independent lifted critical points. However, the case for deeper networks is unclear to us because the linear independence result is much more complicated. Very few work, e.g. [2] discusses this problem, and even with this result it is hard to explicitly characterize these critical points.

For sample-dependent lifted critical points, we mentioned in Remark 4.5 that it takes the form (2). However, the case for deeper networks is much more complicated. We hope to find a way to characterize them in the future.

[2] L. Zhang, “Linear Independence of Generalized Neurons and Related Functions”, arXiv:2410.03693, 2024.

We thank the reviewer for providing a detailed review and giving valuable suggestions on improving our paper. Below we first give a comment on motivation and importance of the paper, then address the reviewer’s questions.

$**Motivation and importance of paper:**$ Understanding the global convergence and training dynamics of neural networks remains a fundamental challenge. A key obstacle is the prevalence of non-global critical points and manifolds, which hinder efficient training and convergence to global minima. Although recent work has identified high-dimensional critical manifolds—embedded from narrower networks' critical points—the geometry of these sets and their dependence on training data are still poorly characterized. Without this understanding, it is difficult to analyze the distribution of local minima, saddles, and strict saddles, or to estimate escape probabilities, convergence rates, and acceleration strategies near them.

Our work takes a significant step toward characterizing critical sets geometrically in two ways. First, building on existing work, we demonstrate the prevalence of low-complexity critical points lifted from narrower networks, which exhibit favorable generalization properties. The tendency of training dynamics to stagnate at these low-complexity critical points—combined with early stopping—may help networks generalize well regardless of sample noise. Second, we show that saddles exist among sample-dependent lifted critical points, thus establishing a foundation for further studying escape dynamics from these saddles. We particularly emphasize that for one hidden layer networks, all sample-dependent lifted critical points are saddles, thus narrowing the potential presence of local minima to the sample-independent subset.

$**Is it necessary to discover all sample-independent lifted critical points:**$ Yes from the motivation of our paper. Currently, under a mild assumption on activation, we are able to discover all sample-independent lifted critical points for one hidden layer neural networks, as they are all produced by critical embedding operators. The case for deeper networks is unclear to us yet.

$**Concerning the sample dependence of critical points, it is not clear to me why your work complements the previous studies: **$ Several works, such as [1, 2], discover that embedding operators can produce sample-independent lifted critical points. We include this as Proposition 4.1.1 in our paper. However, they do not notice 1. the sample (in)dependence property of critical embedding, and 2. the operators cannot produce all sample-independent lifted critical points for deep neural networks. We address them in our paper. We also discover sample dependent critical points which are not discovered by previous works.

[1] Y. Zhang, Z. Zhang, T. Luo, et al., “Embedding Principle of Loss Landscape of Deep Neural Networks”, NeurIPS, 2021.

[2] B. Simsek, F. Ged, A. Jacot, et al., “Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances”, ICML, 2021.