We sincerely thank you for your detailed review and for the positive feedback on the clarity and rigor of our paper. Your questions are insightful and get to the heart of our work's motivation and technical contribution. We are confident that we can provide satisfactory answers and are grateful for the opportunity to clarify these points.

On the Motivation: Why Study Neuron Splitting?

This is a fundamental question. The importance of studying neuron splitting is not about the operation itself, but about using it as a precise, mathematically tractable model for understanding overparameterization and generalization.It provides insights into why overparameterized models can generalize well detailed as follow: A fundamental question in deep learning theory is how a highly complex neural network can generalize well. The embedding principle uncovered in our work provides a key mechanism:

• Our KKT embedding principle formally guarantees the existence of low-complexity solutions (those embedded from smaller, simpler models) within the vast solution space of overparameterized networks. Prior work has shown that gradient flow converges in direction to a KKT point of the max-margin problem (Lyu & Li, 2019), but a significant gap remained: large networks can possess many KKT points with varying complexities. Our work bridges this gap by proving that simple, generalizable solutions are structurally preserved as network width increases.
• This has immediate practical implications. The co-existence of both low-complexity (embedded) and high-complexity KKT points suggests that initialization and hyperparameter tuning play a critical role in selecting a solution. Our framework provides a theoretical basis for future studying this selection process and designing training procedures that explicitly steer optimizers toward these desirable, low-complexity solutions to improve generalization.

On Technical Necessity: Why is KKT Point Embedding Needed?

This is an excellent point that clarifies the core technical challenge. Your intuition that one could prove this by "showing the neurons remain splitted all the time" is correct, and our dynamic trajectory preservation result (Theorem 5.2) can be proved by this way.

However, Our contribution is best understood in two interconnected parts for homogeneous networks:

1. A Static KKT Point Embedding Principle: This proves that the solutions (KKT points) of a smaller network's max-margin problem are isometrically embedded within the solution set of any wider network derived from it. It uncovers the hierarchical nature of the KKT points of the max-margin problem. Moreover, it opens doors to further studies that identifies all KKT points from low to high complexity, thus obtaining all candidate solutions of the neural network training.
2. A Dynamic Trajectory Embedding Principle: This part is a further extension to the previous results, in which we realize that the embeddings we use to characterize KKT points can be further employed to characterize the trajectory of training dynamics: the entire gradient flow training trajectory is similarly preserved under our transformation.This result provides us a very powerful tool to charaterize (i) the embedding of non-global solutions (training data not perfectly separated) of neural networks and (ii) embedding of solutions for nonseparable training data. Note that the trajectory embedding principle in it self doesnot imply KKT embedding principle because there is no guarantee that all KKT points are neural network solutions .

A central concern in neural network theory is the connection between the final parameters of a trained network and the solutions to a corresponding KKT problem. To address this, our Static KKT Point Embedding Principle fundamentally relies on techniques related to the KKT conditions. This analytical approach is non-trivial and necessary to establish our results. Subsequently, the Dynamic Trajectory Embedding Principle, which is an extension of the static problem, naturally leverages the same KKT-based framework for its analysis.

On the Generality of the Conditions in Theorem 4.5

This is a very sharp observation. The conditions in Theorem 4.5—that the transformation $T$ must preserve the network's output and subgradient structure—are indeed strong, and this is by design.

A generic linear transformation would completely change the function computed by the network, making a comparison of the dynamics meaningless. Instead, our conditions define precisely what it means for a transformation to be structure-preserving, ensuring that the function and its optimization landscape are isometrically embedded.

To directly address your question about applications in other scenarios, we have formally extended our framework to homogeneous Convolutional Neural Networks (CNNs). This required designing a novel 'channel-splitting' transformation, meticulously constructed to respect the architectural hallmarks of CNNs, such as weight sharing and locality. The transformation isotropically splits a single feature channel into $m_{split}$ new channels via coefficients $\{c_i\}$ that satisfy $\sum_{i=1}^{m_{split}} c_i^2 = 1$. The specific construction is as follows:

• At Layer k: The filter $F_j^{(k)}$ that produces channel $j$ is replaced by $m_{split}$ new filters, where the i-th new filter is defined as $c_i F_j^{(k)}$.
• At Layer k+1: In turn, every filter in the next layer that takes channel $j$ as input has its j-th input slice replaced by $m_{split}$ new slices, where the i-th new slice is scaled by the corresponding coefficient $c_i$.

Crucially, we prove that this transformation satisfies the strong conditions of our core theory in Theorem 4.5. This is a powerful validation of our framework, showing it provides the rules for designing such embeddings in other complex, homogeneous architectures. Given its significance, we will integrate this CNN extension as a new main result in Sections 4 and 5 of our revised manuscript, with an eye toward future applications in architectures like Transformers.

Minor Writing Issues

Thank you for your careful reading and for spotting these issues. We appreciate the detailed feedback and have corrected all of the issues in the revised manuscript:

• The notation for norms and vectors has been made consistent throughout the paper, as you suggested (e.g., using $L^2$ notation).
• The chain rule expression in Definition 3.2 has been clarified to resolve the ambiguity, reflecting the structure you proposed.
• The overloaded symbol $L$ has been resolved. We now use $L$ only for the order of homogeneity and have adopted your excellent suggestion to use $\omega(\cdot)$ for the ω-limit set.
• The stray parenthetical note, which was an internal remnant from a previous draft, has been removed.
• The typo has been corrected to the proper mathematical notation in Line 300.

We wish to express our sincere gratitude for your exceptionally thorough and insightful review. Your constructive comments were invaluable; they helped us clarify the core contributions of our work and have significantly strengthened the manuscript. We hope these detailed responses address your questions, and that our revised paper will merit your support.

References

• Lyu, K., & Li, J. (2019). Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv:1906.05890.

Thank you for your positive feedback and for raising your score. We are particularly encouraged that you found our elaboration on the work's significance to be clear and that you appreciated the new channel-splitting framework. We genuinely appreciate your willingness to raise the score for our work.

We are grateful to the reviewer for the thorough and insightful feedback on our manuscript. Your comments on the related work, narrative structure, and significance of our results have been instrumental in strengthening the paper and have also highlighted promising future research directions. We have undertaken a substantial revision based on your detailed guidance.

On Related Work and Significance

Broadened Context of Related Work

Following your guidance, we have substantially revised our discussion of the related literature in two key areas:

1. “Optimization dynamics and implicit bias”: We have updated this section's title. We expanded our discussion to cover recent developments, including the analysis of margin in non-homogeneous models (Kunin et al., 2023)、(Cai et al. 2025), and the connection between KKT conditions and benign overfitting (Frei et al., 2023).

2. “Network embedding principles”:we have broadened the discussion to better situate our work, citing foundational research on hierarchical structures that is highly relevant to our embedding principles (Fukumizu & Amari, 2000), as well as recent studies on permutation symmetries that connect different global minima (Şimşek et al., 2021). we carefully considered the suggested literature on network pruning and the Lottery Ticket Hypothesis. We agree this is an important field, but we concluded that its core principle is conceptually distinct from our work. Specifically, pruning focuses on identifying and removing less important neurons to create a sparse sub-network. In contrast, our method focuses on merging neurons that have a specific structural relationship to create a smaller, equivalent network.

Articulating the Significance of Our Work

Our contribution is best understood in two interconnected parts:

1. A Static KKT Point Embedding Principle: This proves that the solutions (KKT points) of a smaller network's max-margin problem are isometrically embedded within the solution set of any wider network derived from it.
2. A Dynamic Trajectory Embedding Principle: This shows that the entire gradient flow training trajectory is similarly preserved under our transformation. The static principle is the essential foundation that guarantees this dynamic preservation.

While the invariance of the network function under neuron splitting may seem intuitive, the core of our contribution lies in rigorously proving that this intuition extends to the optimization landscape's critical points (the KKT points) and, crucially, to the training dynamics (the gradient flow trajectories). This formalization yields significant insights into why overparameterized models can generalize well.

A fundamental question in deep learning is how a highly complex neural network can effectively learn a low-complexity classification boundary. Our work provides a direct theoretical answer:

• Our KKT embedding principle formally guarantees the existence of low-complexity solutions (those embedded from smaller, simpler models) within the vast solution space of overparameterized networks. Prior work has shown that gradient flow converges in direction to a KKT point of the max-margin problem (Lyu & Li, 2019), but a significant gap remained: large networks can possess many KKT points with varying complexities. Our work bridges this gap by proving that simple, generalizable solutions are structurally preserved as network width increases.
• This has immediate practical implications. The co-existence of both low-complexity (embedded) and high-complexity KKT points suggests that initialization and hyperparameter tuning play a critical role in selecting a solution. Our framework provides a theoretical basis for understanding this selection process and for future work on designing training procedures that explicitly steer optimizers toward these desirable, low-complexity solutions to improve generalization.

On Narrative Flow and Major Questions

On the Narrative Flow of Sections 3 and 4

Following your valuable suggestions, we have improved the narrative flow by:

• Deferring technical definitions (formerly Def. 3.1, 3.2, Asm. 3.3) to the appendix.
• Splitting Section 4 into two separate sections: one establishing the general KKT embedding framework and another dedicated to its application to neuron splitting and channel splitting.

On the Major Questions

1. On the relation between Theorem 4.6 and 4.9: We agree that the two-layer network transformation is a special case of the deep network one. To streamline the paper, we have removed the separate theorem for two-layer networks and now present only the general deep network result as our main theorem. The two-layer construction is now included as a pedagogical example in the appendix.

2. On extending the framework to CNNs: We have formally extended our framework to homogeneous Convolutional Neural Networks (CNNs) by generalizing "neuron splitting" to "channel splitting" ans included this as a new main result. The transformation splits a channel into $m_{split}$ new channels using coefficients $\{c_i\}$ that satisfy $\sum_{i=1}^{m_{split}} c_i^2 = 1$. It is defined as follows:

• Layer k: Replacing the filter $F_j^{(k)}$ that produces channel $j$ with $m_{split}$ new filters, where the $i$-th new filter is defined as $c_i F_j^{(k)}$.
• Layer k+1: For every filter $F_p^{(k+1)}$, replacing its $j$-th input channel slice with $m_{split}$ new slices, where the $i$-th new slice is scaled by the corresponding coefficient $c_i$.

Successfully proving that this general transformation satisfies the conditions of our core theory (Theorem 4.5) is a powerful validation of our framework. It shows that the principles of isometric and output/subgradient preservation are not confined to simple network topologies but provide a robust blueprint for analyzing a wider class of homogeneous networks. This strongly suggests that our framework can be adapted to other architectures with specific structural invariances.

3. On mapping KKT points from larger to smaller networks: Applying our framework to the larger-to-smaller network mapping via neuron merging is a valuable direction that connects our work to network pruning.

To map Karush-Kuhn-Tucker (KKT) points from a larger network (parameters $\eta$) to a smaller one ($\theta$), we can define a "neuron merging" operator $M$ where $\theta = M(\eta)$. For this operator to be KKT-preserving , it must satisfy conditions analogous to our Theorem 4.5, namely:

1. Output Preservation: $\Phi(M(\eta); x) = \tilde{\Phi}(\eta; x)$. This is analogous to the output-preserving condition for splitting.
2. Subgradient Preservation: $\partial_{\theta}^{\circ}\Phi(M(\eta); x)$ must maintain a specific relationship with $M(\partial_{\eta}^{\circ}\tilde{\Phi}(\eta; x))$, analogous to the subgradient condition for splitting.

The output preservation condition imposes a strict structural constraint. For instance, merging several neurons into one ($\sum_{i}a_i\sigma(b_i^T x) \to a\sigma(b^T x)$) requires the weight vectors $\{b_i\}$ of the neurons to be collinear.

While this requirement seems restrictive, it connects to the practical phenomenon of neural condensation, where neuron weights are observed to align and become collinear during training (Zhou, et al.,2022). This suggests that valid opportunities for such a theoretically-grounded merge may arise naturally.

Our future work will explore the precise KKT point mapping from larger to smaller networks under valid merging operations.

Response to Minor Comments

We have incorporated all of your minor comments:

• Figure 1 has been resized and given a more detailed caption, and it is now referenced at the end of the introduction to serve as a roadmap.

• The technical definitions have been moved to the appendix.

• Section 4 has been split into two separate sections.

• We have clarified the definition of ω-limit sets and added a formal citation.

Once again, we would like to express our gratitude for your thorough and constructive review. Your comments have been instrumental in helping us strengthen the manuscript.

References

• Kunin, D., et al. (2023). The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks. International Conference on Learning Representations (ICLR).
• Cai, Y., et al. (2025). Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks. Proceedings of the 42nd International Conference on Machine Learning.
• Frei, S., et al. (2023). Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization. arXiv preprint arXiv:2303.01462.
• Fukumizu, K., & Amari, S. (2000). Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural Networks.
• Şimşek, B., et al. (2021). Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances. Proceedings of the 38th International Conference on Machine Learning (ICML).
• Lyu, K., & Li, J. (2019). Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv:1906.05890.
• Zhou, H., et al. (2022). Towards understanding the condensation of neural networks at initial training. Advances in Neural Information Processing Systems, 35.
• Hirsch, M. W., Smale, S., & Devaney, R. L. (2013). Differential equations, dynamical systems, and an introduction to chaos. Academic Press.

We are sincerely grateful to the reviewer for their careful assessment and for recognizing that our theoretical results are correct, novel, and useful for future research. Your feedback on the paper's structure and clarity has been invaluable, and we have undertaken a significant revision to address your concerns and enhance the manuscript's impact.

On Contribution Density and Paper Structure

We understand and appreciate your central concern regarding the density of the theoretical contribution and the overall structure. To address this, we have substantially restructured the paper to improve both its clarity and theoretical density.

1. Streamlined Theoretical Presentation: We have consolidated our main theoretical results to create a more focused narrative. This includes deferring highly technical definitions (formerly Def. 3.1, 3.2, and Asm. 3.3) to the appendix and splitting Section 4 into two more focused parts: one establishing the general KKT mapping framework, and the other detailing its application to neuron splitting. Furthermore, to resolve the feeling of redundancy you noted, we have consolidated the application theorems. The two-layer network transformation, being a special case of the deep network one, has been moved to the appendix as a pedagogical example.

2. Enhanced Contribution Density with a New Major Result: To further address your point on contribution density and to showcase the power of our framework, we will include a new major result in the revised manuscript: the formal extension of our framework to homogeneous Convolutional Neural Networks (CNNs). This is a non-trivial extension that required designing a new "channel splitting" transformation that respects the architectural constraints of CNNs, such as weight sharing and locality. The transformation splits a channel into $m_{split}$ new channels using coefficients $\{c_i\}$ that satisfy $\sum_{i=1}^{m_{split}} c_i^2 = 1$. It is defined as follows:

• Layer k: Replacing the filter $F_j^{(k)}$ that produces channel $j$ with $m_{split}$ new filters, where the $i$-th new filter is defined as $c_i F_j^{(k)}$.
• Layer k+1: For every filter $F_p^{(k+1)}$, replacing its $j$-th input channel slice with $m_{split}$ new slices, where the $i$-th new slice is scaled by the corresponding coefficient $c_i$.

Successfully proving that this general transformation satisfies the conditions of our core theory (Theorem 4.5) is a powerful validation of our framework. It shows that our principles are not confined to simple network topologies but provide a robust blueprint for analyzing a wider class of homogeneous networks.

3. On Providing Empirical Contribution: we have conducted a comprehensive suite of experiments to empirically validate our theory and will add a dedicated section for these results to the main paper. Our experiments are presented as a three-step narrative, moving from ideal conditions to realistic scenarios, to systematically test the robustness and generality of our Trajectory Embedding Principle.

1. Verification under Ideal Conditions: First, we verified our theory in an idealized setting (homogeneous MLP, GD, separable data). As predicted, the trajectories of the smaller and larger networks remained perfectly aligned, with a minuscule error of 4e-13 attributable solely to numerical precision drift over one million steps.

2. Robustness to Relaxed Assumptions: Next, we tested the principle's robustness by relaxing these ideal conditions, using SGD, non-separable data, and even applying the theory to homogeneous CNNs via our "channel splitting" transformation. In all cases, the trajectory error remained vanishingly small, confirming that the principle holds even without continuous-time optimizers or linearly separable data.

3. Generality on a Practical Task (MNIST): Finally, to demonstrate practical relevance, we scaled our experiments to the MNIST dataset, training both homogeneous MLPs and CNNs that achieved high accuracy (>99%). The trajectory preservation principle held with remarkable precision (1e-14 error), confirming its applicability to modern architectures during meaningful learning on a real-world task.

Conclusion: Crucially, the tiny, non-zero errors observed are an expected consequence of floating-point arithmetic. This comprehensive validation provides strong evidence that our theoretical framework is not just mathematically sound but also a robust and generalizable tool for understanding practical neural networks.

Full experimental details will be added in the appendix, and our code will be made publicly available to ensure full reproducibility.

Response to Minor Issues

We thank the reviewer for catching these important details. We have addressed all of them in the revised manuscript.

• On Undefined Mathematical Symbols: We apologize for the lack of clarity. We have carefully revised the abstract and introduction to ensure that all key mathematical notations (such as $P_{\Phi}$, $L(\theta(0))$, and $T$) are properly introduced and briefly defined upon their first appearance, improving the paper's accessibility.

• On Missing References: We are grateful for the excellent suggestions on related work. We have substantially revised our literature review to provide a more comprehensive context. This includes broadening our discussion on ‘Network embedding principles’ to include the foundational work on hierarchical structures and symmetries (Fukumizu & Amari, 2000; Şimşek et al., 2021) and the lottery ticket hypothesis (Malach et al., 2020). Additionally, we updated our discussion on ‘Implicit Bias’ to cover recent advancements in non-homogeneous models and connections to benign overfitting (Kunin et al., 2023; Frei et al., 2023). We believe these additions now situate our work much more clearly within the landscape of contemporary research.

• On the Stray Comment: We sincerely apologize for this oversight. The comment on line 197 was indeed a leftover note from an internal draft and has been removed from the final version.

Once again, we would like to express our gratitude for your thorough and constructive review. Your comments have been instrumental in helping us strengthen the manuscript.

References

• Fukumizu, K., & Amari, S. (2000). Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural Networks.
• Şimşek, B., et al. (2021). Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances. Proceedings of the 38th International Conference on Machine Learning (ICML).
• Malach, E., et al. (2020). Proving the Lottery Ticket Hypothesis: Pruning is All You Need. Proceedings of the 37th International Conference on Machine Learning (ICML).
• Kunin, D., et al. (2023). The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks. International Conference on Learning Representations (ICLR).
• Frei, S., et al. (2023). Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization. arXiv preprint arXiv:2303.01462.

We are very grateful for your insightful review and constructive feedback. We especially appreciate that you recognized the novelty of our work in showing the preservation of the entire gradient-flow path, which is a core part of our contribution. Your feedback has been instrumental in helping us bridge the gap between our theory and its practical implications.

Our Main Response: A Comprehensive Empirical Demonstration

In direct response to your excellent suggestion to bridge the gap between our idealized theory and practice, we have conducted a comprehensive suite of experiments to systematically test the robustness and generality of our Trajectory Embedding Principle. We present these findings as a three-step narrative, moving from ideal conditions to more realistic scenarios. Our theory predicts that the maximum deviation between the two trajectories, $\max_{k \in \{0, 1, \dots, N\}} ||\eta_k - T\theta_k||_2$, will be negligible.

Step 1 & 2: Verification and Robustness on Toy Models

We began by verifying our theory under ideal conditions and then tested its robustness when key assumptions were relaxed.

• Setup: We constructed 2-layer homogeneous MLPs (with a hidden layer of 1 neuron vs. 2 neurons) and simple 2-layer CNNs (consisting of one convolutional of 2 channels vs. 4 channels and one fully-connected layer). We initialized them such that the initial parameters of the larger network were a direct mapping of the smaller one via our transformation $T$ (i.e., $\eta_0 = T\theta_0$). This transformation T implements a neuron splitting mechanism for MLPs and a channel splitting mechanism for CNNs. The latter is described in detail in our response to Question 2.The latter is described in detail in our response to Question 2. Both networks were then trained under various conditions for 1 million steps on a two-dimensional linearly separable dataset.

• Conclusion: The results, summarized below, confirm that the trajectory error remains extremely small across all setups.

The small, non-zero errors (on the order of 1e-13 to 1e-10) are an expected consequence of numerical precision. Over a million steps, minuscule floating-point rounding errors inevitably accumulate.

It is crucial to note that for the SGD experiment (Exp. 2), both networks were trained using an identical sequence of mini-batches. We also ran a control experiment where the two networks were trained with different, randomized batch orders. In this scenario, the maximum trajectory error increased dramatically to 1.5316e-01. This result is expected, as a network's training path is highly dependent on its specific data sequence.

Step 3: Demonstrating Generality on MNIST

Finally, to address the scope and demonstrate the framework's applicability to more realistic architectures, we extended our tests to the MNIST dataset.

• Setup: We scaled up to the MNIST dataset (classifying 3s vs 5s), training both two-layer homogeneous MLPs (128 vs 256 hidden units) and two-layer CNNs (10 vs 20 channels). We used SGD with a learning rate of 0.001 and a batch size of 64 for 1000 epochs. The models achieved high classification accuracy (e.g., 99.99% for MLP, 99.79% for CNN), ensuring we observed the trajectory error during a meaningful learning phase.

• Conclusion: The results, summarized below, confirm that the trajectory preservation principle holds with remarkable precision even on this more complex task. Both the MLP and CNN models maintained a near-zero trajectory error throughout training, demonstrating that the training dynamics of the wider network are highly predictable from its narrower counterpart.

This comprehensive suite of experiments confirms that our theoretical principle is not only correct in its ideal setting but is also a robust and generalizable framework for understanding the relationship between homogeneous networks of different sizes.

We will add a dedicated section for these results to the main paper. A detailed description of all experimental setup will be provided in a dedicated appendix section in our revised manuscript. Furthermore, we will make the code publicly available on GitHub to ensure full reproducibility.

On Assumptions, Scope, and Specific Questions

On Strong Assumptions and Limited Scope: We agree that our theoretical setup involves strong assumptions. However, we wish to clarify a critical distinction: the assumptions of full-batch gradient flow and perfectly separable data are not required for our KKT and Trajectory Embedding Principles.

These conditions are primarily needed for the well-known result from prior work that guarantees the training direction converges to a max-margin KKT point. In contrast, our embedding principles (Theorems 4.5 and 5.2) describe a more fundamental structural relationship between networks. This relationship holds regardless of whether the training converges to that specific max-margin solution.

Our new experiments explicitly confirm this. The trajectory embedding holds with near-perfect precision even when using Stochastic Gradient Descent (Exp. 2 & MNIST) and on non-separable data (Exp. 3). This demonstrates that our framework is robust and not confined to those idealized convergence scenarios. Furthermore, to broaden the scope, we are happy to clarify that our revised manuscript introduces a new main result extending our framework to Convolutional Neural Networks (CNNs).

Question 1: How hard would it be to include bias terms? This is a challenging but important question. Extending our framework to include bias terms is non-trivial because biases break the network's strict homogeneity, which is a cornerstone of our current analysis and the associated max-margin dynamics. Specifically, this extension (with bias) would require a fundamental redesign of the corresponding linear transformation T, which currently relies on this homogeneous property.

However, very recent work has begun to bridge this gap. For instance, Cai et al. (2025) extend the implicit bias analysis to a broad class of near-homogeneous networks. Their framework is powerful because it formally includes architectures with components that break strict homogeneity, such as bias terms and residual connections. They show that for these networks, gradient descent still converges in direction to a KKT point of a corresponding margin maximization problem. This development provides a promising theoretical pathway for our own work. A natural future direction is to investigate whether our KKT embedding principle can be generalized to this near-homogeneous setting, thereby extending our results to practical networks that include biases.

Question 2: How hard would it be to handle weight sharing?

To demonstrate the generality and power of our approach, we have introduced a new main result extending our principle to Convolutional Neural Networks (CNNs).This required designing a novel 'channel-splitting' transformation, meticulously constructed to respect the architectural hallmarks of CNNs, such as weight sharing and locality. The transformation isotropically splits a single feature channel into $m_{split}$ new channels via coefficients $\{c_i\}$ that satisfy $\sum_{i=1}^{m_{split}} c_i^2 = 1$. The specific construction is as follows:

• At Layer k (Outgoing Weights): The filter $F_j^{(k)}$ responsible for generating channel $j$ is replaced by $m_{split}$ new filters. The i-th new filter is simply a scaled version of the original: $c_i F_j^{(k)}$.
• At Layer k+1 (Incoming Weights): Consequently, every filter in the next layer that receives channel $j$` as input has its j-th input slice replaced by $m_{split}$ new slices, with each new slice being scaled by the corresponding coefficient $c_i$.

Successfully proving that this transformation satisfies our core theory (Theorem 4.5) validates our framework as a flexible blueprint applicable to a broad class of homogeneous networks, with future work poised to explore architectures like Transformers.

Minor Issues

The "Theorem ??" placeholders were indeed artifacts from our drafting process and have all been corrected in the final version. We apologize for the oversight.

Thank you again for your valuable and pragmatic feedback. We believe the new empirical demonstrations, a broader scope that now includes CNNs, and our detailed clarifications directly address your main concerns. We hope these additions will convince you of the paper's value and merit your support.

References

• Cai, Y., et al. (2025). Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks. Proceedings of the 42nd International Conference on Machine Learning.

Thank you for your detailed experiments and revisions. I am increasing my score. Good luck!

We sincerely thank the reviewer for their positive feedback on the clarity and careful writing of our manuscript. We appreciate the thoughtful critique and would like to address the primary concerns regarding the paper's significance and originality, which we believe are more substantial than perceived.

On the Significance and Originality of Our Work

We respectfully argue that the significance of our findings is substantial, as it provides a foundational, structural explanation for key questions in deep learning generalization. Our contribution is best understood in two interconnected parts:

1. A Static KKT Point Embedding Principle: This proves that the solutions (KKT points) of a smaller network's max-margin problem are isometrically embedded within the solution set of any wider network derived from it.
2. A Dynamic Trajectory Embedding Principle: This shows that the entire gradient flow training trajectory is similarly preserved under our transformation. The static principle is the essential foundation that guarantees this dynamic preservation.

While the invariance of the network function under neuron splitting may seem intuitive, the core of our contribution lies in rigorously proving that this intuition extends to the optimization landscape's critical points (the KKT points) and, crucially, to the training dynamics (the gradient flow trajectories). This formalization is not merely a "good exercise"; it yields significant insights into why overparameterized models can generalize well.

A fundamental question in deep learning is how a highly complex neural network can effectively learn a low-complexity classification boundary. Our work provides a direct theoretical answer:

• Our KKT embedding principle formally guarantees the existence of low-complexity solutions (those embedded from smaller, simpler models) within the vast solution space of overparameterized networks. Prior work has shown that gradient flow converges in direction to a KKT point of the max-margin problem (Lyu & Li, 2019), but a significant gap remained: large networks can possess many KKT points with varying complexities. Our work bridges this gap by proving that simple, generalizable solutions are structurally preserved as network width increases.
• This has immediate practical implications. The co-existence of both low-complexity (embedded) and high-complexity KKT points suggests that initialization and hyperparameter tuning play a critical role in selecting a solution. Our framework provides a theoretical basis for understanding this selection process and for future work on designing training procedures that explicitly steer optimizers toward these desirable, low-complexity solutions to improve generalization.

Regarding Originality: While neuron splitting has been studied, our work's originality lies not in the operation itself, but in the development of a formal KKT Point Embedding Principle that uses such operations to create a rigorous link between the solutions and training dynamics of networks of different sizes. We are the first to establish this comprehensive static and dynamic correspondence under a unified theoretical framework.

Regarding the proof techniques, we agree that they are direct applications of established tools like the Clarke subdifferential. We see this as a strength, lending clarity and rigor to our results. The novelty lies not in the techniques themselves, but in their application to establish a new and significant principle that was previously unformalized.

Revisions to Enhance Contribution and Clarity

We appreciate your perspective on the paper's limitations of our core results. To better demonstrate the significance of our framework and the depth of its contributions beyond the initial intuition, we have made the following substantial revisions

1. Broadened Scope with Extension to CNNs: We acknowledge that focusing solely on fully-connected homogeneous networks could be perceived as a limitation. To demonstrate the generality and power of our approach, we have introduced a new main result extending our principle to Convolutional Neural Networks (CNNs). This required designing a novel 'channel-splitting' transformation, meticulously constructed to respect the architectural hallmarks of CNNs, such as weight sharing and locality. The transformation isotropically splits a single feature channel into $m_{split}$ new channels via coefficients $\{c_i\}$ that satisfy $\sum_{i=1}^{m_{split}} c_i^2 = 1$. The specific construction is as follows:

• At Layer k: The filter $F_j^{(k)}$ that produces channel $j$ is replaced by $m_{split}$ new filters, where the i-th new filter is defined as $c_i F_j^{(k)}$.
• At Layer k+1: In turn, every filter in the next layer that takes channel $j$ as input has its j-th input slice replaced by $m_{split}$ new slices, where the i-th new slice is scaled by the corresponding coefficient $c_i$.

Successfully proving that this transformation satisfies our core theory (Theorem 4.5) validates our framework as a flexible blueprint applicable to a broad class of homogeneous networks, with future work poised to explore architectures like Transformers.

2. Improved Narrative and Contribution Density: To address the impression that the paper was overly detailed or like an "exercise," we have significantly restructured the manuscript for a more focused and impactful narrative. This includes:

• Consolidating Theorems: We have consolidated the application theorems. The two-layer network case, being a special case of the deep network one, has been moved to the appendix, allowing the main text to focus on the more general and powerful deep network result.
• Streamlining Presentation: We have deferred highly technical definitions (formerly Def. 3.1, 3.2, and Asm. 3.3) to the appendix to improve readability and focus on the core contributions.

Response to Minor Issue

• Comment on line 197: We sincerely apologize for this oversight. This was a remnant from an internal draft and has been removed in the revised version.

Once again, we thank you for your time and valuable feedback. We hope these clarifications and the significant revisions to the manuscript, including the new CNN result, will help you re-evaluate the significance and originality of our contribution more favorably.

References

• Lyu, K., & Li, J. (2019). Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv:1906.05890.