Enhancing Performance of Multilayer Perceptrons by Knot-Gathering Initialization

The major assumption and limitation of this manuscript is the [Weight-Act] pipeline, as described in Eq. 1. However, nowadays, it became standard to use normalization layers such as batch normalization or layer normalization with the [Weight-Norm-Act] pipeline. I think that the use of a normalization layer would yield different initialization methods in their theory, especially in Eqs. 1-6; in other words, I wonder that the proposed KGI with these equations would increase the number of knots as desired in practice.

For example, the normalization layers consist of a normalization step and an affine step that adopts gamma and beta parameters in the form of y=ReLU(z), z=gamma*zhat+beta, and zhat=normalize(Wx+b). For this scenario, I conjecture that initialization should be applied to gamma and beta, not the weight and bias. Indeed, the use of a normalization layer provides normal distribution, which can be exploited to sample xhat in place of Eq. 2.

In summary, there are plenty of practical and interesting theories that should be considered when adopting the normalization layer, which, however, is not discussed in the current manuscript. Not to mention that there are numerous architectures, such as skip connection or self-attention, which are not sufficiently targeted for the subject of KGI in this manuscript. I found that Section 5.1 provides some discussion, such as variants of weight configuration and future works with normalization layers; nevertheless, the current manuscript still sticks to the [Weight-ReLu] pipeline without normalization layer. I evaluate that the current form of KGI seems not compatible with modern architecture, which uses normalization layers. Indeed, the authors targeted MLP for CIFAR-10; MLP for muon spectroscopy; MLP for DeepONet; MLP for VAE; and MLP for GPT-2. All these examples rather seem to indicate the limitation and archictectural restriction of KGI onto MLP without normalization layers.

Minor comment: template of ICLR 2024, not ICLR 2025, was used. In the 2025 template, there should be numbering for all lines. Although this issue might be minor, the violation of format should not be allowed in principle.

Section 3.2 states, “our objective is to increase, rather than to maximize, knot density within the input domain.” The rationale for not maximizing knot density could be better explained, potentially with supporting experimental data.

Equation 6 introduces a hyperparameter to balance this method with other initialization strategies. It would be beneficial to explore how this hyperparameter impacts MLP performance, ideally through comparative experiments.

• Most of the experiments are shown using MLP; it is not clear if this will translate to CNNs/ViTs.

• With the current trend of foundational models, where the expressive power is increased by increasing the model parameters count, there is a possibility that KGI will not give performance gain. Further discussion on this would be beneficial.

• In section 2, it is not clear how the input ranges are determined for different activation functions. Why is ReLU bounded between 0 and 1?

• In section 3.3, the use of mutual information does not make much sense. Since the transformation is deterministic, the MI will always be equal to the entropy H(Y). Why is the entropy maximized at the root of z=0. Further explanation would be helpful.

• There is no discussion of how increasing the knots improves the convergence rate and why this will not lead to overfitting.

• KGI can only be used when training a model from scratch/initialization and cannot be used fine-tuning/transfer learning.

• Accuracy for datasets is quite low as MLP is used. It is not clear if KGI will improvement when using larger and complex sota models.

• Is KGI helpful in truly deep networks? I see that most networks used in the experiments are toyish. It has a maximum depth of 5, and the networks used in the real world data seems even shallower (disentangled uses 2 layered networks?). GPT is the only larger scale model explored, and here, the advantage seems very small. Can authors comment on if KGI still holds its advantage if the network is deep enough - let's say 15 layers or so. You can, for example, use a Resnet style architecture for CIFAR10 (replacing conv layers with nn.Linear to support your framework). I do note that authors have mentioned that for sufficienctly sized models the final loss values are comparable, and only the convergence is faster - I'm trying to see if this holds for truly deep networks (for eg. the difference in slowness for GPT-2 is negligible and within confidence interval).

• What are the insights from experiments in Section B? How do the number of knots change as we train? Can authorize summarize any patterns they are seeing across different network sizes and activation functions? Can authors also look at how the number of knots as we train? Are deeper networks able to increase their number of knots as training progresses? Does more knots generally mean better training and test error?

• Authors outline in Section 5.1 that KGI is compatible with architecture as CNNs and GNNs. It will considerably strengthen the paper if authors can show that performing KGI on SoTA architectures in ImageNet etc improves performance.

Overall, while it is an interesting observation and useful contribution, I am not sure if KGI really moves the needle on practical application. If my concerns, particularly on KGI being valuable for sufficiently complex networks, is addressed, I am happy to increase my rating.

I view the above contributions as relevant but consider the paper as not yet ready for publication. Firstly, some aspects concerning the clarity of the paper, especially section 4 on “Experiments”, require some additional refinement to strengthen the contribution. I cover these aspects in W1, W3 and questions Q1 and Q2. Secondly, in my opinion, the quality of the paper can be improved with respect to the positioning of the method and the empirical evaluation, which I elaborate on below. In more detail, I see the following main weaknesses:

W1. Relationship between expressiveness / universal approximation / capacity measures and “local expressiveness”/ initialisation / normalisation: The paper positions itself as a work on the expressive power of neural networks, and the authors reference the claim that “[t]he number of knots in piecewise linear neural networks has been explored as an intuitive and interpretable measure of their expressive power.” (page 9) I view these claims as slightly overstated and a potentially detrimental shift in the contribution's focus. I see the contribution as follows. The main idea of KGI can be viewed as an initial weight perturbation that mostly influences the following optimisation steps during training. An MLP without KGI should be as expressive as one with KGI, but might not converge as quickly (see W3.2). As the authors mention in the related work section, I believe the domains closest to their proposal lie in normalisation techniques, work on adaptative activation functions, and other weight initialisation approaches. Therefore, “local expressiveness” should mostly be understood in this context and related to optimisation aspects. In this direction, considering “optimisation budgets”/training steps and KGI remains underexplored, in my opinion (see W3.2 on this below). However, KGI does not affect the expressiveness of MLPs or the capacity of the modelled hypotheses class in any way.

W2. Impact compared to related approaches is difficult to assess: While innovative, KGI is not be sufficiently distinguished from previous strategies. The statement that “[s]ince KGI introduces a mechanism distinct from previous studies and is compatible with most existing techniques […], we do not use a static baseline in our experiments.” (page 6) should be supported with clear empirical evidence and comparisons. KGI’s goal of enhancing “local expressiveness” is similar to other weight initialisation and general normalisation techniques. Running the same experiments with an additional comparison to related approaches, e.g. LSUV (Mishkin & Matas, 2016) or batch normalisation, for instance, and e.g. different initialisation strategies would allow for a more conclusive assessment of the relevance and benefits of KGI. This is currently missing.

W3. Empirical evidence and interpretation: Several aspects make the interpretation of the experimental results challenging.
W3.1. Train error / test error / overfitting: In section 4.1, it appears that the focus is on training MSE, while in section 4.2 in some cases test results are considered. It would be helpful to provide more clarity on the considered metrics and compare both training and test curves. In that context, the potential risk of overfitting due to increased knot density at initialisation (mentioned in the discussion) should be evaluated and discussed, too.
W3.2. Comparison of non-converged results: A major issue I currently see is that most of the losses have not converged, in particular for the “no KGI MLPs” (see Figures 4, 5, and 6). It appears that implicitly, a fixed number of optimisation steps or an “optimisation budget” is assumed and “enhanced local expressiveness” is mostly attributed to attaining lower losses at the same number of optimisation steps. However, consideration of the “optimisation budget” / training steps is currently missing. It could beneficial to justify the choice of the optimisation budget. Additionally, as the “no KGI MLPs” have not yet converged, the actual improvement in general error or accuracy is not assessed, but rather the error or accuracy after a particular number of optimisation steps. One could expect “no KGI MPLs” to attain similar test errors or accuracies at a later stage of training. This is why I currently see the main improvements of KGI only concerning convergence speed. Extending the experiments to carefully cover converged results might provide additional insights and more rigorously evaluate the impact of KGI on final performance.

W4. Issues with the definition of “knots”: The definition of “knots” is natural for ReLU activation functions but becomes increasingly fuzzy for other activation functions. Two issues related to section 3.3 on “Smooth activation functions” may illustrate this well.
W4.1. Incorrect mutual information: The displayed mutual information in Figure 2 (middle panel) is incorrect and likely stems from a frequent misconception when extending the notion of entropy to continuous random variables. In the derivation provided in Appendix A, it is stated that "[s]ince $Y = tanh(Z)$ is a deterministic transformation of $Z$, the conditional entropy $H_\mu (Y \vert Z)$ vanishes”. This statement is true when $Y$ and $Z$ are discrete random variables but false for continuous random variables. In the latter case, we need to consider the differential conditional entropy, which is $h(Y \vert Z)=-\infty$, leading to $I_\mu(Z;Y) = \infty$.
W4.2. Mixed results for tanh: tanh is highlighted as a particular example of a smooth activation function in section 3.3. However, the curve and surface fitting results illustrated in Figures 3e, 3f, 5d, and 6d show that KGI does not lead to consistent and substantially different results.

Minor remark: Page 1, 4th paragraph. The authos might want to correct the sentence: “We are generalizing the concept of knots to smooth activation functions (such as GELU and Tanh) is straightforward but rigorous.”

Based on the different aspects raised in W1 to W4, I view the submission as not yet ready for publication and suggest a major revision of the paper. However, I invite the authors to address my objections and clarify potential misunderstandings.