Reply to the reviewer G246

Thanks for your valuable comments and suggestions. We are pleased with your support for our paper.

Response to Question 1

Thanks for figuring out this typo. We indeed want to express "better optimization property" with the words "optimization capacity". A better optimization property of a module means that, adding the module to a network may improve conditioning, numerical stability and training efficiency in the optimization process. The optimization property of normalization is the main reason why normalization can be widely applied in various deep neural networks, as we described in introduction of this paper.

Here we take PLN-2 as an example. When LN acts on $R^2$, LN constrains the mean and variance both and it outputs only two values. This constrain is compact for only two neurons, ensuring that the output is nearly constant---assuming the input vector is $(x_1,x_2)$ and $x_1>x_2$, we can identify that the output must be a constant vector $(1,-1)$ as long as $x_1>x_2$. Consequently, the derivative of the output with respect to $x_1$ is zero, which can lead to stagnation in parameter updates during optimization. This observation explains why PLN-2 has " worse optimization property", despite our theoretical proof of its strong approximation capabilities. Therefore, we believe that there is a trade-off between optimization and optimization, as shown in Section 4.

Response to Question 2

In Section 4, we aim to provide a comprehensive understanding of the theoretical and practical aspects of activation functions. Specifically:

1. Theoretical Comparison: In Section 4.1, we analyze the approximation bounds of different activation functions, including PLN and ReLU. Our theory aims to provide quantitative results of different activation functions, which can be regarded as a theoretical reference when we explore further in experiment.

2. Practical Insights: In practical scenarios, the parameters of the network is obtained by training, resulting the network may perform much worse than its theoretical case. Figure 5(c) is a supportive example for this. Therefore, it is necessary to consider the optimization property of activation functions.

Our ultimate goal is to demonstrate that a network with PLN is not only a universal approximator but also inherits the optimization benefits of normalization. This dual property of PLN—combining strong approximation capabilities with improved optimization dynamics—makes it a promising candidate for simplifying deep neural network architectures and advancing their theoretical understanding.

We also add approximation experiments with 512 random inputs and labels on $R^8\times R$ to explore the high-dimensional case, following the same experimental settings in section 4.1. Here we give the MSE loss using a network with depth 1 and width 256.

We find PLN-4 and PLS-4 performs much better than the other activation functions, beyond the results of $R \to R$ approximation experiments in this paper.

Additionally, we have included experiments on high-dimensional data in Section 5. While these experiments are not focused on approximation tasks, they further highlight the practical advantages of PLN in deep learning contexts.

Response to Question 3

In Section 4.1, our theoretical analysis provides quantitative insights into the approximation capabilities of different activation functions. However, as demonstrated in Section 4.2, the practical performance of a network is also heavily influenced by its optimization properties. As you can see, our experiment is conducted on a simple network. Although we applied various training techniques (line 252-262) to attempt to find the better parameters, it is still a hard task---referring to the bad performance in Figure 5(c) . We find the optimization problem exists, even in such a tiny network (depth 1, width 16). Therefore, We further explore the relationship between optimization and approximation in deep neural networks, namely the experiments in section 5.

The propositions in Section 4.1 give us a good reference of the approximation capacity of a network. Only based on Section 4.1, we can contribute some bad performances of a network to the optimization problem rather than its approximation capacity confidently---based on that we know the theoretical performance of this network is outstanding, but we have not exploit it in a certain training process.

About Essential References

Upon carefully reviewing, we find the two references provide are supportive work for this paper. These will be added to citations in revision. We sincerely appreciate the reviewer for mentioning them.

Much thanks for your support again.

Reply to the reviewer 1Q2E

Thanks for your valuable comments and suggestions.

The main concern raised by the reviewers is the correctness of our proof. After re-examining our proof, we are confident that there are no errors in our reasoning. Here, we attempt to clarify why the reviewer might have misunderstood our proof.

Clarification on the Misunderstanding of the Proof

The reviewer pointed out that the operation $\sigma$ does not act element-wise. We trace back to lines 132–155 and clarify the following:

In line 150, $\sigma(\boldsymbol{w^\top_j x} + b_j)$ acts element-wise. Here, both $\boldsymbol{w}$ and $\boldsymbol{x}$ are vectors, and $b_j$ is a scalar. Therefore, $\boldsymbol{w^\top_j x} + b_j$ is a real number rather than a vector. Consequently, $\sigma(\boldsymbol{w^\top_j x} + b_j)$ is applied element-wise. As we mentioned in line 153, $\sigma(x) = (x / |x| + 1) / 2$ is a function on $R$. Besides, the form in our proof (line 150) aligns with Cybenko's work (line 65, Eqn. 1), ensuring the correctness of our proof.

If this is not the source of confusion, another possible reason is the discrepancy between the input and output dimensions of the Layer Normalization (LN) operation. In our proof, we construct $G(x) = \sum\limits_{j=1}^{N+1} \boldsymbol{\alpha}_j^\top LN(\boldsymbol{W_j x} + b_j)$, where $\boldsymbol{W}_j$ and $\boldsymbol{b}_j$ belong to the first linear layer, and $\boldsymbol{\alpha}_j$ belongs to the second linear layer. We set $d = 2$ in this proof, as mentioned in the paper.

The input to the $N+1$ LNs has $2(N+1)$ neurons, and the output of the LNs also has $2(N+1)$ neurons. However, the final output of Eqn.5 is the linear combination of $N$ neurons. This is because:

1. We merge the $(N+1)$-th term into the previous $N$ terms.
2. We set $\boldsymbol{\alpha}_j = [\hat{\alpha}_j, 0]^\top$, as shown in line 136. While the previous $N$ terms output $2N$ neurons, only $N$ of them are allowed to pass—because the other $N$ neurons are multiplied by zero in $\boldsymbol{\alpha}_j$. This is why the reviewers might have misunderstood our proof.

If the reviewer still has trouble understanding our proof. Please feel free to point it out, and we would like to clarify it.

Clarification on the Experimental Design

The experiments are not solely designed to support the theory, they also lead to a discussion on approximation and optimization. In practical scenarios, the parameters of the network is obtained by training, resulting the network may perform much worse than its theoretical case. Figure 5(c) is a supportive example for this. Therefore, it is necessary to consider the optimization property of activation functions. In this paper, we find that PLN can be seen as a combination of normalization and activation—possessing both good approximation properties (as we propose) and optimization properties (inherited from normalization). As we concluded in Section 5, deep neural networks with only linear modules and PLN can perform well. These experiments identified both the approximation and optimization properties of PLN.

Reply to the reviewer Guee

Thanks for your valuable comments and suggestions.

Response to Weakness 1

To begin with, we list our contributions below to clarify the novelty of this paper.

1. We are the first to consider LN (and LS) as activation functions and provide the mathematical proof of its universal approximation property by constructing proper parameters in the linear layers.
2. We are the first to propose the concept of PLN and PLS. Although the similar concepts like GN or LN-G have been proposed in the previous work, we discussed their differences with our PLN in our paper.
3. This paper also discussed the universal approximation property of normalization for the first time.

Besides, we point out that the proof is not direct as the reviewer commented. As we can see in the proof of Lemma 1, LN is not equivalent to sigmoidal functions in the network. We construct proper weights and biases in the linear layer and then obtain a similar result to the linear combination of sigmoidal functions.

We also construct piece wise step functions to prove the universal approximation capacity of PLN, as shown in Appendix A.3. We initially try to show our theory of PLN in this way, but this method is more complex than the current version. Considering the readability of this paper, we decided to give the proof based on Cybenko's work.

As for the reviewer's concern about Corollary 2, we provided details in the supplementary material, as we mentioned in line 197. The relationship between LS and PLS is similar to LN and PLN, as shown in Figure 1. When we put LS parally, we then obtain PLS. Compared with PLN, norm size 1 is suitable for PLS, while PLN requires the norm size of at least 2. Besides, note that LS is RMSNorm, which is also widely used in LLMs (e.g., LLama, Qwen2).

Response to Weakness 2

Due to the limited words for the reply, could you please refer to the Response to Question 2 in the Reply to the reviewer G246, for understanding Section 4 better?

We admit it is hard to distinguish the approximation and optimization properties of different activations. Although we applied various training techniques (line 252-262) to attempt to find the better parameters, it is still a hard task---referring to the bad performance in Figure 5(c) . We find the optimization problem exists, even in such a tiny network (depth 1, width 16). Therefore, We further explore the relationship between optimization and approximation in deep neural networks, namely the experiments in section 5.

We will adjust our descriptions and expressions in the revised version for better readability.

Response to Weakness 3

This paper does not aim to show that PLN can outperform other normalizations either activation functions. We aim to show that PLN can be seen as a combination of normalization and activation---it has both good approximation property (as we propose) and optimization property (inherit from normalization). Furthermore, it may help simplify the structure and of DNNs and then provide a convenient platform to study DNNs.

Response to Question 1

There are also results for high dimensional input in the form with the sign $O(\cdot)$, as shown in the references mentioned by the reviewer G246. But it is hard to give the precise bounds like that in Section 4.1. Therefore, we add experiments to explore the high-dimensional case. Could you please refer to the Response to Question 2 in the Reply to the reviewer G246 for the detailed results?

Response to Question 2

We further discussed PLN-2 in the Response to Question 1 in the Reply to the reviewer G246, could you please turn there for our response? Here we can see that better approximation capacity may suffer from optimization problem. As for the practical scenarios, following our experiments in section 4 and 5, we recommend that $d=4$ for shallow networks, and $d=8$ or lager for deep networks.

Response to Question 3 and 4

In figure 10, the "Identity" term means that we use PLN-8 as normalization and Identity as activation. The "PLN-8" term does not introduce additional nonlinearity based on the "Identity" term---it essentially add PLN-8 (as activation) after another PLN-8 (as normalization). Here we provide the results on ViT for classification with PLN-8 as Normalization.

From the classification task, we find that "PLN+ReLU" is much better than "PLN+Identity", it indicates that the architecture will affect the conclusion. On the other hand, in the time-series tasks (Figure 11), we find the performance of "Identity-Identity" is not so bad, indicating that the task will also affect the conclusion.