Neural Characteristic Activation Value Analysis for Improved ReLU Network Feature Learning

Thanks for your comments on our paper! We will respond to your comments point by point below:

Can the analysis apply to the existing advanced batch normalization improvements like IEBN [1], SwitchNorm [2], layer norm [3]? These missing works should be considered and added to the related works or analysis.

Thanks for pointing out these related works. Please note that LayerNorm is already discussed in the related work section (Section 2) under “Other normalization methods”. We have added [1,2] to the related work section in the updated manuscript.

Regarding the applicability of our analysis, the proposed characteristic activation value analysis is designed to analyze neural network parameterizations. We showed in Section 2 that BatchNorm can be viewed as a kind of weight normalization, so our analysis will produce the same result for BatchNorm as that for weight normalization. Furthermore, since BatchNorm, IEBN, SwitchNorm, LayerNorm, and almost all existing normalization techniques are different variants of the same normalization technique which operate in the weight-space parameterization, they will have the same instability property as standard parameterization and weight normalization, as shown in Equations 12 and 14. We have clarified this in the updated paper manuscript.

We hope that this has sufficiently addressed all your concerns. Please let us know if you have any further questions or comments.

Thanks for your comments on our paper! We will respond to your comments point by point below:

The main contributions of this paper are to propose one geometric parameterization for ReLU networks and input mean normalization. But the input mean normalization proposed in this paper is very similar to the mean-only batch normalization (Salimans & Kingma, 2016). What’s the advantage of the former against the latter?

Regarding our contributions, please note that geometric parameterization and input mean normalization are only part of the contributions in our paper, which are natural products from our main contribution - the characteristic activation value analysis for neural network parameterizations. We used the proposed novel analysis to identify the instability of common neural network parameterizations such as standard parameterization and weight normalization. Motivated by this, we proposed geometric parameterization which has theoretical stability guarantees as shown by our analysis in Theorem 3.3. We verified the theoretical result with three illustrative experiments in Section 3.5.

Regarding input mean normaliztion vs mean batch normalization, as discussed at the beginning of Section 4.2, the input mean normalization is a technique to address the covariate shift problem in multi-layer networks with geometric parameterization. We subtract the mean from the input to each layer because our proposed analysis assumes that input distribution is centered around the origin. The mean-only batch normalization (Salimans & Kingma, 2016) would not address this problem because it is applied to the pre-activation values (i.e., in this case the zero mean distribution will be destroyed after going through the ReLU activation, so the input distribution to the next layer will not be zero mean anymore). We have clarified this in the updated manuscript.

The experimental results are not convincing. The authors should compare the performance of the proposed algorithm on more models and datasets.

We have conducted extensive experiments with

• three representative models: MLP, VGG, and ResNet;
• several representative regression and classification benchmarks: ImageNet (ILSVRC 2012) classification dataset, ImageNet-32 classification dataset, 6 UCI regression datasets, 2D Banana classification dataset, and 1D Levy regression dataset.

We believe that these experiments have verified the usefulness of the proposed algorithm with different model architectures and on benchmarks from different domains.

Both the English language and equations in this paper need to be improved.

We’d be grateful if the reviewer could point out the texts or equations that are confusing, and we will try our best to clarify them and improve our presentation. That said, we noticed that Reviewer n38z actually appreciated our clear writing in an easily comprehensible manner.

We hope that this has sufficiently addressed all your concerns. Please let us know if you have any further questions or comments.

Thanks again for your effort during the reviewing process! We believe that we have addressed your stated concerns in our response and would like to ask if the reviewer thinks this is the case as well. If our response is affirmative, we would like to ask the reviewer if they are happy to increase their scores.

Thanks for your comments on our paper! We will respond to your comments point by point below:

The authors should talk more about why they prefer optimization in the angular space instead of the weight normalization space since they are equivalent as shown in Eq. 9

Section 3.4 talks about why optimization is preferred in the angular-space rather than the weight-space, which is the main theoretical contribution of this paper. In summary, Theorem 3.1 shows that the change of angular direction under small perturbation is bounded by the magnitude of perturbation, which is robust against noise during stochastic gradient optimization. In contrast, although Eq. 9 shows an equivalency between weight normalization and geometric parameterization, Eq. 14 shows that they have entirely different stability properties, i.e., weight normalization is as unstable as standard parameterization under small noise perturbation during stochastic gradient optimization. Furthermore, this further implies that any weight-space parameterization will suffer from this issue because they will all lead to Eq. 14. We have clarified this in the updated manuscript.

The theoretical proof only shows that the change in the angular direction under perturbation is bounded by the magnitude of perturbation. Using $\Delta\phi$ as evidence for generalization is not theoretically justified. I think loss function values should be added as supporting evidence too.

We did not use $\Delta\phi$ as evidence for generalization. Instead, we used it in our illustrative experiments in Section 3.5 to verify that our theoretical results about the stability of different parameterizations from Section 3.4 hold in practice. The evidence we used for generalization is the final test performance (RMSE and accuracy), i.e., Figure 2(d)-(g), Figure 3(j)-(m), Figure 4, and Tables 1, 2 and 3. The loss function values are consistent with these test performance, which we didn’t report in the paper due to page limits.

I found some of the experimental settings unsatisfactory. For example, the training settings of ResNet-50 are very different compared to standard settings. I expect to see a comparison of different methods with standard learning rate decay, i.e. first 30 epochs: 0.1, 30-60 epochs: 0.01, 60-90 epochs: 0.001. Under this setting, BN should achieve around 76.1% top-1 accuracy. Or, the authors should provide a convincing explanation why not use the standard training setting.

This is because different parameterizations have different convergence speeds as shown in our ablation study (Figure 4). The fixed learning rate decay scheme mentioned by the reviewer may be useful for BatchNorm, but it can be suboptimal and thus slow down the convergence of geometric parameterization (because it converges faster and will reach the plateau faster at each stage than BatchNorm). Therefore, we used an adaptive learning rate decay scheme, Reduce LR On Plateau, for geometric parameterization. Then, we had to use Reduce LR On Plateau for BatchNorm and weight normalization as well for a fair comparison.

In Eq.16, only input mean normalization (IMN) is used, why not further normalize the input features with its variance?

IMN is used to address the covariate shift problem in multi-layer networks with geometric parameterization, which ensure that the input distributions to the intermediate layers of multi-layer networks are centered around the origin, because this is an assumption of our proposed analysis. There is no need to further normalize the input features with its variance because our analysis does not assume that the input distribution has unit variance. That said, it would be harmless to normalize the variance in terms of the final test performance, but calculating the variance for the input to each layer in each forward pass would increase the computational cost and slow down training, and therefore we chose not to do that. We have clarified this in the updated manuscript.

We hope that this has sufficiently addressed all your concerns. Please let us know if you have any further questions or comments.

Thanks again for your effort during the reviewing process! We believe that we have addressed your stated concerns in our response and would like to ask if the reviewer thinks this is the case as well. If our response is affirmative, we would like to ask the reviewer if they are happy to champion our paper.