DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS

We are very grateful to the reviewer for their careful reading of our paper, thoughtful feedback, and constuctive suggestions. Below we respond to the concerns and/or questions raised.

Comment:

The main weakness I find in the paper is that while the motivation for the paper comes from a practical perspective, namely building neural networks with sparse activations that can be implemented more efficiently in practice, it seems that the applicability of the results is not clear. To my understanding, the results only apply for very deep neural networks (the experiments use 100-layer networks). The authors should clarify whether or not their results apply to networks of reasonable depth. Specifically, it would be good to show some experiment for networks of reasonable depth and show how the activation choice affects the behavior. It seems that in this setting depth is only hurting performance, so while it is in theory interesting to analyze how such deep networks should be trained, it seems that the applicability of this method is limited.

Response:

While we agree that our paper provides primarily a theoretical contribution -- insofar as the experiments presented are on relatively simple tasks, and were primarily designed to verify the theory -- we believe that the contributions are important and of interest to the community nonetheless. The key goal of EoC initialisations is to preserve signal propagation and avoid vanishing and exploding gradients, and the best proof of the ability to do this is to show that we can train networks with very many layers. This is what motivated our choice of experiments. We should also note that our work is not unique in this regard, and similar experimental setups are relatively common in experiments which develop EoC theory.

Having said that, we think that your comment is fair and important, and that it is worthwhile to test how the comparisons play out in practice when the networks are not quite so deep. To explore this, we have repeated the DNN experiments from the paper, but with depth=30 instead of depth=100. The results are shown in the table below, and are now included in Table 4 and 5, in Appendix H.

We are very grateful to the reviewer for their careful reading of our paper, thoughtful feedback, and constuctive suggestions. Below we respond to the concerns and/or questions raised.

Comment:

"The stddev in table 2 are a bit odd to me, either a higher numerical precision is needed, or it needs to be explained why the are many cases of zero std."

Response:

Indeed most of the 0.00 standard deviations were simply because the results in the table were rounded to 2 decimal places. We have amended Table 2 to include higher precision to make the results clearer. 0.00 standard deviation was only accurate to higher precision as well in the cases when all of the seeds completely fail to train at all. Thank you for this suggestion.

Comment:

"Results on ST are only shown in the appendix. They should be shown and discussed in the main paper, given that they are an essential part in the rest of the manuscript."

Response:

Thank you for this suggestion. Putting $ReLU_\tau$, $ST_\tau$, and $CST_{\tau, m}$ experimental results in the Appendix was originally done purely due to space constraints, but we acknowledge and agree that it is important for all key experimental results to appear in the main paper. In order to make space, we simplified and combined Figures 2 and 3, and so we have now included all the important experimental results in Table 2.

Comment:

"The higher sparsity regime in table 2 (s=0.85) needs to be explored/explained more, there are interesting things happening."

Response:

We agree that the high sparsity regime is very interesting! But we suggest that our analysis of the impact of $s$ and $m$ on the shape of the corresponding variance maps and the related failure modes accounts for the results quite nicely. In particular, we see that initially increasing $m$ is necessary to achieve good accuracy at high sparsity, as expected. However, as predicted by the theory, increasing $m$ too much results in $V'(q^*)$ and $V''(q^*)$ together being too large, causing $q^*$ to fail to be stable in practice. This relates to your other question about Figure 4. The phenomenon described here is exactly what is illustrated in Figure 4, which shows the Variance maps of $CReLU_{\tau,m}$ and $CST_{\tau, m}$ in the high sparsity, large $m$ regime. When the variance map curves of to trace or even cross the $q=V(q)$ line, the result is that $q^l$ does not converge to $q^*$, and instead remains larger than $q^*$ layer on layer. Figure 4 shows that the corresponding $\chi_1$ value in this case is larger than one, which causes exploding gradients and training failure. We have tweaked the wording at the bottom of page 8 in Section 4 to make it clearer which point from the main text Figure 4 is illustrating.

If you still think something is not clear, could you perhaps explain what analysis you think is missing?

Comment:

No source Code provided, implementation details on experiments only in the appendix.

Response:

The implementation details were left to the appendix due to space constraints. We ask that this please not be considered a weakness of our paper. We are working to make the source code available and will alert you if this is possible before the discussion period concludes.

We are very grateful to the reviewer for their careful reading of our paper, thoughtful and positive feedback, and constructive suggestions. Below we respond to the concerns and/or questions raised.

Comment:

"Is this the first time that the variance map equations are being derived for these non-linearities?"

Response:

Yes, to the best of our knowledge this is the first time they have been derived.

Comment:

"Plots of Figure 2 are missing axis labels and the plot legends are not readable on printed paper."

Response:

Thank you for highlighting this. We have added axis labels and enlarged the legend font.

Comment:

"There could have been a study of the computational efficiency since that is the main motivation of the work."

Response:

Unfortunately it was not possible for us to perform an empirical study of the computational efficiency gains due to sparser activations, because the sparse operations necessary to leverage sparse activations are not yet well supported on the accelerator hardware on which we are running our experiments. However, better support for efficient sparse operations is a high priority and ongoing research area for both deep learning hardware and software developers, given the potential performance gains. The number of flops in each matrix-vector multiplication $AB$ for $A \in \mathbb{R}^{m \times n}$, $B \in \mathbb{R}^{n \times d}$ in the forward pass could in theory shrink from $\mathcal{O}(mnd)$ to $\mathcal{O}(msd)$.

We are very grateful to the reviewer for their careful reading of our paper, thoughtful feedback, and constructive suggestions. Below we respond to the concerns and/or questions raised.

Comment:

"I am not entirely sure why sparse ReLU fails to train. Table 2 only shows results for magnitude clipping for CReLU and the usual ReLU with $\tau=0$." and "What is the criterion on $V_\phi$ for successful training? For example, let's see Figure 2. Which of the shapes are good and expected to train well vs which are the ones that are expected to fail? I see that for $\tau=1$ the curves have higher curvature. In particular, the blue curve intersects the line x = y at one point where the derivative is non-zero but the curvature is positive. Is this expected to fail because the derivative is non-zero? I found the explanations in the text somehow repetitive and hard to parse. Can the authors explain the criterion on $V_\phi$ in words just from Figure 2?"

Response:

According to the theory, two things are required for stable training:

1. we require that $\chi_1 = 1$, which guards against vanishing and exploding gradients as well as instability to input perturbations at initialisation.

2. we require that the variance map $V(q)$ is sufficiently stable around $q^*$ such that $q^l$ does in practice converge to and remain at $q^*$. This is a necessary requirement for the EOC initialisation in practice since $q^*$ is used in the calculation of the values of $\sigma_w$ and $\sigma_b$ in order to ensure $\chi_1 = 1$. If in practice $q^l$ does not stably converge to $q^*$, and instead grows larger, then we no longer have $\chi_1=1$ and experience the associated training failure modes of exploding or vanishing gradients.

The problem we have shown with $ReLU_\tau$ and $ST_\tau$ is precisely that it is impossible for them to satisfy both criteria $\chi_1=1$ and $q^l$ converging stably to $q^*$.

In light of the above, we agree that the original Figures 2 and 3 could be misleading, since they included variance maps for $ReLU_\tau$ and $ST_\tau$ which did not correspond to EOC initialisation. To address your question and to improve clarity on this point we have combined Figures 2 and 3 into a single figure, now showing only the variance maps for both activation functions corresponding to $\chi_1=1$. We have also edited Section 2 of the paper to make this point more clearly (in particular see paragraphs 2, 3, 4, and 5 of Section 2 in the updated version). We think it reads better now, so thank you for this helpful feedback.

Comment:

"Why is there only one row for ReLU?"

Response:

Standard ReLU was simply included in Table 2 as a baseline, a commonly used activation function against which we can compare the accuracy and activation sparsity in our experiments.