Fully Identical Initialization

My biggest concern is the Cifar-10 performance compared to the simple Kaiming initialization.

• Table 2 shows that using SGD optimizer (which gives the best performance all across), Kaiming init obtains almost the same performance (93.36 v/s 93.41 and 94.06 v/z 94.04) as IDInit, while being only slightly slower.
• This brings into question the practical utility of the proposed initialization.

Thanks for the reviewer’s comment. While both IDInit and Kaiming initialization methods perform similarly in certain settings, our research indicates that IDInit offers superior stability and adaptability across a wider range of scenarios. To verify this claim, we have conducted additional experiments with varied hyperparameters, particularly focusing on weight decay and learning rate, using ResNet-56.

RTable 1. Hyperparameters for IDInit on ResNet-56.

RTable 2. Hyperparameters for Kaiming on ResNet-56.

As shown in RTable 1 and RTable 2, these experiments reveal that IDInit consistently surpasses Kaiming initialization in almost every setting tested. Most notably, in scenarios with smaller learning rates (e.g., 1e-2 and 1e-3), we observe a significant drop in the performance of models initialized with Kaiming. In contrast, IDInit maintains high accuracy levels under these conditions. This stability of IDInit is attributable to its ability to achieve isometry dynamics in the residual stem, a crucial factor for maintaining performance stability.

Further, we have extended our experimentation to include ResNet-32 on the CIFAR-10 dataset, as detailed in Section F.1 of the uploaded revision PDF. These additional results consistently demonstrate that IDInit not only matches but often exceeds the performance of Kaiming initialization, especially under varying and challenging hyperparameter settings.

In summary, while Kaiming initialization performs comparably in certain standard scenarios, IDInit exhibits greater performance consistency and stability across a broader spectrum of conditions. This enhanced stability and adaptability make IDInit a more practically valuable choice for diverse neural network applications. We will ensure that these findings are clearly presented in our revised manuscript to underscore the practical utility and advantages of IDInit over traditional initialization methods like Kaiming.

(clarity) The idea of dynamical isometry has been introduced in (Saxe et al., 2014).

We acknowledge the seminal work presented in [1] regarding the concept of dynamical isometry in the context of non-residual networks. The approach outlined in [1] focuses on achieving dynamical isometry by orthogonalizing network weights to ensure that the mean squared singular value of the Jacobian matrix remains close to 1, a significant advancement for non-residual network initialization.

However, our work with IDInit diverges from [1] in its application to residual networks, which are among the most powerful and prevalent structures in current deep learning architectures. In technique, IDInit emphasizes the maintenance of identity transit through both the main and residual stems, and empirical results show the effectiveness.

In light of this, while IDInit and the method in [1] both target dynamical isometry, IDInit's contribution lies in its novel application to residual networks. This extension not only broadens the understanding of dynamical isometry but also showcases its practical utility in some of the most advanced neural network architectures today. We will include a detailed discussion of these distinctions and contributions in the revised manuscript.

Please, include the references listed in the weaknesses section.

Thank you for your suggestion. $1$ pioneered the integration of dynamic isometry in nonlinear networks, enhancing training efficiency. Specifically targeting Convolutional Neural Networks (CNNs), Reference $2$ formulated a mean field theory for signal propagation adhering to dynamic isometry principles, enabling the training of networks with up to 10,000 layers. Different from these approaches, the proposed IDInit maintains identity in both the main and residual stems to preserve dynamic isometry to achieve good performance and fast convergence. Furthermore, we adapt the transformation on the identity matrix to accommodate various scenarios, such as nonsquare matrices and higher-order tensors. We will include these references in the revision.

(clarity) I would argue that the patch-maintain convolution is not very well motivated. I believe the problem is that I do not understand how this relates to Ghostnets (Han et al., 2020).

What is the link between Ghostnet and the patch-maintain scheme?

Thanks for the useful comment. Ghostnet $3$ highlights the critical role of channel diversity in enhancing network performance. This insight serves as a foundational motivation for our patch-maintain scheme. In contrast to the channel-maintain approach, which tends to replicate channel features and thus might limit diversity, our patch-maintain scheme introduces a novel strategy to increase channel differentiation. By shifting features, the patch-maintain scheme aims to add variability and uniqueness across channels. This method is designed to capitalize on the principle articulated in Ghostnet – that channel diversity is beneficial for performance. This linkage between the need for channel diversity (as emphasized by Ghostnet) and our patch-maintain scheme is a key aspect of our work. We will include this discussion in the revision.

(clarity) It also took me some time to realise that the channel-maintain convolution (as it is called in the appendix) which is described in the first paragraph of Section 3.3.1 is something different from the proposed patch-maintain setting. Note that this channel-maintain setting has also been used in (Xiao et al., 2018).

(clarity) The ablation experiment in Section 4.4 is claimed to explain why channel-maintain is not as good as patch-maintain. However, the explanation in section 4.4 seems to indicate that this is just an ablation of the different components of the proposed solution.

Do you have a direct comparison between patch-maintain and channel-maintain schemes for IDInit?

Sorry for the confusion. In this work, we introduce the patch-maintain scheme (i.e., IDIC$_{\tau}$) as our novel method for transforming a matrix into a convolutional format.

On the other hand, the channel-maintain method, referenced from prior studies [2][4], serves as a comparison baseline in our experiments, denoted as 'w/o IDIC$_{\tau}$'. In Section 4.4, we present this comparative analysis, where we demonstrate that the patch-maintain scheme significantly enhances the performance of IDInit. To address the clarity issues raised, we will revise our manuscript for more explicit elaboration.

Thank you for your observation regarding Table 1. This table was designed to address the convergence issues highlighted in reference [5], where the authors analyzed a network comprising entirely linear layers without biases and optimized it using Gradient Descent (GD). In line with their setting, we constructed a 3-layer fully linear network without biases as our experimental model. This choice was not an attempt to contrive a specific outcome but rather to adhere closely to the conditions set out in [5]. Therefore, this is a fair assessment.

Our primary objective is to demonstrate that the convergence problem identified in [5] is readily solvable. To this end, any evidence that contributes to solving this convergence problem aligns with our goal. Consequently, our findings that SGD with momentum aids in resolving convergence issues, and that momentum can further expedite this process, are pertinent. Similarly, your observation that incorporating bias parameters might facilitate convergence is equally valuable and does not contradict the purpose of our experiment.

In summary, we appreciate your contribution to this discussion, as it helps uncover various potential solutions to the convergence problem. These findings, including the use of SGD with momentum, smaller batch sizes, and the addition of biases, do not contradict the central premise of Table 1; rather, they provide supporting evidence for the ease of resolving convergence issues. We find these insights intriguing and plan to include them in our next revision to offer a more comprehensive understanding of the IDInit approach.

[5] Bartlett, Peter, et al. "Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks." International conference on machine learning. PMLR, 2018.

Theoretical analysis seems incorrect and its proof lacks details. In Theorem 3.1, the author claims IDI breaks rank constraint such that its residual has rank more than D_0. However, in the proof in the appendix, the author only shows the full matrix has rank more than D_0, which is not the same as the residual. Please provide more details in the proof to justify your claim.

We appreciate the reviewer's feedback regarding the theoretical analysis and acknowledge the need for further elaboration. However, our claim is indeed correct, and we would like to provide additional details for clarity.

Firstly, it is important to understand that the rank of $\theta^{(0)}$ exceeding $D_0$​ directly leads the distinct values of the feature $\{x^{(k)}\}_{k=1}^{L-1}$ to be larger than $D_0$​. Then, since the gradient of $\theta^{(k)}$ is calculated as $\frac{\partial \mathcal{L}}{\partial \theta^{(k)}}=\frac{\partial \mathcal{L}}{\partial x^{(k)}}\circ x^{(k-1)}$ for $k\in \{1, 2, \dots, L-2\}$, the rank of $\frac{\partial \mathcal{L}}{\partial \theta^{(k)}}$ will be larger than $D_0$. As a result, with a learning rate $\mu$, the subsequent $\theta^{(k)}$ updated as $\theta^{(k)} = \theta^{(k)} - \mu \frac{\partial \mathcal{L}}{\partial \theta^{(k)}}$, will also have a rank greater than $D_0$​. This effectively resolves the rank constraint issue. Furthermore, this is also supported by the empirical evidence supporting as presented in Figure 4, where IDInit's success in surpassing the rank of $D_0$​ is demonstrated.

We believe that this additional explanation clarifies the theoretical foundation of our approach and addresses the concerns raised. We will revise the proof in the revised manuscript to ensure a comprehensive understanding of our methods and findings.

The authors claim that the rank constraint can be broken by IDI even when non-linearity like ReLU is not applied. It seems contradict approximation theory which emphasizes the importance of non-linearity to ensure expressivity. It would be great for authors to provide more insights on this point.

We appreciate the reviewer's point regarding the possible contradiction between IDInit and the approximation theory. To clarify, our claim is that IDInit has the capacity to break the rank constraint even without traditional non-linearities such as ReLU. This does not negate the significance of non-linearities in enhancing expressivity, as emphasized by approximation theory. In more detail, our work demonstrates that IDInit's mechanism allows for an increase in the rank of learned representations through padding the identity matrix repeatedly., independent of non-linear transformations. When non-linearities like ReLU are introduced, as per approximation theory, they indeed augment the expressivity of the IDInit. In summary, IDInit, in conjunction with non-linear activations, provides a compounded benefit to the model's expressivity. Therefore, IDInit's impact is additive to the benefits conferred by non-linearities, rather than contradictory.

Authors mention that dead neurons problem happens when batch normalization set to 0 or downsampling operation cause 0 filled features. However, these cases are not common in practice and it's better to motivate more on why IDIZ is important.

Thank you for the comment on IDIZ in addressing the issue of dead neurons, particularly in relation to batch normalization (BN) and downsampling operations. However, we contend that the scenarios we described are common in practice.

• Batch Normalization: The practice of setting the gamma parameter of the last BN layer in a ResNet block to zero is not only recommended for enhanced performance as indicated in references $1$

$2$

$3$ , but also a default setting in widely-used packages like timm $4$ . This approach, while improving performance, can inadvertently lead to dead neurons, a problem IDIZ aims to mitigate.

• Downsampling: In common downsampling practices, there are two prevalent approaches: (i): Using pooling (e.g., avgpool) to reduce feature resolution, coupled with zero-padding to expand channel size (as shown in Line 40 of $5$ and option 'A' in Line 66 of $6$ ). (ii): Employing convolution to simultaneously reduce resolution and expand channel size (evident in Line 241 of $7$ and option 'B' in Line 72 of $6$ ). In particular, for the widely-used datasets like CIFAR-10, option (i) is frequently preferred.

IDIZ is designed to generalize the utilization of IDInit by solving dead neurons of above conditions which is also common settings. So that IDIZ can be used more widely for addressing the disharmonious nature of identity-control methods, which avoids potential risks in compatibility with other techniques. We believe this is important characteristic to an initialization algorithm.

We will revise our manuscript to better discuss these points, ensuring that the relevance and utility of IDIZ are clearly and concisely presented.

In other words, $\frac{\partial \mathcal{L}}{\partial \theta^{(1)}}$ consists of $N$ rank-1 matrices, the summation of which can not exceed $D_0$. Thus, I don't agree the authors' claim that "the rank of $\frac{\partial \mathcal{L}}{\partial \theta^{(k)}}$ will be larger than $D_0$".

The authors mention the empirical showcase in Figure 4. Is the 3-layer network in Figure 4 a linear network or a non-linear network? It should be a linear 3-layer network if the authors want to demonstrate Theorem 3.1 empirically.

Thank you for your insightful observations. We acknowledge your point regarding the limitation of $\frac{\partial \mathcal{L}}{\partial \theta^{(k)}}$ to $D_0$. ​However, we maintain that Theorem 3.1 remains valid. In a scenario using full-batch gradient descent for a single round, $\theta^{(k)}$ indeed retains a rank of at most $D_0$. Nevertheless, when employing Stochastic Gradient Descent (SGD), the weight update process involves iteratively adding gradients of rank $D_0$. As noted in reference [9], the addition of lower-rank matrices can increase the overall rank. This implies that the rank of $\theta^{(k)}$ can be increased, provided the gradients are sufficiently independent, achievable through the use of ample training samples.

To clarify this concept, we utilized a 3-layer network for analysis. Following the notations from Section A.3 in the appendix and assuming $D_h=2D_L=2D_0$​, we demonstrate the process using SGD. After the initial training step with a sample $x_1^{(0)}$​, the gradient is calculated by

\mu\Pi & \mu\Pi \\\\ \mathbf{0} & \mathbf{0} \end{pmatrix*},$$ where $\Pi = \frac{\partial L}{\partial x_1^{(3)}} \circ x_1^{(0)} \in \mathbb{R}^{D_L\times D_0}$. Continuing with a second sample $x_2^{(0)}$​, there is $$\frac{\partial \mathcal{L}}{\partial \theta_2^{(1)}}=\begin{pmatrix*} (I-\mu\Pi)M(I-\mu\Pi)x_2^{(0)} & (I-\mu\Pi)Mx_2^{(0)} \\\\ -\mu\Pi M(I-\mu\Pi)x_2^{(0)} & -\mu\Pi Mx_2^{(0)} \end{pmatrix*},$$ where $M=\frac{\partial \mathcal{L}}{\partial x_2^{(3)}}$. Thus, the residual step can be calculated as $$\hat{\theta}^{(1)}=\theta^{(1)}-I=\begin{pmatrix*} -\mu\Pi - (I-\mu\Pi)M(I-\mu\Pi)x_2^{(0)} & -\mu\Pi - (I-\mu\Pi)Mx_2^{(0)} \\\\ \mu\Pi M(I-\mu\Pi)x_2^{(0)} & \mu\Pi Mx_2^{(0)} \end{pmatrix*}.$$ Without loss of generality, assuming $rank(\Pi)=D_0$, the rank of $\hat{\theta}^{(1)}$ should exceed $D_0$​. Regarding your query about Figure 4, the network used in this experiment is indeed a linear 3-layer network, as per the requirements to empirically demonstrate Theorem 3.1. There are no non-linear activation functions involved, ensuring the network’s alignment with the theorem's conditions. We appreciate your feedback and will ensure to include a comprehensive revision and discussion of this aspect in our next manuscript update. We hope this response addresses your concerns effectively.

Dear Reviewer FbDy,

We appreciate your feedback and have taken the opportunity to clarify and revise the proof of Theorem 3.1. This revised proof, along with a detailed explanation, is also included in Section A.3 in the appendix in the uploaded revision PDF. In this revised analysis, we focus on batch data rather than single data points.

Assume weights are updated with the stochastic gradient descent (SGD). Without loss of generality, we set $D_h = 2D_0 = 2D_L$. Given two batches of inputs as $\{x_1^{(0, 1)}, x_1^{(0, 2)}, \dots, x_1^{(0, N)}\}\in \mathbb{R}^{D_0}$ and $\{x_2^{(0, 1)}, x_2^{(0, 2)}, \dots, x_2^{(0, N)}\}\in \mathbb{R}^{D_0}$, where $N \ge D_0$ is the batch size. Therefore, the initial gradient of $\theta^{(1)}$ are

$$\frac{\partial \mathcal{L}}{\partial \theta^{(1)}} = \begin{pmatrix*} \Pi & \Pi \\\\ \mathbf{0} & \mathbf{0} \\ \end{pmatrix*},$$

where $\frac{\partial \mathcal{L}}{\partial \theta^{(0)}} \in \mathbb{R}^{D_h \times D_0}$, $\frac{\partial \mathcal{L}}{\partial \theta^{(1)}} \in \mathbb{R}^{D_h \times D_h}$, $\frac{\partial \mathcal{L}}{\partial \theta^{(2)}} \in \mathbb{R}^{D_L \times D_h}$, $\Pi = \frac{1}{N}\sum^N_{i=1}\frac{\partial L}{\partial x^{(L)}} \circ x_1^{(0, i)} \in \mathbb{R}^{D_L\times D_0}$, and $\mathbf{0}$ is a zero values. $\circ$ denotes outer production.

After training with the second data batch, the gradient is calculated as follows:

$$\frac{\partial \mathcal{L}}{\partial \theta^{(1)}}= \begin{pmatrix*} (I-\mu\Pi)M(I-\mu\Pi)K & (I-\mu\Pi)MK \\\\ -\mu\Pi M(I-\mu\Pi)K & -\mu\Pi MK \end{pmatrix*},$$

where $M=\frac{\partial \mathcal{L}}{\partial x_2^{(3)}}$ and $K=\frac{1}{N}\sum^N_{i=1}x_2^{(0, i)}$. This leads to the following residual component:

$$\hat{\theta}^{(1)} = \theta^{(1)} - I = \begin{pmatrix*} -\mu\Pi - \mu(I-\mu\Pi)M(I-\mu\Pi)K & -\mu\Pi - \mu(I-\mu\Pi)MK \\\\ \mu^2\Pi M(I-\mu\Pi)K & \mu^2\Pi MK \end{pmatrix*},$$

Without loss of generality, assuming $rank(\Pi)=D_0$, we can conclude

$$rank(\hat{\theta}^{(1)}) \ge D_0.$$

Therefore, IDInit can break the rank constraint by achieving the rank of $\hat{\theta}^{(1)}$ larger than $D_0$.

We sincerely hope that this revision addresses your concern. If there are any further questions or clarifications needed, please do not hesitate to let us know.

Kind regards,

Authors

Thank you for the great insight. Indeed, it is accurate that the rank of the outer product of two non-zero vectors is at most 1. However, for batch data with size $m \ge D_0$, the matrix $\Pi$ can still achieve a rank of $D_0$. This is due to the aggregation of multiple rank-1 matrices arising from each instance in the batch. As these rank-1 matrices are derived from different instances, their cumulative effect can lead to an overall increase in the rank of the aggregated matrix $\Pi$.

Furthermore, we also empirically validate that even with a batch size of 1, through iterative training steps, the rank of $\hat{\theta}^{(1)}$ can exceed $D_0$. This phenomenon occurs due to the successive accumulation of gradients over multiple steps.

Thanks again for your thoughtful question! We are happy to discuss more if you have any other questions.