IDInit: A Universal and Stable Initialization Method for Neural Network Training

Response to "This paper is very hard to follow...."

We appreciate the reviewer’s feedback and the time taken to evaluate our work. While we understand the reviewer's concerns, we believe the paper is generally well-structured and well-written. This perspective is supported by the feedback from the other three reviewers. However, we recognize the importance of addressing any potential clarity issues to ensure accessibility for a broader audience. Below, we provide specific responses to the points raised:

Why change the notation from Figure 1 to Equation 1? What do you mean when applying the activation function to the weights / transformation in Equation 1? This notation is not defined or explained.

We use the notation $W_1$ in Figure 1 as a convenient way to represent matrices, which helps illustrate the identity-control initialization in Figure 1 and the motivation presented in Figure 2. On the other hand, the notation $\theta^{(i, 0)}$ in Equation 1 is central to defining our method and providing the derivations of the theorems and proofs.

Equation 1 serves as a general formulation for the residual network. The activation function $a(\cdot)$ is introduced as a placeholder to suggest its applicability in this context. It is intentionally left unspecified to keep the discussion general and not tied to a particular function, such as ReLU or others.

Many aspects of Figure 2 are unclear. Does this network have activation functions? Why is the learning rate not tuned per method? Why is the rank issue so significant here despite the matrices being almost square? I find it hard to believe that a difference of 40 features out of 280 can cause such a large difference between the methods. Why use Adagrad and not a more standard optimizer like SGD or Adam?

Thank you for your thoughtful comments. We address each point below to clarify and justify our experimental setup and the insights from Figure 2.

1. Activation Functions and Learning Rate

The experiments in Figure 2 follow the setup described in [1] using their open-source code. As detailed in Appendix C.5 (referenced in the caption of Figure 2), we use the ReLU activation function. The learning rate of 0.01 and the Adagrad optimizer are also adopted from [1]. Adagrad adaptively adjusts the learning rate during training, which mitigates the need for manual tuning of a fixed learning rate. These choices ensure consistency and fairness in the experimental setup.

2. Rank Constraint and Matrix Design

The focus of Figure 2 is not directly on rank constraints but rather on motivating the design of $W_1$ when $W_2$ is set to $\mathbf{0}$.

• Figure 2(b): In Figure 2(b), none of the initialization methods encounter rank issues. However, Identity-1 can surpass Random and Hadamard by maintaining identity. We speculate the worse performance of Random and Hadamard is caused by initializing $W_1$ with random values or partial values of Hadamard matrix, as this is the only difference between them and Identity-1. To address this, we consider initializing $W_1$ as $I$ to preserve the inductive bias of Identity-1, which leads to improved results.

• Figure 2(c): Here, we introduce rectangular $W_1$ and $W_2$ to explore cases where a real identity matrix cannot be applied (requiring square matrices). Among the methods, only "Partial Identity" encounters rank constraints due to the matrix dimensions, leading to slightly worse performance. This degradation, caused by the loss of fewer than 40 features (as noted by the reviewer), is evident when compared to Random and Hadamard. In contrast, the Default initialization performs the worst, as it does not align with the training dynamics. IDInit avoids the rank constraint problem and effectively preserves the inductive bias of Identity-1, resulting in superior performance.

3. Choice of Adagrad

Adagrad, chosen in accordance with [1], dynamically adapts the learning rate based on gradient history, making it particularly suited for this experiment. While SGD and Adam are more common optimizers, the use of Adagrad ensures consistency with the original study and highlights the generality of our method.

4. Motivation and Future Exploration

Figure 2 effectively motivates the design of IDInit by demonstrating how identity-based initialization maintains structural bias and avoids pitfalls like rank constraints. While the exact mechanism underlying the improvement remains an open question, these results strongly suggest the potential of identity initialization. We plan to include a discussion section in the manuscript to encourage further exploration of this phenomenon and inspire the community to investigate advanced initialization strategies.

Response to "Some of the experiments are not convincing...."

Section 4.1: I don’t find this convincing. The granularity of the sweep is too high and the best performance for the Kaiming initialization occurs at the edge of the sweep.

Thank you for your comment. To address your concern, we have extended the hyperparameter sweep range in Appendix G.3 of the uploaded PDF. Specifically, we expanded the learning rate sweep from $10^{-3}$ to $10^{1}$ and the weight decay sweep from $10^{-8}$ to $10^{-1}$. This ensures that the best-performing hyperparameters are not located at the corners or edges of the grid.

As shown in Figure 22 of the uploaded PDF, IDInit consistently delivers strong performance across this expanded range of settings and achieves the best results among all initialization methods tested. This demonstrates the robustness of IDInit and supports its effectiveness under a fair and comprehensive comparison framework.

Section 4.2: I also don’t find this very convincing. In general the hyperparameters need to be tuned per method for a fair comparison. How do we know that your hyperparameters don’t favor your method?

Thank you for raising this concern. We show a grid search for ResNet-110, as detailed in Appendix G.4 of the uploaded PDF. Specifically, we explored the hyperparameters across a grid of learning rates $\{1, 0.2, 0.1\}$ and weight decay values $\{10^{-4}, 5 \times 10^{-4}, 10^{-3}\}$ on the baseline Kaiming initialization and the Fixup method. As shown in Figure 23, both Kaiming and Fixup achieve their best accuracy with a learning rate of 0.2 and weight decay of $5 \times 10^{-4}$. Notably, Fixup fails to train with a learning rate of 1. Based on this analysis, using a learning rate of 0.2 and weight decay of $5 \times 10^{-4}$ as the training hyperparameters in Section 4.2 is justified.

We hope this addresses your concern, and we welcome any further suggestions for improvement.

[1] Bachlechner, Thomas, et al. "Rezero is all you need: Fast convergence at large depth." UAI, 2021.

[2] Zhu, Chen, et al. "Gradinit: Learning to initialize neural networks for stable and efficient training." NeurIPS, 2021.

[3] Zhang, Hongyi, et al. Fixup initialization: Residual learning without normalization. ICLR, 2019.

Figure 3: I think this would be clearer if you used one symbol for the number of features on the skip path and another for the intermediate dimension. Here D_i is used twice to mean different dimensions.

Thank you for pointing this out. To clarify the notation, we will use $D_h$ to represent the intermediate dimension (the middle dimension between two matrices) and retain $D_i$ to denote the number of features on the skip path. This adjustment will ensure consistency and eliminate any ambiguity in the figure. We will update Figure 3 accordingly in the revised manuscript.

Figure 4: I feel that when comparing momentum to no momentum the learning rate must be taken into account. Using momentum in SGD increases the effective learning rate so it is not surprising that the difference is larger.

The primary contribution of Section 3.1.1 and Figure 4 is not to highlight the acceleration of differences, but to demonstrate that SGD effectively resolves the symmetry problem caused by identical initialization. Momentum further amplifies this effect, providing a supplementary benefit, as it is a common enhancement used with SGD.

We also analyze the influence of the learning rate in Appendix G.7 of the uploaded PDF. Specifically, when $x_1,x_2,y_1,y_2 \in \mathbb{R}^d$ are random vectors whose entries are i.i.d. Gaussian random variables with $N(0, \sigma^2)$, the magnitude of asymmetry can be bounded by $(6\eta^2+3)d^2\sigma^2$. This shows that a higher learning rate promotes greater asymmetry, further explaining the observed differences. However, a high learning rate can affect training stability. Therefore, while using a higher learning rate to reduce symmetry, it is crucial to carefully select its magnitude to maintain stability.

Lines 314-318: Here you introduce the “loose” condition where you add random noise to your leading matrices. However, this is not mentioned again. Are the later experiments conducted with this change and does it make a difference in practice?

Thank you for your observation. We have addressed the "Loose" condition by including additional experiments in the newly added RTable 1.

RTable 1. Analysis of components.

The extended analysis in RTable 1 shows that the "Loose" condition, along with the components 'IDIC' and 'IDIZ,' contributes independently to performance improvements. Furthermore, the combination of these components yields the most significant results. Across all comparison pairs—specifically, settings 1/4, 2/6, 3/7, and 5/8—the "Loose" condition consistently demonstrates performance improvements. This highlights its practical value and its role in enhancing the overall effectiveness of the initialization methods.

Section 4.3: Here I feel you could explain the exact setup a bit better, e.g. what exactly do you do without IDIC?

Thank you for your comment. In Section 4.3, IDIC refers to the use of the patch-maintain scheme (as described in Eq. (9)) instead of the channel-maintain scheme (outlined in Lines 325–327 of the original paper). This adjustment enables IDInit to better increasing channel diversity during initialization, leading to a notable improvement of 3.42% in accuracy. We will revise the manuscript to provide a clearer explanation of the setup and explicitly contrast the configurations with and without IDIC to enhance clarity.

We hope these revisions will address the reviewer’s concerns while maintaining the strengths highlighted by the other reviewers. Thank you again for your valuable feedback.

Dear Reviewer 96yw,

We hope this message finds you well.

We have addressed the comments and concerns raised in the reviews to the best of our ability and are keen to receive any further feedback or clarification. With the decision deadline approaching, we would like to kindly check if there are any remaining questions or points you would like us to address.

We greatly appreciate the time and effort you and the other reviewers have dedicated to this process, and we are happy to provide any additional information if needed.

Thank you for your attention, and we look forward to hearing from you soon.

Best regards,
Authors of Submission #6753

The manuscript may need to elaborate the discussion on the momentum part, e.g., by providing some theoretical justifications.

Thank you for the suggestion. To provide further theoretical justification for the role of momentum, we conducted additional analysis, which is detailed in Appendix G.7 of the uploaded PDF. Specifically, when $x_1,x_2,y_1,y_2 \in \mathbb{R}^d$ are random vectors whose entries are i.i.d. Gaussian random variables with distribution $N(0, \sigma^2)$, the magnitude of asymmetry can be bounded by $(6\eta^2+3)d^2\sigma^2$.

Since momentum is accumulated from gradients, it inherently amplifies the asymmetry present in the gradients. Consequently, when momentum is incorporated into weight updates, it promotes an increasing magnitude of asymmetry over time. This amplification effect provides a theoretical explanation for the enhanced performance observed when momentum is used, as it helps break the symmetry more effectively during training.

We will update the manuscript to include this theoretical insight, offering a clearer understanding of momentum’s role in promoting asymmetry and its implications for improving training dynamics.

Some design/hyper-parameter choices are not justified by some references, e.g., many parts in Sec 4 (line 434-line 439, line 449-line 454, line 464-line 474, line 504-line 509, line 517-line 523)

Thank you for your thoughtful feedback. We appreciate your reminder and will carefully review and supplement the missing references in the identified sections of Section 4 in the next version of the manuscript. This will ensure that all design and hyperparameter choices are properly justified and supported by relevant literature.

We hope this addresses your concern, and we welcome any further suggestions for improvement.

suboptimal: In certain instances, IDInit remains suboptimal. For example, in Table 1, the final accuracy of a 110-layer neural network does not surpass the baselines.

Thank you for your comment. We demonstrate consistently superior performance in most scenarios compared to various baselines, as shown in Table 1 and Table 4. In Table 1, we also highlight the faster convergence speed, which is another advantage of IDInit. Furthermore, as illustrated in Figure 7 and Figure 12, IDInit exhibits robustness across a wider range of hyperparameters (e.g., learning rate and weight decay). Consequently, IDInit significantly outperforms baseline initialization methods.

More experiments on self-supervised learning methods and diffusion models: The proposed IDInit has not been validated across diverse learning paradigms, such as self-supervised learning methods [1] and diffusion models [2]. Could you include additional experimental results involving self-supervised learning methods and diffusion models?

Thank you for your suggestion. We conduct experiments on GPT-Base-MOE in Appendix G.5 and DiT-S/4 in Appendix G.6 of the uploaded PDF. Both experiments demonstrate that IDInit enables faster training compared to the default initialization.

Convergence rate: The paper lacks theoretical analysis regarding the convergence rate of IDInit. Could you provide further theoretical analysis on the convergence rate of IDInit?

We share an interest in this question; however, analyzing the convergence rate theoretically is challenging. To the best of our knowledge, no initialization methods currently provide a theoretical analysis of it. The convergence process of deep neural networks is iterative and influenced by numerous factors beyond the initialization method, including the network architecture, optimization algorithm, learning rate, batch size, and data distribution. Further exploration is needed in this area. We will include a discussion in the paper to encourage the research community to focus more attention on this important problem.

We hope this addresses your concern, and we welcome any further suggestions for improvement.

How do the results in Figure 2 change with varying matrix sizes?

We conducted an experiment with a larger model size, as detailed in Appendix G.2 of the uploaded PDF. With the increased size, the model converges faster across all initialization methods. Among them, IDInit consistently achieves the fastest convergence and delivers the best performance.

A suggestion: remove the word “Some” from the caption of Figure 8, as it diminishes the perceived significance of the experiment.

We have removed the word "Some".

The term “residual stem” was initially unclear. Introducing and defining this term earlier in the paper might help readers understand it more quickly.

We have defined it as follows: "Consider an $L$-layer residual network, each residual block of which consists of a residual connection and a residual stem that refers to the component excluding the residual connection."

Lines 134–135: The phrase, “In this paradigm, the input-output Jacobian is calculated as…” is somewhat confusing. Equation 2 simply defines the input-output Jacobian, and its definition does not change based on whether it is viewed through the lens of dynamical isometry. I recommend removing the phrase “In this paradigm” for clarity.

Thanks for the valuable suggestion. We have revised this part as "Before introducing the mechanism, we first consider the input-output Jacobian which is defined as...."

We hope this addresses your concern, and we welcome any further suggestions for improvement.

[1] Bachlechner, Thomas, et al. "Rezero is all you need: Fast convergence at large depth." UAI, 2021.

[2] Zhu, Chen, et al. "Gradinit: Learning to initialize neural networks for stable and efficient training." NeurIPS, 2021.

[3] Zhang, Hongyi, et al. Fixup initialization: Residual learning without normalization. ICLR, 2019.

[4] Zhao, Jiawei, et al. Zero initialization: Initializing neural networks with only zeros and ones. TMLR, 2022.

[5] Bartlett, Peter, Dave Helmbold, and Philip Long. "Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks." ICML, 2018.

[6] Hardt, Moritz, and Tengyu Ma. "Identity matters in deep learning." ICLR, 2017.

detail of Figure 2: Firstly, I have concerns regarding Figure 2...

Thank you for your valuable feedback. We construct the experiment following the methodology outlined in [1], utilizing their open-source code. The classification task is conducted on the Cifar10 dataset. The model employs a residual structure with 64 residual blocks, each comprising two matrices, $W_1$ and $W_2$, along with a classification layer at the top. Each block is represented as $Y = \text{ReLU}(W\_2 \text{ReLU}(W\_1 X)) + X$.

The term "Square" refers to $W_1 \in \mathbb{R}^{240 \times 240}$ and $W_2 \in \mathbb{R}^{240 \times 240}$, while "Rectangle" denotes $W_1 \in \mathbb{R}^{280 \times 240}$ and $W_2 \in \mathbb{R}^{240 \times 280}$, as described in the caption of Figure 2. Unlike "Square" matrices, "Rectangle" matrices cannot be initialized to an identity matrix since identity matrices are defined for square matrices. "Square Loss" and "Rectangle Loss" represent the losses observed during the experiments with "Square" and "Rectangle" matrices, respectively.

The lines in Figures 2(a) and 2(b) are averaged over three runs. To provide additional clarity, we include Figure 20 in Appendix G.1 of the uploaded PDF, which includes standard deviation.

"Random" refers to initializing $W_1$ using Xavier initialization and setting $W_2$ to $\mathbf{0}$, a strategy adopted in Fixup initialization. We use "Random" instead of "Fixup" to align with the "Random" matrix shown in Figure 2(a).

More experiments on ImageNet: Figures 8 and Table 3 are potentially valuable...

Thank you for your suggestion. To further compare with other identity-control methods, we conducted experiments on ResNet-50 using Fixup and Zero. As shown in RTable 1, IDInit outperforms both Fixup and ZerO, demonstrating its superior performance on large-scale datasets.

RTable 1. Results of ResNet-50 on ImageNet.

Regarding multiple runs, we acknowledge that repeated experiments provide a more precise assessment of performance. However, large-scale experiments are computationally intensive, and the extensive amount of data involved minimizes the likelihood of significant fluctuations in the empirical results. As a result, prior studies on ImageNet ([1], [2], [3], and [4]) often rely on single runs for large-scale experiments.

For ImageNet, we performed five experiments, all of which demonstrated the superior performance of IDInit. Specifically, for ResNet-50, IDInit achieved an accuracy of 76.72%, surpassing the 76.2% reported in [2], which confirms the feasibility of our settings.

Therefore, we believe our validation on ImageNet and the tested models is both robust and reliable.

Imprecise claims: Lines 110–114 state...

We have revised the claims in the uploaded PDF for more precise, e.g., "we have significantly improved accuracy by 3.42% and 5.89% respectively." -> "we have significantly improved accuracy of classifying Cifar10 on ResNet-20 by 3.42% and 5.89%, respectively.". Please check it.

doubts on dynamical isometry: The definition and understanding of dynamical isometry...

Thank you for the reminder—this is a very interesting observation. The networks discussed in Pennington (2017) are not residual networks, and for residual networks, which are the focus of this paper, the situation can differ significantly. Specifically, in residual networks, such as $Y=\text{ReLU}(W\_2\text{ReLU}(W\_1X))+X$, the non-linearity resides in the sub-stem of the residual block. As explored in [5] and [6], when the weights in the sub-stem are small, residual networks with ReLU can approximate a linear network. This allows the model to follow dynamic isometry, as shown in Figure 17 of the original paper, where IDInit produces most $\chi$ values close to 1.

However, the situation changes if ReLU is placed in the main stem, e.g., $Y=\text{ReLU}(\text{ReLU}(W\_2W\_1X+X))$. In this case, the network cannot maintain isometry, as noted in Pennington (2017). This suggests that placing ReLU in the main stem is not advisable.

We will include this discussion in a dedicated paragraph in the next version of the paper.

break symmetry: In Section 3.1.1's convergence analysis...

Thank you for your insightful comment. We extending the analysis on the magnitude of asymmetry in Section G.7 of the uploaded PDF. According to Eq.(4), the asymmetry in the gradient arises from:

Here, we assume $x_1, x_2, y_1, y_2 \in \mathbb{R}^d$ are random vectors with entries that are i.i.d. Gaussian random variables, following $N(0, \sigma^2)$. The magnitude of asymmetry can be calculated as

This shows that a higher learning rate promotes greater asymmetry, further explaining the observed differences. However, a high learning rate can affect training stability. Therefore, while using a higher learning rate to reduce symmetry, it is crucial to carefully select its magnitude to maintain stability.

Detial of Appendix A3: In Appendix A3, what is the upper index in $x_i^{(0)}$...

In Appendix A.3, $x_i^{(0)}$ is similar to $x^{(0)}$ in Eq. (1), representing the activations within the layers. Since the network consists of three layers, the final output is denoted as $x_i^{(3)}$, where $i$ indicates the $i$-th training batch. For the first batch, $x_1^{(3)}$ is used to calculate the gradient.

Regarding the batch size, it is set to $N = D_0$, ensuring that each sample in the batch is linearly independent. This guarantees that the rank of $\Pi_1$ and $\Pi_2$ in the gradient reaches $D_0$, a crucial condition for demonstrating Theorem 3.1.

Identity-1: In Figure 2, I am not sure if I understand what Identity-1 is...

Identity-1 serves as a baseline consisting of residual blocks, each of which contains only a square matrix initialized to $\mathbf{0}$. This setup is equivalent to a non-residual layer with a square matrix initialized to $I$. The baseline is designed to demonstrate the properties of a precise identity maintenance scheme.

Specified $\tau$: In Equation (3), how is $\tau$...

We have already specified the setting in Line 798-800 of the original paper of the beginning of Appendix B, as "In this paper, for ReLU activated networks, $\tau$ is set to $\sqrt{2}$ for the first layer in a network and 1 for other $\text{IDIC}\_{\tau} / \text{IDIC}\_{\tau}$ initializing layers, while for tanh-activated networks, all $\text{IDI}\_{\tau}$ is set to 1, and $\varepsilon$ is $1e-6$ for all $\text{IDIZ}\_{\varepsilon}/\text{IDIZC}\_{\varepsilon}$ initializing layers."

More extension of Section 4.1: I liked the hyperparameter experiments from Section 4.1...

We extend the search grid of Figure 7 in Appendix G.3 of the uploaded PDF. As shown in Figure 22 of the uploaded PDF, we scanned the learning rate from 1e-3 to 1e1 and weight decay from 1e-8 to 1e-1, ensuring that the best-performing hyperparameters are not at the corners or edges of the grid. As illustrated in Figure 22, IDInit consistently performs well across a wide range of settings and achieves the best performance among all the initialization methods tested.

We also add search of ResNet-110 in Appendix G.4. We perform a detailed hyperparameter analysis for ResNet-110, evaluating the learning rates {1, 2e-1, 1e-1} and weight decays {1e-4, 5e-4, 1e-3} on the standard baseline Kaiming and the more fragile Fixup method. As shown in Figure 23, both Kaiming and Fixup achieve optimal accuracy with a learning rate of 2e-1 and a weight decay of 5e-4. However, Fixup fails to train with a learning rate of 1. Consequently, selecting a learning rate of 2e-1 and a weight decay of 5e-4 as the training hyperparameters in Section 4.2 is justified.

Thank you for detailed answers. I appreciate the time and effort the authors undertook to address my concerns. However, I still have some concerns regarding the work. I divided them into “Major” (i.e. influencing my decision) and “Minor” (i.e. easy to fix, or just suggestions on the style of writing, does not have an influence on my decision but are more of an advice on how to improve the paper). Given the short amount of time remaining for the rebuttal the authors might like to focus on the “Major” issues.

Major:

Ad. Point 4) [doubts on dynamical isometry]

I still have problems with this point. First of all, yes, residual connections that maintain the identity along the main stem make it possible for the Residual Network to attain dynamical isometry for any type of activations. To the reference provided by the authors I would suggest adding [1], which investigates just this matter, providing theory. My problem is that the authors never properly define what is dynamical isometry, despite using this term. Again, achieving $X=1$ is not the definition of the dynamical isometry. Moreover, the discussion on how the residual connections influence that property (as provided by the authors in the rebuttal) should be included in the paper.

[1] Tarnowski, Wojciech, et al. "Dynamical isometry is achieved in residual networks in a universal way for any activation function." The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.

Ad Point 5) [break symmetry]

Thank you, but I have now significant issues with the provided argumentations and equations. Basically, you are giving an upper bound on you asymmetry metric $\mathbb{E}(||\Omega-\Omega^T||_{F}^2)\le (6\eta^2+3)d^2\sigma^2$

and argue that by increasing $\eta$ the asymmetry will increase as well. But, since you provide only an upper bound, this does not necessarily have to be true. I.e. upper bound can be arbitrarily large, but the exact value of the asymmetry metric might not change. To make the point, the authors should show a lower bound in terms of $\eta$ - something of type $f(\eta) \le \mathbb{E}(||\Omega-\Omega^T||_{F}^2)$. Then indeed, with the increase of the learning rate the asymmetric metric will for sure increase as well. Otherwise the entire argumentation is baseless.

Question 3. - I am sorry, maybe I was not specific enough - Yes, I have seen that in the paper you give the values of $\tau$ for different activations (e.g. $\tau_{RELU}=\sqrt{2}$). My question was not what is the value of $\tau$, but why is the value equal to this specific quantity? I.e. how what basis was that determined or calculated?

Minor:

Ad. Point 1) [Figure 2]. Thank you for addressing this, the description is better, but I still have some minor suggestions:

We construct the experiment following the methodology outlined in [1], utilizing their open-source code. [...]

Thank you, that clarifies the setup for me. I have seen appendix C.5 in which you add some of the information, but please, just add the entire paragraph you wrote to me to that description in the appendix. I.e. it is meaningful that you have used the code of ReZero for reproducibility.

"Random" refers to initializing $W_1$ using Xavier initialization and setting $W_2$ to $\mathbf{0}$,

Thank you, maybe I was not precise. My question was not about “what is “Random”” but rather “where is “Random” in the plot? In the plots in Figure 2b and 2c the “Random” approach is supposed to be denoted using the yellow line, but this line is not visible in the plots. Hence my question. From Figure 20 I suspect that the results for Random overlay with Hadamard, but it would be nice if the authors add a comment about this in captions, otherwise it is confusing.

Ad Point 2) [experiments on ImageNet]

Thank you for the additional table with comparison, this looks interesting. If it is possible, I would also like to see those comparisons depicted on a plot as in Figure 8b (i.e. the learning curve). It is clear that IDInit gives the best final accuracy, but up to 60% it goes neck-to-neck with Fixup. So it has to surpass Fixup at some later epoch in the training, and it would be interesting to see what that epoch is.

Regarding multiple runs, [...]

I understand that many papers on ImageNet use a single seed and that ResNet models typically show small standard deviations (e.g., Table 2 of ReZero). While I’d be surprised if IDInit increased the standard deviation—since it aids signal propagation and might improve stability—I believe repeating experiments is good scientific practice. This ensures robustness against unexpected impacts or unusually favorable seeds. Though the authors may lack time for this during the rebuttal, I recommend reporting results from multiple runs if the paper is accepted to enhance reliability.

Dear Reviewer jXZf,

We hope this message finds you well. As the review period approaches its end, we would like to kindly follow up on your feedback regarding our rebuttal. We would greatly appreciate it if you could let us know whether your concerns have been adequately addressed. If there are any remaining issues or points requiring clarification, we would be happy to provide further details.

Thank you very much for your time and efforts in reviewing our submission.

Best regards,
Authors of Submission #6753