We appreciate the reviewer's thoughtful evaluation and generous score, and are glad to see that our work was found to be valuable. Below, we address the reviewer’s question.

Request to Explain RHS of Eq. (20)

We thank the reviewer for their engagement with this result. The RHS of Eq. (20) is intended to show a lower bound on the growth rate of the hidden state's norm. Taking $t = \alpha \sqrt{n}$, we approximate the involved sum using an integral as follows:

where we defined a small $\epsilon>0$ to avoid the singularity, substituted $x := k / \sqrt{n}$, and defined the discretization set $X_{\left[\epsilon, \alpha\right]} := \left[\{ \frac{k}{\sqrt{n}} \}\right]$ for $k \in \left[ \lfloor \epsilon \sqrt{n} \rfloor, \dots, \lfloor \alpha \sqrt{n} \rfloor \right]$. We then further approximate this integral using the substitution $y := x^2 / 2 = k^2 / 2n$, which yields:

We thank the reviewer for highlighting this equation, as it led us to identify a minor typo in the current version of the formula: the denominator incorrectly included only $t$ instead of $t^2$. Importantly, this typo does not affect the main conclusion of the section regarding the exponential growth rate and the resulting exploding behavior of the hidden state. We will correct this in the revision and include a more detailed version of the derivation.

We appreciate the reviewer’s comments. Below, we address the concerns raised and provide clarifications regarding the contributions and empirical findings of our work.

1. Response to Weaknesses

1.1 Instability of Glorot in Previous Works

The reviewer asked why vanilla Glorot initialization leads to exploding behavior in our Table 1, while such behavior was not reported in prior works using Gaussian initialization on Long Range Arena benchmarks. The explanation here is straightforward: prior works typically applied additional, implicit rescaling to the Glorot weights (often chosen empirically) to avoid instability, but this detail was rarely highlighted. In particular:

• "Resurrecting Recurrent Neural Networks for Long Sequences" [1]: This influential paper introduced the LRU initialization scheme and demonstrated its empirical advantages over Glorot initialization. However, the actual implementation (based on the default Vanilla RNN module in Haiku, as stated in Footnote 3 of [1]) used a truncated normal initialization over the range $[-2, 2]$. This truncation significantly alters the spectrum of the initialized weight matrices. In particular, it effectively rescales Glorot initialization by a factor of approximately $0.8$, which greatly reduces the probability that any eigenvalue exceeds one in modulus. This implicit rescaling is not discussed in [1], but our results show that it plays a crucial role in preventing signal explosion. This highlights the importance of careful spectral analysis when evaluating initialization schemes.
• "Efficiently Modeling Long Sequences with Structured State Spaces" [2]: This work, which introduced the widely used S4 model, was more explicit about their rescaling of Glorot initialization baseline. As stated in Section 4.4, they empirically scaled down the variance of the Gaussian initialization until the model no longer exhibited instability.

These examples demonstrate that the problem of exploding behavior under Glorot initialization is well-known and relevant in modern RNN literature. However, most previous works addressed it only empirically, by tuning scaling factors, without providing theoretical analysis or principled justification.

We also offer a more detailed discussion of these two works [1,2], highlighting how the instability of Glorot manifests in their setups, in our response to Reviewer rSLB (par. 1, item (2): "Theoretical gaps in high-impact RNN works"). We will make this point more explicit in the revision to clarify the motivation behind our theoretical analysis and rescaling approach.

1.2 Clarification of Contributions

We would like to clarify the core contributions of our work, as the reviewer’s summary may not fully capture our results in two important respects:

• Stability and instability of Glorot initialization: The reviewer mentioned that we show stability of hidden states under Glorot initialization in both the infinite-width and infinite-width-and-length regimes. In fact, our analysis demonstrates stability in the infinite-width regime, but instability in the infinite-width-and-length regime. In addition, we (1) identify the precise sequence length scaling threshold $t=\Theta(\sqrt{n})$, which is sufficient to cause instability, and (2) provide a more general finite-width result for the hidden state norm, which supports both limiting cases. These results offer a theoretical explanation for why long-sequence tasks pose unique challenges for initialization, and allow us to quantify how long is "long enough" for instability to arise.
• Empirical validation: The reviewer raised a concern that our experiments do not clearly show the benefits of our rescaled initialization compared to standard Glorot. However, our experiments in Table 1 and Fig. 1 directly illustrate that, consistent with our theory, vanilla Glorot leads to exploding hidden states and training divergence, whereas our rescaled variant avoids this issue and enables stable training.

In summary, the main contribution of our work is a theoretical understanding of the behavior of Glorot initialization in linear recurrent architectures, especially in the long-sequence regime. Given Glorot’s wide use as a default baseline, we believe that shedding light on its limitations and offering a principled alternative is both timely and impactful. We also discuss the significance of our findings in more detail in our response to Reviewer rSLB (par. 1, “Relevance and Significance”).

2. Response to Questions

Q1. Performance Comparison

While it is true that all considered initialization schemes successfully prevent hidden state explosion, preventing instability alone does not guarantee equivalent downstream task performance. Different initializations influence not only the magnitude of hidden states but also the model's training dynamics and convergence behavior. Consequently, subtle differences in initialization can significantly affect generalization properties. This explains why, despite all schemes effectively mitigating hidden state explosion, performance differences between our rescaled Glorot, Glorot/2, and Uniform initialization can still occur.

Q2. Density and Hidden States for Glorot/2 and Uniform Initialization Schemes

Eigenvalue modulus density: Since matrices initialized using Glorot/2 correspond exactly to standard Glorot-initialized matrices scaled by a factor of $1/2$, their eigenvalues are simply those of the standard Glorot matrices divided by two. Thus, the eigenvalue density for Glorot/2 initialization is identical to that shown in the upper row of Fig. 1, but scaled horizontally by a factor of $1/2$. For matrices initialized uniformly, the eigenvalues are uniformly distributed on the complex unit circle. Therefore, the modulus density of these eigenvalues follows a linear profile forming a triangular distribution from $0$ to $1$. This distribution closely resembles the Glorot distribution depicted in Fig. 1 but without the small tail extending beyond modulus one.

Hidden states behavior: For both Glorot/2 and Uniform initialization, the norms $||W^k x||$ decrease as $k$ increases, and the corresponding hidden-state norms exhibit logarithmic growth with respect to $k$. This behavior is analogous to what we observe for the rescaled Glorot initialization in the lower row of Fig. 1. However, for Glorot/2 initialization, the decrease in $||W^k x||$ is significantly faster compared to Uniform initialization and our rescaled Glorot variant, resulting in notably smaller hidden-state norms.

Q3. Error Bars in Table 1

The results in Table 1 are averaged over three runs, with error bars representing the standard deviation across these repetitions.

Q4. Behavior on Short Sequences

The eigenvalues of the transition matrix remain unchanged regardless of sequence length, as the same matrix is used at each time step. For shorter sequences, any deviation of the spectral radius from 1 has a limited effect due to the reduced number of recurrent applications. As a result, both the Glorot and our proposed Rescaled Glorot initialization are unlikely to cause either exploding or vanishing hidden states in this regime. In general, short sequences are not considered challenging from a stability perspective, and our method is designed to address long-sequence dynamics without compromising performance on shorter ones.

We would be happy to address any further questions or provide additional clarification, and hope that the provided responses may support a more positive evaluation of our work.

References

[1] Orvieto et al. "Resurrecting recurrent neural networks for long sequences." ICML (2023).

[2] Gu et al. "Efficiently modeling long sequences with structured state spaces." arXiv preprint arXiv:2111.00396 (2021).

Dear Reviewer,

We hope that our rebuttal has addressed the main concerns raised in your review. In particular, we aimed to clarify that rigorously establishing the instability of Glorot initialization and demonstrating it empirically are central contributions of our work. We also emphasized that our theoretical results show the contrast between the infinite-width limit (where Glorot is stable) and the joint infinite-length-and-width limit (where it becomes unstable).

If these clarifications were helpful, we would be grateful if you would consider updating your score. We would also be happy to continue the discussion if any questions remain.

We appreciate the reviewer’s comments and respond to their concerns below.

1. Relevance and Significance

The reviewer raised concerns about the significance of our study of Glorot initialization in linear RNNs, particularly in light of the many alternatives for stabilizing signal propagation. To clarify the relevance and impact of our work, we highlight the following points:

(1) Recent surge in interest for linear RNNs: The success of LRU [1] and SSMs [2] for long-range sequence modeling has inspired a recent surge of interest in linear RNNs, due to their competitive performance and computational efficiency. The linearity of these models enables the use of efficient parallel scan operations, allowing for fast inference and training. Notably, they avoid the use of LayerNorm or complex nonlinear gating mechanisms (as in LSTMs and GRUs) within the recurrent block, since such components would compromise their efficiency. Thus, the LRU's recurrent block differs from vanilla linear RNNs only in its initialization and parametrization. Given that such architectures are becoming increasingly common in recent literature, and frequently adopt Glorot initialization as a baseline, we believe our contribution is both timely and important for informing best practices in this field.

(2) Theoretical gaps in high-impact RNN works: While we fully acknowledge that Glorot initialization is a classical method, we emphasize that misunderstandings about its properties, and lack of theoretical analysis, continue to appear in high-impact recent work. We highlight two such examples:

• "Resurrecting Recurrent Neural Networks for Long Sequences" [1]: This influential paper introduced the LRU scheme for initializing and parametrizing linear recurrent layers, which has proven highly effective for long-range reasoning. While the method works well empirically, the (non-rigorous) justification presented in Section 3.2.2 of [1] is based on a misconception: namely, that the eigenvalue moduli of Glorot-initialized matrices remain strictly below one. The authors even illustrate this by Figure 2 and state the strong circular law in their Theorem 3.1. However, the Glorot initialization baseline used to verify LRU's benefits is actually truncated Glorot, which roughly rescales the spectrum by $0.8$ (see our response to Reviewer qFV8, 1.1). This mismatch between the cited spectral theory and the actual behavior in long-range tasks highlights the need for more precise analysis, which our work provides. Based on our results, the authors could have: (1) directly explained the instabilities they likely encountered prior to applying the truncation, and (2) provided a more accurate foundation for their initialization scheme. While our work does not yet offer a full theoretical explanation for the success of LRUs, such an explanation remains a central and timely open question. We believe our paper takes a necessary first step by clarifying misconceptions about Glorot that appear in [1].

• "Efficiently Modeling Long Sequences with Structured State Spaces" [2]: This paper introduced the influential S4 model, which also involves a combination of initialization and parametrization for discretized linear SSMs (applying only linear temporal operations). To validate their strategy, the authors compare it against Gaussian initialization. Unlike in [1], they explicitly state in Section 4.4 that they scale down the Gaussian variance until the result stops exploding. In doing so, they are empirically addressing the same type of instability that our work analyzes theoretically. Specifically, we consider eigenvalues $\lambda_i$ of a Gaussian matrix $A$, which translates to the eigenvalues $(1 + \Delta/2 \cdot \lambda_i)/(1 - \Delta/2 \cdot \lambda_i)$ in discretized SSMs. This makes the spectral behavior of Gaussian initializations directly relevant to the paper's experiments setup. We view this as another example of recent impactful work where theoretical understanding of Glorot initialization was lacking, leading to ad hoc empirical fixes rather than principled best practices. Our work aims to fill this gap.

(3) Broader mathematical context: Recent advances in deep learning theory have led to significant interest in the behavior of large Gaussian matrices, particularly in the infinite-width limit, where neural networks converge to Gaussian processes, and related developments such as Neural Tangent Kernel (NTK) theory. However, the vast majority of this literature assumes that the network's depth is fixed, meaning the number of layers (or matrix multiplications) remains finite. From a mathematical perspective, the setting where both depth and width grow simultaneously is substantially more challenging, as it often involves open questions in random matrix theory. To the best of our knowledge, existing work has only explored this joint limit in the special case of fully-connected networks with independent weight matrices across layers [3,4]. In contrast, our work takes a first step toward understanding the infinite-depth-and-width regime in a more difficult setting of linear RNNs, where the same matrix is repeatedly applied, leading to strong dependencies across layers.

We hope that these considerations help clarify the significance of our contribution. The points outlined above are currently discussed in Sections 1 and 2 of our submission. In the revision, we will rework the introduction to highlight the motivation and relevance of our results more clearly, incorporating the examples presented here.

2. Empirical Validation

The reviewer expressed concern that our empirical validation is incomplete, particularly because we do not compare our rescaled Glorot scheme against: (1) vanilla Glorot with LayerNorm, (2) gated architectures such as LSTM or GRU, and (3) modern architectures like LRU or SSMs. We address each point below.

(1) LayerNorm: We thank the reviewer for suggesting this experiment and have added results evaluating accuracy and training time for a linear RNN with LayerNorm on the IMDB dataset. The results show that, while LayerNorm prevents divergence, it increases training time and slightly reduces performance compared to RNN with rescaled init but without LN. As noted above, incorporating LayerNorm disrupts the linear recurrent structure that enables the hardware-friendly efficiency of recent SSM and LRU architectures. Moreover, the theoretical behavior of RNNs with LayerNorm remains considerably less well understood than that of vanilla linear RNNs. For these reasons, we believe our approach is both more efficient and more transparent.

(2) Gated architectures: Since gated architectures have a fundamentally different structure from the linear RNNs studied in our work, we believe they fall outside the intended scope. These models rely on nonlinear gating mechanisms and multiple interacting components, which complicates fair and meaningful empirical comparison.

(3) Modern architectures: Our initial evaluation was based on the LRU architecture, using the same overall structure as in [1] but without the LRU-specific parametrization. We agree with the reviewer that including results with the LRU parametrization would make the evaluation more comprehensive. Accordingly, we have added these results to the table below.

Overall, we believe the primary contribution of our work is theoretical, which is why we initially kept the empirical section minimal and focused on validating our analysis. That said, we appreciate the reviewer’s suggestions and will expand the empirical evaluation in the revision to include the additional comparisons shown here.

3. Reliance on Gaussian Inputs

The reviewer notes that our results assume i.i.d. Gaussian inputs and raises concerns about their applicability to real-world signals. While real signals indeed often exhibit temporal dependencies, we believe this limitation does not significantly impact the relevance of our analysis, especially for the main question we address: whether the signal explodes over time.

Since we study linear RNNs, we can decompose the input as a structured signal component and additive noise: $x_t = s_t + n_t$. Due to linearity, the RNN output at time $t$ becomes a sum of two terms: one driven by the signal, $h_s^t := \sum_{k=1}^t W^k s_{t-k}$, and one by the noise, $h_n^t := \sum_{k=1}^t W^k n_{t-k}$. Assuming the noise $n_t$ is i.i.d. Gaussian, our theoretical results show that $h_n^t$ explodes under the specified sequence length scaling. Therefore, even if $h_s^t$ remains bounded, the exploding behavior of $h_n^t$ alone is sufficient to cause the total output $h^t = h_s^t + h_n^t$ to explode in norm.

We agree that extending the analysis to more general or structured input distributions could yield deeper insights. We see this as an important direction for future work and discuss it in the Limitations section of our paper.

References

[1] Orvieto et al. "Resurrecting recurrent neural networks for long sequences." ICML (2023).

[2] Gu et al. "Efficiently modeling long sequences with structured state spaces." arXiv preprint arXiv:2111.00396 (2021).

[3] Hanin & Nica. "Products of many large random matrices and gradients in deep neural networks." Communications in Mathematical Physics (2020).

[4] Roberts & Sho Yaida. "The principles of deep learning theory." Cambridge University Press (2022).

We thank the reviewer for their thoughtful feedback and appreciate the positive assessment of our theoretical contributions.

We agree that recurrent stability plays a particularly important role in sequence generation and forecasting tasks. Due to the limited rebuttal period, we were unable to incorporate new benchmarks into our current experimental setup. However, we will include an empirical evaluation of our initialization strategy on these tasks in the camera-ready version.