Thank you for providing constructive feedback and for your praise of our detailed analysis and thorough review of related work. We are also glad you agree that identifying advantages of SSMs in learning long-term memory is important. We will now address your concerns about the novelty and scope of our work and our engagement with prior time series literature.

One major argument concerns the applicability of our work to nonlinear models. In the paper, we begin by referencing prior work that has taken a similar approach to studying the linearized versions of deep learning architectures, which already exhibit non-trivial properties, and later connect them with their nonlinear counterparts. We think this approach is beneficial for fundamental theory work as we can reduce the phenomenon of study to its simplest form, in our case learning long-term dependencies with deep SSMs. We develop our theoretical framework for studying how these models learn long-term dependencies and then connect our findings with experiments. We study the linear system identification task where a teacher model is used to generate input-output pairs to train a student model. Figure 3A-B shows experiments with the linear models described by the theory, however, Figure 3C extends this setting to include models with nonlinear activations between layers, which shows qualitatively the same training dynamics and outcomes for the nonlinear models.

Re prior work comparison The goal of our paper is to understand how deep SSMs learn long-range dependencies. While our work shares some common ground with Zucchet and Orvieto’s (2024) paper, our contributions extend significantly beyond their analysis. Specifically:

• We extend their curse of memory analysis for a single layer by incorporating the evolution of autocorrelation of inputs across many layers of a deep SSM
• We introduce novel analysis of learned temporal dependencies across layers of deep SSMs, moving beyond the single-layer case that restricts analysis to eigenvalues of the dynamics (A) matrix
• We demonstrate that analyzing single-layer dynamics gives an incomplete picture for deep SSMs, as we show that group delay affects the long-range dependencies learned by deep SSMs. We show this delay is affected by both the depth of the model and the B and C matrices, explaining their role in learning long-range dependencies. To our knowledge, this has not been addressed by any prior work.

Re depth of models in experiments We agree it is important to include experiments with deeper models to solidify our analysis. We first note that Figure 5 in the appendix of the original submission shows one such experiment in which we trained deep SSMs with up to 6 layers, showing the decomposition of timescales across layers. The experiments we have done so far indicate that no new behavior beyond what we observed for the 3 layer case in the main paper is observed for deeper models. We have now included 12-layer deep SSM experiments in Figures 5 and 6 in the appendix.

Re engaging with prior work You point out an existing connection between the analysis of input autocorrelation functions (ACF) and state space models. We recognize that in conventional time-series analysis, the ACF provides a useful summary of a linear model, which can aid in model selection, fitting, and interpretation (Shumway & Stoffer, “Time Series Analysis and its Applications”, Fourth Edition). Here, we use the ACF to study the properties of deep SSMs, namely, the connection between the ACF across layers and the evolution of the hidden state variance across layers of the deep SSM.

Re duplication of content You raised concerns about duplication of content from prior work, namely, the paper Zucchet and Orvieto (2024). Note that our Lemma 1 differs from that of Zucchet and Orvieto (2024) by the assumption of symmetry of the inputs $u_t$, which also changes its proof. We included the full derivation so that the proof of proposition 1 is fully contained in the main paper. That being said, we have now included a citation in Lemma 1 to clarify that it is a variation on the lemma introduced in Zucchet and Orvieto (2024).

Re figure 5 In the original submission, Figure 5 showed six separate deep SSMs trained on the same linear system identification task generated by a one-layer teacher SSM. We show the learning trajectory of models with varying numbers of layers, starting at six, ending at one, with each model converging to a solution. The deeper SSMs learn eigenvalues that are generally smaller than those of the teacher, with the exception being the one-layer student, which converges to the teacher parameters exactly.

Thank you again for your feedback. We believe we have addressed the concerns raised in the review and hope you will reconsider your assessment. Please let us know if we can provide further clarifications.

Dear reviewer m1Wn,

Thank you again for taking the time to review our paper and provide detailed feedback. Please let us know if we can answer any more questions before the paper revision deadline on November 27th. The deadline for you to submit any further comments is December 2nd.

Thank you for taking the time to read our paper and provide a detailed review. We will now respond to the concern that the paper lacks cohesion between individual theoretical analyses.

We are interested in providing a theoretical account of how deep SSMs learn long-range dependencies, which we develop in order of increasing complexity.

• First, we examined how stacking multiple layers in a deep SSM affects hidden state variance. Larger hidden state variance leads to sharp loss landscapes which hinders learning. We show that deeper architectures intensify the variance explosion problem (Zucchet and Orvieto (2024)) due to increased cross-layer signal correlation.
• This naturally leads to our analysis of group delay in Section 3.2 to characterize signal propagation through stacked LTI layers. This reveals previously unrecognized mechanisms of memory beyond individual layer timescales that enable deep SSMs to capture long-range dependencies. To our knowledge, this is a novel contribution to the deep SSM literature.
• Section 3.3 refines our analysis by proving the equivalence between deep and shallow SSMs. This framework yields two key insights:
  1. Timescales are shared across layers in deep SSMs, suggesting the falsifiable prediction that they can learn long-range dependencies without always requiring near-unity eigenvalues at each layer.
  2. The B and C matrices mediate cross-layer interactions. While diagonalizing the A matrix in the wide, shallow form decouples the time evolution of each state dimension, the input-output map remains unchanged—revealing that B and C matrices mediate these cross-layer interactions. Though they don't affect A's eigenvalues, they influence group delay—and thus, the system's memory characteristics.

Re shallow-deep equivalence in training We showed two equivalent representations of a deep SSM encode the same temporal dependencies. This equivalence allows us to more readily study the specific differences in training between these two representations, specifically a single-layer SSM and a deep multi-layer SSM. Our experiments show that when learning long-range dependencies encoded by a teacher model, the multi-layer SSM has favorable training dynamics. This is to be expected and can be analyzed through the lens of prior learning dynamics literature (Saxe et al., 2014). In the paper, we explain this discrepancy through the sensitivity of parameter initialization. Specifically, in a deep SSM, B and C are initialized within the standard range (around 1) and then stay within this range across all layers throughout training. If we took this learned deep SSM and applied the transformations discussed in Section 3.3, we would see that the diagonal one-layer (wide) SSM representation will have B and C with significantly larger values, which may explain the difficulty of training the model in this (temporally) equivalent representation.

Re typos Thank you for pointing out the typo in Equation (5), indeed k should be used throughout the equation. We have also corrected the K=2 statement that was present in a previous version of the paper. The current version includes a general analysis for any number of layers K.

Re Figure 1C and layernorm Thank you for pointing out a possible source of confusion in Figure 1C. Indeed, introducing a layernorm would mitigate the signal explosion to some extent, although it’s important to note that layernorm is applied with respect to the latent state dimension, not with respect to time, which is why even with layernorm, deep SSM architectures use techniques like input regularization (LRU) and discretization (S4, S5) to regulate hidden state variance across layers.

Re consistency of theoretical predictions The lines you referenced in the main paper were specifically related to the prediction that in the teacher-student experiment in Figure 3A, the two-layer SSM learns eigenvalues that are smaller at each layer compared to those of the one-layer teacher. This is consistent with the role of group delay in learning long-range dependencies for deep SSMs. While there are still some open questions, we have provided an initial theoretical framework through which to study the role of B and C in more depth, afforded by the equivalence between deep and shallow linear SSMs and the P matrix transformation from equation (9). The implication of this is that B and C can regulate the group delay across layers, effectively allowing the model to regulate the effects of exploding hidden state variance, as analyzed in Section 3.1.

Our analysis has so far focused on real-valued SSMs, which have become common-place for working with language data. Nevertheless, we are happy to include analyses of complex-valued models in the appendix in our final submission.

Thank you again for providing thorough and constructive feedback. We hope we have addressed your major concerns. Please let us know if we can provide additional details or clarifications.

Dear reviewer ZW4U,

Thank you again for taking the time to provide detailed feedback on our paper. We have now added additional figures showing experiments with complex-valued SSMs to the appendix. Please let us know if we can answer any more questions before the paper revision deadline on November 27. The deadline for you to submit any further comments is December 2nd.

Thank you for the detailed and constructive feedback. You highlighted the contribution of this work to the understanding of how deep SSMs learn long-range dependencies. We are glad you found our analysis of how different components of deep SSMs contribute to learning long-range dependencies insightful. We will now address the individual comments and questions.

Re teacher-student experiments The goal of the teacher-student experiments is to study how known dynamical systems are learned by deep SSMs of varying latent state size and layer depth. The task is to map white noise inputs $u_t$ to the teacher-generated targets, $y_t$. The teacher is a fixed SSM with parameters chosen such that it encodes a dynamical system with long-term dependencies, i.e., with eigenvalues λ≈1. Importantly, this experimental setting allowed us to show that a deep SSM student can learn smaller timescales at each layer while still capturing long-term dependencies, thanks to the effects of group delay. This shows that understanding deep SSMs requires considering both group delay and layerwise dynamics.

Re model depth in experiments Thank you for the suggestion on increasing the depth of the models used in our experiments. We have now extended our experiments to 12 layers in the appendix, showing similar effects of depth (and group delay) on learning long-term dependencies with deep SSMs.

Re Section 3.1 Yes, your definition of R^{(k)}(\Delta) is what we used in the paper. We will make sure to clarify this definition in the final version.

Re Figure 1 and 2 The role of Figures 1 and 2 is to study the general properties of signal propagation in deep SSMs, supplementing the theoretical analysis in Section 3. The models used to produce these figures were not trained on any particular task, their parameters were fixed manually. The model outputs in Figure 2A-B were produced by passing an impulse input (one-hot vector) and a white noise input through the fixed model in Figure 2C.

• Figure 1 grounds our understanding of the curse of memory problem for deep SSMs, extending the analysis from (Zucchet and Orvieto, 2024). Specifically, we show that the inputs to layers in a deep SSM become more correlated regardless of the initial input autocorrelation function. To connect this to our experiments with white noise inputs, this shows that for sufficiently deep SSMs, changing the correlation of the inputs would not have a significant effect on the nature of the experimental setup. We will include this in our final submission.
• Figure 2 then shows the effect of stacking multiple state space layers on the propagation of the signal for a simple impulse and white noise, both showing the effective delay that's introduced, complementing our theoretical analysis of group delay.

Re LRU experiments The final experiment in our paper trains varying parameterizations of the Linear Recurrent Unit (LRU) model on the teacher-student task, adapted from Zucchet and Orvieto (2024). The LRU is a deep SSM architecture with input regularization designed to ameliorate the explosion of hidden state variance across the network. We trained multiple LRU models, varying the number of layers and their latent state size while keeping the number of trainable parameters roughly equal. The goal of this experiment was to reproduce the setting described in Figure 3B while using a deep SSM architecture used in practice. This experiment reproduces the same result shown in Figure 3B for the LRU, highlighting the differences between training parameter-equivalent representations of a deep SSM. We will include these details in our final submission.

Re proposition 2 Proposition 2 describes the form of a deep SSM in the frequency domain, specifically through the frequency response. The frequency response is later used to derive the form of the group delay, upon which we build our theoretical analysis of how deep SSMs learn long-range dependencies. We recognize that the relevance of proposition 2 might not be obvious without considering the derivation of group delay found in the appendix. We will make this connection more clear in the final version. We also thank the reviewer for requesting more clarity around this section. The parameters can generally vary across layers, in which case we would use the ^{(k)} superscript to distinguish parameters at layer k.

Thank you for taking the time to review our paper. We hope this response addresses your questions. Please let us know if you have remaining questions we can clarify.

Dear reviewer XgTv,

Thank you again for taking the time to provide detailed and constructive feedback. Please let us know if we can answer any more questions before the paper revision deadline on November 27th. The deadline for you to submit any further comments is December 2nd.