Dear reviewer thank for your time in assessing our paper, and for the valuable comments.

Reviewer[bKH5] Point #1: "Theoretical analysis of why the proposed initialization scheme enables SSMs to achieve better performance remains limited. In Proposition 1, the authors demonstrate that a previous method, S4D-Lin, fails to solve the continuous copying task when the discretization parameter is selected inappropriately. However, the authors do not provide a theoretical analysis of the learning dynamics of SSMs under their proposed initialization. As a result, it remains unclear why their method enables SSMs to converge to better optima and achieve improved generalization performance."

In fact, as shown in the center panel of Figure 4, our method actually performs marginally worse than S4D‑Lin under the optimal configuration of $\Delta$, which, as demonstrated in Proposition 1, converges to the optimal solution for S4D-Lin. The advantage we want to highlight in that motivating experiment on the Continuous Copying task is that our initialization S4D-DFouT does not require tuning of $\Delta$, achieving competitive results where previous initializations are severely sensitive to the selection of $\Delta$ hyperparameter.

To really motivate with a real example the hardness of selecting $\Delta$, we also discuss in Section 5.3 the following case. In the literature, S4D-Lin has repeatedly underperformed, failing to solve challenges like PathX-128 that other initializations could handle. We show that this shortfall is not an intrinsic limitation of the linear initialization, but rather of the wrong selection of the $\Delta$ hyperparameter in the literature. To capture a period of 128, according to our insights, we require $\Delta > 2/128 \approx 0.015$; however in the literature [2] a range of [0.0001, 0.01] was used, just by increasing the range over the calculated threashold we suceed in learning the task (marked with "†" in Table 1).

Reviewer[bKH5] Question #1: Typos and erratas. It seems that there is a typo in Eq.(8). In the proof of Proposition 1, the definition of $d_n$ seems to be confusing.

• The typo on the sign on Eq.(8) will be corrected.

• Thank you for pointing this mistake in the Proposition. We corrected the statement to emphasize the assumption that is no decay in the poles. Therefore, the Proposition 1 is as follows:

Proposition 1.
Let the S4D‑Lin convolutional kernel under zero-order hold (ZOH) discretization be defined as
$\overline{K}[l] = 2\mathrm{Re} \left( \sum_{n=0}^{N-1} C_n \, \overline{\lambda}_n^\ell \right)$, where $\overline{\lambda}_n$ are the discrete-time eigenvalues and $C_n \in \mathbb{C}$.

If we choose $\Delta = \frac{2}{\tau}$ and $\alpha = 0$ (no decay in the poles), for some $\tau \in \mathbb{Z}_{>0}$, and set $C_n = 1$ for all $n$, then $\overline{K}$ exhibits a spike at $l = \tau$, satisfying $|\overline{K}[l]| < |\overline{K}[\tau]|$ for all $l \ne \tau$.

Moreover, if we treat $C = (C_0, \dots, C_{N-1})$ as trainable parameters, then the associated Vandermonde matrix defined by $V_{\ell n} = \overline{\lambda}_n^\ell$ is well-conditioned, and gradient descent on $C$ converges at a linear rate.

Dear reviewer thank for your time in assessing our paper, and for the valuable comments.

Reviewer[eFzT] Point #1: Consider alternative SSMs models: The paper heavily emphasizes discretization issues in diagonal state space models but does not consider alternative state space methods that avoid discretization entirely, such as LRU (Resurrecting Recurrent Neural Networks for Long Sequences, Orvieto et al.).

Thank you for pointing out the reference LRU [4]. Although we included LRU in the results comparison in LRA (Table S4 in Supplementary Material), this work definitely requires better coverage in the related work.

We briefly recall the main findings of the LRU paper. As the Reviewer points, LRU does not rely on the discretization of a latent continuous-time system but directly optimizes the magnitude and oscillation frequencies independently. LRU method preserves (and in a sense is motivated by) the Glorot initialization from RNN, which in the limit behavior is de-facto uniform spectral initialization in the complex unit disk.

Yet our contributions are complementary to this work, both in the findings and in the motivation. We first provide a mechanistic interpretation of S4D models from the frequency perspective. Furthermore, we incur on the analysis on other eigenvalue distributions rather than uniform. We show the importance of the frequential coverage aligned with the spectrum of the data and point out some design flaws of previous S4D initializations as the aliasing and Hermitian symmetry. On the practical side, we provide a layer‑by‑layer analysis of what these models learn (Figure 5), observing that initial layers focus explicitly on capturing the frequency present in the data, revealing emergent “attention” patterns in these initial layers. Leveraging these insights, we propose our initialization.

[4] Orvieto, Antonio, et al. "Resurrecting recurrent neural networks for long sequences." International Conference on Machine Learning. PMLR, 2023.

Reviewer[eFzT] Point #2: Selective State Space Models: Discretization concerns are particularly relevant in selective state space models like Mamba (Mamba: Linear-Time Sequence Modeling with Selective State Spaces, Gu and Dao) where discretization is still used, yet the paper overlooks these types of models.

In Mamba's "selective" state‐space formulation, the initialization of the state matrix A plays a much smaller role than in earlier SSMs architectures. While Mamba does discretize a latent continuous‑time state matrix A, its input‑dependent selection operator $\Delta(x_t)$ dynamically adjusts the effective timestep, making this model less dependent to the initialization scheme. Particularly, in that work [3], authors experiment with three initializations of the SSMs: the real variant is S4D-Real, the complex variant is S4D-Lin, and a random initialization, achieving very similar performance in all cases (see section 4.6.2 of the original work). In Mamba (and more generally in S6 models) the matrix A is not tied to a single fixed discretization step; instead, selectivity produces a context‑dependent state update at every step.

In that non‑stationary domain, providing model interpretability complicates due to the input‑dependent dynamics that evolve at each timestep. Therefore, we limit our work to time-invariant dynamics.

[3] Gu, Albert, and Tri Dao. "Mamba: Linear-time sequence modeling with selective state spaces." arXiv preprint arXiv:2312.00752 (2023).

Reviewer[eFzT] Question #1: Evolution of poles throughout the training: While it may be outside the current scope of the paper, I’m curious about how the state matrix weights evolve throughout training. Analyzing this could provide further insight into why the proposed initialization leads to better performance, and whether its benefits persist or diminish as training progresses.

Thank you for the question. We depict in Figure S9 in the Supplementary Material the final distribution of the poles layer-by-layer after training in sCIFAR for the different initialization schemes (our proposed and the baselines). Our major observation is that in all initializations, the angles of the poles (resonant frequencies) remain largely unchanged from their initial values. However, poles exhibit a significant change in the decay, presenting a smaller radius. In the particular task of sCIFAR, we believe that some poles are learned to decay rapidly to fine-control the initial coefficients of the kernel, which, as reported in Figure 5, are particularly relevant for performance in this task (especially the first 32 coefficients). Furthermore, the initial distribution is preserved: with S4D-Lin showing the entanglement between decay and frequency, resulting in spiral-like clustering of the poles; S4D-Inv spreading poles evenly around the circle; and S4D-DFouT maintaining the original half-plane distribution of the poles throughout training.

In the updated manuscript, we will provide an extension of Figure S9, covering the distribution of the poles (with corresponding norms) for different stages throughout the training (current regulations prevent us from sharing figures in the rebuttal).

We appreciate the reviewer’s comments, which helped us clarify both the motivation and broader implications of our work. We believe these revisions better position the paper’s contributions and respectfully invite reevaluation based on the new context.

Dear reviewer, thank for your time in assessing our paper, and for the valuable comments.

Reviewer[YT22] Point #1 : "Mechanistically how the initialization leads to better models through training?"

We understand the concerns of a circular justification—“better initialization implies better performance, therefore the initialization must be inherently superior.” Here are two findings we observed:

- Preservation of the initial spectral distribution. We depict in Figure S9 in the Supplementary Material, the final distribution of the eigenvalues of A layer-by-layer after training in sCIFAR. We observe that in all initializations, the angles of the poles (resonant frequencies) remain largely unchanged from their initial values. However, poles exhibit a significant change in the decay, presenting a smaller radius after training. At the end of the training, S4D-Lin still shows the entanglement between decay and frequency, resulting in spiral-like clustering of the poles; S4D-Inv spreads poles evenly around the circle; and S4D-DFouT keeps the original half-plane distribution of poles throughout training. This observation indicates that the model preserves its initial spectral basis and primarily refines modal amplitudes during optimization. Consequently, we argue that a comprehensive frequency coverage at initialization contributes ultimately to elevated final performance. This reasoning thereby avoids the circular argument.

- Enhanced utilization of the spectral domain. We also observe in Figure S9 that compared to S4D-Lin and S4D-Inv, the poles in DFouT that contribute most significantly (as measured by their $\mathcal{H}_\infty$-norm) are distributed more uniformly across both quadrants of the upper half-plane, rather than being concentrated at low frequencies. Consequently, DFouT more effectively leverages high-frequency modes that capture fine-grained input features.

Although these observations do not fully explain the learning dynamics, they contribute to a more precise understanding of the role of initialization beyond mere final performance. We appreciate the reviewer’s suggestion to undertake this deeper analysis.

Reviewer[YT22] Point #2 : "Can we design a task with known frequencies and show that poles approach known poles more quickly and reliably across pole distributions?"

Actually, the Continuous Copying task was motivated by a similar reason. In the Continuous Copying, we know a priori the frequencies (fundamental and harmonics) required to generate a delay kernel exhibiting a spike at the desired $\tau$.

One insight we have observed during training—as introduced in Point 1—is that the pole angles (resonant frequencies) remain largely unchanged throughout training. In this context, we also note that succeeding in the Continuous Copying task requires spectral coverage of those fundamental and harmonic frequencies from initialization. However, as illustrated in Figure 3, the spectral distribution of previous S4D initializations is strongly influenced by $\Delta$, making it difficult for this initialization to learn that kernel unless the order $N$ of the model is sufficiently large (we tested up to $N=1024$). In the case of S4D‑Lin, the hyperparameter $\Delta$ dictates the pole distribution: small $\Delta$ values cluster poles at low frequencies, while large $\Delta$ values suffer from aliasing. By contrast, S4D‑DFouT is more robust because it provides a uniform distribution of poles across the spectrum, providing initial coverage of these relevant frequencies otherwise hard to optimize during training.

Reviewer[YT22] Point #3-#4 : "Can we look at raster-scan like tasks (sCIFAR) and preferentially add poles around the image width multiples?"

Thank you for the insightful question. While our primary aim was to design a data‑agnostic initialization that avoids some of the pitfalls of prior schemes, the question of using the model insights to exploit the spectral structure of the data is very on the point.

To test this hypothesis, we have designed an initialization S4D-Spectrum where the real part is preserced from DFouT and the imaginary part matches the discretize spectrum distribution (normalized) of the data.

Results. Regarding sCIFAR, its spectrum presents a fundamental frequency $\Omega=2\pi/32$ and clear harmonics at multiples of that fundamental (Figure S7 in the Supplementary Material). However, using this distribution S4D-Spectrum results in a noticeably less performant model. The insights we provided in Section 5.2 of the main paper can be useful in explaining such behaviour. In that experiment, we measure how much each SSM actually contributes to the output by computing its H-norm (Figure 5, right). We observe that the SSMs operating in the first layer close to the fundamental frequency, present noticeably larger H-norms. However, in successive layers, the frequencies of importance move toward lower bands. The signal experiences progressive temporal mixing throughout the layers toward lower frequencies. In this regard, placing poles only at the marked frequencies in -all layers- does not benefit to the model. It seems to us, though, that setting a temperature hyperparameter to smooth that distribution could lead to better results.

Reviewer[YT22] Point #5 : "Inspired by the better distribution of frequencies, is there a way to design even better models for multiple sampling frequencies given the Nyquist/Hermitian observations (cf. SC35)?"

We extended the experiment by training S4D-Spectrum on SC35. In this setting, the spectrum of the data is more evenly distributed, yielding a more uniform pole coverage compared to sCIFAR. As a result, we observe performance on par with other initialization methods. S4D-Spectrum already incorporates Nyquist and Hermitian symmetry. We design its frequencies to span only the positive half of the spectrum—sampling the DFT from $[0,\pi]$—therefore, it does not suffer from aliasing.

Reviewer[YT22] Point #6 : "What is the distribution of learned eigenvalues across layers? As the computation is fundamentally distributed across layers."

As previously introduced in Point #1, we already depict in Figure S9 in the Supplementary Material the final distribution of the eigenvalues of A layer-by-layer after training in sCIFAR for the different initialization schemes. Our major observation is that in all initializations, the angles of the poles (resonant frequencies) remain largely unchanged from their initial values. However, poles exhibit a significant change in the decay, presenting a smaller radius after training. In the particular task of sCIFAR, we believe that some poles are learned to decay rapidly to fine-control the initial coefficients of the kernel, which, as reported in Figure 5 (left), are particularly relevant for performance in this task (especially the first 32 coefficients).

Reviewer[YT22] Point #7 : "If we were to measure the total change of eigenvalues across training (or limit it with a specific trust region) do you get better performance?

Empirically —as commented in Point #1—we find that the vast majority of change on the poles occurs in the decay (radius); and even due to the time limitation, we could not complete the experiment setting a trust region, we expect no substantial improvement in the accuracy. Our observations suggest that ensuring good spectral coverage upfront is key.

Reviewer[YT22] Point #8 : "Why is S5 excluded as a diagonal variant in Table 1? This is a fairly major point because S5 does use a diagonal parameterization, and outperforms DFouT-initialized models in most cases."

We agree that the composition of Table 1 is not fair to S5 [6], since this method also uses a diagonal state matrix. We will complete Table 1 with S5 results, but also include our proposed initialization with S5 architecture. However, we would like to point an important observation concerning this method that highlights our main findings. Namely, the authors of S5 also reported in Table 7 of their work, results for different initializations as S5-Lin and S5-Inv in addition to S5 (which follows HiPPO-N matrix)—results that we have also collected in our Table S7. Interestingly, S5-Lin again underperforms on tasks as PathX-128, as the authors noted in their Appendix E.3. Therefore, rather than focusing on overall performance, we propose rewriting that table to highlight that previous Fourier-based initializations such as S4-Lin or S5-Lin have repeatedly underperformed in the literature due to some of the limitations we identify in this paper.

Reviewer[YT22] Question #1 : "Doesn’t the locality of the receptive field influence the decay rate?

We agree that the discussion in the decay is also very relevant for the work. We compiled some insights about the decay in the Supplementary Material (Figure S9, commented in Point #6); yet those conclusions should have a space in the main document.

Yes, if the eigenvalues responsible for capturing 32-pixel do not decay (i.e., if they preserve energy), then we would also see repeating peaks in the kernel—not only at 32, but at 64, 96... Indeed, this is exactly what we observe in the kernels of Figure S7. However, those later peaks are much attenuated due to the decay of the corresponding poles. We also provide an explanation for that behaviour. Learning a locally constrained receptive field can be beneficial for these models, as it mimics how CNNs operate: where early convolutional layers specialize in detecting localized features, while deeper layers progressively integrate these patterns into more abstract global representations.

Dear reviewer thank for your time in assessing our paper, and for the valuable comments.

Reviewer[V8Gu] Point #1 : Limited ablation beyond SSMs: While the paper provides thorough comparisons within the diagonal SSM family, it lacks ablations against non-SSM models like Transformers. Such comparisons would help clarify the broader applicability of the proposed method.

In fact, in Table S4 in Supplementary Material, we already include a comparison on Long Range Arena tasks with standard Transformer architecture and some of its variants as Performer, Reformer, and MEGA. In the revised version of the paper, we plan to include these variants in the main article, Table 1.

Reviewer[V8Gu] Point #2 : Motivation could better connect to practical use cases: Although the motivation is theoretically compelling, it would benefit from examples or discussion showing potential industrial applications or real-world scenarios where spectral control via initialization is especially valuable, especially compared to other models, including transformers, MoE.

We actually provide two tasks that pertain to industrial applications or real-world scenarios. First, we evaluate on the Speech Commands dataset (SC35) for audio signal classification, where our proposed initialization scheme achieves state-of-the-art in zero-shot resampling (Table 2). Second, we assess physiological signal prediction using the BIDMC dataset, targeting respiratory rate, heart rate, and blood oxygen continuous time-series data. As shown in Appendix Table S3, a variant of our decoupling method also achieves state-of-the-art results in this domain.

Prior work [1] notes that SSMs often underperform in tokenized or language tasks but excel in continuous domains such as signals or time series. Therefore, we are confident that our method has a big potential in industrial applications.

[1] - Dao, Tri, and Albert Gu. "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality." International Conference on Machine Learning. PMLR, 2024.

Reviewer[V8Gu] Question #1 :Extension to Low-rank: Can the proposed initialization strategy (S4D-DFouT) be extended to low-rank or more general structured SSMs beyond the diagonal case? If so, what challenges or adjustments would be required?

Introducing off‑diagonal or low‑rank components does make the spectrum, and thus mode behavior, less interpretable than in the purely diagonal case. In structured/low‑rank SSMs as S4, modes become linear combinations of eigenvectors, and low‑rank terms $UV^T$ perturb the poles. As a result, it is difficult to attribute specific frequency behavior to individual parameters—for example, pinpointing which row captures a given frequency or understanding how the amplitude and phase of that mode are governed by $C_n\bar{B}_n$. So, to answer the question, we believe the extension is possible, but we do not know how to do it in an "explainable" way.

Reviewer[V8Gu] Question #2 :Discussion on underperformance: What types of tasks or data characteristics might cause S4D-DFouT to underperform? Are there known limitations where this initialization is less effective?

In Section 6, we briefly outline the current limitations of SSMs—a discussion we will expand in the final draft. We show that many Long Range Arena benchmarks regarding sequential images can be “solved” by exploiting local spectral biases (for example, capturing frequencies corresponding to pixel offsets within image rows), rather than genuine long‑range modeling. In contrast, tasks that span the full frequency spectrum—such as ListOps or permuted sequential images like psCIFAR (see the flattened spectrum in Supplementary Figure S7)—remain far from solved. Although SSMs still outperform other architectures as Transformers on the latter problems, these are far from being considered solved.

Reviewer[V8Gu] Question #3:Computational cost: Could you provide a comparison of computational cost (e.g., training time, memory usage) versus accuracy to better illustrate the efficiency gains of S4D-DFouT relative to other models?"

A brief mention of the computational cost is done in the introduction, but we would increase the information in the manuscript. Diagonal SSMs present a significant cost reduction w.r.t. non-diagonal variants. As deeply inspected in [2], by materializing the Vandermonde matrix of A, the computation cost of computing the kernel in diagonal SSMs can be obtained in O((N+L) log2(N+L)) arithmetic operations. Our work builds upon these efficient diagonal variants; we do not propose architectural modifications, but rather aim to provide a mechanistic interpretation of those models. As such, the computational/memory cost remains unchanged.

[2] Gu, Albert, et al. "On the parameterization and initialization of diagonal state space models." Advances in Neural Information Processing Systems 35 (2022): 35971-35983.

We appreciate the reviewer’s comments, which helped us clarify both the motivation and broader implications of our work. We believe these revisions better position the paper’s contributions and respectfully invite reevaluation based on the new context.

Thanks to the explanation provided, I believe making the motivation more clear in the introduction by including industrial examples that can better benefit from your model can attract broader audience. Also, I believe the comparison with other models including transformers should not be in supplementary and it should appear in the main paper. Overall, I find the paper interesting and think it has its own merits. I have modified my score