MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling

We sincerely thank the reviewer for their thoughtful evaluation and constructive feedback. We address the reviewer’s concerns below.

1. Limited applicability to selective scan models

We would like to clarify that the MS-SSM (S6) implementation used in our experiments does, in fact, use Mamba-style input-dependent parameterization as the underlying SSM. Specifically, in our architecture, the parameters of each SSM at each scale are computed based on the original input $x_t$ (detailed in the subsection “Input-Dependent Parameterization”)​, mirroring the selective scan mechanism introduced in Mamba. We did not observe any stability issues during training. As stated in Section 2.2, we maintain a consistent overall state size by setting the effective state dimension of our multi-scale array—$(S + 2) \times N$—equal to the recurrent state size of the baseline Mamba model.

2. No evaluation on long-context NLP tasks

We agree that extending our evaluations to NLP tasks is an interesting next step. For this initial paper, our goal was to validate the core hypothesis of the MS-SSM framework across a diverse set of foundational domains where long-range and hierarchical dependencies are critical such as Long Range Arena (LRA) and hierarchical reasoning tasks like ListOps. As noted in our conclusion, we view NLP benchmarks, such as language modeling, as important future applications to further validate the generality of MR-SSM.

We sincerely thank the reviewer for their thoughtful and constructive feedback. We address your concerns below and will revise the final version of the paper accordingly.

More rigorous comparisons with non-SSM baselines (e.g., attention, RNNs) would better demonstrate its advantages.

We would like to gently clarify that our comparison tables also include a range of non-SSM baselines (e.g., attention, RNNs). All experiments report results for Transformer-based models, and Table 1 includes various RNNs (LSTM, HIPPO-RNN, GRU etc.) and Convolution baselines, Table 2 includes an extensive list of Transformer variants (BigBird, Longformer, etc.), Table 3, 4 also include RNN baselines (e.g. LSTM and LRU). While these results provide broad baselines, we chose to focus our analysis on the comparison with state-of-the-art SSMs like Mamba, as our contribution focuses on architectural improvements to these SSM (RNN) architectures.

The paper focuses primarily on accuracy, but efficiency in sequence modeling involves more than just accuracy…

We appreciate the reviewer's point on efficiency. In our paper, we refer to “efficiency” in terms of asymptotic complexity. MS-SSM maintains the same favorable complexity and scalability as the underlying SSMs (e.g., Mamba), with linear-time inference and parallelizable training. As discussed in the "Complexity" subsection, our multi-scale convolution layer adds only a minimal overhead of $\mathcal{O}(LKS)$ in computation and $\mathcal{O}(KS)$ in parameters per layer, where $L$ is the sequence length, $K$ is the kernel size, and $S$ is the number of scales. In practice, we typically use a small kernel size $K \in [3, 5]$ and set $S=3$, keeping this overhead negligible.

To further support our claim of memory efficiency, we introduce the mean mixing distance (Section 3.1) as a measure of the effective receptive field (ERF). As reported in Table 6, MR-SSM achieves a significantly higher ERF than Mamba, indicating its stronger ability to attend to long-range dependencies.

Additionally, the performance gains are inconsistent: Table 1 shows marginal improvements, while Table 2 reports large gains.

We thank the reviewer for highlighting this observation across different tasks. We see this not as a weakness, but as an important finding that validates our core hypothesis: multi-scale modeling is most impactful in tasks with long-range or hierarchical dependencies. Tasks with Large Gains: The substantial improvements are seen on tasks that are inherently hierarchical (ListOps in Table 2, where MS-SSM achieves a significant gain over Mamba ($63.04$% vs. $38.02$%) or require modeling very long-range dependencies (LRA in Table 4. MS-SSM achieves $14.42$% absolute improvement over Mamba). Tasks with Smaller Gains: Image classification tasks such as sCIFAR, ImageNet-1K in Table 1, are included to demonstrate the generalizability of MR-SSM beyond standard sequence benchmarks. Given the already high baseline accuracy in these tasks, achieving large improvements is inherently more difficult. Nevertheless, MR-SSM still achieves consistent improvement over Mamba.

A clearer justification for why existing SSMs fail at multi-scale tasks would strengthen the contribution. The writing is clear, but the experimental results need deeper discussion

Thank you for this suggestion. As discussed in the motivation of the paper, SSMs have limited, effective memory requiring larger state sizes for improved recall. We support this by introducing the mean mixing distance metric, which quantifies the effective receptive field in Experiments section and we empirically show that MS-SSM achieves a significantly higher ERF than Mamba in Table 6.

We hope the clarification above addresses your concerns.

We sincerely thank the reviewer for the thoughtful feedback. Below, we address the reviewer's suggestions and concerns in detail.

It is recommended to extend the theoretical analysis to parameterization theorems, similar to the theorem presented in StableSSM

We appreciate the reviewer’s interest in a stronger theoretical justification. Our approach to architectural design is motivated by well-established signal processing theory (e.g., Stationary Wavelet Transform), which supports the use of multi-resolution representations for capturing long-range structure. We agree that a theoretical analysis of the parameterization power of multi-scale SSMs, similar to what StableSSM explores for continuous-time linear time-invariant (LTI) SSMs, would be valuable. However, extending such theoretical tools to multi-resolution settings—especially when combined with data-dependent parameterizations—poses substantial technical challenges and is an exciting direction for future work.

That said, we do attempt to bridge the theoretical-experimental gap through:

• the mean mixing distance introduced in Section 3.1, which quantifies the effective receptive field and supports our hypothesis about multi-scale modeling,
• and comparisons with larger models (e.g., Mamba with 2× state size and parameter count), where MR-SSM still outperforms, suggesting that its gains are not simply due to scaling capacity but to improved inductive bias in capturing long hierarchical structures.

Moreover, our scale-dependent initialization scheme is a practical application of the theory connecting an SSM's eigenvalues to its effective memory (Agarwal et al., 2023). That is, SSMs for coarse-grained scales are initialized with eigenvalues closer to 1, prioritizing long-range memory needed to capture global trends, while fine-grained scales are initialized with smaller eigenvalues to prioritize shorter effective memory.

A minor issue is the absence of substantial NLP tasks for language modeling

We agree that extending our evaluations to NLP tasks is an interesting direction. In this work, our main focus was to assess the core capability of multi-resolution SSMs on long-context modeling and hierarchical structure learning, which is directly measured via Long Range Arena (LRA) and hierarchical reasoning tasks such as ListOps. As noted in our conclusion, we view NLP benchmarks, such as language modeling, as important future applications to further validate the generality of MR-SSM as it is fully compatible with causal language modeling architectures.

We appreciate the reviewer for the positive feedback and for recognizing the simplicity and effectiveness of our proposed approach. We address the questions and suggestions below:

Can you quantity “minimal computation and model parameters increase”?

The additional computation primarily stems from the 1D dilated convolutions and scale mixing MLPs, which are lightweight and parallelizable. As stated in the "Complexity" paragraph at the end of Section 2, the multi-scale convolution operation adds a computational overhead of $\mathcal{O}(LKS)$ and $\mathcal{O}(KS)$ parameters per layer, where $L$ is the sequence length, $K$ is the kernel size, and $S$ is the number of scales. To make this concrete, for most of our experiments (e.g., on ImageNet-1K, sCIFAR, ListOps), we typically used a small kernel size of $K \in [3, 5]$ and $S=3$ scales, which introduces only a marginal increase in parameter count. We will highlight these quantitative parameters and computational overhead in the appendix.

Please highlight best (and second best) results in Table 4

Thank you for the suggestion. We will update Table 4 to visually highlight both the best and second-best results for each task in bold and underline, respectively, for improved readability. We also wish to clarify the main takeaway from this table. As noted in the footnote, our primary goal was not necessarily to achieve state-of-the-art results across the entire LRA benchmark, but to conduct a fair and direct comparison against a similar SSM-based architecture, Mamba. On the LRA benchmark average, MS-SSM achieves a $14.42$% absolute improvement over Mamba ($86.73$% vs. $72.30$%).

Additional Note: We agree with the reviewer that extending our evaluations to NLP-related downstream tasks is an interesting future direction. In this work, our primary focus was to assess the core capability of multi-resolution SSMs on long-context modeling and hierarchical structure learning, which is directly measured via Long Range Arena (LRA) and hierarchical reasoning tasks such as ListOps. As noted in our conclusion, we view NLP benchmarks, such as language modeling, as important future applications to further validate the generality of our method.