Response to Reviewer c4kg

We appreciate the reviewer’s thoughtful review and constructive comments. In our responses, we address the following concerns: a detailed comparison with SSM-based methods, the generalizability of VMamba, sensitivity to hyper-parameters, and potential for integration into various frameworks.

Detailed Comparison with SSM-based Methods

In Table 1 of the main submission, we have already compared our method to S4ND [2] and Vim [4] in terms of the number of parameters, train throughput, and the Top-1 accuracy on ImageNet-1K. To provide a more comprehensive evaluation, we additionally conduct comparison on FLOPs and the memory usage, and the results are reported in the following table.

Moreover, we also compare the performance (both effectiveness and efficiency) change with increasing input resolution in Figure 1 in the attachment. For qualitative comparison, we visualize the ERF of S4ND and Vim, and the results are shown in Figure 2 in the attachment. We will include these results and the associated analysis in the revised manuscript.

[Performance comparison between VMamba and benchmark methods. $\dagger$ indicates the value is measured with mix-resolution while Vim does not support training with mix-resolution (values in the brackets are results obtained with fp32).]

Additional Related Studies

We thank the reviewer for bringing these inspiring studies to our attention. We will include references to these papers in the revised version.

Versatility of VMamba

Due to our limited computational resources, we have focused on conducting experiments on benchmark tasks in vision modeling. However, we recognize the importance of illustrating the potential of the proposed method in more generalized tasks.

A preliminary literature review of recently proposed SSM-based approaches in vision tasks, along with our private communications with researchers in the field, highlights the potential of the 2D selective scan technique (SS2D) introduced in this study. SS2D does not make specific assumptions about the layout or modality of the input data, which allows it to be generalized to various tasks. For example, SS2D can process video data by traversing a spatial-temporal plane of frame patches. To our knowledge, recent studies leveraging scanning patterns analogous to SS2D have shown success in various tasks, including image restoration and multimodal data understanding, in addition to those mentioned in the question. We will add these results to the final version and cite their works if they are published by then.

Despite not being inherently prohibited, we anticipate challenges in directly migrating SS2D to diverse downstream tasks due to varying requirements. Bridging the gap between SS2D and these tasks, along with proposing a more generalized scanning pattern for vision tasks, is a promising research direction. We will include this discussion in the revised version, hopefully to provide readers with some inspiration.

Clarity in Mathematical Derivations

Due to limited space, we have included detailed proofs in the appendix and will provide more rigorous and clearer derivations in the revised version. We also recognize the significance of providing more intuitive and accessible explanations, and will include them in the revised version.

Sensitivity ot Hyper-parameters

According to our experience, we have not found any hyperparameter to which VMamba is particularly sensitive. This observation is also supported by the ablation results on single hyper-parameters (initialization approach in Table 11 and activation function in Table 15) as well as different combinations (Tables 12, 13, and 14) included in the Appendix.

We conducted additional experiments on the influence of the learning rate, and the results are reported in the following table. We will include this discussion in the revised version.

[The performance of VMamba-T with different learning rate. Results marked by $\dagger$ is the default setting used in the submission. All the models here are trained on [SERVER 2].]

Potential of Integrating into Various Frameworks

The core of VMamba lies in the design of the SS2D module, which aims to bridge the gap between 1D sequence scanning and 2D plane traversing, rather than specific architectural configurations. SS2D can function as an end-to-end token mixer, allowing it to be integrated into various mainstream backbone networks in computer vision.

Indeed, integrating SS2D into existing frameworks requires additional considerations. One critical aspect is the numerical precision settings in the model, which significantly impact performance and computational speed. Another important factor is the inclusion of normalization layers to stabilize the training process. We will include these points in the revised version to assist researchers who may want to build upon our work.

Response to Reviewer ZfkW

We appreciate the reviewer’s thoughtful review and positive comments about our study. In the following sections, we address the reviewer’s primary concern regarding the lack of ablation on design choices and clarify several other issues raised.

More Ablation on Design Choices

First of all, we would like to clarify that our primary reason for modifying multiple hyper-parameters simultaneously is to ensure that the number of parameters and FLOPs remain comparable, facilitating a fair comparison between different model variants. In Table 5 (corresponding to Figure 3 (e) in Section 4.3 of the main submission), we detail the configurations used to optimize the overall performance of VMamba, balancing both effectiveness and efficiency rather than isolating the impact of each hyperparameter.

However, we sincerely acknowledge the importance of analyzing the significance of each individual hyperparameter and architectural design choice on the overall performance. We plan to conduct more comprehensive experiments to extend the results of experiments isolating each hyperparameter in Table 5, and include those results in future versions of this study.

To address the issue mentioned in the reviewer's comment, we have conducted additional experiments to analyze the influence of changing a single variable. The results are reported in the following table. Values for Step (e.1) and Step (e.2) are copied from Table 12 and Table 14 in the appendix, respectively, while Step (d.1) and Step (d.2) present new results obtained during the rebuttal process.

Details of accelerating VMamba. $\dagger$ and $\ddagger$ indicate the value is obtained from [SERVER 1] and [SERVER 2], respectively. All other experiments are conducted on [SERVER 0].

Clarification of the Statement

There is a typo in the mentioned statement, and the correct version is "such modification prevents the weights from being input-dependent, resulting in a limited capacity for capturing contextual information" (i.e., change from "input-independent" to "input-dependent"). We will fix this typo and conduct thorough proofreading to prevent further errors in the revised version.

Detailed Explanation of the Referred Statement. S4ND [2] extends S4 [1] to higher-dimensional contexts through a straightforward outer product, with the essential condition being that the SSM in S4 is implemented using 'accelerated convolution'. Specifically, S4 utilizes a global convolutional operation to compute the output of the SSM, denoted as $\mathbf{y}$, given the input data $\mathbf{u}$ and the kernel function $\mathbf{K} = \mathbf{C}e^{\mathbf{A\Delta}}\mathbf{B}$.

Efficient computation is achieved if $\mathbf{A}$ has an 'Normal Plus Low-Rank' (NPLR) form and $\Delta$ is constant, enabling the low-rank approximation of $\mathbf{K}$ in the spectral domain, allowing the convolution to be efficiently computed with Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (IFFT). Conversely, if $\Delta$ is input-dependent or context-aware, the kernel function will no longer maintain a low-rank form in the spectral domain, leading to a substantial increase in the convolution computation time.

The efficacy of a recurrent model is significantly limited by its capacity to effectively compress context [3]. By leveraging the task of selective copying and employing Induction heads, Mamba [3] illustrates that LTI models lack content awareness. Consequently, it is concluded that a fundamental principle in developing sequence models is selectivity: the context-aware capability to emphasize or disregard specific inputs within a sequential state.

Repetitive Reference Entries

We will address the issues mentioned in the comment and conduct thorough proofreading to prevent further errors in the revised version.

Thanks to the authors for providing a thorough and solid response to all my concerns. Based on all the reviewers' comments and the rebuttal, I am happy to keep my rating as a Strong Accept (8).

Response to Reviewer dvTH

We thank the reviewer for the constructive comments and are glad they appreciate the performance of VMamba. Below, we clarify the reviewer’s concerns regarding the detailed structure and the influence of hyper-parameters on VMamba.

Usage of Positional Embedding

To clarify, VMamba does not use positional embedding. Sorry for any confusion caused, and we will make this clear in Section 4.1 Network Architecture (lines 129-136) of the revised version as follows:

"Subsequently, multiple network stages are employed to create hierarchical representations" $\rightarrow$ "Without further incorporating positional embedding, multiple network stages are employed to create hierarchical representations."

Explanation of Performance Improvement

In step (e) shown in Figure 3 for Section 4.3, we manage to save parameters and FLOPs by reducing the expansion ratio and eliminating the entire multiplicative branch. This allows us to increase the number of layers from [2,2,2,2] to [2,2,5,2], resulting in the observed performance improvement. Similarly, in step (g), lowering the expansion ratio enables us to increase the depth of the model with additional layers. For step (f), the performance improvement is due to the larger expansion ratio and the addition of extra DWConv blocks. By using a smaller d_state value, we keep parameters and FLOPs comparable. We will provide more details on these points in the revised version.

Influence of d_State. In Section H.3 of the Appendix, we explore the impact of adjusting the d_state parameter on VMamba. Table 12 shows that increasing d_state from 1 to 4 yields only marginal performance gains while significantly reducing throughput, indicating a substantial negative impact on VMamba's computational efficiency. To mitigate this, we propose lowering the ssm_ratio parameter to reduce overall network complexity. We find the best performance at (d_state=8, ssm_ratio=1.5).

Influence of ssm_ratio. We also analyze VMamba's sensitivity to the ssm_ratio parameter, with results presented in Table 13 of Appendix H.4. The results clearly indicate that lowering the ssm_ratio significantly reduces performance but also greatly increases the inference speed. On the other hand, adding more layers boosts performance but also decelerates the model.