DAMamba: Vision State Space Model with Dynamic Adaptive Scan

We sincerely appreciate your valuable and encouraging comments on our technical contributions and performance.

Q1: Dynamic adaptation schemes for information aggregation paths are common in the development of deep neural networks nowadays. Examples include Deformable Convolution [1-4], Kernel [5], Transformer [6,7]. The authors have neglected the relevance of dynamic design in these important works to this paper, and lacked the necessary analysis and comparisons.

R1: We thank the reviewer for raising this point. Our paper primarily focuses on the differences and design of scan strategies within Vision SSM architectures, and therefore does not provide a systematic analysis of the connections with Deformable Convolution, Kernel methods, and Transformers.

In fact, Dynamic Adaptive Scan (DAS) shares a similar core idea with these methods—that is, dynamically determining the feature extraction regions based on the input content—although the modeling mechanisms differ:

Deformable Convolution, Kernel methods, and Transformers mainly learn dynamically aggregated receptive regions; in contrast, DAS reorganizes the sequence input at the scan level, learning not only the dynamic aggregation of regions but also their global ordering. This endows Vision SSMs with an enhanced ability to adapt to the 2D image structure.

We will supplement the final version with relevant citations (e.g., [1–7]) and provide additional necessary analysis and comparisons.

Q2: In the related work and background introduction, the authors have also neglected important elements such as Dynamic Neural Networks [8] which are closely related to this paper.

R2: Thank you for the reminder. Works on Dynamic Neural Networks (DNN) [8] indeed share conceptual connections with our proposed DAS method, especially regarding structural dynamicity. We will include this survey paper and some of its referenced works in the related work section.

Q3: As a visual model based on sequence modeling strategy, the development of other existing sequence modeling strategies such as [9-11] is also neglected in the discussion and analysis.

R3: We acknowledge that the current related work section primarily focuses on Vision Transformer and Mamba’s vision adaptation approaches, and does not systematically cover emerging sequence modeling architectures such as RWKV [9], RetNet [10], and xLSTM [11]. These methods have shown strong performance in language tasks and are gradually being adapted for vision applications, making them worthy of attention. We will expand the background introduction on sequence modeling to include these recent advances in the final version.

Q4: In ablation experiments, the performance of ConvPos and ConvFFN can only be treated as a re-validation of the validity of the existing components, and cannot be used as a performance contribution point in the model design. So it seems somewhat unreasonable to treat them as additional independent components for ablation. It might be more appropriate to merge the whole into the baseline as a more standard baseline.

R4: We agree with the reviewer’s opinion that ConvPos and ConvFFN are standard modules in popular vision models. Indeed, we should incorporate them into the “standard baseline configuration” for ablation studies. We will redo the ablation experiments accordingly in the final version as per your suggestion.

Q5: Why does the current baseline model perform so well? From the experimental results, it has surpassed the well-designed Vim-Ti, LocalVim-Ti and EffcientVMamba-T.

R5: This is because our DAMamba adopts a pyramidal hierarchical structure, whereas LocalVim and Vim use an isotropic structure (LocalVMamba and VMamba adopt a pyramidal hierarchical structure). In vision backbones, pyramidal structures typically outperform isotropic structures. For example, the pyramidal VMamba outperforms the isotropic Vim. Additionally, although EfficientVMamba-T also uses a pyramidal structure, under the same parameter size (6M), the baseline used in our DAMamba-P ablation study consumes more FLOPs (1.2G vs. 0.8G), which leads to better performance.

Q6: The ablation experiment also lacks a comparison and discussion of the scanning strategies that have been proposed in the current Vision Mamba-related studies (e.g., those introduced in the reviews [12,13,14]).

R6: To ensure a fair comparison, we did not use ConvPos and ConvFFN, taking DAMamba-P as the baseline and replacing its scan method with other scanning strategies for comparison. As shown below, our method clearly outperforms these alternatives. Due to limited time during the rebuttal, we only compared a few scanning strategies, but we will include more comparisons and discussions of scanning methods from surveys [12], [13], and [14] in the final version.

REF:

1.Deformable Convolutional Networks, 2017

2.Deformable ConvNets v2: More Deformable, Better Results, 2019

3.InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions, 2023

4.Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications, 2024

5.Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation, 2020

6.Vision Transformer with Quadrangle Attention, 2023

7.DAT++: Spatially Dynamic Vision Transformer with Deformable Attention, 2023

8.Dynamic Neural Networks: A Survey, 2021

9.Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence, 2024

10.Retentive Network: A Successor to Transformer for Large Language Models, 2023

11.xLSTM: Extended Long Short-Term Memory, 2024

12.Visual Mamba: A Survey and New Outlooks, 2024

13.A Survey on Visual Mamba, 2024

14.Vision mamba: A comprehensive survey and taxonomy, 2024

The authors have addressed my concerns in their responses. Combined with the better model performance reported for their method, these improvements have led me to raise my overall score.

Nevertheless, the manuscript still suffers from an incomplete survey of related work, which is a weakness that regrettably pervades much of the current neural network design. Specifically, the manuscript still neglects a substantial body of related work on

• deformable operations,
• dynamic network architectures,
• recent sequence-modeling methods in vision, and
• alternative scanning strategies within the Mamba family.

A more complete discussion would help reviewers and readers better appreciate the proposed method’s advantages and situate it against existing approaches.

We sincerely appreciate your valuable and encouraging comments on our technical contributions and performance.

Q1: It is unclear how the initial points are chosen, are they learned?

R1: The reference points are not learned; they are predefined. Specifically, each patch in the image serves as an initial point, with its center position taken as the initial coordinate.

Q2: It is unclear why multiple points are derived from a single sample point and why they thus need to be ordered from left to right and top to bottom?

R2: Each reference point has only one offset point because scanning from top to bottom and left to right better aligns with human intuitive vision. Sorting the points according to their original coordinates—from top to bottom and left to right—helps preserve local adjacency and enhances semantic continuity. Moreover, the top-to-bottom, left-to-right scanning order is also a commonly used, simple, and effective approach in Vision Mamba models.

Q3: The equations are improperly introduced/explained for anyone not familiar with how SSM's work.

R3: We appreciate this feedback. Currently, the method section indeed assumes that readers have some background knowledge of State Space Models (SSM). To improve readability, we will, in the final version:

1.Provide a more detailed introduction and explanation of the SSM working principles; 2. Include pseudocode or clearer module descriptions covering offset generation, offset point feature computation, and sequence rearrangement.

Q4: The method section is insufficient, it is unclear how certain things work (see Questions).

R4: Thank you for your suggestion. In the final version, we will further improve the methodology section, especially by clarifying the data flow for initial point selection, offset calculation, offset point sampling, sorting, and aggregation processes. We will also introduce pseudocode and module diagrams for key components to reduce abstraction and enhance clarity.

Q5: The detection and instance/semantic segmentation results do not include the best competitors from the classification results (like TransNext). Looking at the TransNext paper for example, they seem to get higher performance for these tasks. Unless there is a good reason to leave them out, .it should be clearly noted that on dense downstream tasks the proposed method is not state-of-the-art.

R5: Thank you for your suggestion. Although the TransNext paper appears to achieve higher performance on these tasks, our method attains better accuracy in image classification. Additionally, for models of comparable size such as DAMamba-T, our inference speed is nearly twice that of TransNext-T. We will clearly state in the final version that our proposed method is not state-of-the-art on dense downstream tasks.

Q6: Although the papers mostly centers around the dynamic scan, the convolutional inductive biases seem to be more impactful in the ablation study (+0.8) than the dynamic scan (+0.6).

R6: Yes, for SSM models that excel at global modeling, convolutional local modeling is very important—especially for vision tasks where capturing local details is crucial.

We sincerely thank the reviewer for the constructive comments and positive interaction. Below are our detailed responses to each question.

Q1: The authors indicate that TransNext performs better on dense tasks, while worse on global classification. Could they provide an explanation why that is?

R1: This is because TransNext uses significantly more FLOPs for dense vision tasks on high-resolution images. As shown in the table below, we tested the FLOPs of TransNext-T in object detection and instance segmentation tasks under the Mask R-CNN framework, with all models evaluated at a resolution of 1280×800. The results show that, given the same number of parameters, TransNext-T has far higher FLOPs than our DAMamba-T. Therefore, the original TransNext paper reported only the parameter counts for vision downstream tasks without providing FLOPs.

Q2: Also, the authors mention their method is faster than TransNext, however inference speed is not reported in the paper. Based on the efficiency misnomer by Deghani et al, it might be deceiving to solely rely on parameters and flops. Therefore, it would help if the authors could provide FPS, so that it becomes clear that their method still provides a better trade off between speed and accuracy on dense tasks compared to prior methods.

R2: In Section A.1 Model Efficiency Comparison (line 425 of the paper), we report a trade-off comparison between inference throughput and performance with other popular models. The results show that our model achieves the best balance between performance and speed.

Q3: Furthermore, it would be helpful to see the impact on FPS for the added convolutional inductive biases, as these make the model no longer purely SSM based and might therefore slow it down.

R3: Below are the ablation results of inference throughput for ConvPos and ConvFFN. It can be seen that adding convolutional inductive biases leads to a slight decrease in inference speed.。

We sincerely appreciate your valuable and encouraging comments on our technical contributions and performance.

Q1: Limited Theoretical Discussion: The paper’s justification for DAMamba’s improved spatial modeling remains primarily empirical. While the practical benefits of DAS are evident, the work does not rigorously dissect or quantify how much spatial adjacency (or other higher-order image structure) is preserved versus prior scan methods, beyond anecdotal visualizations and gains in accuracy—this limits mechanistic insight.

R1: We appreciate the reviewer’s suggestion for a deeper theoretical exploration. At present, the paper primarily demonstrates the advantages of Dynamic Adaptive Scan (DAS) through experimental accuracy and visualizations. To further enhance the understanding of its mechanism, we will incorporate the following improvements in the final version:

Introduction of quantitative evaluation metrics: We will introduce a metric based on spatial position reconstruction error to quantitatively assess DAS’s ability to preserve the original spatial structure, in comparison with other scanning strategies such as Sweep Scan or Local Scan.

Inter-class comparative analysis: We will analyze the overlap between DAS scanning regions and ground-truth object boundaries across different image categories to quantify how effectively DAS learns to focus on target regions.

Q2: I believe that DAMamba’s performance may stem from overfitting to a fixed resolution. I’m genuinely curious—what would happen if the authors extended DAMamba to different resolutions? If it fails to maintain strong performance across varying resolutions, then I would consider the overall architectural design to lack sufficient insight.

R2: We thank the reviewer for their attention to the generalization ability of our model. In our image classification experiments, we did use a fixed resolution of 224×224 for both training and evaluation, which is the standard practice adopted by most current methods.

However, in downstream vision tasks such as object detection and instance segmentation (using the Mask R-CNN 1× setting), the image resolutions during both training and testing are dynamically varied. Specifically, the shorter side of each image is resized to 800 pixels, while the longer side is capped at 1333 pixels, with the original aspect ratio preserved.

In addition, we also adopted the more advanced Mask R-CNN 3× training strategy in our experiments. This setup introduces stronger data augmentation mechanisms, including multi-scale training. Concretely, during training, each image is randomly resized to one of 11 predefined scales, with the short side ranging from 480 to 800 pixels and the long side limited to a maximum of 1333 pixels. These dynamically varying input resolutions pose a higher challenge to the model's scale robustness.

Under such settings with dynamic input resolutions, our model still achieves performance significantly superior to mainstream baselines in both object detection and instance segmentation tasks. This strongly validates the robust generalization capability of our model when handling vision tasks across varying resolutions.

Q3: While major design elements are described, there are some notational ambiguities—e.g., in the formalism in Section 3.2 around offset prediction and feature sampling, more explicit pseudocode or implementation details would aid reproducibility for non-experts.

R3: We thank the reviewer for their concern regarding reproducibility. Although we have included implementation code in the supplementary materials, we acknowledge that this may not be sufficient. In the final version, we will provide the following additions:

Pseudocode for the DAS module: We will include PyTorch-style pseudocode that outlines the forward pass of the DAS module, covering key components such as offset computation, coordinate transformation, and bilinear interpolation sampling.

Additional implementation details: We will provide more comprehensive information on aspects such as the initialization of the offset learning range, the gradient flow path, and the interface between DAS and the SSM module.

Q4: While Figure 5 in the supplement visually contrasts regions of attention, the main methods for inspecting what is actually learned by the scanning process are limited to qualitative renderings, lacking systematic quantitative evaluation (e.g., are certain classes of object consistently mis-scanned?)

R4: We thank the reviewer for pointing out the limitations in our interpretability analysis. In the current version, we primarily rely on visualizations of DAS scanning paths on ImageNet samples as intuitive evidence. To enhance the systematic nature of our analysis, we plan to include the following additions in the final version:

Category-level statistical analysis: We will compute the overlap between DAS scanning regions and ground-truth objects across different ImageNet classes, analyzing how the model attends to objects of varying shapes and sizes.

Failure case analysis: We will examine samples with degraded performance (e.g., misclassified images) to investigate whether the DAS scanning regions deviate from the key object areas.

We sincerely appreciate your valuable and encouraging comments on our technical contributions and performance.

Q1: On Line 181, it is mentioned that sample feature vectors of interest are arranged in the order of top to bottom and left to right. This is unclear in the sense that you are doing 4-way scanning, or it's just one scan path with flattening in the mentioned order?

R1: Thank you to the reviewer for pointing out the ambiguity in terminology. In line 181 of the paper, what we meant is: the sampled patches in DAS are ordered from top to bottom and left to right based on their original positions. This ordering is solely used to construct a one-dimensional sequence for SSM processing, which is scanned only once—not a four-way scan in the traditional sense.

Unlike the 4-way scan used in VMamba, our method employs a single scan path for flattening. However, the patch order is dynamically predicted and non-fixed, and then sorted according to their original coordinates (top to bottom, left to right). This preserves local adjacency and enhances semantic continuity. We will further clarify the definition of the scan path in the final version and emphasize that our method does not use multi-directional scanning.

Q2: The effect of DAScan, Convpos, and ConvFFN is only discussed on the smallest version of the model. It would be helpful to see the effect of all three on at least DAMamba-S or DAMamba-B.

R2: We agree with the reviewer’s point. Due to limited computational resources and time constraints during the rebuttal period, we will conduct additional ablation studies on DAMamba-S and DAMamba-B in the final version.

Q3: Other questions are discussed in the weakness section above.

R3: Due to the limited time and computational resources during the rebuttal period, we conducted experiments on DAMamba-T using the same channel and block configurations as VMamba-T. The results are shown below, demonstrating that our model still significantly outperforms both VMamba and LocalMamba:

Due to the same constraints of time and computational resources during the rebuttal, we also performed a comparative study on different scan methods using DAMamba-P. To ensure fairness, we disabled both convpos and convfffn, used DAMamba-P as the baseline, and replaced its scan mechanism with the 4-way VMamba scan and 4-way Local scan. The results, shown below, clearly demonstrate the superiority of our DAS method over the other two scanning strategies.

I thank the authors for the rebuttal. Most of my concerns are addressed. I will raise the score.

Please report the new results and a more thorough analysis of them for the Small model size in the camera-ready.