V2M: Visual 2-Dimensional Mamba for Image Representation Learning

W1 and Q1: Analysis of the simplification

Thanks for this constructive suggestion. In terms of computational accuracy, we admit that calculating $h^2_{i,j}$ with only the vertical component is less precise than using the full component as specified. However, the iterative calculation process of $h^2_{i,j}$ in Eq. 8 in the revised manuscript also includes the input term $x_{i,j}$, which is derived from Eq. 5 in the revised version and inherently incorporates modeling information from the horizontal direction. This term also represents a distinct difference between the horizontal and vertical components. Therefore, the calculation of $h^2_{i,j}$ only reduces the weight of the horizontal input to some extent without completely losing all horizontal information, and the network may also strengthen the learning of the horizontally low-weighted components through the optimization of $\mathbf{A}_2$. Additionally, the direct consequence of this simplification is that it enables the use of hardware optimization algorithms for 1D SSM to achieve acceleration, which facilitates the training process. In the future, we will also explore methods for performing precise 2D SSM calculations on hardware. (Please see Section 3.2 in the revised manuscript marked in red.)

W2: Performances

Thanks for this comment. We believe that the performance improvements on the ImageNet-1K and downstream tasks should not be considered insignificant. For instance, LocalMamba achieved a ~0.2% improvement over the baseline in downstream object detection tasks.

In Table 4, we employ 2D SSM in four directions to ensure that the class token can obtain effective information from all four directions. As we place the class token in the middle position in the main experiment, reducing the modeling in one direction would result in the class token lacking information from the tokens that precede it in that direction, as shown in Figure 4. This design is closely related to our 2D modeling of the image. For a more fair ablation, we conducted experiments using the mean pooling to obtain the class token and obtained an accuracy of 75.5% as shown in Figure 4. We further evaluated the performance of only using the upper-left direction and achieved a classification accuracy of 75.3%. This shows that reducing the number of directions hardly affects the performance under this setting.

W3 and Q2: Ablation and computation speed

Thanks for this nice suggestion. Our 2D SSM block is an integrated unit and cannot be disassembled into modules (for example, removing parts of the structure within V2M would prevent obtaining the final classification results). However, we have supplemented the model speed indicator (throughput) in addition to FLOPs in the revised version. We admit that V2M may slow down the model to a certain extent, which corresponds to the increase in FLOPs as we have illustrated (refer to our limitation part). However, V2M also enhances the classification performance. We will further optimize the speed of V2M in the future. (Please see Section 4.3 and Table 5 in the revised version marked in red.)

Dear reviewer 5M99,

I hope this message finds you well. We greatly appreciate the valuable feedback and suggestions you have provided so far. As the deadline approaches, we are eager to receive your feedback on our response and revisions. If possible, we kindly request an update on the progress of the review to ensure we can address any further comments or revisions promptly.

Should you require any additional information or assistance from our end to help facilitate the review process, please do not hesitate to let us know. Your insights are highly valuable to us, and we genuinely appreciate your time and effort in reviewing our paper.

Thank you for your patience and cooperation. We are looking forward to hearing from you soon.

Warm regards,

Submission2798 Authors.

W1: Character corners

Thanks for this suggestion and we apologize for possible misunderstanding. The top right corner of the characters in Figure 1 (1 and 2) represents the dimensions of the state space equation. We have extended the one-dimensional state space equation to two dimensions, modeling it from both the horizontal and vertical aspects (1 denotes horizontal and 2 denotes vertical). The bottom right corner (i and j) represents the spatial position of the state, for example, the horizontal state at position (i, j) is determined by the horizontal state and vertical state at position (i, j-1). We have provided these explanations in the revised version. (Please see Figure 1 in the revised manuscript marked in red.)

W2: Function names in formulas

Thanks for this comment. We have adopted the Roman font for function names in formulas in the revised version. (Eq. (9-11) in the revised version.)

W3: Citations

Thanks for this nice suggestion. We have updated the citations to their final publication in our revised manuscript.

The authors' rebuttal resolves my concerns. And I keep my scores.

W1: Novelty

We want to clarify that although both V2M and VMamba claim to preserve the two-dimensional structural information of visual data, VMamba and other vision Mambas adopt similar measures by unfolding the images row by row and column by column, followed by modeling in these directions. Therefore, in essence, they still largely destroy the two-dimensional information during the modeling process, merely presenting a combination of one-dimensional modeling results. In contrast, V2M alters the modeling logic at the lower level, utilizing two-dimensional state space equations for feature extraction of image data. Although we have simplified the modeling of the two-dimensional state space equations in our implementation to obtain some one-dimensional results, we do so only to leverage the acceleration process of one-dimensional SSM and have eliminated the operation of flattening images by rows and columns (we no longer involve the operation of concatenating tokens that are spatially distant from each other). In general, V2M fundamentally represents a more suitable modeling method for two-dimensional images

W2 and Q2: Simplification

Thanks for this comment. In terms of computational accuracy, we admit that calculating $h^2_{i,j}$ with only the vertical component is less precise than using the full component as specified. However, the iterative calculation process of $h^2_{i,j}$ in Eq. 8 in the revised manuscript also includes the input term $x_{i,j}$, which is derived from Eq. 5 in the revised version and inherently incorporates modeling information from the horizontal direction. This term also represents a distinct difference between the horizontal and vertical components. Therefore, the calculation of $h^2_{i,j}$ only reduces the weight of the horizontal input to some extent without completely losing all horizontal information, and the network may also strengthen the learning of the horizontally low-weighted components through the optimization of $\mathbf{A}_2$. Additionally, the direct consequence of this simplification is that it enables the use of hardware optimization algorithms for 1D SSM to achieve acceleration, which facilitates the training process. We will also explore methods for performing precise 2D SSM calculations on hardware. (Section 3.2 in the revised manuscript marked in red.)

W3: Performances

We believe that an improvement of 0.2%-0.4% in classification results on the ImageNet-1K dataset should not be considered marginal. Many existing works (e.g., LocalMamba) have also achieved similar improvements over the baseline as our method. Additionally, our method is plug-and-play and compatible with existing Vision Mamba backbones. We also conducted comparative experiments by incorporating our method into EfficientVMamba. We achieved a 0.4% improvement (to 76.9%) over EffVMamba-T (76.5%) on the ImageNet-1K dataset, demonstrating the applicability of our method. (Please see Table 1 in the revised manuscript.)

W4 and Q1: Computation speed

Thanks for this nice suggestion. In terms of memory, we exercised strict control during the model design phase, resulting in a parameter count consistent with the baseline and thus no significant difference in memory usage. As for the speed-related metrics of the model, we have provided the corresponding FLOPs in our paper, which to some extent reflects the complexity of our algorithm. In addition, we have conducted speed assessments for V2M under both Vim and VMamba backbones, with throughput as the metric. We admit that V2M may slow down the model to a certain extent, which corresponds to the increase in FLOPs as we have illustrated (refer to our limitation part). However, V2M also enhances the classification performance. We will further optimize the speed of V2M in the future. (Please see Section 4.3 and Table 5 in the revised version marked in red.)

W5: Presentation

We apologize for possible misunderstanding. Splitting Equation 7 (Eq. 3 in the revised manuscript) into Equations 9 (Eq. 5 in the revised manuscript) and 10 (Eq. 6 in the revised manuscript) merely consists of calculating the first and third terms, as well as the second term of Equation 7 separately. That is, Equation 9 corresponds to the first and third terms of Equation 7, while Equation 10 corresponds to the second term. For the transition from Equation 11 to Equation 12, we focus on the computation of $h^2_{i,j}$, which is where the simplification of our method lies—in restricting the calculation of $h^2_{i,j}$ to be related only to the vertical aspect of the state and not to $h^1_{i,j}$. Therefore, Equation 12 is the derivation of the evolution of $h^2_{i,j}$. We first obtain the result of the first term in Equation 11 through Equation 12, and then add it to the second term of Equation 11 to get the final output. We may not have described this process very clearly in our paper, and for this, we apologize again and have provided further elaboration in the revised version.

W4: Performance -- Reproduced results of Vim-T

Thanks for this comment. We have conducted a detailed examination of the previous training configurations. Due to the limited number of GPUs at our disposal, in order to make more efficient use of the limited resources to verify the effectiveness of the scheme, we reduced the original batch size of 128 to 64 per card when conducting all Vim-T and V2M-T experiments. This may have led to a potential decrease in performance. However, we ensure that the hyperparameters used for Vim-T and V2M-T are exactly the same, making the comparative experiments fair. Additionally, we will also retrain and evaluate under the original settings and make the necessary modifications in the subsequent updated version of the paper. We deeply apologize for possible confusion in our paper.

W5: Performance -- Improvements over baselines

Thanks for this comment. V2M is a plug-and-play method that can be applied to various vision Mamba frameworks, not limited to Vim and VMamba, among others. We conducted experiments on Vim, VMamba, and LocalMamba because they are representative works in the vision Mamba series and have attracted considerable attention. Additionally, we have demonstrated that V2M can further enhance the performance of EfficientVMamba as you have mentioned and acknowledged, and we will also explore applying V2M to more baselines in the future.

W6: Computational speed

Thanks for this comment and your interest of our work. We are modifying the underlying CUDA code to perform a 2D parallel scanning process, which helps in precise calculations and accelerates the V2M framework.

W7: Presentation

Thanks for this suggestion. We have further modified our presentation of these equations in the revised manuscript.

Summary

In general, we are grateful for your valuable comments on our work and the interesting aspects you mentioned about the V2M solution. We have provided as detailed an explanation and analysis as possible for the shortcomings you pointed out, and we will continue to explore further in the future. Therefore, we hope that you can reassess our work, which will also encourage and support our subsequent explorations.

Dear reviewer 7MsE,

I hope this message finds you well. We greatly appreciate the valuable feedback and suggestions you have provided so far. As the deadline approaches, we are eager to receive your feedback on our response and revisions. If possible, we kindly request an update on the progress of the review to ensure we can address any further comments or revisions promptly.

Should you require any additional information or assistance from our end to help facilitate the review process, please do not hesitate to let us know. Your insights are highly valuable to us, and we genuinely appreciate your time and effort in reviewing our paper.

Thank you for your patience and cooperation. We are looking forward to hearing from you soon.

Warm regards,

Submission2798 Authors.

W1: V2M's performance on long sequences

Thanks for this suggestion. We conducted long sequence fine-tuning with reference to Vision Mamba (Vim), ensuring a training epoch of 30 for comparison purposes. Specifically, our V2M-T model achieved an accuracy of 78.8% compared to 78.3% by Vim-T in the long sequence setting for ImageNet-1K, while V2M-S further reached an accuracy of 82.0% as opposed to 81.4% by Vim-S. This demonstrates the effectiveness of the V2M framework on long sequence data. Due to time constraints, we plan to explore the long sequence settings with more backbone architectures in the future. (Please see the revisions in Section 4.3 and Table 4 marked in red.)

W2: Computation speed

Thanks for this suggestion. We have conducted speed assessments for V2M during the rebuttal under both Vim and VMamba backbones with throughput as the metric as Table 5 in the revised version. We provided the corresponding FLOPs in the orginal paper, which to some extent reflects the complexity of our algorithm. We acknowledge that V2M may slow down the model to a certain extent, which corresponds to the increase in FLOPs as we have illustrated (refer to our limitation part). However, V2M also enhances the classification performance on the ImageNet-1K dataset. We will further optimize the speed of V2M in the future. (Please see the revisions in Section 4.3 and Table 5 marked in red.)

Q1: Equation references and background formulation

Thanks for this nice suggestion. We have utilized the correct equation references and shortened the background formulation part in the revised version of our paper.

Q2: Positional embedding

Thanks for this comment. We have verified the effectiveness of V2M on spatial information modeling by removing the positional embedding compared with Vim. Specifically, we have tested the corresponding performance drops of V2M-T and V2M-S without the positional embedding, which is 76.0% (-0.2%) and 80.3% (-0.1%) on ImageNet-1K, respectively. This verifies the spatial modeling ability of our V2M.

Q3: Classification head

Thanks for this comment. We employ a common form of classification head. Specifically, we first use an MLP to project the feature dimensions to 1000 (the number of classes in the dataset), and then after operations such as log softmax, we constrain it with the ground truth through cross-entropy.

Q4: V3M

This is an exciting suggestion. V3M can be considered as a direct extension of V2M in the dimension of state. While V2M performs state inference in both horizontal and vertical directions, V3M, tailored for video data, requires the addition of a temporal dimension for more complex modeling processes. In the computational process, we can also refer to the approach of V2M, simplifying the three-dimensional inference into the form of three one-dimensional SSMs, which facilitates the acceleration of the training process. As for the specific results of V3M on video data, we will delve into this in the future, and we appreciate again for raising this valuable suggestion.

Dear reviewer RmnR,

I hope this message finds you well. We greatly appreciate the valuable feedback and suggestions you have provided so far. As the deadline approaches, we are eager to receive your feedback on our response and revisions. If possible, we kindly request an update on the progress of the review to ensure we can address any further comments or revisions promptly.

Should you require any additional information or assistance from our end to help facilitate the review process, please do not hesitate to let us know. Your insights are highly valuable to us, and we genuinely appreciate your time and effort in reviewing our paper.

Thank you for your patience and cooperation. We are looking forward to hearing from you soon.

Warm regards,

Submission2798 Authors.