We sincerely thank Reviewer tmKq for the valuable suggestions. We make responses as follows.

Q1: Evaluation is only carried out for image classification on a single benchmark dataset (ImageNet-1K).

A1: Thanks for the helpful suggestion. In response, we add semantic segmentation as a downstream task to further validate the generalization ability of our method beyond image classification.

Semantic segmentation：

Setting: As no downstream-task checkpoints of Vim are publicly available, we follow the official codebase settings to fairly evaluate both Vim and AMVim. ToMe is used as the token reduction strategy, with training-free pruning results reported under identical conditions.

results:

Analysis:

• AMVim achieves better performance than Vim in the non-pruned setting, indicating stronger representational capacity.
• Under token reduction, AMVim consistently outperforms Vim by a significant margin (up to +6.7 mIoU), validating its robustness and generalization ability in dense prediction tasks.

Object detection： Given the lack of publicly available Vim checkpoints for object detection, we would need to re-train both Vim and AMVim from scratch. Given the high resolution (1024×1024), each run takes ~8 days on 4× RTX 4090 GPUs.

Due to limited computational resources and other experiment priorities, this evaluation could not be completed within the rebuttal period. However, we acknowledge its importance and plan to include it in the final version.

---

Q2: Somewhat limited technical novelty.

A2: Thanks for the question. We agree that both varying scanning directions and window-based mechanisms have been explored individually. However, our key novelty lies in introducing the multi-scale concept into SSM-based modeling, which enables emergent architectural capabilities that cannot be achieved by either technique alone. We clarify this from two perspectives:

1. Pruning robustness:

Setting: We choose Vim and LocalVim (window-based scanning) as baselines. All models are evaluated using ToMe as the pruning strategy, and we report their training-free Top-1 accuracy across various token reduction ratios.

Top-1 Accuracy (%):

Analysis:

• Although LocalVim employs a window-based scanning strategy, AMVim consistently achieves higher accuracy across all token reduction ratios, particularly under aggressive pruning (e.g., 46% reduction). This indicates that window-based scanning alone is insufficient for ensuring robustness.
• As shown in the ablation studies (Section 4.4), simply varying scanning directions yield limited improvement.

2. Efficiency:

Setting: All experiments are conducted fairly on a single NVIDIA GeForce RTX 4090 GPU with an image resolution of 224×224. We use ToMe as the default token reduction strategy.

Throughout (image/s):

Analysis:

• AMVim incurs a modest throughput drop (~180 img/s) due to window-based scanning.
• AMVim remains ~2.5× faster than LocalVim, despite both using window mechanisms.
• This demonstrates that AMVim offers a superior trade-off between robustness and efficiency for practical deployment.

---

Q3: Table 1: The 75.3 and 79.5 accuracies are reported for AMVim with ToME? If so, which token reduction ratio?

A3: Thanks for the question. The Top-1 accuracies of 75.3% and 79.5% reported in Table 1 correspond to AMVim with ToMe under a token reduction ratio of 0.17, and are obtained after fine-tuning. These results are derived follow the Hidden State Alignment setting, as mentioned at the end of the first paragraph in Section 4.2. In contrast, all other results presented in the tables and figures (outside of Table 1) are based on training-free evaluations.

To avoid confusion, we will revise Table 1 in the final version for greater clarity. The updated table is shown below:

---

Q4: Table 4 and section 4.4: Is the random scanning order kept the same between the different token reduction ratio runs?

A4: Thanks for the question. For a fair comparison, the random scanning orders are generated in advance and kept fixed throughout training and during evaluation across all token reduction ratios.

---

Q5: Have the authors also experimented with different scanning configurations per layer?

A5: Thanks for the insightful suggestion. Indeed, LocalVim explores per-layer scanning variations. We address this question from both the robustness and efficiency perspectives:

• Robustness: As shown in the robustness results (refer to the top-1 accuracy table in A2), this strategy does improve robustness compared to the baseline Vim. However, it still lags significantly behind our method in terms of pruning robustness.
• Efficiency: More importantly, adopting layer-wise scanning variations typically incurs higher computational costs. This is reflected in the throughput results (see the throughput table in A2), where LocalVim shows a ~2.5× lower than our AMVim. Such a design conflicts with our core objective of maintaining high efficiency.

---

Q6: Have the authors also considered applying the method to Vision-LSTMs (ViL, Alkin et al., ICLR, 2025)?

A6: Thanks for the valuable suggestion. Our window-based multi-scale selective scanning mechanism is model-agnostic and can be flexibly integrated into ViL-style architectures. Specifically, in ViL, the original flip operation in the even-numbered blocks could be replaced with our window-based scanning strategy to enhance multi-scale information exchange between global and local contexts.

In this work, we focus on Vim as a representative Mamba-based backbone to enable a controlled and fair comparison within the family of state-space models. While we have not incorporated our method into ViL within the limited rebuttal timeframe, we plan to explore a ViL-based variant in the camera-ready version to further demonstrate the generality of our approach.

---

Q7: Minor observation: The multiscale scanning mechanism is fist mentioned on line 59 as an advantage in the introduction, but at that point it is not explicitly linked to the window-based nature of the scan.

A7: Thanks for pointing out this potential ambiguity. To improve clarity and prevent misinterpretation, we will revise the beginning of the seventh paragraph (Line 50) in the introduction as follows:

"In this work, we propose AMVim, a novel Asymmetric Multi-scale Vision Mamba for pruning robustness. To enrich spatial information diversity, we introduce a multi-scale scanning mechanism that integrates a window-based scanning strategy in one branch while retaining sequential scanning in the other."

This revision explicitly links the multi-scale behavior to the window-based design from the outset, making it clear that the multi-scale perception arises directly from the window-based scanning mechanism, rather than from an additional or explicit multi-scale module.

---

Q8: The experimental section is confusion at times.

A8: Thanks for pointing out these sources of confusion. We address each issue in detail below:

Inconsistent terminology: We will explicitly change the label "prune" to "ToMe" in Table 3 to clearly distinguish token merging from random pruning and ensure consistency across all tables.

Inconsistent precision: We will unify the formatting of reported FLOPs to one decimal digit throughout the paper for clarity and consistency.

Missing standard deviations: We acknowledge this oversight and will include standard deviation values for experiments involving randomness in the final version to improve the statistical reliability of the results.

We sincerely thank Reviewer f6Uw for the constructive comments. Below, we provide our detailed responses.

Q1: There is no evaluation of real-world latency or memory.

A1: Thanks for the valuable suggestion. In response, we include throughput-vs-accuracy experiments as well as latency scaling tests across different image resolutions to better reflect real-world performance.

1. Throughput vs. Accuracy:

Setting: All experiments are conducted fairly on a single NVIDIA GeForce RTX 4090 GPU with an image resolution of 224×224. We use ToMe as the default token reduction strategy. We choose Vim and LocalVim (window-scan based) as baselines.

Throughout (image/s):

Top-1 Accuracy (%):

Analysis:

• Efficiency: AMVim shows only a slight throughput drop (~180 img/s) compared to Vim, yet delivers ~2.5× higher throughput than LocalVim.
• Robustness: AMVim consistently achieves the highest pruning robustness across all reduction ratios with a substantial margin.
• Conclusion: AMVim strikes a significantly better balance between robustness and efficiency, making it more practical for real-world deployment.

2. Latency scaling tests:

Setting: All experiments are conducted fairly on a single NVIDIA GeForce RTX 4090 GPU. These evaluations are performed under non-pruning settings.

Throughput (image/s):

Analysis:

• Throughput drops for both methods as resolution increases, reflecting Mamba’s linear complexity.
• AMVim incurs only a modest throughput drop compared to Vim, and this gap gradually narrows as the resolution increases.

---

Q2: Reliance on pretrained Vim checkpoints.

A2: Thanks for the helpful suggestion. To assess whether the performance gains of AMVim are intrinsic or merely inherited from Vim, we retrained AMVim from scratch under the official settings.

Setting: We followed the default training configurations provided by Vim. ToMe was adopted as the pruning strategy, and we report training-free results.

Results:

Analysis:

• AMVim trained from scratch shows some degradation in robustness, confirming the benefit of pretraining for pruning stability.
• Despite this, AMVim trained from scratch still outperforms Vim by ~25% on average across all pruning ratios, confirming that its core robustness primarily stems from the proposed multi-scale selective scanning mechanism rather than inherited weights.

---

Q3: Restricted model scale.

A3: Thanks for the valuable suggestion. We include the AMVim-Base results to validate the scalability of our approach on larger model capacities.

Setting: As no official training configuration publicly available for Vim-Base, we fine-tune the Vim-Base model for 150 epochs using a learning rate of 1e-5. ToMe was adopted as the pruning strategy, and we report training-free results.

Results:

Analysis: Our method achieves competitive accuracy at full capacity and shows notable robustness (+1.9%) under token reduction, confirming the scalability and effectiveness of our approach when extended to higher-capacity models.

---

Q4: No downstream-task evidence.

A4: Thanks for the helpful suggestion. In response, we add semantic segmentation as a downstream task to further validate the generalization ability of our method beyond image classification.

Semantic segmentation：

Setting: As no downstream-task checkpoints of Vim are publicly available, we follow the official codebase settings to fairly evaluate both Vim and AMVim. ToMe is used as the token reduction strategy, with training-free pruning results reported under identical conditions.

results:

Analysis:

• AMVim achieves better performance than Vim in the non-pruned setting, indicating stronger representational capacity.
• Under token reduction, AMVim consistently outperforms Vim by a significant margin (up to +6.7 mIoU), validating its robustness and generalization ability in dense prediction tasks.

Object detection： Given the lack of publicly available Vim checkpoints for object detection, we would need to re-train both Vim and AMVim from scratch. Given the high resolution (1024×1024), each run takes ~8 days on 4× RTX 4090 GPUs.

Due to limited computational resources and other experiment priorities, this evaluation could not be completed within the rebuttal period. However, we acknowledge its importance and plan to include it in the final version.

---

Q5: Is there a reason why the figures for AMVim in Table 1 and Table 3 are different?

A5: Thanks for the question. The Top-1 accuracies in Table 1 correspond to AMVim with ToMe under a token reduction ratio of 0.17, and are obtained after fine-tuning. These results are derived follow the Hidden State Alignment setting, as mentioned at the end of the first paragraph in Section 4.2. In contrast, all other results presented in the tables and figures (outside of Table 1) are based on training-free evaluations.

To avoid confusion, we will revise Table 1 in the final version for greater clarity. The updated table is shown below:

---

Q6: Visual analysis via ERF or activation maps.

A6: Thanks for the insightful suggestion. In response, we employed Effective Receptive Field (ERF) visualization to examine the activation patterns of Vim and AMVim.

Setting: We use random noise as input to probe the model’s receptive behavior. The visualizations are generated using a jet colormap for clarity.

Results: Due to the official restriction on uploading figures during the rebuttal phase, we will include the comparative ERF visualizations in the final version.

Analysis:

• Vim exhibits activation primarily at the four corners, especially top-left and bottom-right, with almost no activation in the center, indicating a biased and spatially sparse response.
• AMVim shows significantly broader and more evenly distributed activation, particularly in the center region, demonstrating enhanced spatial coverage and perception.

---

Q7: The term “multi-scale” seems to be appropriate.

A7: Thanks for the thoughtful advice. While we use “multi-scale” to reflect the use of different window sizes for capturing varied contextual ranges, we acknowledge the potential ambiguity. We will revise the term in the final version to better reflect the design for improved clarity.

def save_erf_map(erf_map, filename, colormap='jet'):
    """
    Save an ERF map as an image file.

    Args:
        erf_map: numpy array, shape (H, W)
        filename: path to output image, e.g., "erf_hot.png"
        colormap: color mapping, e.g., 'gray', 'hot', 'jet', 'viridis', 'plasma'
    """
    import matplotlib.pyplot as plt

    plt.figure(figsize=(6, 6))
    plt.imshow(erf_map, cmap=colormap)
    plt.axis('off')  # Remove axes
    plt.colorbar(fraction=0.046, pad=0.04)
    plt.tight_layout()
    plt.savefig(filename, dpi=200, bbox_inches='tight', pad_inches=0)
    plt.close()
    print(f"ERF map saved as '{filename}'")

We sincerely thank Reviewer S9i4 for the insightful suggestions and constructive feedback.

Q1: This paper only evaluated the new model from the perspective of model accuracy, but it should also be important to test its real performance and speedup, not just by using FLOPs. And it'd be better to analyze the performance overhead of the proposed WMS3 block

A1: Thanks for the valuable suggestion. In response, we provide a detailed analysis of real hardware performance by reporting throughput-vs-accuracy results.

Setting: All experiments are conducted fairly on a single NVIDIA GeForce RTX 4090 GPU with an image resolution of 224×224. We use ToMe as the default token reduction strategy. We choose Vim and LocalVim (window-scan based) as baselines.

Throughout (image/s):

Top-1 Accuracy (%):

Analysis:

• Efficiency: AMVim incurs only a modest throughput drop (~180 img/s on average) compared to Vim. This slight overhead mainly stems from the directional switching introduced in one branch of the WMS3 block. However, AMVim achieves approximately 2.5× higher throughput than LocalVim, which applies per-layer scanning configurations that introduce greater computational overhead.
• Robustness: Across all token reduction ratios, AMVim consistently achieves the highest accuracy, significantly outperforming both Vim and LocalVim. This confirms the robustness of the proposed architecture under aggressive token pruning.
• Conclusion: These results demonstrate that AMVim strikes a significantly better balance between efficiency and pruning robustness, making it a more practical and deployable solution for real-world applications.

---

Q2: The evaluations are only conducted on the image classification task. It would be better to show more results on other benchmarks.

A2: Thanks for the helpful suggestion. In response, we add semantic segmentation as a downstream task to further validate the generalization ability of our method beyond image classification.

Semantic segmentation：

Setting: As no downstream-task checkpoints of Vim are publicly available, we follow the official codebase settings to fairly evaluate both Vim and AMVim. ToMe is used as the token reduction strategy, with training-free pruning results reported under identical conditions.

results:

Analysis:

• AMVim achieves better performance than Vim in the non-pruned setting, indicating stronger representational capacity.
• Under token reduction, AMVim consistently outperforms Vim by a significant margin (up to +6.7 mIoU), validating its robustness and generalization ability in dense prediction tasks.

Object detection： Given the lack of publicly available Vim checkpoints for object detection, we would need to re-train both Vim and AMVim from scratch. Given the high resolution (1024×1024), each run takes ~8 days on 4× RTX 4090 GPUs.

Due to limited computational resources and other experiment priorities, this evaluation could not be completed within the rebuttal period. However, we acknowledge its importance and plan to include it in the final version.

---

Q3: As for the window-based multi-scale selective scanning mechanism, there is no overlap between different windows. Will it be helpful to add some new modules to exchange the information between different windows? Just like the shift in the swin transformer.

A3: Thanks for the insightful suggestion. To investigate the potential benefits of cross-window information exchange, we incorporated a Swin-style shift module into our architecture.

Setting: The shift is applied to the window-based branch. Specifically, odd-numbered WMS3 blocks perform cyclic shifts to enable cross-window interaction, while even-numbered blocks retain standard window-based scanning. We fine-tuned this modified variant from our AMVim checkpoint using the same training setup. ToMe is adopted as the pruning strategy, and we report training-free results.

Results:

Analysis:

• Integrating the Swin-style shift results in a clear drop in both accuracy (↓2.6%) and pruning robustness (↓6.2%). This may stem from a misalignment between the uniform cyclic shift and the spatially structured design of our multi-scale mechanism, potentially disrupting intended token interactions.
• We acknowledge the importance of cross-window communication. However, directly transferring the Swin shift from Swin Transformer is suboptimal in our setting. We will investigate more tailored and compatible strategies in future work.

---

Thanks again for the professional, detailed, and valuable reviews! We have done our best to address each of the above concerns and hope our response can resolve them. Please let us know if there are any other questions. We will actively join the discussion until the end of the rebuttal period.

We sincerely thank Reviewer FyAa for the valuable comments and suggestions regarding the presentation of Table 1. Below, we provide our detailed response and corresponding modifications.

Q1: Inappropriate comparison setting in experiments. In Table 1, it is not fair to compare the performance of AMVim with token-pruned results of Vim. A more appropriate comparison would involve evaluating AMVim against various Vim variants without pruning to establish its baseline performance. For pruning robustness, it would be more convincing to compare both Vim and AMVim under the same pruning strategies—such as Token Recognition and Hidden State Alignment. Including such results would strengthen the empirical support.

A1: Thanks for pointing out this confusion. We clarify the setting and motivation behind Table 1 as follows:

Setting: The Top-1 accuracies reported in Table 1 correspond to AMVim with ToMe under a reduction ratio of 0.17, and are obtained after fine-tuning. These results are derived follow the Hidden State Alignment setting, as mentioned at the end of the first paragraph in Section 4.2. In contrast, all other results presented in the tables and figures (outside of Table 1) are based on training-free evaluations. To avoid confusion, we will revise Table 1 in the final version for greater clarity. The updated table is shown below:

Analysis:

• Instead of relying on complex, task-specific pruning techniques, improving the architectural resilience of Mamba enables genric pruning strategies (e.g., ToMe, originally designed for ViTs) to achieve promising performance.
• This finding suggests that resolving Mamba’s architectural fragility offers a more flexible and scalable solution for pruning robustness.

---

Q2: Inconsistency in experimental results. In Table 1, the top-1 accuracy of AMViM-Tiny is reported as 75.3, while in Table 3, the same model is shown to achieve 76.3. Since these refer to the same architecture, this discrepancy should be clarified.

A2: Thanks for the question. As clarified in A1, the 75.3% top-1 accuracy reported in Table 1 refers to AMVim with ToMe, under a token reduction ratio of 0.17, and is obtained after fine-tuning.

In contrast, Table 3 reports the 76.3% accuracy as the non-pruned (full-token) baseline performance of AMVim, without applying any token reduction. Additionally, the 72.8% result in Table 3 reflects the training-free accuracy of AMVim with ToMe under the same 0.17 reduction ratio.

---

Thanks again for helping us improve the paper, and we hope the response can resolve these concerns! Please let us know if there are any further questions. We will be actively available until the end of the rebuttal period.