Q1.Details about the lightweight prediction module

Thanks for your valuable advice. We will depict the prediction module more clearly in the revised manuscript. In our design, each QuadVSS block has its specific prediction module that determines the informative regions of the current layer. The potential reason is that each layer needs to attend to different regions as the feature resolution and network depth change, given that features of different depths and resolutions are known to have different responce patterns in terms of regions and context. The shift scheme also helps the model select the informative regions more flexibly, which can enhance the feature representation quality.

Q2.The effects of more window-level partitions

Thanks for your thoughtful question. The downstream tasks (detection and segmentation) need the pre-training weight of the ImageNet classification. Our two-level window partition strategy mainly considers the input image resolution (224x224) of the image classification task. The input image resolution in dense prediction tasks is generally higher than that in the image classification task. Thus, our module design may not be optimal for downstream dense prediction tasks involving very high resolutions. We will further improve this window partition scheme in future work. A similar phenomenon is observed in Table 4 of the main text: A moderate local window size (14x14) perform better than overly-small (2x2) and overly-large (28x28) window sizes.

Moreover, we conduct a simple experiment in semantic segmentation to explore the proposed question. We split the image into sub-quadrants for all windows using fixed local window sizes. The size is increased four times in the table below. It is worth noting that semantic segmentation results are highly related to the pre-training weights from the ImageNet Classification. Since we do not train each model variant on ImageNet classification, the results may not truly reflect the effects of window size in the segmentation experiment. In the table below, the 8x8 window setting outperforms the 1x1 setting. This indicates that more window partitions can improve the segmentation performance in high-resolution inputs. It is hard to find the optimal local window size for a specific downstream task. The quadtree variant without pertaining achieves the results in between the 8x8 setting and 32x32 setting. It indicates that the two-level quadtree window partition setting may not be flexible enough to handle the high-resolution downstream task. It is because we designed our hyper-parameter by mainly considering the image classification task (lower image resolution). Thus, we will be devoted to designing more flexible window partition schemes to handle various downstream tasks in the future.

Q1.More illustrations of hyper-parameters

Thanks for your valuable advice. We will add more analysis on the hyper-parameters. The impacts of the increased number of window partition levels are explored in Table 5 of the main text. We examine the choice of partition resolutions in the bi-level quadtree-based partition strategy. Experimentally we deduce the optimal resolutions to be {1/2, 1/4} for the coarse- and fine-level window partition, respectively. It is worth noting that the design space is handcrafted and may be extended to more levels. We will explore a more flexible scheme in the future.

Q2.More analysis of learnable parameters

Thanks for your insightful opinion. We will include more analysis on the learnable parameters in the revised manuscript. To show the relationship between model complexity and performance, we plot the performance against flops for different methods and models in Figure 2 of the attached PDF file. We also conduct an ablation experiment on the local and global context embedding for the prediction module. The method of combining both the local and global contexts outperforms the one using only the global context vector.

Q3.More details about training settings

We will add more illustrations of training details in the supplementary materials. To compare fairly with previous vision Mamba methods, our training settings strictly align with VMamba, whose training configurations were inherited from Swin Transformer. The training details for image classification are as follows: QuadMamba models are trained from scratch for 300 epochs, with a 20-epoch warm-up period, using a batch size of 1024. The training process utilizes the AdamW optimizer with betas set to (0.9, 0.999), a momentum of 0.9, an initial learning rate of $1\times 10^{-3}$, a weight decay of 0.05, and a cosine decay learning rate scheduler. Additional techniques such as label smoothing (0.1) and EMA (decay ratio of 0.9999) are also applied. No other training techniques are employed.

For object detection and semantic segmentation, our method also strictly follows VMamba and Swin Transformer. The training details are as follows: Object detection and instance segmentation experiments are conducted on COCO 2017, which contains 118K training, 5K validation and 20K test-dev images. An ablation study is performed using the validation set, and a system-level comparison is reported on test-dev. For the ablation study, we consider four typical object detection frameworks: Cascade Mask R-CNN, ATSS, RepPoints v2, and Sparse RCNN in mmdetection. We utilize the same settings as in previous works: multi-scale training (resizing the input such that the shorter side is between 480 and 800 while the longer side is at most 1333), AdamW optimizer (initial learning rate of 0.0001, weight decay of 0.05, and batch size of 16), and $3 \times$ schedule (36 epochs).

Q1.Clear and fair model comparisons

Thanks for your valuable advice. We regret that our Li/T/S/B naming system may confuse readers compared with other methods. To clearly show our method's advantages, we rename the QuadMamba model variants (Lite -> B1, Tiny -> B2, Small -> B3, and Base -> B4). Our B1 and B2 variants are compared with efficient models, such as lite CNN models and efficient ViT models. Our B3 and B4 maintain similar model sizes as the tiny/small variants in transformer and vision Mamba backbone methods. Thus, our B1/B2 should be compared with those under 10M model parameters. Our B3 should be compared to tiny variants whose model parameters are around 30M, and our B4 to small variants around 50M. For instance, VMamba-T (9.1 GFLOPs) does not correspond to QuadMamba-S (5.5 GFLOPs) but QuadMamba-B (9.3 GFLOPs) instead. To make the comparison clearer, we highlight several comparisons under similar model sizes to show the advantages of the QuadMamba. Our model variants (Lite/Tiny) demonstrate clear superiority on extreme light model levels (under 10M). Moreover, our small and base models achieve comparable and better results compared to methods of similar model sizes.

The detailed plots and comparisons are found in the attached PDF file. We highlight several comparisons under fair conditions to show the advantages of QuadMamba:

Q2.Reported inference time

In Table 9 of the supplementary materials, we benchmark the throughput of the QuadMamba model variants on an A800 GPU platform. The results are also shown below. Compared to the vanilla VMamba model, our QuadMamba model has negligible inference latency. Moreover, the throughput of our lite and tiny models is much higher compared to the efficient CNN and Transformer models, which implies better inference efficiency.

Q1.Significance

Q1A1 Comparing to LocalMamba: How to effectively preserve 2D spatial dependencies is an important challenge in adapting sequence models into the vision domain. Though it has been partially explored by previous works such as PlainMamba [1] and LocalMamba [2], it is non-trivial that our proposed QuadVSS is a data-adaptive and light-weight design for the locality module.

• Data-adaptive: LocalMamba opts for a differential neural network search during training for the optimal locality strategy for each layer. Once trained, the architecture of LocalMamba is fixed. For varied input images, LocalMamba used the same fixed locality strategy. In contrast, our QuadMamba has a learnable data-dependent locality strategy, which means it can dynamically and adaptively generate the optimal scanning sequences for different input data and various downstream tasks.

• Light-weight: LocalMamba, which requires handcrafting the network search space and extra optimization losses, brings much more complexity during training. Differently, our method brings minimal additional complexity in terms of training and optimization, as the quadtree-based scanning module is lightweight.

Q1A2 Gains and improvement to existing Mamba: We believe this is a misunderstanding, and the gain of our method over existing ones is actually significant. We apologize for the possible slight confusion in Table 1's main results, which is due to misalignments between how our model variants and others (like VMamba) are named. For instance, VMamba-T (9.1 GFLOPs) does not correspond to QuadMamba-S (5.5 GFLOPs) but QuadMamba-B (9.3 GFLOPs) instead. To make the comparison clearer, we highlight several comparisons under similar model sizes to show the advantages of QuadMamba. Our model variants (Lite/Tiny) show clear superiority on extreme light model levels (under 10M). Moreover, our small and base models achieve comparable and better results compared to methods of similar model sizes. It is also worth noting that our method of bringing locality into Mamba is generalized, practical, and easy to implement. compared to other complex methods. More details are found in the attached PDF.

Q1A3 Applicable to ViT architecture: Our quadtree-based window partition module and sequence construction strategy are specially designed for the recent vision Mamba models. The idea of our QuadMamaba originated from the coarse-to-fine feature representation philosophy, found in many prior arts such as InceptionNet [3], Multiscale Transformer [4], Focal Transformer [5], and Quad-attention Transformer [6]. However, the casual sequence modeling scheme in the Mamba model is completely different from the non-casual attention scheme in vision transformers. The constructed sequence for Mamba has to casually scan each token from the input. The length of the token sequence in Mamba has to be kept the same for multi-layer feature processing. Thus, it brings much difficulty in preserving spatial adjacency in Mamba. Differently, due to the flexibility of the attention scheme, the quantity of the output tokens can easily maintained if the query tokens remain unchanged. Thus, our method is highly customized for Mamba.

Q2.Experimental results

Q2A1 More comparisons with the baseline: To demonstrate the effectiveness of the proposed QuadVSS block, we will add additional ablation studies on different model sizes. The table below shows the improvement in tiny model levels/FLOPS thresholds.

Q2A2 Training overheads: In the attached PDF files, we plot the GPU memory vs. Batch size and training curve. Our QuadMamba has an affordable GPU memory overhead compared to the baseline. We find that our method does not result in any difficulty in training convergence.

Q2A3 Plots with Flops vs. Performance: We plot the detailed comparison of Flops and Performance in the attached PDF file. To clearly show the advantages of our method, we rename the QuadMamba model variants (Lite -> B1, Tiny -> B2, Small -> B3, and Base -> B4).

Q2A4 Block numbers in different stages: According to our observation, QuadVSS blocks work better in early model stages, where the feature resolution is higher. However, stacking more blocks in the second stage (high resolution) brings significant computation overload, especially for high-resolution images. To strike a balance of FLOPs between different stages, we allocate more QuadVSS blocks in the third stage (low resolution) instead of the second stage for larger models such as QuadMamba-Small and -Base (channels=144). For smaller models like QuadMamba-Lite and -Tiny, allocating more QuadVSS blocks to the second stage is more affordable because of fewer number of channels (channels=48), so it is unnecessary to defer the QuadVSS blocks to the third stage.

Q3.Clarity and questions

Q3A1 Window hyper-parameter: We always pick the 2x2 windows for two-level window partitions. Considering the 224x224 image resolution and four-time downsampling ratio, the two-level window partition strategy is enough for image classification. A more level window partition will be explored.

Q3A2 Patch hyper-parameter: The image features, which are partitioned by learnable modules, are then reshaped to a 1D token sequence for the Mamba block. For the Mamba, the 1D token sequence is completely the same in terms of computation flows. Thus, the computing efficiency remains unchanged with different patch numbers.

Q3A3 Label of Eq.7: As the QuadMamba brings no extra training complexity, there is no training label for the softmax in Eq. 7. We design a differential sequence construction strategy and apply the Gumbel-Softmax trick, which can help optimize Eq.7.

Q4.Minor nits and typos

We will revise the main text carefully.

Thank you for the additional graphs and figures. While QuadMamba performs well compared to baselines in terms of #param, Fig 2 shows that performance is incremental wrt MACs (FLOPS). To me, actual model sizes are not as important, FLOPS (or model latency on same hw) are the primary criterion, and on this one the gains are quite small. So the graphs actually reinforce my initial statement - I do not think there is a naming misalignment - I was comparing similar flops to similar flops and there was not much notable improvement there.

Q2A1 The table below shows the improvement in tiny model levels/FLOPS thresholds.

I do not see FLOPS listed in the table, just #params. I explicitly asked for quality comparisons for same FLOPS.

Q1A3

Your answer seems to be "Mamba adds a bunch more complicated constraints to the process" but you did not actually answer my question of -- can I do sampling and apply the Gumbel-Softmax idea to ViTs?

As the QuadMamba brings no extra training complexity, there is no training label for the softmax in Eq. 7.

Softmax usually suggests a classification objective with training data. It seems what actually is going on is Eq 9. But at the point where you introduce Eq 7, the context is missing and is quite confusing.

I still lean positive but the explanations do not change my general takeaways, so I would keep my rating.

Thanks for your responses and for maintaining positive ratings. We apologize for not addressing the concerns precisely due to the word limit. We hope the following explanation can partially address your concerns.

• Our QuadMamba can achieve similar gains with a reasonable and simple learnable module compared to the complex methods (e.g., LocalViM).

• We feel sorry that we forgot to provide the Flops in Q2A1. The Glops for Mamba-T is 1.7G, and the Flops for QuadMamba-T is 2.0G.

Regarding Gumbel-Softmax:

1. Application to Vision Transformers (ViTs) and Other Architectures:

The Gumbel-Softmax technique can be integrated into Vision Transformers (ViTs) and various other architectures or tasks, such as detection and super-resolution. Unlike the Softmax, which outputs continuous values in the range (0,1), Gumbel-Softmax provides discrete outputs in {0,1}. This capability is particularly beneficial for differentiable decision-making. Several ViT studies, like A-ViT[7] and SparseViT[8], utilize Gumbel-Softmax primarily for sparsifying feature computation. However, there's currently no exploration of using Gumbel-Softmax for patch partitioning in ViTs, which we believe is a promising area for future research.

2. Supervision and Learning:

Gumbel-Softmax here does not require explicit supervision. It receives gradients from the overall objective, such as the 1000-way cross-entropy loss in ImageNet classification, and learns to predict coarse-to-fine partitions in an end-to-end manner. The supervision for the partitioning process is implicit.

[7] A-ViT: Adaptive Tokens for Efficient Vision Transformer. Hongxu Yin, etc. In CVPR22.

[8] SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer. Xuanyao Chen, etc. In CVPR23.