HRVMamba: High-Resolution Visual State Space Model for Dense Prediction

1. This paper feels more like a technical report instead of an academic paper. The challenges of Vision Mamba models are obtained from previous works, and the solutions are also based on current techniques. Besides, some observations and methods are not properly cited. For example, the problem where multi-way scanning approach disrupts 2D spatial dependencies has been proposed in MambaIR[1], and the solution of Multi-scale Depthwise block has also been introduced in SMT[2]. Given these considerations, this paper lacks introducing any new insights or techniques to the community.

2. The experiments are not valid enough. For example, for the semantic segmentation, only results on Cityscapes and PASCAL Context are reported. The results on the widely used ADE20K are missing. The authors may consider reporting results with the same framework like Uppernet on ADE20K and compare with publicly available results like VMamba. Complementing these results and comparing with publicly available results will make this paper more solid.

3. Some experimental data lack further explanation. In Table 7, the authors report the performance on the COCO val set. As the COCO dataset has many benchmarks, this "val" set is ambiguous. If it refers to the pose estimation, the best results here are 74.2, which is not coherent with the results in Table 3. Additional explanation for the setting differences is needed. Besides, there are some very weird data such as the one in L412. The authors mentioned the poor performance of LocalVMamba on the PASCAL-Context dataset, but no further explanation for the reason is provided.

4. The detailed efficiency comparisons are missing, including the inference speed and memory cost on different datasets.

5. Some minor errors: In L213, Fig.3 refers to Fig.2 by mistake. Besides, the contents from L445 to L450 are not appropriate for the ablation section.

6. This work does not give any discussion about the limitations.

Reference:

[1]. Guo, Hang, et al. "MambaIR: A Simple Baseline for Image Restoration with State-Space Model." arXiv e-prints (2024): arXiv-2402.

[2]. Lin, Weifeng, et al. "Scale-aware modulation meet transformer." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

• Limited Novelty: The HRVMamba model mainly combines existing methods, including the VSS block, DWConv, DCN, and the HRNet architecture. This integration-based approach may not meet ICLR’s high standards for innovation.
• Concern for Limitation 1: While the paper addresses the lack of 2D inductive bias in previous visual Mamba models, it raises concerns about whether introducing such bias could restrict the scaling ability of HRVMamba. Vision Transformers (ViTs) have demonstrated that reduced inductive bias can facilitate better scaling, so incorporating strong inductive biases might limit HRVMamba's scalability and performance on larger-scale models.
• Concern for Limitation 2: The paper uses Deformable Convolutions (DCN) to mitigate the long-range forgetting issue observed in previous visual Mamba models. However, there is a concern about whether DCN can effectively address this problem as the sequence length scales up. The efficacy of DCN for maintaining high-level feature relationships over significantly longer sequences remains uncertain, raising questions about its robustness as a scalable solution for long-range dependencies.
• The paper references preprints and arXiv versions of significant works, such as Mamba (COLM), Vision Mamba (ICML), and VMamba (NeurIPS). The authors should update these citations to their final published versions to reflect the current state of the literature.

1. This paper's novelty is limited. It incorporates the VMamba block[1] into the HRNet architecture[2], including DCNv4, making it lack sufficient novelty and insights. In addition, similar ideas have been explored in HRFormer[3,4]. The novelty of this paper is below the threshold for ICLR publication.
2. The motivations for using DCNv4 and DVSS block in this paper are unclear. For example, this paper lacks a specific analysis of the long-range forgetting problem of Vmamba in vision and why DCNv4 can solve such a long-range forgetting problem of mamba. Whether from the theoretical or experimental analysis perspective, the authors need to provide exact evidence to present.
3. Lack of comparisons with recent vision mamba works, such as MambaVision[5].
4. In Fig.1, why are there neat blocks in the activation map? How can this be explained? Is it related to the scans in different directions of VMamba? Image activation is usually continuous, and I'm confused about it.
5. How do the feature maps of different resolutions fuse, downsample, or upsample?
6. How about the inference latency of the proposed HRVMamba, which includes VMamba blocks, multi-scale depthwise convolution blocks, and DCNv4 blocks?
7. In Tab.7, adding a 3x3 convolution has no effect. However, adding larger depth-wise convolutions, such as 5x5, 7x7, or 9x9, improves a little (0.3 AP), but this also introduces many additional parameters. It's unclear here whether the effect comes from extra parameters, larger convolutions, larger receptive fields, or multi-scale convolutions.
8. The performances of baseline methods (such as HRFormer) on Cityscapes and PASCAL Context are too low, which are far inconsistent with the original paper[3], for example, HRFormer-B obtains 82.6 mIoU (Cityscapes) and 58.5 mIoU (PASCAL Context) while achieving 77.3 mIoU (Cityscapes) and 42.6 mIoU (PASCAL Context) in this paper. I think a fair comparison is very important. However, the results of the current comparison methods are obviously much lower than those of the original methods.

References
[1] Liu et al. VMamba: Visual State Space Model. NeurIPS 2024.
[2] Wang et al. Deep High-Resolution Representation Learning for Visual Recognition. TPAMI 2020.
[3] Yuan et al. HRFormer: High-Resolution Transformer for Dense Prediction. NeurIPS 2021.
[4] Gu et al. Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation. CVPR 2022.
[5] Hatamizadeh et al. MambaVision: A Hybrid Mamba-Transformer Vision Backbone. NeurIPS 2024.

Limited Novelty: In my view, this paper incorporates techniques from CNN networks, such as DCNv4 and multi-resolution structures (from FPN and HRNet), into the Mamba block to enhance network performance. I am somewhat skeptical about whether such an innovation alone is sufficient for a publication at ICLR.