MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone’s Potential with Masked Autoregressive Pretraining

• The effectiveness of auto-regressive training has already been verified by a previous paper [A]. It significantly reduces the novelty and contribution of the paper's findings.
• Although the paper shows various findings, the findings lack general applicability and theoretical evidence. More experiments are needed to verifies the findings can be applied to general cases, which can contribute to the community.
• The proposed method, MAP, seems more like a mixture of heuristics and hyper-parameter tuning than a novel training method. It is incremental.
• Performance improvement in the 3D domain looks marginal. Considering that HybridMT3D requires additional # params and FLOPs, it is not clear that it is better than other methods.
• The writing quality is not good. It is hard to understand the main argument through the texts

[A] Autoregressive pretraining with mamba in vision, 2024

1. Lack of in-depth analysis on why MAP works. Although the author has included ablations on various aspect of the training and architecture design, most of them are pure tuning without analyzing the reason behind it. It would be more interesting adding some analysis on why, for example, row-wise causal decoder is working better.
2. Lack of more downstream transfer experiments on dense predictions, such as semantic segmentations on ADE20K. Existing reported experiments for 2D task is only reported on ImageNet1K classification.
3. Limited exploration on the AR scan pattern. ARM[1] explores cluster-based raster scan, which performs better than row-wise of column-wise.
4. Lack of hybrid mamba-transformer baseline with MAP applied. The comparison in 2D task lacks the baseline method with MAP. I'm not sure if the reported performance is because of the architecture change or the proposed pretraining.

[1] Autoregressive Pretraining with Mamba in Vision

• The paper’s contributions need to be stated more clearly, and some aspects lack novelty. While it proposes both a pre-training method -- Masked Autoregressive Pre-training (MAP) -- and a hybrid architecture (i.e., Mamba-Transformer), their individual contributions are not clearly distinguished, and introducing hybrid architecture itself is not novel. The authors begin by highlighting MAP's effectiveness by showcasing the enhancements to the Mamba-Transformer hybrid architecture (while the pilot study only employed handled Mamba -- ViM --). The reviewer suggests a more logical flow: MAP should first be tested on a standalone Mamba, followed by evaluating it on a hybrid architecture. Furthermore, most evaluations seem to focus on hybrid architectures, which obscures MAP's real aspect and results in a blending of contributions between the method and architecture. For example, there is insufficient reasoning as to why MAP would outperform existing methods like MAE and AR, along with a missing logical link explaining why MAP would be effective specifically for the hybrid architecture. The reviewer recommends that the authors distinctly separate their contributions and provide systematic support for each, suggesting that focusing on MAP would lead to more productive outcomes.

• As mentioned above, there is a lack of evidence and intuition explaining why MAP is effective for standalone Mamba or Transformer models (actually, we need the intuition as to why MAP does not fall behind MAE for Transformers). Furthermore, the rationale for using random masking (why random masking?) in Mamba-pre-training is also lacking. Without preliminary studies testing random masking in the pilot study (only the last tokens are masked), it is unclear why this approach would work and how it can be effectively combined with Mamba training because detailed explanations are absent.

• Furthermore, the motivation for introducing a hybrid architecture immediately after MAP is unclear, causing the paper’s main contributions to diverge by adding an architectural focus that may detract from emphasizing the primary contribution.

• The pilot study does not fully align with the proposed pre-training method. While it successfully indicates that autoregressive pre-training is beneficial for Mamba, it does not clearly explain why random masking is necessary. Additionally, the reviewer feels that details like the optimal scanning direction and masking ratio for tokens at the end of the sequence, while potentially informative, are not prerequisites for the pilot study. Furthermore, the pilot study fails to provide any foundational rationale for a hybrid architecture.

• In experiments, the effectiveness of MAP on non-hybrid architectures is neither impressive nor fairly compared with competing methods: 1) improvements for ViTs are minimal, and 2) competing methods like AR are not directly compared to MAP (e.g., MambaR-B/L + AR vs. MambaR-B/L + MAP in Table 8).

• Missing related works:

  • TokenUnify: Scalable Autoregressive Visual Pre-training with Mixture Token Prediction
  • MambaMIM: Pre-training Mamba with State Space Token-interpolation
  • Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations
  • CMViM: Contrastive Masked Vim Autoencoder for 3D Multi-modal Representation Learning for AD classification

• Many details are missing, including the experimental setup and the baseline used for the experiments:

  • It is unclear how Mamba can handle random masking (rather than using masks only at the end of the sequence) during pre-training.
  • The baselines and experimental setups for some experiments are missing, such as the experiments in Tables 2, 3, and others.
  • The differences between the proposed method and a simple combination of masking strategy and autoregressive modeling are not clearly presented. The reviewer only notes a causal decoder.

• Minors

  • Ablation studies, such as the full results of the masking ratio of MAP, are not sufficiently presented.
  • The authors are encouraged to add more spacing after tables and figures to improve readability.
  • Found duplicated references at lines 554-559, 560-566, 657-664, 696-701.
  • In Fig. 2(b), the masking position seems somewhat strange.

1. Limited Other Tasks Validation: Most 2D vision experiments are conducted on standard IN1K datasets, I wonder if the downstream dense prediction tasks, i.e., classification, detection, could benefit from the proposed pre-training techniques.
2. Out-of-data Citation: The paper references preprints and arXiv versions of significant works, such as Mamba (COLM), Vision Mamba (ICML), and VMamba (NeurIPS). The authors should update these citations to their final published versions to reflect the current state of the literature.

1. Quite limited performance gain compared with large ViT models, even with the hybrid MMMT design, e.g. 85.9 original MAE baseline vs. the strongest 86.2 result with also a much lower throughput.
2. The original promise of mamba was on the long sequence or long context modeling, but the design in this paper does not fully demonstrate this advantage. See the question section for more discussion here.
3. If I am understanding correctly, the goal of enabling mamba pre-training was to extend the method to longer sequences. (If this is not the case, then within the ViT-supported sequence length, we could perfectly use ViT-L with various MAE variants but not MAP). In this case, some of the design choices, to some extent, were not compatible with the original mamba design, e.g. the transformer layers in the hybrid design and the transformer decoder layers. These may not work on long sequences and may prevent the context scaling as follow up.