GlobalMamba: Global Image Serialization for Vision Mamba

This manuscript highlights a limitation in current vision mambas, which utilize tokenization and a sequential flattening process. The process overlooks the inherent 2D structural correlations present in images, making it challenging to extract global information effectively. To address these limitations, the proposed GlobalMamba method strategically organizes pixels according to their corresponding Discrete Cosine Transform (DCT) frequencies. Subsequently, it transforms each set of pixels with the same frequency back into the spatial domain, resulting in a longer image series for the subsequent tokenization in a mamba baseline. The experiments tackling tasks of image classification, object detection, and semantic segmentation are conducted on ImageNet-1K, COCO, and ADE20K datasets.

The work builds on Mamba state space models (SSMs) in vision tasks. State space models have been proposed as sequence models that can improve the limitations of quadratic complexity in self-attention by leveraging a framework that allows for parallelism in training, with sequential predictions in autoregressive tasks. In the field of vision, research on SSMs for vision has focused on remediating a central limitation of sequential modelling of spatial data; optimal ways for aligning data in a flattened two-way message passing setup.

The authors bring about an important point; given that spatial data have less of an inherent ordering than general sequence modelling tasks, it is not clear how a global receptive field can be achieved as handily as in vision transformers (ViTs). For transformers, a bidirectional approach simplifies to a global view where all tokens interact in parallel per self-attention block. Contrarily, bidirectionality in more classic sequence models still implies two-way message passing in a flattened sequence. Hence, work on SSMs for vision have rediscovered classic scan orders over patches to determine an appropriate extension to modelling spatial data as sequences. Currently, SSMs use more layers with lower throughput to achieve typically slightly worse results than ViTs.

The authors go on to propose an approach to provide global context for SSMs in vision by introducing tokenization by a pyramidal structure of low-pass filtered tokens from the top down. While the paper touches on a central modelling component in SSMs for vision, the authors end up modelling the pyramidal tokens at each level with classic raster scan ordering, referred to as “causal tokens”. The resulting method ends up being better, however, it is very unclear if this is due to the arguments in the paper. Instead, the gains could feasibly come from having many more tokens in the sequence, which generally improves performance, but increases complexity. In other words, showing that the gains come from the ordering of low-pass filtered images rather than the feature pyramid with increased sequence length becomes imperative to demonstrate, given the central argument provided by the authors.

In summary, it seems that the tokenization method has benefits to other Mamba-based approaches in vision tasks. However, in this reviewer's opinion, the current draft does not adequately delineate the source of these performance gains. As it stands, it seems possible that gains can be attributed to more tokens, rather than a causal ordering of frequency bands in the image.

Most of existing Vision Mamba frameworks employ patch-based image tokenization and then flatten them into 1D sequences for causal processing, which is not effective in processing 2D structural correlations of images. With a frequency-based image serialization method, this paper proposes a new vision mamba model, GlobalMamba, which can better exploit 2D structural correlations of image token sequences. Experiments on image classification, object detection and semantic segmentation are conducted to demonstrate the effectiveness of the proposed method.

The paper introduces GlobalMamba, a visual model designed to improve image representation by arranging image tokens in a frequency-based, causal sequence. The model leverages the Discrete Cosine Transform (DCT) to decompose images into different frequency bands, which are transformed back to the spatial domain to form image sequences progressing from low to high frequency. This sequence is then processed by a Mamba-based model to enhance feature extraction by focusing first on global, low-frequency information before finer details.

This paper tries to address the inherent limitations of current Vision Mamba models, which achieve linear complexity by processing image tokens sequentially but rely on patch-based tokenization that flattens 2D images into 1D sequences, thereby neglecting essential 2D structural correlations and hindering the extraction of global information. To overcome these challenges, the authors introduce a novel global image serialization method that utilizes the Discrete Cosine Transform (DCT) to convert images from the spatial to the frequency domain, organizing pixels based on their frequency ranges. By transforming each frequency band back to the spatial domain, they generate a series of images that preserve global information for effective tokenization. Building on this approach, the proposed GlobalMamba model employs a causal input format that better captures the causal relationships within image sequences. Extensive experiments demonstrate that GlobalMamba significantly outperforms existing Vision Mamba models and Vision Transformers in key vision tasks, including image classification on ImageNet-1K, object detection on COCO, and semantic segmentation on ADE20K. This work presents a meaningful advancement in leveraging global image information within Vision Mamba architectures, enhancing their applicability and performance across diverse high-resolution and complex vision tasks.

I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.