RoMA: Scaling up Mamba-based Foundation Models for Remote Sensing

Q1: Poor writing obscures the contributions and includes grammar issues, repetition, and unclear sections.

A1: We sincerely appreciate the reviewer's feedback regarding writing clarity. We acknowledge that certain sentences may lack fluency or appear ambiguous. In response:

• For the caption of Figure 2, we agree that the phrasing can be improved. The current version indeed contains a typo ("different" → "difference", "autogresive" → "autoregressive").

  In the revised version, we will rephrase it as:

  Comparison between our autoregressive pretraining strategy and the standard MAE method. (1) RoMA encodes all patches using a Mamba encoder, whereas MAE encodes only a randomly sampled subset. (2) RoMA predicts the next token in a sequence to capture continuity, while MAE only reconstructs masked patches.

• For lines 154-156, we acknowledge the original phrasing is grammatically flawed and may seem vague. In the revised version, we will rephrase this section for better clarity:

  Limitations of MAE-based Pretraining for Mamba. We begin by analyzing why MAE-based pretraining may be suboptimal for Mamba-based RSFMs, given the unique characteristics of RS imagery and the architectural mismatch between MAE and Mamba.

• For lines 75-80, which enumerate the contributions, the repetition noted by the reviewer may stem from syntactic redundancy. We will streamline the phrasing to make each contribution more concise, while preserving the technical depth. For example, we will replace the first contribution with

  (1) We introduce RoMA, the first autoregressive pretraining framework for Mamba in remote sensing, which supports high-resolution imagery and demonstrates consistent performance gains with increasing model and data scales.

  While the second contribution will be replaced by:

  (2) We propose a rotation-aware mechanism that combines adaptive region cropping with angular embeddings to enhance rotation-invariant representation learning. This design effectively handles orientation variability and sparsely distributed targets in remote sensing scenes during autoregressive pretraining.

• Regarding the "Advantage of Autoregressive Pre-training of Mamba" paragraph, our intent was to highlight the architectural alignment between Mamba and autoregressive modeling, as both employ sequential processing. We have revised this section to more clearly explain how this alignment brings practical benefits to the pretraining process. The paragraph will be revised to:

  Autoregressive pre-training aligns naturally with the sequential nature of Mamba's architecture, which processes input tokens in a temporally ordered manner. Specifically, autoregressive modeling constructs image patches sequentially and predicts the next token based on previous context, mirroring Mamba's token-by-token scanning mechanism. This architectural alignment facilitates more coherent temporal dependencies and better token transition modeling, allowing Mamba to learn more structured and semantically meaningful representations. Therefore, the autoregressive training paradigm not only complements Mamba's design but also enhances its ability to model spatial continuity and visual context in remote sensing imagery.

Q2: Section 3.2.3 is unclear, especially regarding patch tokenization, token order, multi-scale prediction, and the role of the Mamba encoder.

A2: We thank the reviewer for pointing out the ambiguity in Section 3.2.3. We appreciate the opportunity to clarify the technical design regarding patch tokenization, sequence ordering, the Mamba encoder, causal decoding, and the multi-scale prediction mechanism.

Notably, many of these technical components follow widely adopted practices in the community (e.g., ViT-style patching, raster ordering, and standard Mamba configuration). To avoid redundancy, we did not elaborate on them in detail in the manuscript. However, we acknowledge the need for clearer exposition and will revise the manuscript accordingly.

Below, we provide a detailed response:

• Patch Tokenization: Each input image is divided into non-overlapping patches (e.g., 16×16 pixels), which are then linearly projected into visual tokens via a trainable embedding layer, following the standard practice in ViT-based models [1].

  [1] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021.

• Token Sequence Ordering: To support autoregressive modeling, we arrange the patch tokens into a 1D sequence using a raster-scan ordering (left to right, top to bottom), consistent with the Vision Mamba design [2].

  [2] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model, ICML 2024.

• Mamba Encoder: The backbone of RoMA is a Mamba-based encoder used for extracting image features, as stated in lines 216-217 of the manuscript.

  Our contribution focuses on designing an effective pretraining strategy tailored for remote sensing, rather than altering the encoder’s architecture.

• Causal Cross-Attention: During pretraining, we introduce a decoder to perform causal supervision via cross-attention. Specifically, the decoder comprises a sequence of virtual query tokens, each designed to predict a target patch token based only on encoder outputs from earlier positions (as defined by raster-scan order). This setup enables autoregressive learning while maintaining strict causality.

  Although this cross-attention mechanism introduces quadratic complexity during pretraining (with respect to the decoder’s token length), we emphasize that:

  This decoder is only used during training to provide autoregressive supervision. It is discarded after pretraining. Thus, it does not affect the efficiency of the final foundation model at inference time.

• On Computional Complexity: We respectfully believe there may be a misunderstanding. The foundation model we develop and ultimately deploy is the Mamba encoder, whose linear complexity [2] is one of its primary advantages:

  $$\mathcal{O}(N \cdot d)$$

  where $N$ is the number of tokens and $d$ is the hidden dimension. This contrasts with conventional ViT-based encoders used in many remote sensing foundation models, which exhibit quadratic complexity:

  $$\mathcal{O}(N^2 \cdot d)$$

  Hence, our efficiency claims pertain strictly to the Mamba encoder, and the decoder’s quadratic operations during training do not impact inference-time cost.

  In summary, RoMA's computational efficiency claims are based solely on the Mamba encoder, which is used at inference. The decoder contributes only during training and does not affect the model’s practical deployment cost.

  [2] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model, ICML 2024.

• Multi-Scale Prediction Strategy Thank you for pointing this out. We clarify that the multi-scale prediction in RoMA is implemented by aggregating spatially adjacent decoder tokens and reconstructing their corresponding higher-level patches in parallel. As described in lines 241-243 of the paper.

  Specifically, we construct larger-scale features by grouping and concatenating adjacent decoded tokens based on their spatial positions (e.g., 2×2 neighboring tokens for a 32×32 patch if the token represents 16×16), and then map these features back to pixel space using an MLP. This enables RoMA to predict target pixels at multiple spatial granularities (e.g., both 16×16 and 32×32 patch levels) simultaneously, thereby enhancing its ability to model objects with diverse scales in RS images.

  We will revise both the text and Figure 3 in the updated manuscript to make this multi-scale strategy more intuitive and visually clear.

Q3: The novelty of using a rotation augmentation with angular embedding for the pretraining method seems fairly limited

A3: Thank you for your comment. Our contribution extends well beyond the use of rotation augmentation and angular embeddings. Specifically, our goal is to address a fundamental challenge in current remote sensing foundation models: the inefficiency of ViT-based architectures in handling high-resolution inputs due to their quadratic complexity.

To address this, we propose RoMA, a novel autoregressive pretraining framework built on a Mamba-based encoder with linear computational complexity, making it inherently more scalable to high-resolution remote sensing imagery.

To support effective pretraining under the unique characteristics of remote sensing imagery, we introduce a series of tightly integrated strategies:

• (1) Sparsity of foreground objects: We design an adaptive region cropping strategy to identify regions with high knowledge density, which often correspond to meaningful land objects. This ensures that the model focuses its learning capacity on informative regions.

• (2) Variation in object orientation: We introduce angular embeddings that encode rotation angles into the token representations. This facilitates rotation-invariant feature learning and improves robustness to orientation diversity.

• (3) Extreme variation in object scales: We propose a multi-scale prediction strategy, which reconstructs token representations at different spatial granularities (e.g., 16×16 and 32×32 patches), enabling the model to capture both fine-grained and coarse spatial patterns.

These components are not isolated techniques, but are integrated cohesively into an autoregressive pretraining pipeline, which aligns well with Mamba’s sequential architecture. As a result, we successfully construct a remote sensing foundation model based on the Mamba framework, tailored to both the computational efficiency and unique characteristics of remote sensing imagery.

Q1. Literature on rotation-invariant training could been included.

We will incorporate relevant citations and expand the discussion accordingly in the revised version of the paper. Specifically, lines 175-182 will be revised as follows (new content highlighted in bold): RS images often contain redundant airport runway pixels, while airplanes appear at various orientations, leading to different postures and shapes. Such directional diversity has been extensively addressed in supervised object detection through rotation-equivariant or rotation-invariant designs~\cite{han2021redet, yang2020arbitrary, han2022align}. However, autoregressive...

---

Q2. Difference between Scale-MAE with multi-scale prediction strategy

While both RoMA and Scale-MAE involve multi-scale reconstruction strategies, their mechanisms and motivations are fundamentally different.

Scale-MAE employs a Laplacian-pyramid-style decoder that progressively upsamples feature maps to produce multi-level representations. Each decoder stage is tasked with reconstructing the low-frequency or high-frequency component of the original image. The corresponding supervision consists of low-resolution and high-resolution versions of the input image, approximating its low- and high-frequency content, respectively.

In contrast, RoMA's multi-scale prediction strategy does not rely on progressive decoding. Instead, it forms larger-scale features by concatenating decoder token representations that are spatially adjacent in the original image layout. This process enables RoMA to effectively reconstruct the input at multiple patch granularities, such as 16×16 and 32×32 patch-level representations, in parallel. In the revised version of the paper, we will provide a clearer comparison with Scale-MAE to highlight the novelty of RoMA's multi-scale design.

---

Q3.Clearly state why the loss is not applied in the token space. Where possible within the given framework?

We fully acknowledge that applying reconstruction loss in token space, as opposed to pixel space, has been shown in prior works such as BEiT[1] and CAE[2] to improve semantic representation learning in MAE-style frameworks. This is largely because token-space loss operates on semantically rich latent features, rather than low-level raw pixel intensities.

However, such approaches typically depend on a pre-trained tokenizer (e.g., dVAE[3], CLIP[4]) or an auxiliary feature encoder to provide supervision in the latent space. In our work, we deliberately adopt pixel-space reconstruction loss in order to isolate the contribution of our RoMA framework itself, without introducing additional variables such as tokenizer quality or auxiliary networks. This design choice ensures a fair and transparent comparison with existing baselines under a unified pretraining setting.

That said, token-space loss is fully compatible with our framework. To demonstrate this, we conducted a preliminary experiment comparing pixel-space and token-space reconstruction objectives. For the token-space setup, we adopt a frozen CLIP model as the teacher, and apply LayerNorm to normalize its output features. The RoMA framework serves as the student, augmented with a lightweight MIM head (fully connected layers) to predict the teacher’s latent representations. Both teacher and student receive the same full input image, and we compute the Smooth-L1 loss between their latent features across all spatial positions. The overall architecture is illustrated below:

Input ──► Teacher (Frozen CLIP) ─► LN ─┐
        │                              ├─► Smooth-L1 Loss (all tokens)
Input ──► Student ───► MIM Head ───────┘

This setup enables supervision in the token space, allowing the student to align with the high-level semantic representations produced by a strong teacher encoder. Compared to pixel-level objectives, this strategy encourages the model to learn more abstract and transferable visual features during pretraining. Due to time constraints, we pretrained both versions (pixel-space and token-space) for 50 epochs, and transferred the resulting models to the UCM and AID remote sensing classification benchmarks, following the same protocol as in our main paper. The results are as follows:

These results confirm that token-space loss indeed yields stronger performance, further validating its effectiveness. More importantly, this experiment illustrates the flexibility and compatibility of the RoMA framework with different pretraining objectives.

We consider this a promising future direction. To maintain a focused and methodologically clean evaluation of our core contributions, we plan to include this experimental comparison, along with implementation details, in the supplementary material of the revised version. [1] BEiT: BERT Pre-Training of Image Transformers, ICLR. [2] CAE: Context AutoEncoder for Self-Supervised Representation Learning, IJCV. [3] DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors, NeurIPS. [4] Learning Transferable Visual Models From Natural Language Supervision, ICML.

---

Q4. Comparisons with more recent approaches that already shown the limitation of MAE-only based pretraining, like CROMA and AnySat.

We acknowledge the importance of including recent advances such as CROMA and AnySat, and we will incorporate citations, discussions, and comparisons with these works in the revised version of the paper. Specifically, lines 98–102 of the first paragraph in the related work section will be revised as follows:

MA3E integrates remote sensing angle factors into the MIM training architecture for both the encoder and decoder. Recent efforts have also explored hybrid pretraining strategies that combine reconstruction and contrastive objectives. CROMA \cite{CROMA} jointly leverages masked autoencoding and cross-modal contrastive learning over spatially aligned SAR and optical imagery, aiming to improve both unimodal and multimodal representations. AnySat\cite{AnySat}, in contrast, focuses on learning a universal model across heterogeneous inputs by incorporating contrastive learning into a joint embedding predictive architecture, demonstrating strong performance on multi-resolution and multi-modal datasets. While these methods...

To further address this point, we conducted preliminary comparisons. We have completed the evaluation of AnySat and CROMA on scene classification datasets using the same protocol as our main paper. However, CROMA is tailored for multimodal SAR-optical inputs with a backbone designed for multispectral imagery. To adapt it, we added a 1×1 convolution to project the 3-channel RGB input into the 12-channel space required by CROMA, with the layer updated during linear probing.

These results highlight the strong performance of our RoMA framework. We also note that AnySat and CROMA may not perform optimally in this setting, possibly due to its pretraining being oriented toward multi-resolution, multi-sensor data, whereas AID and UCM are single-modal RGB datasets. With more extensive fine-tuning and adaptation, its performance may improve further.

---

Q5. Experiments about standard deviations, e.g. linear prob

Following your advice, we conducted linear probe classification experiments three times using different random seeds for initialization, while keeping all other settings consistent with the protocol described in our main paper. The results are summarized in the table below.

As shown, RoMA exhibits consistently strong performance with low variance, indicating better robustness compared to other competing methods. Due to time constraints, we report standard deviations for two representative baselines (ARM and SatMAE++). We will include standard deviation results for additional methods in the revised version.

---

Q6. Typos and Minor Issues

1. The font size in Figure 1 will be enlarged in the revised manuscript.
2. The term "object detection" will be corrected to "change detection" in the revised manuscript.
3. We will cite CROMA in the related work section and briefly discuss its relevance in the revised manuscript.
4. We will include a citation to AnySat in the related work section and briefly discuss its connection to masked image modeling methods in the revised manuscript.
5. The term "high-value patches" refers to image patches with high information density, as computed using the LBP metric described in lines 180-187. In the revised version, we will provide a clearer explanation of this term in the caption of Figure 3.
6. In the revised version, we will include the full name of LBP (Local Binary Pattern) in the caption of Figure 4 for clarity.
7. The missing hyperlink to Section 1 will be added in the revised manuscript.
8. We acknowledge that the wording in line 239 may be misleading. To clarify, the comparison described indeed refers to the pixel-level loss computation, as explicitly defined in Equation 2. The phrase "corresponding tokens in the original image" simply refers to the decoder outputs being aligned with their corresponding image regions, rather than indicating token-level supervision. We will revise this sentence for clarity in the updated version.
9. We apologize for the mistake: the accuracy values for HOG and LBP were mistakenly swapped in Table 3(b). This will be corrected in the revised version of the manuscript. We also appreciate your suggestion regarding figure placement, and we will adjust the layout so that figures appear at the top or bottom of each page to improve readability.

---

Thank you again for your constructive feedback.

Q1. Remote sensing targets are typically small. Could the autoregressive pre-training method lead to the neglect of local information, thereby resulting in insufficient learning of small targets?

A1: We agree that small object representation is a critical challenge in remote sensing. However, we believe that the potential loss of local information is not inherently caused by the autoregressive pretraining paradigm, but is more related to the choice of patch size and whether the target region is adequately included in the input.

To assess the ability of our RoMA model to capture small targets, we fine-tune RoMA-Base with UperNet [1] on the iSAID dataset [2], which contains a large number of small target categories. We compare its performance with several state-of-the-art models, reporting both the overall mIoU and the IoU for representative small-object categories, including Ship, Small Vehicle, Swimming Pool, and Plane. The average object widths for these categories are around 15-53 pixels.

The results are shown in the table below. Our method achieves the highest overall mIoU and also shows notable improvements on small-object classes, especially Small Vehicle (average width 15 pixels), where RoMA surpasses all other backbones. These results suggest that the proposed adaptive region cropping strategy effectively increases the visibility of small foreground objects during pretraining, allowing the model to learn more discriminative representations.

Fine-tuning performance on iSAID (following RingMo [3] settings):

[1] Unified Perceptual Parsing for Scene Understanding, ECCV 2018.

[2] iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images, CVPR Workshops, 2019.

[3] RingMo: A Remote Sensing Foundation Model With Masked Image Modeling, IEEE TGRS 2022.

---

Q2. How can we ensure that the selection of top k tokens includes the target? If the target is not in a relatively central position, it may still lead to the situation shown in Figure 4(b).

(1) How can we ensure that the selection of top k tokens includes the target?

A2-(1): This is an interesting question. To ensure that the selected tokens include the target, we incorporate a three-stage safeguard mechanism (see lines 183-195 of the main text):

(1) Coarse localization via information density: We first compute an information density map using LBP, which highlights regions rich in texture and structure, often corresponding to land-cover targets. This enables us to localize potential object regions while filtering out most of the uninformative background.

(2) Candidate region selection: To further ensure that land objects are included, we fix the size of candidate regions and generate all possible regions that fully contain the previously identified coarse localization points. Among these candidates, we select the region with the highest average information density, under the constraint that it must exceed the global average of the entire image. This improves the likelihood of capturing informative targets.

(3) Adaptive region shrinking: If none of the fixed-size candidate regions selected in step (2) satisfies the threshold, i.e., its average information density is not higher than the global average of the entire image, we reduce the region size and regenerate all candidate regions that still include the coarse localization point. We then repeat the selection process described in step (2). This adaptive refinement increases the likelihood of isolating smaller yet informative targets that may otherwise be diluted by surrounding background.

Through this three-stage safeguard, our framework maintains strong coverage of targets, mitigating the potential risk of information loss.

(2) If the target is not in a relatively central position, it may still lead to the situation shown in Figure 4(b).

A2-(2): Thank you for your question. To evaluate the performance of our proposed adaptive region cropping strategy, we conducted a quantitative analysis using the full SpaceNet V1 building segmentation dataset.

Specifically, we uniformly rescaled all images and their corresponding segmentation masks to a fixed resolution of 224×224, and divided each image into non-overlapping 16×16 patches. We then applied our adaptive region cropping strategy to each image to select a representative region.

The results show that random cropping yields only 38.56%, whereas our method achieves a significantly higher foreground object capture rate of 75.09%, indicating that in over three-quarters of the samples, the selected region overlaps with at least one labeled object in the segmentation mask. This demonstrates the effectiveness of our adaptive cropping strategy.

Foreground Object Capture Accuracy (SpaceNet V1):

This substantial improvement demonstrates that our adaptive cropping strategy effectively increases the likelihood of capturing foreground objects, which is crucial for training autoregressive models to learn semantically meaningful representations. While a 75% overlap rate does not guarantee full object inclusion, it still provides a much denser learning signal compared to random cropping, and we found it sufficient to yield superior performance over existing methods in downstream tasks.

We acknowledge that further improvement is possible. In future work, we plan to investigate more advanced region selection strategies, such as leveraging auxiliary weak annotations, to further enhance target coverage and representation learning.

---

Q3. In Figure 3, were larger blocks reconstructed using different orders?

A3: Thank you for your question. When reconstructing larger blocks, we did not employ different ordering strategies. Instead, we followed the spatial layout of the original image patches. For example, a 2×2 block would be arranged as:

[1 , 2]      [1 , 2]

[5 , 6] $\Rightarrow$ [5 , 6]

This is consistent with the description in lines 241-242 of the main text:

"Building on MSE loss function, we concatenate token representations from each decoder block at a higher scale following a predefined raster order".

We appreciate your comment and will revise Figure 3 to more clearly reflect this ordering scheme for better clarity.

---

Q4. Was angle prediction included? How does angle embedding function within the network?

A4: Thank you for your question. RoMA does not include an explicit angle prediction task. Instead, angle embeddings are incorporated to provide directional prior information, which helps guide the network toward learning rotation-invariant representations during pre-training. These embeddings are injected alongside patch tokens to condition the model with orientation cues, without requiring supervision from angle labels. The effectiveness of this strategy has been demonstrated in MA3E [1].

[1] Masked Angle-Aware Autoencoder for Remote Sensing Images, ECCV 2024.

---

Thank you again for your constructive feedback. If you have any further questions, please don’t hesitate to let us know, we would be happy to continue the discussion.

Q1. While the proposed framework shows promising results on standard optical RS datasets, could the authors comment on or provide evidence regarding the applicability of RoMA to other types of remote sensing data such as SAR or hyperspectral imagery? Since these modalities have very different data characteristics, it is important to clarify whether the method generalizes beyond the current experimental scope.

A1: Thank you for your insightful question. In principle, RoMA can potentially be extended to other types of remote sensing data such as SAR and hyperspectral imagery.

For SAR data, although it differs from optical imagery in imaging mechanism (active microwave vs. passive optical), it still follows a similar top-down observation geometry. This allows the rotation-invariant operations and multi-scale prediction strategies in RoMA to remain applicable. Moreover, SAR images often exhibit high-intensity backscatter in regions corresponding to man-made structures (e.g., buildings, ships), which aligns well with RoMA's adaptive cropping strategy based on information density. In terms of channel compatibility, most SAR images contain 1–4 channels; for single-polarization data (e.g., VV), we can replicate the image into a 3-channel format to match the input requirements of the Mamba backbone.

For hyperspectral data, although it consists of a large number of spectral bands, the core components of RoMA, can be extended to 3D spatial-spectral cubes. Inspired by SpectralGPT [1], the hyperspectral image cube can be partitioned into spatial-spectral subcubes. We can then compute a 3D information density map by evaluating each subcube's spatial saliency and spectral variance. Based on this map, the most informative subcubes can be selected, and adaptive cropping strategy can be applied to their surrounding 3D regions. The multi-scale prediction strategy can also be extended by considering multiple scales in both spatial and spectral dimensions. For the rotation operation, unlike traditional 3D image volumes, hyperspectral cubes consist of two spatial dimensions and one spectral dimension without a well-defined geometric structure across all three axes. Therefore, we do not apply geometric 3D rotations to subcubes. Instead, we enhance data diversity using hyperspectral-specific augmentations such as spectral band shuffling. These operations preserve the semantic consistency of spatial-spectral mappings while improving the model's generalization ability.

In summary, while current experiments focus on standard optical datasets, the architecture and principles of RoMA are general enough to be extended to SAR and hyperspectral imagery.

[1] SpectralGPT: Spectral Remote Sensing Foundation Model, IEEE TPAMI 2023.

Q2. The paper claims advantages in handling high-resolution images. However, the highest resolution used in the experiments appears to be 1248×1248, which is relatively modest compared to typical operational remote sensing images that can reach tens of thousands of pixels per side. Could the authors either include experiments on higher-resolution datasets or clarify the limitations of the current approach in truly large-resolution scenarios?

A2: Thank you for the valuable question. To address the reviewer's concern, we conducted additional experiments on higher-resolution images using RoMA-Base and ViT-Base. Each model was evaluated over 200 forward passes to ensure statistical reliability. We report peak GPU memory usage (in MB) and average inference speed (samples per second) across a range of input resolutions. All experiments were conducted on a single NVIDIA A100 GPU with a batch size of 1. (Note: Figure 1 in the main text uses an RTX 4090 GPU with a batch size of 2.)

Peak memory usage (MB):

Inference speed (samples/sec):

These results demonstrate the scalability of RoMA to higher-resolution inputs. In contrast, ViT-Base encounters out-of-memory (OOM) errors at resolutions beyond 2048×2048. This highlights RoMA's superior computational efficiency and its potential applicability to operational remote sensing scenarios involving ultra-high-resolution images.

---

Overall, thank you again for your constructive feedback. If you have any further questions, please don’t hesitate to let us know, we would be happy to continue the discussion.

Dear reviewer ezvU,

Can you please share your thoughts post rebuttal, once you've had a chance to see the author responses and also the other reviews?

Best, AC