Benchmarking Robustness of Foundation Models for Remote Sensing

1. Lack of novelty: From what I received in this paper, this paper lacks new algorithm/architecture/methodology with fundamental contribution. All works are limited to testing and applying of architectures or metrics from previous works. Correct me if I am wrong.
2. New loss? New architecture?: This paper did a good job in demonstrating current models are not competitive in generalization to different resolutions and band combinations, but gives out no new solutions or possible proposals. For example, will it be helpful to develop a new loss mechanism or a new architecture? Or re-design current models to adapt to your generalization needs?
3. Multi/hyper-spectral data: Recent studies more focus on how to retrieve spectral representations across all bands instead of transferring the knowledge from, say RGB, to NIR. Correct me if I am wrong.
4. Pre-training, Fine-tuning: These are not new concepts at all and have been studied extensively, the conclusion of fine-tuned backbone impact generalization and frozen backbone preserves generalization ability is intuitive and familiar to most researchers and has been widely accepted as a fact, repeatedly conduct these trainings seems redundant.

Only scene classification and change detection are explored. There are many other tasks within remote sensing which are highly relevant and may have dramatically different feature relevancy (i.e. more spectrally dominant vs. structurally).

I disagree with the authors' stance that images at inference time are more likely to be lower-resolution. Very often the opposite is true- there are huge amounts of publicly available low resolution imagery. However, with the introduction of private satellite, planes, and drones, into the remote sensing space, people are trying to use these high-resolution sources for a very specific task while leveraging the decades of low resolution data that exists. Similarly, satellite imagery is only going to get better over time so being able to adapt the low resolution data to a new high resolution task is paramount.

For tasks like change detection and scene classification in general, there is substantial labeled data out there- foundation models would hopefully boost performance, but they're still usable. In contrast, there are many novel remote sensing tasks (especially in ecology/climate and agriculture) which are blocked by a lack of good labels and desperately need improved foundation models. I think exploring some of these would dramatically improve the impact of the work.

Additional analysis and discussion is warranted around why contradictory results are seen for different tasks/datasets. This type of result has been a key challenge (particularly in remote sensing), so I think it warrants more discussion.

1. Although the paper aims to establish a benchmark for evaluating foundation models in remote sensing, the scope of experimentation is limited. The study only explores a small subset of possible variations and factors within the remote sensing domain, which weakens its potential to serve as a comprehensive benchmark for ICLR readers. For instance, the study could have broadened its scope by including more downstream tasks, such as object detection and segmentation. Additionally, it fails to address important variables such as geographical location, times of the day, or seasonal variations, all of which could significantly impact the models' performance in real-world applications. By focusing on just a few experimental parameters, the study provides only a partial view of the potential applications and robustness of these foundation models in remote sensing.
2. Remote sensing data includes numerous factors that could influence model performance, such as resolution, spectral band variability, and environmental conditions. However, the current study only examines resolution and spectral bands, omitting several critical factors that users in the field of remote sensing would consider valuable. For instance, controlling for environmental factors (e.g., time of year, weather conditions) or ablation studies on model and dataset size could provide more meaningful insights. Additionally, benchmarks could have varied the type of pretraining datasets used, comparing options like FAIR1M and MillionAID, which cater specifically to different data characteristics. This limited approach leaves significant gaps in understanding the models' broader adaptability and robustness.
3. One of the most significant weaknesses of the paper is its lack of coverage of remote-sensing-specific foundation models, particularly those developed with unique properties for remote sensing challenges. Models such as Scale-MAE and SatMAE are designed to address the exact generalization challenges discussed in the paper, such as resilience to spatial, temporal, and spectral variability. Scale-MAE, for instance, is specifically designed to handle resolution variations, a critical factor in satellite imagery, while SatMAE incorporates temporal and locational encoding, making it suitable for applications that require adaptability across time and space. By not including these specialized models, the paper misses a critical opportunity to benchmark the very architectures that are purpose-built to handle the complexities of remote sensing, thus limiting the practical relevance and depth of the benchmark.
4. While the paper discusses self-distillation-based models, it does not sufficiently evaluate how different pretraining methods, such as DINOv2 and EVA, might impact model robustness and generalization capabilities. Advanced pretraining techniques are known to affect the performance of foundation models differently, especially when fine-tuning for specific tasks like those in remote sensing. Including a comparative analysis of these methods could offer insights into how various approaches to pretraining influence downstream task performance and generalization in remote sensing contexts. This omission reduces the value of the benchmark, as it does not provide a complete picture of how alternative pretraining methods might improve or hinder model performance in real-world scenarios.

References

• [SatMAE] Cong, Yezhen, et al. "Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery." Advances in Neural Information Processing Systems 35 (2022): 197-211.
• [Scale-MAE] Reed, Colorado J., et al. "Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
• [EVA] Fang, Yuxin, et al. "Eva: Exploring the limits of masked visual representation learning at scale." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
• [DINOv2] Oquab, Maxime, et al. "Dinov2: Learning robust visual features without supervision." TMLR. 2024.

1. I am concerned about the label quality of the datasets used. For example, BigEarthNet V2 [8] provides a significant improvement over BigEarthNet. While I understand that it might not be feasible to use BENv2 given its release date, the authors should consider data and label quality when selecting datasets for benchmarking. The authors shall discuss how they assessed the quality of the datasets used and what impact potential label quality issues might have on their results.

2. How do the authors define the generalizability of foundation models? Given that different foundation models are trained on different datasets or combinations, comparisons may be problematic. For instance, DinoV2 uses data augmentation techniques like random cropping and scaling, which enrich the spatial resolution of the training data. However, not all foundation models use such augmentations, potentially making direct comparisons unfair. The authors should address these inconsistencies and avoid drawing broad conclusions about generalizability without accounting for differences in training data and augmentations. The authors are suggested to provide a clear definition of generalizability in the context of their study and include a detailed analysis of the training data and augmentation techniques used by each evaluated model, and discuss how these differences might impact the comparisons and conclusions drawn from the benchmark results

3. It would be beneficial to also compare the foundation models with smaller models like ResNet50. This comparison could illustrate the advantages and trade-offs of using larger foundation models versus smaller ones.

4. The paper claims that the performance gap between frozen models and full fine-tuning is relatively large for models such as ChannelViT-S, Prithvi, and Clay v1. However, I am concerned about how the learning rate was selected. Using the same learning rate across different models may not be optimal. Moreover, different learning rates for the backbone and the task-specific heads (e.g., classification, segmentation, change detection) should be considered. Without optimizing these hyperparameters for each model, the conclusions drawn regarding their performance are not convincing.

5. The authors conclude that all tested models struggle with generalizability across scales and spectral bands. While foundation models are expected to generalize well to different data sources, this relies on training with large, diverse datasets. It is therefore expected that models trained on aerial images may perform poorly on satellite images due to the inherent differences between these data sources. I recommend that the authors further investigate the relationship between pre-training datasets, data augmentations, and model generalizability.