Galileo: Learning Global & Local Features of Many Remote Sensing Modalities

We thank the reviewer for the thoughtful feedback, for your attention to detail and for acknowledging the strengths of our submission.

1. Technical Novelty

Our work introduces several technical innovations for self-supervised learning in general and pretraining models for remote sensing specifically:

1. We are the first to successfully fuse a diverse set of multiple sensors and products (e.g. weather) across space and time in a pretrained model. This fusion broadens Galileo's real-world applicability — as these inputs make a difference to downstream tasks [2,3,4] — and offers important empirical findings for multimodal self-supervision (e.g. the contributions of modalities to performance in Table 9).
2. We introduce contrastive learning with varied depth targets, which we show is highly effective (8% improvement on the MADOS benchmark, Table 7). Our algorithms construct target tokens from (i) linear projections of the inputs (in our local loss) and (ii) a varying number of target encoder layers (in our global loss). Galileo is the first to exploit early-exited targets in SSL - we show its effectiveness via extensive ablations.
3. Galileo constitutes a novel combination of our dual local and global SSL losses which are each novel on their own. We demonstrate the efficacy of the algorithms in isolation (Tables 6, 7) and when combined (Table 8).
4. A novel combination of existing methods can also constitute technical novelty. For example, CROMA [NeurIPS ‘23] combined MAE and radar-optical contrastive learning: these SSL methods, independently, were not novel to remote sensing or ML in general. However, its joint reconstructive and contrastive multi-modal self-supervisory method was novel.
5. We note that this work is submitted as “Application-Driven Machine Learning” as implemented by the ICML 2025 process. This has different reviewing criteria including “Originality need not mean wholly novel method: It may mean a novel combination of existing methods to solve the task at hand…so as to match the needs of the user”. This is the case for Galileo and users of remote sensing in (1) the flexibility across possible inputs, (2) the accuracy across 10 diverse benchmarks, and (3) the effectiveness in the fine-tuning regime (for resource-rich groups) and kNN & linear regime (for resource-poor groups).

2. Technical Distinction between AllDisc and PatchDisc

AllDisc samples negative examples from all patches in a batch, as opposed to within an instance. AllDisc can yield empirical gains compared to PatchDisc (e.g. 27% improvement on EuroSat, Table 6). We agree that AllDisc is a simple modification of PatchDisc. This simple modification that gives significant improvement is a strength of our method. We will update Section 2.2.1 to clarify this distinction.

3. The rationale behind the choice of PatchDisc and AllDisc

We train Galileo with PatchDisc for local and global learning (as discussed in Section 2.2.3), but we observe significant improvements when using AllDisc for global learning (Table 6), so we include both in the text. We thank the reviewer for the feedback on the choice of PatchDisc vs. AllDisc in the final combined method, and we will make this clearer in the method section.

4. Masking Strategy and Depth Details

Thank you for your attention to these important details. While space-time masking is best when learning global features (Table 6), random masking is best when learning local features (Table 7). Note that the target encoder depth also matters: prior work always used the full encoder to construct targets, but we find that a target depth of 0 is best for learning local features, and depth that varies per modality is optimal for learning global features. We are the first to vary target depth in this way, and the first to show its importance for remote sensing. We will clarify the use of random masking in Section 2.2.2 per this comment.

5. Performance and Complexity vs. CROMA

Galileo-Base outperforms CROMA-Base on image tasks (Table 1) and is less complex (CROMA-Base has 60% more parameters than Galileo-Base). Galileo is architecturally simpler than CROMA, which requires 3 encoders to process a Sentinel-1&2 image pair (an MS optical encoder, a SAR encoder, and a fusion encoder). Galileo leverages a single encoder to process inputs across space, time, spectral bands, modalities, etc. Re: dataset size, CROMA was pretrained on SSL4EO with 1M samples of 264x264 pixel images while Galileo was pretrained on 127K samples of 96x96 pixels at 24 timesteps. In total, CROMA was pretrained on >2x as many Sentinel-2 pixels, so Galileo’s performance is not explained by dataset size.

Thank you for your review, and your detailed questions.

Weaknesses

1. Evaluation on only Sentinel 2 tasks

We agree that benchmark datasets over-represent Sentinel-2 based tasks, providing few opportunities to test the value of Galileo’s many modalities that reflect the wide diversity of input sensors used in the real world [1,2,3]. We benchmark on Sen1Floods11 (includes Sentinel-1 inputs), and CropHarvest (includes highly multimodal inputs, including topography and weather). Galileo excels on both of these datasets.

We ablate all of our input modalities in Table 9 and find that - even for Sentinel-2 tasks - diverse modalities significantly helps performance (e.g. MADOS gains 4% with a model trained on VIIRS night lights compared to without).

The above points show that the variety of modalities is useful, not a hindrance.

2. Technical novelty

Our work introduces several technical innovations for self-supervised learning in general and pretraining models for remote sensing specifically:

1. We are the first to successfully fuse a diverse set of multiple sensors and products (e.g. weather) across space and time in a pretrained model. This fusion broadens Galileo's real-world applicability — as these inputs make a difference to downstream tasks [2,3,4] — and offers important empirical findings for multimodal self-supervision (e.g. the contributions of modalities to performance in Table 9).
2. We introduce contrastive learning with varied depth targets, which we show is highly effective (8% improvement on the MADOS benchmark, Table 7). Our algorithms construct target tokens from (i) linear projections of the inputs (in our local loss) and (ii) a varying number of target encoder layers (in our global loss). Galileo is the first to exploit early-exited targets in SSL - we show its effectiveness via extensive ablations.
3. Galileo constitutes a novel combination of our dual local and global SSL losses which are each novel on their own. We demonstrate the efficacy of the algorithms in isolation (Tables 6, 7) and when combined (Table 8).
4. A novel combination of existing methods can also constitute technical novelty. For example, CROMA [NeurIPS ‘23] combined MAE and radar-optical contrastive learning: these SSL methods, independently, were not novel to remote sensing or ML in general. However, its joint reconstructive and contrastive multi-modal self-supervisory method was novel.
5. We note that this work is submitted as “Application-Driven Machine Learning” as implemented by the ICML 2025 process. This has different reviewing criteria including “Originality need not mean wholly novel method: It may mean a novel combination of existing methods to solve the task at hand…so as to match the needs of the user”. This is the case for Galileo and users of remote sensing in (1) the flexibility across possible inputs, (2) the accuracy across 10 diverse benchmarks, and (3) the effectiveness in the fine-tuning regime (for resource-rich groups) and kNN & linear regime (for resource-poor groups).

Questions

1. AllDisc vs. PatchDisc

AllDisc samples negative examples from all patches in a batch, as opposed to within an instance. AllDisc yields significant empirical gains compared to PatchDisc (e.g. a 27% improvement on EuroSat, Table 6). We will make the following change to the “Loss function” part of Sec. 2.2.1 to clarify this (updates in italics): "To encourage globally discriminative representations, we extend the PatchDisc loss to better discriminate samples in a batch. We achieve this by sampling negative examples from the entire batch, as opposed to within the sample."

2. Maximal input tokens

We trained all the final Galileo models on a single GPU with 80Gb of memory, but smaller mini-batches allowed us to run many of our pretraining experiments on a consumer-grade 24Gb GPU. We agree that a large number of tokens is a challenge when incorporating many modalities, timesteps and spatial dimension: when subsampling the inputs (Appendix B.2) we selected (size, timestep) combinations so that the maximum number of input tokens was 1500. The masking percentage during training was 10% of tokens unmasked and 50% of tokens decoded; this was consistent for both the global and local branches.

Other comments

1. Relating strategies to past papers

Thank you for this feedback. To better balance our own descriptions with relations to other work, we will first provide a self-contained summary of our method at the start of Sec. 2 (and we will move the preamble on motivations to the supplementary material to make space for this suggestion).

[1] https://arxiv.org/abs/2312.03207

[2] https://essd.copernicus.org/articles/15/5491/2023/

[3] https://soil.copernicus.org/articles/7/217/2021/

Thank you for your thorough review; we are glad you recognize the meaningful challenges we address with Galileo, since this was one of the primary objectives of this work.

Balancing the losses

We combine the global and local objectives via a simple average (Section 2.2.3), and demonstrate via ablations that Galileo can learn from this combination effectively (Table 8). There is no balancing of the losses, and no multi-objective tuning to explain.

We also measure token representation similarities in Table 2: combined retains the within-sample diversity of local and achieves between-sample diversity between global and local. The optimal diversity within or between samples is unknown and likely task-dependent; we offer these measurements to complement our benchmark evaluations.

Operations on input scale and shape / resizing + image processing

Galileo is the first to harmonize multi-modal and multi-resolution inputs for remote sensing in this way. Relative to FlexiViT, we modified the model architecture (“Flexible input shapes”, Section 2.1.2) and the training recipe (described in detail in Appendix B.2). Remote sensing analyses may require data as pixel timeseries [1] or imagery [2] across many sizes and shapes; Galileo is the first model that can process any of these diverse inputs for a range of spectral bands and other products (e.g. weather, topography).

Lack of related research in the image frequency domain

We cite remote sensing papers that aim to learn both high and low frequency features, including ScaleMAE and SatMAE++ (which include in our evaluations in Tables 3 and 4). We welcome the suggestion of additional references.

Answers to questions

1.a. Does this construction match the real world?

Yes: different real world applications of machine learning for remote sensing use very different input shapes, which reflects our subsampling. For example, Skylight [2] uses single-timestep S2 imagery or multi timestep S1 imagery, WorldCereal [1] uses highly multimodal (S2, S1, topography, weather) pixel timeseries, and Global Plastics Watch [3] uses both images and pixel-timeseries. Galileo is the first model that can support these diverse, real world use cases.

1.b. Could the authors explain the scale issue semantically?

Galileo is the first pretrained model to incorporate inputs of significantly different resolutions (e.g. ERA5 at ~30km/pixel vs. Sentinel-2 at 10m/pixel). For inputs significantly coarser than 10m/pixel, we treated them as unchanging in space (i.e. as a pixel timeseries).

We use the term “multiscale” to describe the scale of the targets (e.g. vessels, which occupy a few pixels, vs. glaciers which span kilometres). We apologize for the confusion and will clarify this in the preamble to Section 2.

2.a. Is "time" important in this field?

Yes: previous work has found the temporal dimension to be critical, e.g. for agricultural land cover mapping like our PASTIS benchmark (and even more important than the spatial dimension) [4]. Many large scale mapping efforts (i.e. segmentation) model pixel timeseries to focus on the time dimension instead of the spatial dimension [2, 5, 6].

2.b. Are there any experiments on tasks related to time series?

Our paper contains 3 evaluation datasets with a temporal dimension, two of which only contain the temporal dimension. Results for PASTIS (agricultural land cover segmentation) are in Table 4. Table 1 of [7] found that ignoring PASTIS’s time dimension performed significantly worse. Results for the CropHarvest and Breizhcrops pixel timeseries tasks are in Table 5. Galileo is the best or second-best method across time-series datasets.

[1] https://essd.copernicus.org/articles/15/5491/2023/

[2] https://arxiv.org/abs/2312.03207

[3] https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0278997

[4] https://arxiv.org/abs/1901.10503

[5] https://soil.copernicus.org/articles/7/217/2021/

[6] https://esa-worldcover.org/en

[7] https://arxiv.org/abs/2107.07933

Thank you for the thorough review and excellent suggestions.

Claims and Evidence

1. Flexible input shapes

Fig 3 and Tab 11 show Galileo performs well across varying patch sizes. Tabs 3 and 5 show Galileo takes both pixel timeseries and images. We also run MADOS segmentation with a smaller patch size of 2 here.

2. Benchmark task

We agree that EuroSat alone is insufficient. We showcase SoTA results on ten diverse datasets incl. FloodBase Sen1Floods11 and NASA CropHarvest, created by real-world practitioners.

FTW [AAAI 25] is concurrent per ICML policy. FTW cite PASTIS as a “relevant dataset for comparison"; we evaluate on PASTIS in Tab 4.

3. Baselines

We show that Galileo is the best on average against 16 other pretrained RS models, which in turn gain over specialized models, e.g. SatMAE outperforms a fully supervised ResNet50 (their Tab 9) and Presto outperforms a SITS Former (their Tab 6).

We run new finetuning results here with a SwinViT (Satlas) as requested. We update our rankings (Tab 14) here w/ finetuning: Galileo-Base has the best average rank.

SatCLIP [AAAI 25 = concurrent work] is a location encoder (a function of lat/lon coordinates) while Galileo is a visual encoder (a function of image/pixel spatial/temporal inputs). SatCLIP only compares itself to other location encoders.

4. Contrastive learning in the pixel space

This describes our use of targets from linear projections of input pixels (this work) instead of from deep representations (prior work). See Sec 2.2.2 at “Target Depth” and Fig. 2 (step 4). We thank you for highlighting the insufficient clarity of this novelty: we will edit Sec. 2.2.2 to explain. Galileo is the first model to exploit contrastive learning on multiple depths in SSL - we show its effectiveness via extensive ablations (e.g. 8% gain on MADOS: Tab 7).

5. Proposed loss

PatchDisc losses outperform MSE losses by 5.4% on MADOS, 0.5% on Sen1Floods11, 1.6% on CropHarvest, and 2.3% on EuroSat (Tab 8). These controlled ablations verify the value of our loss for pretraining on remote sensing data.

"Extension of FlexiViT and I-JEPA"

Galileo is not simply an extension: it introduces 1) a first-of-its-kind encoder for our set of more general inputs, 2) a new latent masked modelling method with multiple depth targets, 3) a novel combination of two methods, which are themselves novel (our local and global losses). We describe these in detail here.

Questions

1. Local task: Please see "contrastive learning in the pixel space" above.
2. Patch size: Please see “flexible input shapes” above. We provide new results for patch size = 2 as an alternative to patch size = 1 ( size = 1 results in too many embeddings and excessive computation).
3. Galileo vs. DeCUR, CROMA: No one model is best for all tasks, but Galileo is best on average (please see the rankings here). CROMA and DeCUR’s image specialization may help on m-BigEarthNet and m-Brick-Kiln.
4. AllDisc loss temp: We fix the temp and will note this in Sec. 2.2.1.
5. Contrastive shortcuts: Great observation. Our local loss prevents this shortcut, as position/channel/time embeddings are not added to our input projections. Our combined algorithm stabilizes training (100% of runs achieve >80% on EuroSat in Tab 8 vs. 63% in Tab 6). We emphasize these are not simply I-JEPA losses (link).
6. Location inputs: Galileo takes optional location inputs and achieves top or near-top benchmark results (Tab. 3-5) without them (only CropHarvest has location inputs); see our new result (last row) in link. We include locations because they are included in real world applications [1].
7. Batch size effects: AllDisc experiments used the largest batch size that fits in memory (a common heuristic). As requested, we run our global feature learning algorithm with a smaller batch size (link). A smaller batch size doesn't hurt performance.
8. Inputs and targets: Correct, all data sources are inputs and targets for both the global and local tasks. Patch sizes are randomly sampled from the same distribution for the global and local tasks. We will edit Sec. 2.2.1 and 2.2.2 to reinforce these points.

Other comments

1. Hard to parse Figs 1, 2: Thank you for highlighting this. We will edit Fig 1 to relate the (x_1, x_3) paths to the local and global tasks in Fig 2 and simplify Fig 2 by removing the grid in step 5
2. [terms] are used almost interchangeably: Thank you for highlighting this potential confusion. We will standardize terms to use “local/global” for loss targets and branches vs. “shallow/deep” for target depths.

[1] https://essd.copernicus.org/articles/15/5491/2023