Lightweight, Pre-trained Transformers for Remote Sensing Timeseries

Thank you for your review.

• Understanding the proposed method

  Thank you for your suggestion. We would be happy to clarify our framework and our motivations - are there any specific questions you have about the framework which we could answer either in this response or in the paper itself?

  For clarity, we include some of the motivations and descriptions we currently include in the paper:

  In the introduction (Section 1), we motivate the method by describing the following attributes of remote sensing, and desired characteristics:

  • The ingestion of highly multi-modal data: We aim to leverage the different modalities of data being captured by satellites around the Earth. One of the ways we achieve this is by introducing learned encodings to the transformer method which specifically communicate to the model which sensors a token is constructed from (we describe this in Section 2.2). In addition, our model ingests both raw (e.g. optical) and derived (e.g. landcover-maps) products (we describe this in section 2.1). We hypothesize that this allows the model to map raw data to semantic meaning during the pre-training phase.

  • A strong emphasis on the temporal dimension: Unlike comparable MAE methods (such as ScaleMAE or SatMAE) designed for remote sensing data which treat the data as images (and therefore emphasise the spatial dimension), our model ingests pixel timeseries. We do this because for many remote sensing tasks, measuring change over time is critical; modelling remote sensing data as timeseries is therefore a common practice among remote sensing practitioners.

  • An emphasis on computational efficiency: As described in Section 1 (and elaborated in our response to reviewer DF48, (link), computational efficiency is critical to manage costs when deploying remote sensing models. We therefore keep Presto very lightweight (something which is in part possible because timeseries inputs have a much lower dimensionality than images).

  Figure 1 illustrates the specific pre-training pipeline, with an emphasis on demonstrating the use of diverse inputs (which are both dynamic and static in time), and the masking strategies we implement to support a range of downstream tasks.

  We also include more details about the Presto architecture in Table 8.

• Literature reviews and comparison of related work.

  Thank you for suggesting the AgVision dataset. It is at a 10cm/pixel (centimetres, and not metres) resolution, which Presto is not trained to ingest (this is a factor 100 difference with our pre-training data). We discuss this limitation in Section 5 of our paper.

  In addition, thank you for suggesting the Seasonal Contrast Paper - we have added a comparison to it in Figure 3a. In addition, we also compare to Geography-Aware Self-Supervised Learning [1] in the same table.

  In general, we have updated Section 1 to include more discussion of related works, including Transformer-based modelling for remote sensing time series. Thank you for this suggestion.

[1] Ayush, Kumar, et al. "Geography-aware self-supervised learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

Thank you for your review. We address your points below.

1. Presto’s performance on scene classification tasks:

   We agree with your points in Weaknesses 1, 2, and 3 (which are concerned with Presto’s processing of pixel-timeseries as opposed to spatial information). While we highlight these weaknesses in our paper, Presto is still performant on scene classification tasks - in particular, we show the performance vs. parameter / FLOPs tradeoff in Figure 3 of the updated paper, where Presto achieves competitive results with state of the art models for the EuroSat task while requiring orders of magnitude less parameters & FLOPs.

   i. Limited spatial context: We agree that for scene classification tasks where this spatial context is important, Presto may perform sub-optimally. We discuss this in more detail in Section A.7. However, for many remote sensing tasks spatial context is much less important than temporal context, and it is therefore common for remote sensing practitioners to train pixel-timeseries models [1,2,3,4,5] (we discuss this in Section 1 of our paper). Presto addresses the use case of these practitioners explicitly, while remaining performant on scene classification tasks.

   ii. Lossy aggregation of spatial information: We agree that our token aggregation for spatial information is lossy (and bring this point up in Section A.7). As you note in your review, we plan on addressing this limitation in future work.

   iii. Plateauing performance with Increasing input resolution: We agree that Presto may struggle with accurately classifying images where the relevant pixels for a class are a minority of the pixels in the image. We explore this in more detail in Section A.7, including investigating performance for specific classes in Figure 7.

2. Lack of discussion on specific limitations:

   As we note in our point above, we have dedicated significant parts of the paper to discussing potential limitations (including over a page discussing Presto’s performance on tasks where spatial context is important in Section A.7, “Presto’s failure modes”). We have also included a limitations section in Section 5 in the main body of the paper, where we discuss constraints such as Presto’s inability to process very high resolution imagery.

   We would be happy to expand these discussions; is there anything you would specifically like us to explore in more detail?

3. Q1: How does Presto handle missing or unreliable labels in remote sensing datasets?

   We pre-train Presto to learn representations of remote sensing data. We show in Table 3 that models with very few parameters can be learned on top of these representations, allowing Presto to be used even when very few labels are available.

   For example, Presto excels at the CropHarvest task (in which as few as 203 labels are available, as shown in Table 2).

4. Q2: Can Presto effectively handle tasks that require a high level of spatial context, such as scene classification challenges? If not, are there any plans to address this limitation in future work?

   Presto is competitive with SoTA models in the EuroSat scene classification task, while requiring orders of magnitude fewer parameters and FLOPs (Figure 3).

   However, as we discuss in Section A.7, improving Presto’s capability even more in this regime is an area of future work.

5. Q3. How does Presto perform when the input resolution increases? Are there any specific challenges or limitations observed in accurately predicting specific classes at higher resolutions?

   In Section A.7 and Figure 7, we discuss Presto’s failure modes. Specifically, we highlight that in scene classification challenges where the predictions are dependent on the occurrence of objects in subregions of the image, Presto’s performance does not improve as input resolution increases.

   Example classes in the EuroSat class (which we plot in Figures 7 and 8) include the river and highway classes.

6. Q4: Are there any privacy concerns associated with the use of Presto for collecting information about individuals who are unaware that data is being collected? How can these concerns be addressed when deploying Presto in collaboration with local communities and stakeholders?

   In our Ethics statement, we note the privacy concerns raised by [6] for all remote sensing models, which can collect information about individuals who are unaware data is being collected.

   We therefore recommend that Presto (and remote sensing models in general) be deployed in collaboration with local communities and stakeholders, as is discussed in more detail by [7], [8] and [9].

Please let us know if these responses have helped address your questions and concerns, or if not what uncertainties remain.

[1] Marc Rußwurm, Nicolas Courty, Remi Emonet, Sebastien Lefevre, Devis Tuia, and Romain Tavenard. End-to-end learned early classification of time series for in-season crop type mapping. ISPRS Journal of Photogrammetry and Remote Sensing, 2023.

[2] Vivien Sainte Fare Garnot, Loic Landrieu, Sebastien Giordano, and Nesrine Chehata. Satellite image time series classification with pixel-set encoders and temporal self-attention. CVPR, 2020

[3] Charlotte Pelletier, Geoffrey I Webb, and Franc¸ois Petitjean. Temporal convolutional neural network for the classification of satellite image time series. Remote Sensing, 2019.

[4] Sherrie Wang, Stefania Di Tommaso, Jillian M Deines, and David B Lobell. Mapping twenty yearsof corn and soybean across the us midwest using the landsat archive. Scientific Data, 2020.

[5] Tomislav Hengl, Jorge Mendes de Jesus, Gerard BM Heuvelink, Maria Ruiperez Gonzalez, Milan Kilibarda, Aleksandar Blagoti´c, Wei Shangguan, Marvin N Wright, Xiaoyuan Geng, Bernhard Bauer-Marschallinger, et al. Soilgrids250m: Global gridded soil information based on machine learning. PLoS one, 2017.

[6] Devis Tuia, Konrad Schindler, Beg¨um Demir, Gustau Camps-Valls, Xiao Xiang Zhu, Mrinalini Kochupillai, Saˇso Dˇzeroski, Jan N van Rijn, Holger H Hoos, Fabio Del Frate, et al.

• Evaluation (continued)

  • Section 3.2 discusses performance in image based tasks despite the model not containing mechanisms for spatial modelling, it basically operates as a deep fully connected network on single pixel features. It has to be clarified what can be derived from such a comparison.

    We agree that Presto is primarily designed for pixel-timeseries. Note that because Presto always gets a lat/lon token as input, in addition to other channel tokens (like the RGB one), there are always at least 2 tokens that can attend to each other in self-attention layers.

    Our goals when benchmarking Presto on scene classification tasks are:

    • Comparing Presto to comparable self-supervised models.: Since all globally pre-trained self-supervised models for remote sensing ingest image-based datasets, comparing Presto to works like Seasonal Contrastive Learning, SatMAE or ScaleMAE require us to evaluate Presto on image-based datasets.

    • We show that even a pixel-based small, efficient model with a simple spatial aggregation scheme achieves good performance on some scene classification tasks. Rather than outperforming very large models pre-trained specifically for spatial modelling, we offer a new choice that can be optimal for different preferences w.r.t. the performance/efficiency trade-off.

    • Demonstrating the wide applicability of Presto to different types of tasks. Users might be interested in one model that handles both pixel-based and image-based tasks, for ease of use.

  • In Table 4 it appears that both baselines outperform Presto.

    Yes - we discuss this in Section A.3.1, and hypothesize that this is in part due to Presto’s (currently) coarse approach to modelling spatial information.

  • In Table 5 experiments it is shown that Presto outperforms baselines specifically trained on spatial data which is very surprising given that Presto has no understanding of spatial relations and that the MAE family of models are very similar in terms of their pre-training objective. This most probably can be attributed to the fact that there is a train/test input size discrepancy for the baselines. Given that both models operate out of domain not a lot can be derived from this comparison.

    We begin by noting that while Presto approaches the performance of the other MAE models, the best performing model given a full-resolution input is the ScaleMAE model, which achieves an accuracy of 0.960. In terms of train/test discrepancy, we use the (widely used) splits provided by [6], as implemented in the TorchGeo library. These additionally reflect the splits available in the ScaleMAE codebase [link], suggesting we used the same train/test splits as the other models.

    If by input size you are referring to the resolutions: we note that the ScaleMAE includes the exact same experiment (Table 11), where they compare ScaleMAE and SatMAE performance on EuroSat for input resolutions 16, 32 and 64. In addition, the ScaleMAE model has been specifically designed to work for different input resolutions; their abstract states: "we present Scale-MAE, a pretraining method that explicitly learns relationships between data at different, known scales throughout the pretraining process." Hence, we do not agree that the baselines are operating out of domain, at least not to a greater extent than Presto (which has also been pre-trained just on 10m/pixel resolution data).

  • Experimental results should include more performance metrics including pixel based and class based metrics. I propose to move tables from supplementary material to main paper.

    Thank you for this feedback - we have been consistent with the metrics reported by (i) the datasets themselves, and (ii) works which benchmark against them. Which specific metrics would you additionally like to see reported?

  • Also, several captions need to provide more details to make them complete for understanding (Table 1: class or pixel based metric, Tables 4, 5: what are top, bottom?).

    Our apologies for the lack of clarity. Table 1 computes pixel-based metrics.

    • In Table 4, the top and bottom represent different inputs (S1 and S2), as denoted by the “Data” column
    • In Table 5, the top row of Presto represents results with RGB inputs, and the bottom with multispectral (MS) inputs.

    We have updated the tables and the task descriptions in Section 3.2 to make this clearer.

    Please let us know if we have misunderstood which top / bottom you were referring to.

[6] Neumann, Maxim, et al. "In-domain representation learning for remote sensing." arXiv preprint arXiv:1911.06721 (2019).

Thank you for your thoughtful feedback.

• Additional discussion of related works: Thank you for highlighting these papers. We already cite [1] in our introduction, in the paragraph titled “A highly informative temporal dimension”. Based on your suggestion, we add a paragraph in our introduction discussing the use of transformers for remote sensing time-series.

• Importance of model size: Our apologies for not being clearer in the paper. It is indeed surprising, but in practice, a very large number of remote sensing applications are bottlenecked by computation. We've seen this in our own collaborations with public agencies deploying remote sensing algorithms. Even in rich, well-resourced countries, governments often do not have the computational resources (yet) to deploy highly compute-intensive algorithms, and this is far worse in poorer countries. Most such applications are currently using less effective algorithms to save on computation. We highlight quotes below from practitioners which emphasise the importance of model size - not just for latency, but for total cost:

  • "Becoming Good at AI for Good" [3] by authors from Microsoft, has a subsection titled “Model development with resource constraints”. We quote the following specific use case from this paper:

    “ Robinson et al. [51] trained a fully convolutional neural network (CNN) on over 55 terabytes of aerial imagery to create a high-resolution land cover map over the United States. Differences in seconds of running time per batch translate to hundreds of dollars in the cost of the final computation. Here, a larger, state-of-the-art model would incur a ~270% increase in the cost of the final computation for a fractional increase in performance metrics such as accuracy and intersection-over-union, and so a trade-off in favor of lowering the cost was made.
    

  • The European Space Agency’s World Cereal program, which mapped cropland at a global scale at a 10m resolution, shared a blog post of their experience (https://blog.vito.be/remotesensing/worldcereal-benchmarking). They emphasise three key criteria when selecting models:

    “(i) Quality-related (e.g. how accurate are the results?), (ii) Performance-related (e.g. what computational resources are needed?), (iii) Requirements-related (e.g. how much and what kind of training data is required?)”
    

  • The SoilsGrid project [4] mapped multiple soil metrics at a global scale, at a 250m resolution. We quote two areas from their paper where the discuss the computational costs, and modelling considerations as a result of these costs:

    “The total size of all input and output data used to generate SoilGrids exceeds 30 TiB, so that a first step in preparing SoilGrids250m was to obtain a Synology 12-Bay NAS storage server with 60 TiB space. Handling such a large data set presented major challenges considering computational complexity and network bandwidth limitations.”
    

    They omit kriging, which may increase performance, because of the computational costs:

    “... overall kriging of residuals for global land mass does not seem to be necessary nor is it practical to implement for billions of pixels: it would only marginally improve the accuracy of predictions at high computing costs.”
    

  • MOSAIKS [5] develops a feature-database from satellite imagery specifically with the goal of reducing the computational cost of applying machine learning algorithms to satellite imagery. We quote their introduction:

    “The resource requirements for deploying SIML technologies, however, limit their accessibility and usage. Satellite-based measurements are particularly under-utilized in low-income contexts, where the technical capacity to implement SIML may be low, but where such measurements would likely convey the greatest benefit. For example, government agencies in low-income settings might want to understand local waterway pollution, illegal land uses, or mass migrations. SIML, however, remains largely out of reach to these and other potential users because current approaches require a major resource-intensive enterprise, involving a combination of task-specific domain knowledge, remote sensing and engineering expertise, access to imagery, customization and tuning of sophisticated machine learning architectures, and large computational resources.”
    

[3] Kshirsagar, Meghana, et al. "Becoming good at AI for good." Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 2021.

[4] Hengl, Tomislav, et al. "SoilGrids250m: Global gridded soil information based on machine learning." PLoS one 12.2 (2017): e0169748.

[5] Rolf, Esther, et al. "A generalizable and accessible approach to machine learning with global satellite imagery." Nature communications 12.1 (2021): 4392.

Thank you for your thoughtful response.

1. Comparing the proposed method with similar models and pretraining techniques specific to pixel timeseries from existing literature.

   While we agree that comparing the model to existing pre-trained pixel time-series models would be useful, no globally pre-trained models exist. We discuss this in Section 1:

   While several pre-training methods have been proposed for transformers with remote sensing timeseries [1,2,3], these have not aimed at multi-task, global applicability, having been pre-trained and evaluated on highly local areas (e.g., central California) and evaluated only for a single task (e.g., crop type classification).

   In order to comprehensively evaluate Presto, we therefore:

   • Compare Presto to the state of the art model for the task at hand, where possible (e.g. [4,5]).
   • Compare Presto to pre-trained remote sensing models which treat remote sensing data as imagery [5,6,7] (requiring us to run Presto on the EuroSat dataset).

2. Yet, detailed ablations and analysis are necessary to ascertain their impact in contrast to standard architectures.

   We completely agree that ablations are always helpful to better understand how design choices affect a model’s performance. We have included the following ablations for the model:

   • (at your suggestion) ablations on the effect of pre-training, in Tables 5, 7 and 12
   • (at your suggestion) ablations on model size in Table 8
   • ablations on the pre-training masking strategies, in Table 1

   We note that further ablations on model design choices would be tricky in our case, since the model would not function without them (e.g. without the channel encodings, the decoder would not know what to reconstruct).

[1] Yuan Yuan and Lei Lin. Self-supervised pretraining of transformers for satellite image time series classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:474–487, 2020

[2] Yuan Yuan, Lei Lin, Qingshan Liu, Renlong Hang, and Zeng-Guang Zhou. Sits-former: A pretrained spatio-spectral-temporal representation model for sentinel-2 time series classification. International Journal of Applied Earth Observation and Geoinformation, 106:102651, 2022.

[3] Yuan Yuan, Lei Lin, Zeng-Guang Zhou, Houjun Jiang, and Qingshan Liu. Bridging optical and sar satellite image time series via contrastive feature extraction for crop classification. ISPRS Journal of Photogrammetry and Remote Sensing, 195:222–232, 2023.

[4] Gabriel Tseng, Hannah Kerner, and David Rolnick. TIML: Task-informed meta-learning for crop type mapping. In AI for Agriculture and Food Systems at AAAI, 2022.

[5] Steve Ahlswede, Christian Schulz, Christiano Gava, Patrick Helber, Benjamin Bischke, Michael Forster, Florencia Arias, Jorn Hees, Begum Demir, and Birgit Kleinschmit. Treesatai benchmark archive: A multi-sensor, multi-label dataset for tree species classification in remote sensing. Earth System Science Data, 2023.

[6] Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David B. Lobell, and Stefano Ermon. SatMAE: Pre-training transformers for temporal and multi-spectral satellite imagery. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), NeurIPS, 2022

[7] Colorado J Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell. Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning. arXiv preprint arXiv:2212.14532, 2022.

[8] Oscar Manas, Alexandre Lacoste, Xavier Gir´o-i Nieto, David Vazquez, and Pau Rodriguez. Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data. In CVPR, 2021.

Thank you for your review. We address specific points below:

• Resolution constraints: We agree that Presto is only tested at the resolution of freely available satellite imagery (and highlight this as a limitation and area of future work in Sections 5 and A.3.1 of the paper). We note that the EuroSat runs are run at a variety of (decreasing) image resolutions - Presto’s performance remains relatively consistent as image resolution decreases.

• Small performance gains: We emphasise that in many cases, we are benchmarking against state-of-the-art methods for a task. The main contribution of Presto is not to achieve state-of-the-art results on some benchmarks; it is instead to introduce a computationally efficient model which can be applied to a large range of remote-sensing data formats. We emphasise that no other pre-trained model would be capable of running on all of these benchmarks, let alone achieving competitive results on them.

  In addition, we do note that performance gains are considerable for CropHarvest (Table 3), and for the EuroSat tasks (Figure 3) (especially when considering the compute vs. performance trade offs).

• Set of downstream tasks tested: Thank you for this feedback. We have added other models to the EuroSat benchmark, to better illustrate the performance of Presto relative to other self-supervised models and model architectures. Specifically, we have added the Seasonal Contrast (SeCo) and GASSL papers as additional benchmarks for Presto.

  We note that other self-supervised methods cannot ingest the full set of downstream tasks we test against (for example due to their inability to ingest the full timeseries or all the sensor inputs). This ability to process diverse modalities of data (which we fully describe in Table 2) is one of Presto’s key strengths.

• Intuition on TreeSat performance: Our hypothesis is that the information gain from the implicit reconstruction (i.e. from Presto reconstructing S2, ERA5, Dynamic World and topography given S1) is much higher for S1 than from S2.

• Figure 2 caption: Apologies for the confusion; in our caption, we state that “In this experiment, we mask the Sentinel-2 RGB channels". There are no other channels masked nor missing - we have updated the caption to specify this. Please let us know if there are any additional changes you would like to see to make this clearer.