DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications

First, we wish to thank reviewer zT7y for their complete & thorough review of our submission.

1.Unsupported claims

1. Concern (C): The struggle to directly estimate the full vertical structure

Response (A): Reconstructing the full vertical structure (W) requires modeling a complex distribution P(W|Pix). A pixel embedding (Pix) aggregates spectral information over a 2D footprint but lacks the explicit depth-wise cues needed to resolve W directly (e.g. using a decoder head). In contrast, LDMs learn to iteratively denoise samples toward a plausible W, conditioned on aligned pixel embeds. By this process, LDMs implicitly capture the conditional distribution P(W|Pix) in a more expressive manner. We have revised accordingly.

2. C: Pixel-level alignment necessary for dense predictions

A: We agree with the reviewer that in the fine-tuning case, pixel-level alignment is not necessary for dense prediction tasks, as usually a decoder is trained on the generated patch-sized embeddings. We revised accordingly.

2. Methods And Evaluation Criteria

1. C: The use of the weighted F1 score

A: We relied on this metric to remain consistent with the performance scores reported in the literature. The authors of the PF dataset used the weighted F1 score due to high class imbalance. For the PASTIS data set and the CLC+ datasets, OA accuracy has been reported. For these reasons, we opted for the weighted F1 score. Due to lack of time, we only managed to run partial fine-tuning tests. The results indicate that the micro- and weighted- F1 scores have the same order of magnitude. For the macro F1 score, it is several percentage points lower (e.g., ~7% lower for the PF dataset). However, model ranking remains the same regardless of the score.

2. C: Inference runtime speeds of DUNIA and AnySat

A: Due to design choices and data requirements, AnySat is orders of magnitude slower than DUNIA zero-shot. Below are the wall clock times (in seconds) to generate a ~20x20 km area (~4.19M pixels), assuming a retrieval database containing 256K keys/values for DUNIA and 100 NNs. The test excludes data loading times.

For a database with 512K K/V, retrieval increased to 0.4s. For NN = 200, KNN increased to 1.88s

3. Experimental Designs Or Analyses

1. C: Effect of importance of using both S-1 and S-2 data

A: In the limited time that we had for the rebuttal, we only managed to test on heights and land cover classification (CLC+). Below are the results:

The results show that DUNIA leverages both modalities for heights and, in general, using S-2 yields more accurate results than S-1 for heights. For land cover classification, using either modality leads to similar performances. In the revised version, we will include the remaining datasets.

2. C: Reliance on timeseries data could anyway have been explored a bit

A: We have revised Sections 3 (Approach) and 4.1.1 (first paragraph) to better reflect this point. More specifically, while image composites such as a median composite may be richer than single-date images, they are still less informative than a full time series, as they only provide a median reflectance value over a given period. On the other hand, even this simple form of aggregation preserves significantly more information than a single-date image. Below are the fine-tuning results for two products (Heights, and PASTIS) using randomly acquired-in-time S-1 & S-2 imagery.

3. C: Issue raised regarding the effect of image resolution

A: We agree with the raised concern. Our tests show that image size has no effect on the quality of the resulting product. Below are performance results for different resolution images. The tests were performed in the zero-shot setting:

4. Weaknesses

1. C: How the model is supposed to perform reasonably well in cases where it does not have composite images

A: Please refer to 3.2.

2. C: The statement: outperform specialized supervised models, even in low-labeled data regimes is wrong.

A: Yes we agree with the raised concern. We have revised accordingly.

5. Questions For Authors

1. C: statement(s) about future code and/or model availability

A: We thank the reviewer for raising this point. We are currently preparing the code for publication. We will link to the repository in the finalized version.

Other questions

A: For Qs 2,3, and 4, please refer to 2.1, 2.2, and 3.3 respectively.

We thank reviewer hPcT for the thorough review, comments, and suggestions.

1. Claims and Evidence

1. Concern (C): The paper's claim that it "often outperforms specialized supervised models" is partially supported by the evidence presented

Answer (A): We agree and have included performance results in comparison to: Lang, Fayad, Liu and Tolan (et al.). We have also added an additional dataset for this task (i.e., ALS). The results below clearly show that our model outperforms all current specialist canopy height (CH) models even in the zero-shot (ZS) setting.

2. C: The analysis would benefit from more extensive comparisons across different data regime sizes to better understand the model's behavior

A: We chose the sizes based on empirical findings. We presented the lowest data size below which the products were unusable for a given task. In the revised version, we have added an ablation on lower data regimes. All models showed a degradation in performance. However, the model ranking remained the same as that presented in the original submission. Failure cases include overfitting or convergence towards a worse solution.

3. C: There is limited discussion or analysis explaining why the model performs well in low-data scenarios

A: This is mainly due to the self-supervised training with a contrastive objective. In the main text (line 69), we have included two references that discuss it, and our results corroborate their findings.

2. Methods And Evaluation Criteria

1. C: I strongly recommend that the authors compare the proposed method with well-established remote sensing foundational models

A: We had pre-trained DOFA, SatMAE, and DeCUR. Initially, they were not included due to their performance gaps on vertical structure related tasks. Due to time constraints we couldn't pretrain SatMAE++, and we couldn't find any implementations of SkySense. Below are the non-linear probing (MLP) results for our model (DUNIA), DOFA, SatMAE, and DeCUR. All models were probed concurrently on the same dataset. The results show that these models severely underperform (except for the PF data set) compared to our model.

2. C: Benchmarking against recognized benchmarks (e.g., BigEarthNet) to assess robustness across different years and in different geographical regions.

A: We thank the reviewer for this suggestion. First, we would like to point out that BigEarthNet is a scene understanding task (i.e., an image embedding (not pixel) is used to predict a multi-label). As such, patch-based models are expected to perform better than pixel-based ones. The results below show that even though DUNIA's encoder was not pre-trained on the BigEarthNet dataset, and scene understanding is not the intended use case for DUNIA, yet it compares favorably with the other FMs.

3. Essential References Not Discussed

A: Thank you for the suggestion. We have updated our manuscript accordingly. Please see 1.1.

4. Weaknesses

1. C: Restricted comparison with recent specialized models

A: Please see 1.1.

2. C: Limited geographical coverage (mainly French territory)

A: Please see 2.2.

3. C: Insufficient robustness testing across different years

A: We thank the reviewer for this comment. We have performed two test cases:

1. Use the pre-trained model for year 2020 and fine-tune it (MLP) for 2019 and 2021
2. Pretrain the model using data from three years: 2019, 2020 and 2021.

For both cases, performance results on canopy height mapping (below) show no significant differences year-to-year in either scenario.

Test case 1: Below is the fine-tuning performance on height estimation:

Test case 2: Below is the zero-shot performance on height retrieval:

4. C: Limited analysis of low-data regime behavior

A: Please see 1.2.

5. C: Cross-validation across different environmental conditions

A: We compared DUNIA in the Zero-Shot setting (ZS) across France's 10 major ecological regions (GRECOs A to J). These regions span 12 of the 18 Köppen-Geiger climate types found across Europe and geographically range from lowland plains and coastal plateaus to mountainous terrains. Standard deviation on the height, cover, and CLC+ metrics across the 10 GRECOS regions was respectively 0.28 m, 0.26%, and 0.68%. Indicating similar performance across different environmental conditions.

We would like to thank reviewer WDGb for the thorough review and the helpful comments.

1. Methods And Evaluation Criteria

1. Concern (C): There may be at least one baseline that should be compared against for proper contextualization in the literature.

Answer (A): Thank you for this comment, also raised by hPct. We have added four additional baselines for the canopy height mapping task and one additional dataset (canopy height from ALS). Regarding the foundation model baselines (FMs), we have benchmarked against three new recently proposed FMs. We invite reviewer WDGb to read our responses to reviewer hPct regarding the performance results.

2. Essential References Not Discussed

1. C: It does not seem like the paper has a “self-supervised learning for EO” related work section.

A: We agree and thank you for raising this point. Originally, related works on self-supervised models for EO were cited based on their pre-training objectives. In the revised version, we have included a dedicated section for these models. This section now discusses the previously mentioned models, the new models that were included as part of our response to reviewer hPct and Presto, the model mentioned by the reviewer.

3. Weaknesses

2.1 C: There is a lack of intuitive explanations for methodological choices

A: Thank you. We would be grateful if the reviewer could guide us towards the passages where we failed to deliver a convincing argument for a given design choice. In the original submission, we discussed the following design choices:

1. Tokenization layer.
2. Encoder and decoders.
3. Losses.
4. The choices for the two AEs.
5. Waveform generation.

We realized that the reliance on image composites (i.e., median reflectance of a time series) was not entirely clear, instead of using a time series as input. This has been addressed.

2.2 C: Some figures and tables are unclear and difficult to interpret

A: Please see our responses to your concerns 4.3. and 4.4. In particular, we have:

1. Modified Figure 2.
2. Modified Tables 1 and 2.

3. C: Minimal qualitative examples.

A: We agree. In the revised version of our manuscript, we have split Figure 3 into three figures and included additional maps / product. We hope that this change satisfies this concern.

4. Other Comments Or Suggestions

1. C: The methodology section would benefit from more intuitive descriptions

A: Please refer to our response in 3.2.1

2. C: Including a small figure to clearly demonstrate the advantages of pixel-sized vs. patch-sized embeddings

A: Thank you. We have included a new figure in the Appendix of the revised version, which shows the difference between the two variants and the loss of detail induced by patch-sized embedding models.

3. C: Tables 1 and 2 are really hard to read and should be restructured to improve readability

A: We appreciate your suggestion. In the revised version, Tables 1 and 2 are now four tables. Please refer to our response to reviewer 7y1C (Concern 1.2) on this subject.

4. C: Figure 2 has a ton of details which makes it hard to follow

A: We have simplified Figure 2, only keeping the main blocks that help to understand the methodology. The original Figure 2 has been moved to the Appendix as a reference.

5. Questions For Authors

1. C: Table 1 suggests DUNIA underperforms on PASTIS significantly - why do the authors think that is?

A: Our objective with DUNIA is to ensure accessibility and broad applicability. Thus, several compromises had to be made. One of them was not to rely on time series (TS) data as input, but rather a median composite that still retains some phenological information. As an advantage, this would:

1. Alleviate the need to store large volumes of data.
2. Broader applicability, especially over areas with persistent cloud cover, where having a TS over a given area would not be possible.
3. Make the model more efficient, as it only processes a single image instead of a full TS.

The trade-off is that a simple aggregation function like the median inevitably captures less temporal information than a dedicated TS-aware module. However, in six downstream tasks, we show that TS input is not always necessary to achieve strong performance. We believe that our approach has a good balance between accuracy, efficiency, and accessibility, even if it results in lower performance on datasets like PASTIS, which would strongly benefit from richer temporal information.

2. C: Could you provide more intuitive explanations or visual examples that highlight why specific methodological choices were made?

A: In the revised version, we have added an additional figure to illustrate the advantages of pixel-based versus patch-based embeddings. We have also clarified our reasoning regarding the use of composite imagery vs. TS. If the reviewer feels that further justifications are needed, we will be happy to provide additional clarifications.

First, we thank the reviewer for their feedback and their appreciation of our work. The following are our responses to your comments.

1. Questions For Authors

1. Concern (C): The Methods section could be improved by adding a brief subsection that explains the overall flow of the proposed pipeline in a simplified manner. Currently, it relies on a complex figure and separate explanations of individual modules, leaving the actual flow of the method unclear.

Answer (A): This concern was also raised by WDGb.

1. We simplified the Methods section and revised the first paragraph of Section 3 (Approach) to make it easier for the reader to follow.
2. We have simplified Figure 2, only keeping the main blocks that aid in understanding the methodology. The original Figure 2 has been moved to the Appendix as a reference.

2. C: Tables 2 and 3 could be refined; as they currently appear to present a large amount of information without clear organization.

A: This comment was also raised by WDGb. Due to space limitations, we had to format them as they were so that they could be included in the main text. In the revised version, Table 1 has been split into two tables: the first table now includes only the default configuration for the zero-shot classifier (i.e., sample size S and different KNNs), while the second table includes the modifications to the zero-shot classifier. Table 2 has also been split in a similar fashion.

3. C: It is not clear why the authors selected FMs AnySat and CROMA for comparison. Is it because these are the only two models that use - Sentinel-1 and Sentinel-2 data in their pre-training? If other models also utilize this data, an explanation for choosing only these two would be helpful.

A: Although data requirements were a factor in selecting the models we compared against, the decision to include these two models was based on the type of EO applications they target, performance compared to other EO foundation models, and also their recency and novelty. However, following the comments of the reviewer hPct, we added three more models as baselines namely: DOFA, SatMAE, and DeCUR. We invite reviewer 7y1C to see our response to reviewer hPct for the comparison results.

Again, thank you for your time reviewing our paper.