CropNet: An Open Large-Scale Dataset with Multiple Modalities for Climate Change-aware Crop Yield Predictions

While the CropNet dataset itself appears to be a good contribution, there should be a more elaborate discussion about differences to existing datasets in this realm. Additionally, I see several issues with the experiment setting and the conclusions drawn from it.

Dataset Discussion: Your main focus and argument for this dataset is the fact, that it is combines the three data modalities (whereas other works only have a subset of those available) and that it is much larger in terms of storage size in GBs. However, as a "dataset paper" a discussion and comparison around the context in which this dataset is placed needs to be more elaborate in my opinion.

• Potentially missing references to these datasets Iizumi and Sakai 2020 and EarthStat
• while storage size can be a feature of comparing datasets, it is also somewhat arbitrary since it depends on many factors, like the file format, numerical datatype etc. So more GB is not simply better as you sometimes stress with the "terrabytes-sized" phrase. For example, if your stored satellite imagery would contain some additional S2 band, its memory footprint would be larger, but would it better for crop yield prediction? Vice versa, if there was a more memory efficient way to store your dataset, would it be less valuable?
• More important and necessary is a more extensive discussion about other features of the dataset and how that falls in the context of existing works. One factor would be spatial coverage, which is a tremendous issue in the geospatial ML world, where the vast majority of datasets are curated based on datasets in the Western World, but are lacking in areas where they might actually have the largest impact, Other factors, such as temporal coverage, the number of crops included, etc. should have a more central focus in your discussion

Experiments: The experiment section is where I see the most issues and is the dominant factor behind my review rating. To begin with many important details in your evaluation and analysis are missing or inaccurate:

• how do you split your data in train, validation, and test set? Multiple ways exist, but their strengths and weaknesses are of great importance in spatio-temporal data to make statements about model generalization
• How do you obtain error bars in Figure 7 for the results? Are these computed over repeated model runs for example? But then they are missing in Table 3, which does not have any interval indication.
• In section 3.2 you state as your first observation that "First, all models achieve superb prediction performance with our CropNet data". My issue with this statement is two-fold: One, "superb" compared to what? What are existing approaches that are outperformed by these methods. More generally, how is crop yield prediction done at the USDA at the moment to which you could compare your approach. Second, you make it seem like the "superb prediction performance" you are seeing is due to the CropNet dataset, but this is misleading. If you train models on a dataset and see great metrics, that does not mean your dataset is "great". Quantitative evaluation of models and the quality of the dataset in this context should be discussed separately. The quality of the dataset has of course impact to what a ML model can learn, however, just looking at model metrics do not prove that you have a great dataset.
• As your second observation in 3.2 you state that the MMST-ViT performance is due to the transformer attention mechanism, but this is merely speculative. You have no explicit experiment to underline this claim. This is of course not trivial to do, but you should not state it like a fact. You make a similar claim under 3.3 for MMST-ViT's performance (last paragraph). You make another claim like that in 3.4 last paragraph, where you do not show prove, but are speculating.
• In section 3.4 and Figure 8 you seem to argue that SSL helps with overfitting, however, you are giving no experimental support for that conclusion. Overfitting is a phenomena between a model's performance on the training data and some held out data, a validation dataset, where model performance on the training data is "better" than on the validation dataset. Figure 8 or the text does not mention which dataset split the reported values stem from. This is problematic because you seem to suggest that because MM-SSL achieves lower metric values, you have reduced overfitting, but there is no reference to what that "overfitting" gap between training and validation set is. Therefore, section 3.4 is extremely misleading given the current level of detail or even wrong if you suggest that just a lower metric on an evaluation set reduces overfitting
• I am also not sure why you motivate SSL through the lense of overfitting. In Geospatial ML the possibly greatest advantage of a SSL pretraind crop yield model would be its ability to generalize well over other areas where you have little data available but would like to make predictions. Here, models trained on just little data might be prone to overfitting, but that is different from your evaluation you are describing in your paper.

• pure dataset contribution with little methodological components or insights
• also the dataset seems to be focused on quantity consideration, while lacking detail and quality. It is not clear if this dataset was created with domain experts from the fields of remote sensing or crop yield prediction, as design choices are not well-supported (like Sentinel-2 over MODIS) and central papers/references from crop yield prediction (e.g., SCYM) are not mentioned.

1. There is no discussion section.
2. There is no limitation mentioned in the paper.
3. Not enough literature review.

The proposed dataset uses a few assumptions, which might be restrictive for dataset usage:

1. they provide only 4 of 12 bands of Sentinels-2 imagery (the motivation for this assumption is not provided)
2. only Sentinel -2 L1-C processing level is provided (although some researchers might find L2-A processing level more useful)
3. the resolution of Sentinel-2 imagery is decreased from 10x10 meters per pixel to 9x9 km per pixel (the motivation for this assumption is not clearly explained)
4. the comparison of the deep learning approaches with the conventional ,methods is not provided (in case of the yield prediction sometimes models like RF perform better)
5. the accuracy of the provided predictions does not outperform existing methods
6. the novelty of the dataset is not justified, from the provided description the dataset is a combinations of the existing dataset with some automation and reduction in functionality