KidSat: satellite imagery to map childhood poverty

• The biggest limitation of this paper in my opinion is that it is unclear if this problem should be formulated as a satellite ML benchmark at all. Since the measures in the child poverty index are in most cases not directly observable from satellite data (especially at the resolution of Landsat and S2), it's unclear what features the authors expect to be learned. They even say on L385-386: "Satellite imagery is at best a proxy for multidimensional child poverty." Should we be prioritizing the use of ML to predict variables based on satellite data when that data is at best a proxy (at worst, not informative at all?)? What are the risks of the conclusions that can be drawn from such analyses?

• The results in Table 2 show that some useful information might be present in the satellite data because all of the imagery-based models do at least marginally better than the no-imagery baselines. However, it's not clear from the results if more sophisticated methods are needed to improve results, or if more sophisticated data are needed, since all of the models perform in a similar range that is not dramatically better than a no-imagery baseline.

• There is a lot of space in the text spent on redundant descriptions of the analysis, data, and discussion. Much of this space would have been better spent on additional experiments that would help better characterize the results and their utility, and evaluation with more modeling scenarios. For example:

  • The authors only tested self-supervised foundation models that ingest images. I think simpler baselines that use the satellite data but do not learn spatial correlations, like a random forest or other pixel-based methods, would be useful baselines to include to test the hypothesis that spatial structure is actually being learned.
  • The authors say that they only used RGB imagery to "directly compare" DINO with SatMAE. However, these methods are not directly comparable anyway - for example, different setups for the temporal benchmark. This does not seem like a sufficient reason not to test the multispectral versions of the data with SatMAE.
  • Results are reported as an average across the entire test dataset. Disaggregating these results by subgroup (e.g., country or the bins in Figure A.4) would give a more nuanced understanding of model performance and differences in performance. This would also be more useful for stakeholders to understand the utility (or lack thereof) of using the methods measured on this benchmark.

• Why was the SatMAE temporal model used if you did not include temporal information (replicated the same single-timestep input)? There is no temporal information being given to the model (except maybe the year of the data?).

• Questions about the dataset

  • The Sentinel-2 archive is only available since 2015/2016 but the survey data covers 1997-2022. Does this mean less data is used in the Sentinel-2 experiments?
  • For Landsat, data was drawn from Landsats 5, 7, and 8 to cover the survey period. There are some differences between these three datasets that could increase the heterogeneity of observations in the survey period, potentially a source of error that was not discussed in the paper. Did the authors used a harmonized version of these products?

• methodological questions are not investigated in-depth. E.g., what impact does the pre-training algorithm have on downstream performance in general. Are (reconstruction-based) MAE features better than Dino (joint-embedding based) features (see for instance, Shekhar et al., 2023 for a comparison here) in general for poverty mapping? Do takeaways in that direction align with existing papers?

Shekhar, S., Bordes, F., Vincent, P., & Morcos, A. (2023). Objectives matter: Understanding the impact of self-supervised objectives on vision transformer representations. arXiv preprint arXiv:2304.13089.

• The description of the satellite data is incomplete (I am aware that the satellite specifications are listed in the supplementary material but I am referring to the preprocessed satellite imagery used as model input). For example, key information such as the processing level, the selected spectral bands, and the spatial resolution of the satellite imagery are missing. This also applies to the abstract which does not provide any information on the satellite imagery.

• A literature review on relevant remote sensing studies mapping poverty is missing. Instead, the related work section cites several less relevant datasets such as xView2 or SpaceNet 7.

• Since nighttime light data is a key modality for poverty mapping, it is surprising that the authors did not include it. Therefore, I strongly suggest enriching the dataset with nighttime light data from, for example, SDGSAT-1 (ideal due to its high spatial resolution) or VIIRS.

• The visualizations are not useful for understanding the spatial patterns the models produce since they cover extensive areas. Therefore, I request some examples of model outputs over metropolitan areas of cities known to have deprived neighborhoods (e.g., Maputo, Mozambique). This will lead to great insight into the capability of models trained on KidSat.

• For the spatial benchmark, I assume the clusters were randomly split into five folds. However, this split largely overlooks spatial autocorrelations and does not sufficiently test generalization ability across space (see Rolf et al., 2024). This limitation is also indicated by the performance of the Gaussian process regression baseline. Therefore, I recommend adding experiments for a region-based split.

1. Limited Exploitation of Multispectral Data While the "KidSat" dataset includes multispectral satellite imagery from sources like Sentinel-2, the study does not fully utilize the additional spectral bands beyond RGB. The paper mentions that DINOv2 relies solely on RGB bands, potentially limiting the model’s ability to capture information relevant to socioeconomic indicators, such as vegetation health or water bodies. Future work could include experiments with multispectral configurations or foundation models designed to leverage the entire spectral range available in Sentinel and Landsat imagery.

2. Constraints on Temporal Prediction Capability The temporal benchmark reveals challenges in predicting poverty indicators in years beyond the training period (2020–2022). This suggests that the current models may be overfitting to historical data patterns and struggle to generalize to newer datasets. Incorporating temporal encoding or exploring time-series models may improve the framework’s capability to handle time-sensitive socioeconomic changes. Furthermore, training models with time-stamped data could aid in detecting trends, which would be valuable for forecasting.

3. Fixed Input Size and Resizing Effects on Model Performance The study limits DINOv2 and SatMAE’s performance by requiring the imagery to be resized to a specific input size (224×224 for certain configurations). Resizing can lead to information loss, particularly when working with high-resolution satellite images. Evaluating models with variable or adaptive input sizes, or employing patch-based processing for high-resolution images, might preserve more spatial detail and enhance prediction accuracy.

4. Absence of Uncertainty Estimation The current approach is deterministic and does not quantify uncertainty in model predictions, which is crucial for policymakers who may rely on this data to inform decisions. Including uncertainty metrics, such as confidence intervals or probabilistic models, would provide a more comprehensive assessment of model reliability, particularly in areas with sparse data coverage.

5. Limited Comparison with Non-ViT Architectures Although the study highlights the use of Vision Transformers (ViTs), the experiments with alternative architectures, like CNN-based models, are limited and do not include any recent transformer alternatives. Comparing ViTs with other state-of-the-art architectures, particularly those optimized for spatial analysis, would help clarify the advantages and limitations of ViTs in the context of satellite imagery-based poverty estimation.

6. Dataset Coverage and Geographic Bias The dataset, while extensive, is geographically focused on Eastern and Southern Africa. This focus may limit the applicability of the benchmark models to other regions with different socioeconomic and environmental conditions. An expanded dataset that includes diverse regions globally could help generalize the findings and provide a more comprehensive tool for poverty estimation.

7. Lack of Direct Validation with Ground Truth The study’s reliance on DHS survey data as ground truth is practical, but there is limited discussion on validating model predictions with additional sources or cross-referencing with other poverty metrics. Incorporating data from other surveys, censuses, or economic indicators could enhance model robustness and provide a broader validation framework.