How big does your neural network have to be?: A Scaling Law Study in Multi-Spectral Remote Sensing

The methodology description and experiment setup is missing several relevant details (listed under "Questions"). In addition, the methodology has a series of weaknesses as outlined below that only allow for partial or specific statements of scaling laws. This contrasts the author's tendency to make quiet general statements about model behavior that are invalid under the experiment setup.

Methodological weaknesses:

• In line 160 you state that the labels of LoveDA and BigEarthNet are unreliable. While that might be true, what supports your choice of the BVM dataset. From looking at the BVM publication, I cannot tell for example, whether these labels were manually annotated or automatically generated like it often is the case in EO.
• A main weakness is the choice of just a single recently published dataset on which the authors base all of their conclusion for a subselection of models. In your conclusion you state that "In this paper, we have explored the scaling laws governing neural networks applied to large-scale, multi-spectral remote sensing tasks", but you have only explored a single task on a single dataset. Even within EO segmentation, many specific and relevant tasks exist, like flood segmentation, land-cover classification, crop-type mapping etc. While you recognize this limitation in the subsequent paragraph, it stand in contrast to the general statements you make throughout the paper. For example in line 353, you state that "the MS-Swin" models offer additional advantages, making them a better choice for multi-spectral segmentation tasks", which as a general conclusion does not follow your experiment setup
• Another main weakness is the evaluation scheme, mainly that you only employ Overall Accuracy as a metric for segmentation, and moreover, base all your conclusions on this metric. You state in line 284 that it is a "global measure of how well the model correctly classifies pixels across classes". But this is not necessarily true. The issue here is that for segmentation tasks with high class imbalance like also present in BVM, just predicting a background class or an overly present class will often result in high scores, but does not in itself necessitate a skillfull image segmentation model. Other metrics, like Intersection-over-Union (IoU) or F1 score also need to be considered.
• I am unfortunately confused by your "scaling efficiency metrc", as I cannot find a clear definition of how the individual factors G, P, and C are computed. Additionally, I am not sure where the motivation for this metric comes from, and would be interested in a discussion about its edge cases, where it is defined, for example if G=PxC, etc.
• your introduction of a Spectral Dependency Module is not going into detail about relevant prior research in this domain. Ideas about this have been discussed for example here

• The scope of the question the authors claim to answer in the beginning of the paper is mismatched with the scope of their experiments. The general claims made in the paper about scaling laws and architecture preferences are all based on a single dataset, which is large in terms of number of pixels but limited in terms of geographic/task diversity. The claims can only be made about that dataset, not generalized to multispectral remote sensing datasets as a whole.

• The scope of datasets included in Figure 1 is very limited. There are many popular datasets not included in this, such as EuroSat and Functional Map of the World, to name a couple.

• The writing quality is poor throughout the paper. There are many vague or poorly worded statements that obscure the authors' intended meaning, especially in Section 2.1.

• The presentation of the results in Table 1 should be improved. Some rows seem like they are not in the appropriate category, e.g., ResNet-230M is in the 50M-100M parameter category and ResNet-1550M is in the 150M-300M parameter category. The results are given for one model run so we cannot assess the variation due to randomness in the results (multiple runs should be performed with the mean and standard deviation reported).

• The authors claim the results for the MS-Swin models (using their proposed SDM module) are significantly better than without. However, these differences are likely not significant - for most comparisons they are less than 0.1% higher. Thus it is misleading to claim that their proposed model "achieves superior accuracy and scaling efficiency".

• The authors point out some of these limitations in Section 5, which I commend them for. However, the limitations are too great to accept this paper.

• lack of focus on the underlying research question: Is this a scalability study or the proposition of a spectral module?
• Results are not very informative (large models perform better on a large enough dataset). What are more fine-grained takeaways? (Maybe also beyond the remote sensing context)

• The analysis is conducted on a very reduced set of datasets from local areas which limits any findings from this work. Expanding the analysis to other datasets will benefit the work.
• The work is framed as a study on the scaling laws of neural networks on multispectral images however it mostly focus on analysing the performance of the proposed MS-swin tranformer and do not consider other transformer based architectures.
• It is difficult to draw any conclusion from the analysis presented on the scaling laws of neural networks.
• The paper needs some work to make it more cohesive.

While the study of scaling laws for multispectral images may hold interest for the remote sensing community, it may have limited relevance for a broader ICLR readership. The general finding—that increasing model and dataset size improves accuracy—is well-established, and Transformer-based architectures are already known to perform well under such conditions. The primary novelty claimed by the authors is the Spectral Dependency Module (SDM). However, I have reservations about its actual impact on performance. Specifically, I question whether capturing long-range dependencies through attention is necessary in this context, as a linear layer could potentially offer similar computational strength. Furthermore, Figure 4 suggests that the best model is the one without SDM, which raises concerns about SDM’s effectiveness. There is also an inconsistency between Table 1 and Figure 4 that requires clarification. Additionally, the significance of SDM's reported improvements in Table 1 is uncertain, as no confidence intervals, standard deviations, or other statistical measures are provided.

Other minor remarks:

• The abstract mentions that convolutional models typically only accept three channels, but they can be adapted to handle a larger number of channels.

• The accuracy comparisons in Figure 1 are challenging to interpret, and comparing across different datasets may not be entirely fair. Separate plots showing accuracy versus dataset size for a single data distribution and accuracy versus model size would make the findings more digestible.

• Symbols such as $N_p$ and $P$ should be defined for clarity.

• Distinguishing between models in Figure 7 is difficult; improved visual differentiation is recommended.

• The use of boldface in Table 2 (Appendix) could be misleading, as different datasets are compared.