Thank you for your constructive review. While your specific feedback with respect to satellite data is quite valuable, we respectfully disagree with the broader criticism that ”there are too many nuances within the separate domains that can be all included in one paper and allow for the conclusions that the authors draw.” We address this issue in detail in our general response (Clarification 1), where we note that performance on standardized benchmarks is the main way of comparing methods in ML and that evaluating on a large subset of the tasks used by developers of specialized FMs themselves to claim success is sufficient to establish our own main claim. If these tasks do not address relevant settings then our claim is still true but our results imply that better benchmarks need to be developed and agreed upon in the relevant communities.

We address your specific weaknesses and questions in two separate comments below.

Weaknesses

1. [”Earth Observation (EO) data encompasses a wide array of sensors and modalities. Additionally, there are a diverse set of tasks, classification, segmentation and pixel-wise regression just to name a few for optical data. [...] I would argue it is difficult to make your overall claim, if you do not compare across EO tasks, for example including segmentation, as this would align more with the premise of FMs.”]
   Many satellite FMs do not handle all data formats, e.g. SatMAE is built for multispectral Sentinel2 data while Scale-MAE and Cross-scale-MAE are pretrained on RGB. This leads to a lack of unified benchmarks, which we have made a best-effort attempt to handle by focusing on a widely used format (Sentinel2). Furthermore, most satellite FMs highlight their classification result (e.g. it is the first results table in SatMAE, CROMA, DOFA, etc.), so we believe the task is reasonable to focus on. While our results do not imply anything about performance on tasks like segmentation, the fact that FMs are not able to convincingly beat supervised baselines on a collection of tasks that their own authors have decided are important is strong evidence for our claim.
2. [”there are other relevant and easily accessible recent models available like Clay, Scale-MAE, the Satlas-model, or DOFA”]
   Thank you for these suggestions. Because of the lack of consistent evaluations we fine-tuned all FMs ourselves and so were constrained in how many comparisons we could make. Following your suggestion, we evaluated DOFA and Satlas on GEO-Bench (Clay / Scale-MAE have more challenging formatting / data-loaders but we will consider them in a revision, along with the rest of the tasks). As shown below, the best of these new FMs only beats DASHA by about 1.3% on average, which while better than CROMA / SatMAE is still not a convincing demonstration of the utility of large-scale pretraining (for comparison, BERT beats all supervised baselines on GLUE by about 16.8% on average). |Model|m-bigearthnet|m-brickkiln|m-forestnet|m-pv4ger|m-so2sat|EuroSAT|avg| |-|-|-|-|-|-|-|-| |DOFA-base|73.46|98.6|55.6|98.3|61.36|99.2|81.09| |DOFA-large|73.91|98.7|57.4|98.3|61.92|99.19|81.57| |Satlas|71.82 |98.80|52.75|98.2|57.51|99.17|79.68| |DASHA|72.72|98.92| 57.24|97.4|56.28|99.07|80.27|
3. [”I would argue that this is not necessarily new, because there are works that already show strong performances of supervised baselines. For example in the Prithvi by Jakubik et al. Table 4 you observe the competitive UNet performanceand that for subclasses it is often better than the FM. Similarly, in DOFA by Xiong et al., Table 1 and 2 show that fully trained supervised baselines can sometimes beat the finetuning schemes. More general a work by Corley et al. conducted a focused study around pretrained weights and demonstrated that carefully finetuned baselines are very competitive.”]
   As discussed in the following, we do not believe these works undermine our novelty:

• Jakubik et al. show that a supervised UNet is competitive with a single FM on a single non-standard task. In contrast, our claims rely on evidence that supervised methods are competitive with multiple FMs across multiple standard tasks (and across multiple different domains). Their result is only weak evidence for our claim (which they do not make) and so does not affect its novelty.
• Xiong et al. show that supervised models are competitive with a linearly probed FM, whereas we show that supervised models are competitive with a fully fine-tuned FM. Our comparison is with a much stronger method that better reflects what FM practitioners do (given enough compute) when aiming to maximize accuracy.
• Corley et al. show that ImageNet-pretraining is a strong baseline for satellite imagery when properly tailored. We categorize FMs pretrained on out-of-modality data but tailored to in-modality tasks as specialized FMs (c.f. SwinT-base in Table 2 and GPT4TS (OFA) in Table 3) and so we view that work as comparing between specialized FMs. Their unpretrained models are not competitive with the FMs they study.

4. [”the more interesting application of FMs is for fine tuning on datasets with scarce/few labels where training a model from scratch would likely just lead to over fitting”]
   We address concerns regarding data-scarce settings in the general response (Limited data). Notably, we conduct experiments that show that even when both are provided much less data, supervised baselines are competitive with specialized FMs (for satellite the drop in performance is roughly the same). Furthermore, while we view this as an interesting question, we believe experiments on benchmarks chosen by the creators of specialized FMs themselves suffice to justify our claim that the same FMs have not yet been shown to outperform supervised learning.

We’re reaching out to follow up on our rebuttal and to check if you have any further questions. In our response, we addressed your concerns about domain-specific nuances and argued for the soundness of using existing benchmarks to justify our main claim. Following your suggestions, we also conducted new experiments comparing our baselines to additional satellite foundation models and assessing them in low-data settings; we believe the results we reported further validate our main claim. We hope that these responses resolve the weaknesses outlined in your review and would appreciate a prompt response to confirm that or with further questions / clarifications. Thank you again for your thoughtful feedback!

Thank you for your positive response and useful feedback. We are glad that you found our paper relevant, easy to understand, and comprehensive, and we hope to address your concerns below:

1. [“The paper presents two "solutions" from an architectural standpoint, i.e., CNNs for genomics/satellite data and auto-AR for time series.This leads to two questions: What model should a practitioner use when facing a task in a different specialized domain? How come 1D CNNs (or, e.g., RNNs) fail to match the performance of FMs in time series forecasting? Is it just a matter of domain knowledge and data type?”]
   Thank you for the interesting questions, which we are happy to address in some detail. However, we first note that, as discussed in the general response (Clarification 3), the goal of our paper was not to promote tuned CNNs and Auto-AR as general-purpose multi-domain solutions, but rather to use them to show that specialized FMs have not yet outperformed supervised learning in these domains. Thus showing that any one baseline is competitive with state-of-the-art specialized FMs is sufficient to make our claims.

• To address your first question: DASHA, our CNN-based automatic search workflow, is the more general approach and can be generalized to many other tasks, as it can work on any data type that can be processed by a CNN. The results in our Empirical Results section also show that DASHA achieves significant improvement upon the vanilla CNN architectures. Therefore, if people want to use our findings in the paper for stronger benchmarking in other specialized domains, we recommend DASHA. While Auto-AR is a very simple forecasting baseline, it is designed for auto-regressive prediction and so it is unclear if it can be useful even for other 1D tasks.
• To address your second question: we agree that the underperformance of CNNs is an interesting question and one worth exploring further, although the strong performance of Auto-AR is sufficient for the purposes of the claims in our paper. Roughly speaking, the underperformance of CNNs (and possibly of RNNs) may be due to a smaller effective training data size when applied to forecasting datasets. This is because these datasets consist of overlapping points that are chunks of one single sequence, so CNNs will compute very similar representations for many of the data points due to translation invariance. Indeed we’ve found empirically that training CNNs on fewer chunks with less overlap significantly improves performance, despite involving fewer actual data points. Thus it may be possible to improve the performance of traditional neural networks via smarter data processing, which is an interesting direction for future research. However, the primary focus of our paper is to demonstrate that specialized FMs have not convincingly outperformed supervised baselines, so performing a more detailed investigation shifts the focus away from our main objective.
• [”In practice, the paper would benefit from some more convincing justification and evidence for the statement, "While DASHA can be applied to forecasting tasks, it is not competitive with state-of-the-art time series FMs." (Line 240).”]
  We are somewhat unsure about the specific concern here, as this claim is negative about our own method. It is justified by the reported performance in Table 3.
• [”why in Table 3 the evaluation of DASHA for the "Full setting" panel is reported as "unknown" ("-")?”] The DASHA results in Table 3 are incomplete because the computational cost of running it on one of the tasks was prohibitively large due to the large number (862) of output channels. However, as discussed above the Auto-AR results on their own suffice for our claims related to time series FMs, and we provide the (partial) DASHA results only for additional context.

2. [“The authors should add a brief discussion either in the Introduction or Related Work with references regarding simple tuned baselines”]
   Thank you for highlighting these relevant sources. We agree that referencing these papers (as well as similar work in text classification, text representation, RL, and architecture search) is worthwhile and will include them in a revised version of the paper.

I thank the authors for the detailed rebuttal. I still have some doubts regarding the proposed solution, which is split into different approaches.

If "showing that any one baseline is competitive with state-of-the-art specialized FMs is sufficient to make our claims" what is then the purpose of proposing an automated baseline and another approach such as AutoAR if DLinear, which is already a supervised baseline, scores an average RMSE close to AutoAR and FMs on the partial experimental setting (Table 3)? I still think that the actual implementation of the supervised baseline to beat FMs does matter, and indeed I consider the automated DASHA a contribution that could strengthen future FMs benchmarks.

I appreciate that the authors agreed with the related literature and found additional works in other ML fields that are related to their paper. The addition of such works would definitely strengthen the paper.

We’re reaching out to follow up on our response and to check if you have any further questions. In our rebuttal, we addressed your concerns about the choices faced by practitioners and the need for additional references about past work on baselines in ML; for the first point we also discussed and provided some empirical insights into your interesting question about the underperformance of CNNs. We hope that these responses resolve the weaknesses outlined in your review and would appreciate a prompt response to confirm that or with additional questions / clarifications. Thank you again for your thoughtful feedback!

Thank you for your constructive feedback. We are glad that you found our results comprehensive and relevant, and we hope to address your concerns below.

Weaknesses

1. ["The paper lacks an exploration of their applicability to other types of data or more complex, multi-modal tasks."]
   As noted in the general response (Clarification 3), the goal of our paper was not to promote DASHA and Auto-AR as general-purpose multi-domain solutions, but rather to use them to show that specialized FMs have not yet outperformed supervised learning in these domains. We do not make a claim that DASHA and/or Auto-AR are applicable to any tasks beyond those studied in the paper, and so investigating that is out-of-scope.
2. ["The choice of baselines (e.g., Wide ResNet and UNet for genomics and satellite imaging) is reasonable but could be expanded."]
   The central claim of our paper is that specialized foundation models struggle to beat supervised baselines, so it is sufficient to provide just one supervised approach that is competitive with FMs, which we have done for all domains under consideration. While additional baselines may add context to the results, as you mentioned this would mainly serve to reinforce the claims we already show.
3. [“Although the paper effectively highlights the efficiency of supervised workflows, the focus on computational cost may not fully account for the trade-offs in scalability and generalization that FMs can offer.”]
   While we are not sure exactly what tradeoffs this is referring to, you may be interested in the new data-scarcity experiments we report in the general response (Limited data).
4. [“For time series, the paper does not sufficiently address the unique challenges of zero-shot forecasting …”]
   As discussed in the general response (Clarification 2), our goal was to identify a broad trend across multiple domains, and so due to computational and space considerations we could not do very fine-grained investigations of specific subsets of FMs.

Questions

1. [”Could the DASHA and Auto-AR workflows be extended or adapted for multi-modal datasets or tasks involving mixed data types? If so, what modifications might be necessary?”]
   While Auto-AR is an autoregressive method that is specifically suited for time series forecasting, DASHA is likely much more suitable for multi-modal or mixed-type settings; in fact its architecture search component (DASH) was originally developed and evaluated on a very diverse set of domains. For multi-modal / mixed-type settings it is likely that DASHA would need to allow for multiple parallel backbones to process the different inputs modalities / types, whereas the code currently works with one backbone at a time.
2. [”How do the authors envision the role of foundation models evolving in these domains if computational resources continue to scale? Would certain configurations of FMs potentially start to outperform supervised baselines, or do they anticipate the observed trends to persist?”]
   As discussed in our work, FMs in NLP have dominated traditional supervised approaches since the advent of large-scale pretraining. While similar performance gains have not yet been observed in these specialized domains, we think that the availability of compute and large-scale data suggests that with better pretraining approaches and perhaps modeling they will eventually also see (specialized) FMs that outperform supervised learning. Our study highlights the need for better benchmarks and baselines for driving this progress.
3. [”Given the success of NAS in selecting optimal CNN architectures for genomics and satellite imaging, do the authors plan to explore whether reinforcement learning or meta-learning approaches might yield further performance gains?”]
   While the focus of our work was simply to establish the existence of supervised baselines that outperform specialized FMs, we agree that the success of our NAS-based approach suggests that data-driven model development (including as you mention RL and meta-learning) may yield strong performance gains, both for supervised methods and transfer-based approaches. Research in this direction is indeed a likely fruitful area for future work.

We’re reaching out to follow up on our response and to check if you have any further questions. In our rebuttal, we addressed your four concerns about the scope of our paper and the resulting choice of experiments to run. We hope that this resolves the weaknesses outlined in your review and would appreciate a prompt response to confirm that or with additional questions / clarifications. Thank you again for your thoughtful feedback!

Thank you for your constructive feedback. We are glad that you found our work comprehensive and detailed, and we hope that the additional experiments we discuss below and in our general response can address your concerns. Before that, please note that we address your broader concerns that our claims may be overstated due to not considering certain settings (e.g. small-scale data adaptation, data bias, negative transfer) in our general response (Clarification 1). In particular, because the benchmarks we use are exactly the ones used by developers of specialized FMs to claim success, they are sufficient to establish our main claim regardless of whether or not one believes that the standard benchmarks address the appropriate settings. If they do not then our claim is still true but better benchmarks need to be developed in the relevant communities.

1. ["The strong performance of supervised methods through the automated pipeline may be largely attributed to the large-scale training data ..."]
   We have addressed concerns regarding the performance of foundation models in data-scarce settings in the general response (Limited data). As explained there, while we view this as an interesting question and show some results comparing our baselines and top FMs when given limited data, we believe experiments on benchmarks chosen by the creators of specialized FMs themselves are sufficient to justify our claim that the same FMs have not yet been shown to outperform supervised learning.
2. ["... it is unclear whether the automated pipeline is more time- and compute-intensive ..."]
   We agree that this is a valid concern and have addressed it in the general response (Computational cost). As detailed there, we have compared the cost of our baselines to fine-tuning FMs on a representative subset of the tasks; our finding is that running our baselines is not unreasonably more costly and sometimes even cheaper than fine-tuning FMs on the same tasks. We will extend these experiments to all the tasks in the revision.
3. ["The assumption that “their creators make a best-effort attempt to present their own method in the best light” may not be fully reasonable. Due to the high computational costs and complexity of foundation models, obtaining optimal or even sub-optimal hyperparameters is generally challenging."]
   We acknowledge that hyperparameter optimization for specialized FMs presents a significant challenge due to their high computational costs and complexity. However, we believe that the authors of these FMs have likely made considerable efforts to tune both the architectures and hyperparameters of their models, efforts that may not be feasible for a typical practitioner looking to apply these FMs to a different downstream task. While such challenges are far less prominent with simple supervised learning baselines, we view this as an inherent limitation of current specialized FMs. Moreover, our evaluation does not involve selecting specific datasets but focuses on benchmark tasks widely used by several specialized FMs. It is important to note that these specialized FMs have already undergone extensive domain-specific pretraining with the eventual goal of improved downstream performance in these tasks. Therefore, even if hyperparameters are not well optimized, the FMs' pretraining should still reflect their ability to perform across tasks.

We’re reaching out to follow up on our response and to check if you have any further questions. In our rebuttal, we addressed your concerns about the size of the fine-tuning data, the computational cost of our baselines, and hyperparameter optimization; for the first two we also reported new experimental results that confirm our main claim. In addition, we discussed your broader concerns about the claims in our paper in the general response. We hope that this resolves the weaknesses outlined in your review and would appreciate a prompt response to confirm that or with additional questions / clarifications. Thank you again for your thoughtful feedback!

Thank you for taking the time to respond. We believe your concerns stem from (1) a misunderstanding of our goal and (2) a miscomparison when assessing the new cost experiments (although we acknowledge the latter should have been better presented). We hope to resolve this below.

(1) Our goal

Your response states “This paper is not primarily an evaluation paper aimed solely at highlighting FMs’ weaknesses in specific domains. As a technical paper, it should emphasize the performance and technical contributions of the proposed methodology.” We would like to respectfully but strongly push back on this, as the exact opposite is true: our goal is precisely to highlight FMs’ weaknesses in specific domains. This is reflected in

1. Our title
2. The last sentence of our abstract
3. Our intro: “our study’s motivating question: Do these new specialized FMs outperform traditional supervised learning applied to the same tasks?”
4. The first sentence of our conclusion
5. Your initial summary: “This paper aims to challenge the assumption that foundation models (FMs) consistently outperform traditional supervised learning methods. [...] The study highlights that, unlike in general-purpose AI tasks [...] the advantages of FMs in specialized fields remain unproven”
6. Our general response (Clarification 3)

We hope you will reassess the new results in light of our goal being not to propose methods but to evaluate FMs. In particular, to show the main claim of our paper it suffices to show supervised baselines that match FMs with a similar compute budget.

(2) Cost experiments

Your response states “supervised methods generally require significantly more computational resources compared to fine-tuning foundation models.” This assessment comes from comparing the cost of tuning AND training our baselines to the cost of just fine-tuning FMs; as alluded to in our original rebuttal and detailed below, when tuning FM fine-tuning is factored in the cost of our baselines is as good or better:

1. In genomics, fine-tuning FMs takes roughly as long as training our baseline (~0.26 GPU-hours). While developing / tuning our models takes 1.41 GPU-hours, we looked into how the listed FMs were tuned (Section A.1.4 here) and found that it was through 10-fold cross-validation, i.e. 10 runs on 90% of the data each. Thus the tuning costs of the FMs can be inferred to be roughly 2.34 = 10 x 0.9 x 0.26 GPU-hours, more than 1.5x the cost of developing / tuning our models.
2. In satellite, fine-tuning FMs takes 1.5-3 times as long as training our baselines (6.26 and 11.87 GPU-hours for the FMs vs. 3.70 for the baselines). While we do not know how long tuning took in the original evaluations, in our own evaluations we tried 4 different learning rates (Section B), leading to a total cost of 25.04 = 4 x 6.26 and 47.48 = 4 x 11.87 GPU-hours for the FMs, which is also roughly 1.5-3x as long as the total cost of our baselines (15.66 = 11.96 + 3.70 GPU-hours).
3. In time series, our pipeline is >30x faster than fine-tuning FMs.

At the end of this response we present a new version of the tables that accurately reflects these inferred costs. In our revised version we will include these numbers as important context.

Summary

Your response states “Despite this higher cost, [the supervised baselines’] performance is relatively lower or comparable, rather than achieving substantial improvements over FMs.” As discussed in point (2) above, the cost of our baselines is comparable or lower. Furthermore, based on Figure 1 in our paper we believe it is more accurate to say that the performance of our baselines is either relatively higher (genomics) or comparable (satellite, time series). This suffices to show the FMs do not outperform supervised baselines on these tasks, which as discussed in point (1) above is the goal of our paper.

Thank you again for your constructive review of our paper and rebuttal, both of which will improve the presentation of the results. We hope that you will reassess your concerns in light of these clarifications and would be happy to answer any further questions.

Updated cost tables

Genomics (GPU-hours)

Satellite (GPU-hours)

Time series (GPU-mins)

New experiments

We provide new experimental results to address some specific concerns raised by reviewers. Due to time constraints we conduct these experiments only on a subset of tasks, but we will include numbers for all tasks in each domain in the final version.

Computational cost experiments

While we used dataset size and parameter counts as measures of cost, Rev. E1xK raised the valid question of whether our “automated pipeline is more time- and compute-intensive.” To address this, in each domain we take a representative subset of tasks and compare the automated model development and training costs of our baselines to the costs of fine-tuning the top two FMs (i.e. those in Fig. 2). As shown below, for satellite / time series our full pipelines (including tuning) are at worst only slightly slower than fine-tuning the best FM (excluding tuning). For genomics, our training costs are roughly the same as FM fine-tuning costs while our architecture search + hyperparameter tuning costs are roughly 5x more expensive than FM fine-tuning costs. While we do not know how long hyperparameter tuning took for the genomics FMs, we believe the equivalent of trying five different learning rates is a reasonable computational budget. All experiments were run on L40 (L40S for genomics) GPUs and will be extended to all tasks in the revision.

Avg. genomics costs (GPU-hours) across 4 tasks: enhancers, H3, promoter_all_ splice_sites_all

Avg. satellite costs (GPU-hours) across 3 tasks: m-brickkiln, m-bigearthnet, and EuroSAT

Avg. time series costs (GPU-mins) across 4 horizons of 2 tasks: ETTh1 and Weather

Limited data experiments

Revs. E1xK and 9w92 suggest that the “strong performance of supervised methods [...] may be largely attributed to the large-scale training data” and that “the more interesting application of FMs is for fine tuning on datasets with scarce/few labels.” While do not disagree with this, we believe evaluating in the limited data regime is not critical for our study because we chose benchmarks from among those either explicitly proposed by the relevant community for evaluating foundation models (genomics and satellite) or broadly used to do so by the vast majority of its FMs (time series). If the main utility of FMs in a specific domain is in the few-shot regime then its existing benchmarks and evaluations should reflect this (developing new ones is out-of-scope of our paper).

Still, to give a sense of the effect of different amounts of data, we pick the same subset of tasks and FMs as above and give each method only 10% of the original data (in time series we use 20% because the ETTh1 sequence is too short to use 10% while keeping the original context window). As shown in the tables below, we find even in these subsampled regimes that the trend we identified—that our baselines are competitive with leading FMs—remains consistent.

Avg. genomics performance (↑) across 4 tasks: enhancers, H3, promoter_all, and splice_sites_all

Avg. satellite performance (↑) across 3 tasks: m-brickkiln, m-bigearthnet, and EuroSAT

Avg. time series performance (↓) across 4 horizons of 2 tasks: ETTh1 and Weather