Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning

We truly thank the time and effort of reviewer u3i6 in reviewing our paper!

Q1. Without accounting for this added cost of pre-training, the paper's claims about reducing data simulation costs feel vacuous.

Thanks for this great question!

1. We further provide extra experiments on joint unsupervised pretraining on multiple PDEs in Figure 1 of our attached PDF response. We can see that joint pretraining can further improve the performance of fine-tuning on different PDEs. This means fine-tuning on different PDEs can share the same pretrained checkpoint, which reduces the total computational costs. We will include this figure in our camera-ready version.
2. Recent works, like [1], also only considered per-PDE pretraining, even though their work is claimed to target SciML foundation models. Moreover, previous unsupervised pretraining works in computer vision also focused on a single dataset (like ImageNet), and they did not compare the costs of pretraining versus collecting more downstream samples.
3. The most important contribution of our paper is to define and leverage unsupervised data in PDEs and design unsupervised pretraining on top of it. Although pretraining is studied in scientific machine learning, unsupervised pretraining is largely underexplored in the community. Our work is the first kind, and will inspire the community in this research direction. In sum, we totally agree that reducing the total computation costs (training + sample collection) is important, and our promising joint pretraining results (Figure 1 in our attached PDF response) encourage us to keep working on this direction, but this should not weaken the core contribution and target in this paper.

Q2. Baseline of generating extra samples with the same computational cost as pretraining

Actually, since we studied different numbers of samples in Figure 3, we can compare our method with baselines trained with extra generated samples (as indicated by the red arrows showing the savings in the number of samples).

Q3. Contextualize the results of the OOD experiment with meaningful baselines, such as [76, 77]

[76, 77] by Liu et al. are related works, but their works cannot be directly compared with ours because: 1) they did not study 2D time-dependent PDEs; 2) they only scaled up to 5 demos, which is fewer than ours. We will highlight this more in our camera-ready version.

Meanwhile, we further provide another baseline in the OOD in Figure 3 in our attached PDF response. In our paper, our ICL leverages the output (prediction) of neural operators to find similar samples (lines 251–255). In this new baseline, we try to use features extracted by the backbone of the neural operator (high-dimensional features before the final output layer) to find similar samples. As we can see, in general, this baseline is worse than our original ICL method, indicating that the final output of the neural operator can more accurately indicate true similar samples.

Q4. The results of the OOD experiment indicate the OOD task is still a difficult problem for neural operators, but the text does not discuss this difficulty.

Thanks for the suggestion!

Based on our visualization in Figure 14 in Appendix K, we believe OOD is challenging for neural operators because of:

1. Significantly different patterns of solutions under different physical parameters.
2. Different value ranges of solutions under different physical parameters. We will include this discussion in our camera ready.

Q5. Line 132 states "initial snapshot u0(x) ... is sufficient for defining PDE systems." Could the authors explain or qualify this statement?

For a specific PDE with fixed physical parameters and a spatial-temporal domain, if we are given the initial condition and boundary condition, the dynamics of the PDE will be determined since there is no stochasticity in the PDE.

Q6. Line 185 states "Numerical solutions of PDEs are expected to exhibit invariance to filtering blur or different resolutions of inputs." Could the authors explain or qualify this statement?

Our superresolution (SR) objective shares the same motivation as recent works on enhancing scientific data resolutions [2], which is to train SciML models to preserve the inherent physical properties of scientific data while learning the SR process. Given a specific input distribution, after memorizing the input data and fitting the SR objective, solutions of neural operators are expected to preserve the inherent physical properties and exhibit invariance to filtering blur. Moreover, in the context of pretraining, both objectives – MAE and SR – aim to further extract meaningful representations of input samples from current data distributions and provide a better-adapted initialization of network weights.

Q7. The experiment in Appendix E shows encouraging results that the general-purpose self-supervised learning method MoCo v2 does not perform well on the ERA5 task. ... This seems like a nice result and wonder if the authors have intuition about why this is the case.

The main reason MoCo v2 does not perform well is that it was originally designed for object-centric images, and the loss is applied at the instance level (whole image) rather than at the pixel level (per spatial location), making it suboptimal for solving PDEs.

Q8. What are the inputs and outputs of the model for each experiment? As I was reading this paper, I was constantly asking this question and spent time jumping around in the appendix looking for details. Would it be possible to collect all of this information in a table for ease of reference?

Thanks for this suggestion! Here we include another table of details (input/output shapes). Due to the page limit, we will try to squeeze this table into our camera ready.

Q9. For each PDE experiment, what is the distribution of forcing functions (Poisson, Helmholtz) or initial conditions (Reaction-Diffusion, Navier-Stokes)?

For generating forcing functions or initial conditions, we follow previous works [1, 3, 4] and adopt their numerical solvers. We further design the ranges of physical parameters as listed in Table 1.

Q10. How does the "diffusion" physics parameter in Table 1 relate to the definition of the Poisson equation in Equation 1? Are elements of K randomly drawn from this range?

The "diffusion" physics parameter in Table 1 determines the eigenvalue of the diffusion coefficient tensor. For more details about the construction of diffusion coefficient tensors, please consider reading the paragraph “PDE coefficient sampling” in Section 3 of [1].

Q11. How does the Reynolds number relate to the definition of the Navier-Stokes equations (Equation 5)?

The Reynolds number is defined as $Re=\frac{\rho u L}{\nu}$, where $\nu$ is the viscosity of the fluid ($\rho$ is the density of the fluid, $u$ is the flow speed, $L$ is a constant of the fluid’s characteristic linear dimension). $\nu$ is used in Eq. 5. In other words, a smaller $\nu$ indicates a larger Reynolds number.

Q12. Are there spatial boundary conditions for the Reaction-Diffusion equation?

Following PDEBench [3], we consider the Neumann boundary condition for Reaction-Diffusion.

Q13. What do the error bars in Figures 3 and 4 represent?

Error bars in Figures 3 and 4 indicate standard deviations of three independent runs with different random seeds.

Q14. The ERA5 and ScalarFlow datasets have temporal observations; the learning task is to predict the state of the system at time t+16 given a sequence of state observations at times [t,t+15]. What is the unlabeled data used in pretraining for these cases? Is it sequences of state observations at times [t,t+15], for a set of different values t?

Similar to time-dependent PDEs (reaction-diffusion, Navier-Stokes), which are explained in line 131 and in Appendix A, on ERA5 and ScalarFlow, models are trained on individual snapshots without temporal information. The pretraining, fine-tuning, and test sets are separate without overlap. We will provide clearer explanations in our camera-ready version.

Q15. For these experiments, how does using snapshots at t>0 agree with the definition of unlabeled PDE data for time-dependent systems in Section 3.1.1?

In time-dependent PDEs, the simulation can start from any snapshot (i.e., the numerical simulation can take any snapshot as the initial condition and continue simulating the dynamics). In fact, in FNO, the actual simulation of Navier-Stokes starts after the fluid roughly exits the chaotic phases (please refer to the GitHub repo called neuraloperator/physics_informed, line 34 of generate_data.py).

This setting is also similar to starting the collection of climate data at any timestep and continuing the collection. The key is that we do not involve any temporal dynamics during our pretraining.

Q16. Could the snapshots used in the pretraining dataset be used in a supervised learning framework? If so, why doesn't the “random init” baseline have access to this data?

The “random init” baseline can access this data. However, as we consider these pretraining samples as individual snapshots (to simulate the practice where people cannot capture temporal dynamics and can only collect individual spatial climate/flow snapshots), we do not use any temporal information in these samples, which cannot be used for supervised forecasting.

Q17. Are there examples of simulation or experimental settings where it is reasonable to assume we can easily gather snapshots of a system state, but these snapshots could not be used in a supervised neural operator framework?

Collecting snapshots with temporal dynamics in large-scale scenes is significantly more challenging than collecting individual snapshots without temporal dynamics. We have the following examples, and we will try to include them in our camera ready:

1. Weather forecasting: Continuous monitoring of atmospheric parameters requires a network of weather stations, satellites (e.g. large amount of historical satellite data was leveraged in ERA5), and real-time data processing, whereas a single weather report is simpler and less resource-intensive. Long-term climate monitoring involves maintaining observation networks for years, compared to a one-time collection of current climate conditions.
2. Smoke dispersion studies: Tracking smoke spread over time requires multiple sensors (e.g. five cameras with carefully calibrated intrinsics and extrinsics in ScalarFlow) and continuous data processing, unlike a single air quality measurement at a peak pollution event.
3. Ocean currents: The monitoring demands continuous deployment of buoys and sensors, whereas a single measurement of water conditions can be done with minimal equipment like a handheld sensor or a small research boat.

Q18. Do the authors have any intuition for why the Poisson and Helmholtz OOD performance is poor, and the Navier-Stokes performance is better? Is there a connection between the expected smoothness of a particular PDE’s solution and the performance of the ICL method on that PDE? For example, we know that the solution of the Poisson equation should be relatively smooth, but the proposed ICL method outputs highly non-smooth estimates.

To understand the results in the OOD setting, we provide a visualization in Figure 14 in Appendix K. We find that the Helmholtz equation in the OOD setting exhibits much more complicated unseen patterns than the Poisson and Navier-Stokes equations, making it challenging to learn.

The non-smoothness of the ICL method outputs is mainly due to the uncertainty of neural operators on out-of-distribution (OOD) samples. As the sample-wise similarity is measured by the neural operators’ outputs, which are not accurate, the more severe the OOD setting, the less confident the model’s output and similarity measurement will be.

Q19. I see that ICL with more demos improves the scale of the prediction. Are there problem settings where the scale of the solution is of particular interest? Discussing such settings may better contextualize the results on these challenging OOD tasks.

Thanks for this suggestion! In numerical simulations or predictions of PDEs, there are settings where the scale or magnitude of solutions is more important than the exact shape/pattern. We will try to include these examples in our camera ready.

1. Heat Transfer: In large-scale systems, the focus might be on overall temperature and extreme values. For instance, in evaluating a cooling system, the key concern might be the peak temperature rather than the detailed temperature distribution.
2. Fluid Dynamics: For applications like aerodynamics, the overall drag or lift force on an object is often more critical than capturing every detail of the flow pattern, such as in airfoil design.
3. Environmental Modeling: The concentration of pollutants at specific locations or total pollutant transport is often more crucial than the exact distribution, such as in groundwater flow studies.

Q20. Regarding related work: E. Hasani and R. A. Ward, “Generating synthetic data for neural operators.” arXiv, Jan. 04, 2024.

Thanks for bringing this up! We noticed this work after our paper submission. Our method, which works on unlabeled PDE data, is orthogonal to their generation of synthetic PDE data, and the two approaches could potentially be combined into a semi-supervised learning strategy.

[1] “Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior“ Subramanian et al. 2023

[2] “SuperBench: A Super-Resolution Benchmark Dataset for Scientific Machine Learning” Ren et al. 2023.

[3] “PDEBENCH: An Extensive Benchmark for Scientific Machine Learning“ Takamoto et al. 2022

[4] “Fourier Neural Operator for Parametric Partial Differential Equations” Li et al. 2020

Many thanks to the authors for providing detailed responses to all parts of the review. I have specific comments and questions about a few parts of the response.

Q1 and Q9 I really appreciate the design of this extra experiment, and the results look encouraging. However, the lack of detail about the distribution of forcing functions and source functions still obfuscates the significance of these results. I have briefly read [1] to understand the distribution of forcing functions and source functions. To my understanding, the distribution of Helmholtz and Poisson source functions are the same. Is this correct?

Q2 In my opinion, Figure 3 and the relevant text do not give enough detail to compare the proposed method with the baseline of generating extra samples. This is because there is not enough information to evaluate how long it takes to pretrain the model (I could not find this in Section B3), and there is no information about how long it takes to generate samples for the Helmholtz or Poisson problems.

Q4 Thanks for this explanation. Do you plan to update any of the relevant text in the camera-ready version?

Q6 Thanks for this explanation. The explanation makes sense but is quite different from what appears in the original submission. Do you plan to update this in the camera-ready version?

Q8 Could you also add the shapes of the pretraining inputs/outputs to this table?

Thank you for all of the other replies; again, I really appreciate the careful and thorough response.

We truly thank the time and effort of reviewer oEEz in reviewing our paper!

Q1. Overall the paper is quite difficult to read

Thanks for this suggestion! We will try to make our teaser figure (Figure 1) clearer and connect it more to the subsections in methods and experiments, so readers can follow easily.

Q2. Comparison between the different tasks and architectures of the method

1. To focus on pretraining (model initialization), we keep the network architecture the same in each experiment to ensure that our comparisons in each task are fair.
2. We study the contribution of masking vs. super-resolution in Appendix G.1. We find that when training with a low volume of data, we should use much stronger perturbations (high masking ratios and strong blur), whereas a high volume of data only requires mild perturbations.

Q3. The ICL method should be compared to at least one baseline.

Thanks for the suggestion!

We further provide a baseline for the ICL method in Figure 3 in our attached PDF response.

In our paper, our ICL leverages the output (prediction) of neural operators to find similar samples (lines 251–255). In this new baseline, we try to use features extracted by the backbone of the neural operator (high-dimensional features before the final output layer) to find similar samples. As we can see, in general, this baseline is worse than our original ICL method, indicating that the final output of the neural operator can more accurately indicate true similar samples.

Q4. What is the best effective strategy for pretaining? Is it the masking-damasking or super-resolution that yields the best results? Or do you use both?

We find that combined methods (MAE + superresolution) give the best performance. These results are in Appendix G.1, where we exhaustively studied the choices of hyperparameters for MAE and superresolution.

Q5. Did you observe a difference between the video transformer and FNO for this setup?

We tend to avoid recommending specific model architectures, as our method is agnostic to the architecture choices. We have proven that our method can help both FNO and transformers.

We truly thank the time and effort of reviewer 6vSy in reviewing our paper!

Q1. Masked auto-encoding has already been proposed in CV applications. The contribution however is unclear for me.

We would like to clarify and re-emphasize: The most important contribution of our paper is to define and leverage unsupervised data in PDEs and design unsupervised pretraining on top of it. This is mentioned in our first contribution bullet (lines 66–69) and also in line 7 of our abstract. There are previous works that try to pretrain neural operators on simulated solutions [1, 2, 3], but no previous works tried to reduce the simulation costs by defining and leveraging unlabeled PDE data. That means, although pretraining is studied in scientific machine learning, unsupervised pretraining is largely underexplored in the community. Our work is the first kind, and will inspire the community in this research direction.

We do not claim that our MAE method is new, and we also provide clear motivation for why we introduce MAE in pretraining neural operators (lines 160–172).

Q2. Moreover, it is difficult to identify why is the unsupervised pretraining performs best than other baselines.

(Similar question) On figure3, why are the unsupervised training performing better than training from scratch?

We discussed the benefits of our unsupervised pretraining in Figure 4 in Sec. 4.1. There are three reasons:

1. Reduced overfitting, which consistently leads to smaller generalization gaps across all PDEs we studied.
2. Faster convergence, which can accelerate model convergence more than both random initialization and vision-pretrained checkpoints.
3. Meaningful representations, which are extracted from pretrained Video-MAE and are helpful during fine-tuning.

Q3. For the unlabeled data for time-dependent PDE, how does the PDE ICs are sufficient to identify the PDE? How does the model have access to the PDE parameters?

Our models do not have access to physical parameters. Given the initial condition and boundary condition, the dynamics of a PDE are determined since there is no stochasticity in the PDE.

Q4. Is your model trained on multiple physics? Or pretraining are proceeded on one PDE?

Our experiments focus on pretraining on one single PDE.

However, we also further studied joint pretraining. We provide extra experiments in Figure 1 in our attached PDF response. We can see that joint pretraining can further improve the performance of fine-tuning on different PDEs. We will include this figure in our camera-ready version.

Q5. How is proceeded this “random init” training? Is it using data with supervised loss?

In Figure 3, the “random init” and “unsupervised” models share completely the same fine-tuning settings, and they both use the supervised loss (labeled PDE data) during fine-tuning.

The only difference is that “random init” uses random initialization of model weights to start the fine-tuning, while “unsupervised” uses pretrained weights (from our unsupervised pretraining) to start the fine-tuning.

Q6. What is the comparison of other baselines in the OOD setting?

Thanks for the suggestion!

We further provide a baseline in the OOD setting, attached in Figure 3 in our attached PDF response.

In our paper, our ICL method leverages the output (prediction) of neural operators to find similar samples (lines 251–255). In this new baseline, we try to use features extracted by the backbone of the neural operator (high-dimensional features before the final output layer) to find similar samples. As we can see, in general, this baseline is worse than our original ICL method, indicating that the final output of the neural operator can more accurately indicate true similar samples.

Q7. What is the computational cost of training your model compared to other baselines? How many parameters do the baselines contains?

• Pretraining costs. FNO: 20 GPU hours. Video-MAE: 18 GPU hours. (GPU: A100)
• Model parameters. FNO: 67.1M. Video-MAE: 23.4M.

Q8. Societal impact and ecological considerations

Our paper improves the data efficiency of neural operators for solving PDEs, which can significantly reduce computational costs and energy consumption associated with high-fidelity numerical PDE simulations. This improvement can lead to more sustainable scientific research by minimizing the environmental footprint of extensive computational experiments. Moreover, by democratizing access to advanced PDE solutions through more efficient pretraining, our method has the potential to accelerate scientific and engineering advancements in the broader community, benefiting society at large.

[1] “Lie Point Symmetry Data Augmentation for Neural PDE Solvers” Brandstetter et al. 2022

[2] “Self-supervised learning with lie symmetries for partial differential equations“ Mialon et al. 2023

[3] “Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior“ Subramanian et al. 2023

We truly thank the time and effort of reviewer oUb1 in reviewing our paper!

Before providing detailed responses, we would like to make a clarification:

Since our paper focuses on unsupervised pretraining but not SciML foundation models (we never claimed our pretraining leads to such models), we hope reviewer oUb1 will not assume we are pursuing or comparing with the latest SciML foundation model papers. We believe it is unfair to ask us to match their experimental settings.

As the benefits and strategies of pretraining for SciML are still under-explored, both supervised/unsupervised/per-PDE/joint pretraining are meaningful to this research direction. This is similar to what researchers in the computer vision community explored with single dataset/task pretraining [1, 2, 3] before moving on to foundation models.

---

---

Q1. Some important experimental setups are missing

1. Model sizes. FNO: 67.1M. Video-MAE: 23.4M. In each experiment, all compared models are of the same architecture and thus fairly compared.
2. Details about pretraining datasets are clearly documented in Appendix A, which is also clearly referred to in Line 134.
3. In each experiment, we pretrain our model only on one PDE. As we clarified above, our work does not pursue “Pretraining on multiple classes of PDEs”.

Q2. Pretrain different models for different PDEs

1. We provide extra experiments in Figure 1 in our attached PDF response. We can see that joint pretraining can further improve the performance of fine-tuning on different PDEs. We will include this figure in our camera-ready version.
2. Although joint pretraining can be further beneficial, experiments of pretraining on a single PDE are still fundamental and necessary. These should be studied before moving to joint pretraining and cannot be ignored or underestimated. These experiments help analyze the complexity or difficulty of learning each single PDE for neural operators.
3. Recent works, like [4], also only adopted per-PDE pretraining, even though their work is claimed to be more related to SciML foundation models than ours. Previous unsupervised pretraining works in computer vision also focused on a single dataset (like ImageNet).

Q3. Missing baselines

On time-independent PDEs, we mainly focus on the FNO model. There are no publicly available checkpoints of FNO that are pretrained on natural images. Due to limited time during rebuttal, it is also challenging to pretrain an FNO ourselves on large-scale image datasets like ImageNet or JFT-300M. We will try to include this experiment in our camera-ready version.

Meanwhile, on time-dependent PDEs, we studied Video-MAE because we followed previous work [5]. In [5], they also did not consider any image-pretrained MAE models for their claims.

Q4. "in-context learning" in Sec. 3.2

We are happy to rename our method.

Meanwhile, we have further corresponding comments:

1. In recent literature about large language models (LLMs), “context” is mostly referred to in downstream tasks, not in pretrained models, and we did not claim that our context is in our pretrained models;
2. In-context learning is not directly related to auto-regressive models, see [6];
3. Models with self-attention are also not learning anything during inference, since all model weights are fixed. Both self-attention and our models are extracting features during inference.

Q5. Accuracy and standard deviation of few-shot learning

We report standard deviations in Figure 3 in our attached PDF response.

The performance of neural operators in out-of-distribution (OOD) settings is known to be poor and challenging. See Figure B.3 (f) of [4]: performance on Helmholtz (“SYS-3”) is poor (both their errors and ours are close to 1), even with supervised fine-tuning on a few OOD samples.

Q6. Line 95: BERT does not leverage next token prediction for pretraining

Thanks! We will update this sentence with a more precise description.

Q7. Line 142: Multiple benefits of pretraining are listed, but no references are provided

As clearly mentioned in Line 148, these experiments about benefits of unsupervised pretraining are in Figure 4 in Sec. 4.1, which is in the main text and should not be ignored.

Q8. Line 158: I don't think any variant of BERT is scaled to one hundred billion parameters

Thanks! We will cite more papers with models of billion-level parameters that share similar pretraining strategies as BERT.

Q9. Other issues.

Thanks! We will fix these typos.

Q10. Can the pretrained model be applied to other PDEs which are not seen in pretraining?

We provide experiments for fine-tuning on PDEs unseen during pretraining, shown in Figure 2 in our attached PDF response. Fine-tuning on unseen PDEs leads to worse performance, which is expected not only because they have different initial conditions but also due to their mismatched input dimensions. The input dimension of Poisson is four (forcing function and three diffusion coefficients), while the input dimension of Helmholtz is two (source function and the wavenumber). This mismatched input is documented in Section A.

[1] “A Simple Framework for Contrastive Learning of Visual Representations” Chen et al. 2020

[2] “Improved Baselines with Momentum Contrastive Learning” Chen et al. 2020

[3] “Masked Autoencoders Are Scalable Vision Learners” He et al. 2021

[4] “Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior“ Subramanian et al. 2023

[5] “Multiple Physics Pretraining for Physical Surrogate Models” McCabe et al. 2023

[6] “In-Context Operator Learning with Data Prompts for Differential Equation Problems” Liu et al. 2023

We truly thank reviewer oUb1 for reading our response and providing the reply!

Below are our responses to your further questions:

Q1. Pretraining cost and finetuning cost.

1. We reported our pretraining cost in our response to reviewer 6vSy (https://openreview.net/forum?id=MuPlJ9fT4b&noteId=9dXGDW9BRK). We apologize for this omitted response!

• Pretraining costs. FNO: 20 GPU hours. Video-MAE: 18 GPU hours. (GPU: A100)

2. For fine-tuning costs: FNO: 4 GPU hours. Video-MAE: 6 GPU hours. (GPU: A100).

Meanwhile, we hope that reviewer oUb1 does not ignore our two other responses:

1. Extra experimental results: In Figure 1 in our PDF response. We can see that joint pretraining can further improve the performance of fine-tuning on different PDEs. We will include this figure in our camera-ready version.
2. Previously published works also focused on single-dataset pretraining and did not compare the costs of pretraining/fine-tuning with those of data simulation/collection: For example, [1] also only adopted per-PDE pretraining, even though their work is claimed to be more related to SciML foundation models than ours. Previous unsupervised pretraining works in computer vision also focused on a single dataset (like ImageNet).

Q2. “no relevant unsupervised learning baseline comparisons in the FNO experiments”

Since we are the first to introduce unsupervised learning in SciML, there are indeed no previous related works (that also train FNO in an unsupervised manner) with which we can fairly compare.

Q3. “Multiple Physics Pretraining for Physical Surrogate Models, show much more comprehensive baseline comparisons in Table 1.”

The reason McCabe et al. needed to compare with more baselines is that their work proposed both a new architecture and pretraining methods. They needed to demonstrate the benefits of both their architecture (by comparing with models with a comparable number of parameters) and their pretraining (by comparing with models trained on a single dataset).

However, in our case, for each subplot in Figure 3: 1) If we compare with different models, we cannot draw meaningful conclusions because the architecture changes; 2) Since ours is the only work on unsupervised pretraining of neural operators, comparing with supervised pretrained models would be unfair.

Instead, we chose the following approach: 1) To address different architectures, we studied both FNO and transformer models on different PDEs; 2) To address different pretraining methods, we included a vision-pretrained transformer. We believe these two aspects already cover multiple representative baselines that fulfill the purpose requested by reviewer oUb1.

Q4. In-context learning of transformers.

We thank these references, all of them are well awared by we authors. We respect reviewer oUb1’s option. Our definition of “learning” is to update model parameters, and we are open to accepting other definitions.

Q5. Relevant model of billion-level parameters that share similar pretraining strategies as BERT.

One example is the T5 model [2], which is of billion-level parameters and is pretrained with masked language modeling similar to BERT.

Here we quote:

• “we consider an analogous objective to BERT’s “masked language modeling” objective in” in their Sec. 3.
• “we consider an objective inspired by the “masked language modeling” (MLM) objective used in BERT“ in their Sec. 3.3.1.

We again thank reviewer oUb1's reply! We are happy to address any further concerns.

[1] “Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior“ Subramanian et al. 2023

[2] “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer” Raffel et al. 2019