We thank the reviewer for their thorough examination of our work and for the useful suggestions.

• On the very important point of the contribution, we refer the reader to the general comment which explains in detail why there are very few baselines currently capable of tackling the task we address in our paper. Regarding the references you mention, [1,2] assume the PDE class and the PDE coefficients are fixed, [3] builds a class of operators $f_\theta$ that is tested in an ODE setting for only one class of ODE, while our approach defines a class of operators tested in a setting that encompasses multiple PDE classes, [4] assumes that we know whether the context is generated from a different PDE class and PDE coefficients.
• We thank you for the suggestions for complementary baselines. Note, that they make additional assumptions on the contexts.
• This will be corrected in the new version. We did not include coarser results in addition to the 1024, 128x128 and 128x256 resolutions already presented in the paper. However, it is worth noting that experiments conducted at a 512x512 resolution demonstrate the same improved sample efficiency and final accuracy.
• We indeed plan to test our approach on real-world observational data, with climate prediction being an interesting candidate.
• The baselines included in Table 3. (Unet, FNO, AR-diffusion) are known to be the main competitive methods in the case of a single PDE, but we will consider some of your aforementioned suggestions in the next version of the paper.

Answers to your questions.

• We will clarify this point in the next version of the paper.
• We will also clarify this point in the next version of the paper.
• We originally ran experiments with the Dopri5 solver, but these appeared to be significantly slower than a RK4 solver, without necessarily improving performance.

Since my concerns remain entirely unaddressed, and currently the authors do not address them totally, I decide to lower my current score from 5 to 3.

1. I understand that the primary assumption of your work focuses on multiple PDEs. However, several experiments do not entirely align with this focus (e.g., single-dataset training). In such cases, baselines other than AViT are also necessary. Unfortunately, throughout the entire discussion period, the authors failed to address these alternative baselines.

2. 3D PDEs and real-world applications are still not included in the paper.

3. There is no revision for the paper. All the raised questions are not included in the paper during the discussion period.

For the ICLR rebuttal, the authors had the opportunity to revise the paper and add more experiments to convince reviewers. However, during the entire discussion period, they neither revised the paper nor added additional experiments. Their only response was, "We are working on that." I find this lack of engagement disrespectful to us as reviewers. To be frank, I believe most authors of other papers would have already revised their papers by now, given that the modification deadline has passed. I am deeply disappointed, as we have dedicated significant effort to help improve the quality of this work, yet no progress has been made.

Until I see meaningful improvements to this paper (e.g., new experiments or revisions), I will maintain my current score as a form of protest against the authors' attitude.

Dear reviewer,

We thank you once again for your feedback and suggestions, which, rest assured, will be taken into account in the next version of the paper.

Dear reviewer,

We thank you for your comments on our paper, but we need precisions to address them accurately.

• "The selection of numerical schemes should be more diverse"
  In the paper, we rely on a Runge-Kutta 4 numerical scheme, which is widely used for solving PDEs and well implemented in python. Are you suggesting using simpler or higher order numerical schemes? Are you suggesting adapting the numerical scheme based on the dataset?
• "Is the model sufficiently robust?"
  Could you precise the type of robustness you are referring to? (e.g. robustness to noise in the data, to outliers, to adversarial attacks ...).

We thank the reviewer again for their comments on our paper. We would appreciate the opportunity to address the reviewer's points in detail once we have further clarification on their feedback.

We thank the reviewer for their thorough examination of our work and their useful suggestions.

• "the technical contribution is relatively low."
  We strongly disagree on this point, and it is actually reflected by comparing our approach with [1] (which we now do in the paper). Being able to predict a PDE when it is unknown is crucial in many applications, for example, because the exact PDE may be difficult to estimate or derive. Such applications can be addressed with our method but cannot be tackled with [1], since [1] assumes the underlying PDE is known in order to use the appropriate pretrained common weights, denoted as $\theta_c$ in their paper. For the generic task tackled in our paper (which is clarified in the general comment), we cannot rely on pre-identified weights for known PDEs and must build a large network to predict the parameters from the context. This has never been done in the literature, and our paper demonstrates that it not only works but also outperforms approaches that do not explicitly leverage a numerical integrator, such as [2].

• "icl works on quantized tokens"
  We refer the reviewer to the general comment on the use of the term "in-context learning".

• "empirical results:"
  While we agree that more comparisons would be beneficial, there are actually very few baselines that offer a fair comparison. We refer the reviewer to the general comment regarding the task and the novelty of our approach. Specifically, [1] assumes that we know whether the context is generated from a different PDE class and PDE coefficients; [3] is trained on a single PDE class (e.g., turbulent Navier-Stokes, surface temperature, or ocean currents); [4] proposes to learn shared parameters across related tasks, similar to [1], requiring knowledge of the underlying PDE class for a context; [5] assumes that the PDE class is known and also assumes linearity in the parameters to predict on contexts with different PDE coefficients; and [6] also assumes that the PDE class is known in order to learn a unique set of shared parameters.

• "The title is not really appropriate"
  This is an important point. We decided to train a network on different PDEs simultaneously because we had no choice: the task involves learning from multi-physics contexts with a variety of PDEs without knowing the exact PDE. Our method would indeed require fine-tuning on unseen classes of PDEs; however, the layers that need to be trained for each dataset are minimal—the very first layer of the hypernetwork (since the context may have different input channels) and the very last layer of the hypernetwork (because the number of parameters of the predicted operator slightly varies for the same reason). As a consequence, the majority of the weights in our hypernetwork are shared across all contexts. In contrast, [1,3], for example, fine-tune a new set of parameters every time the set of PDE coefficients is changed, even of a small amount.

Answer to your questions

• To be able to change the context length at inference, with minimal loss of performance, we should train on varying context lengths, which we didn't do in the paper.
• We believe the initial clustering in the UMAP space is due to the presence of different types of initial conditions in the dataset, as described in the PDEBench paper [7] (Eq. 17). To achieve the cleanest visualization, we should consider using a dataset with a fixed set of initial conditions drawn from the same distribution, each evolved with a different time operator corresponding to distinct PDE coefficients $\eta$. Such datasets are hard to find, and we opted to rely on the standard PDEBench [7] dataset.
• We are open to any suggestions for more relevant datasets that would be considered convincing and pertinent in the literature. In our paper, the PDE coefficients for most of the datasets we used are available in the PDEBench paper [7]. Each PDE class includes between 4 and 30 PDE coefficients values. For example, the shearflow and Euler datasets (see Table.5) consist of 28 and 20 PDE coefficient values, respectively, with 40 and 500 different initial conditions, and 200-timestep-long trajectories per coefficient for both datasets. When evaluating the model in Tables 2 and 5, we include the same PDE coefficients in the validation set as in the training set. This important information will be mentioned in the next version of the paper.
• The NRMSE is computed at a specific time-step, averaged over enough trajectories to ensure that variance is negligible.

[7] PDEBench - An Extensive Benchmark for Scientific Machine Learning, NIPS, 2022

Dear reviewer,

We thank you for your comments on our paper, but we need precisions to address them accurately.

Our paper addresses the task of predicting the next states of a spatio-temporal physical system, given only the previous $T$ states, across various (i) PDE classes, (ii) PDE coefficients, and (iii) initial conditions -- all without knowledge of the governing PDE.

Few models can tackle this challenging task, as most assume known PDEs and coefficients (e.g., standard solvers) or single PDE with fixed coefficients (e.g., operator learning).

• "I don't see any novelty compared with many other models reported in the literature"
  Could you provide specific examples of the models you have in mind? While we are not claiming novelty in the individual transformer or neuralPDE components, the combination of the two -- using the transformer to predict the parameters of a neuralPDE solver -- is, to the best of our knowledge, novel.
• "the baseline models are out of date"
  Do you also include "AViT" in your statement? This baseline was made public a year ago, and is one of the first models proposed to tackle the above task.

We thank the reviewer again for their comments and suggestions. Below are additional details supplementing our previous response.

• "The major weakness lies in novelty."
  We refer the reviewer to the general comment, which specifically addresses this point.

• "The examples considered to verify the efficacy of the model are too simple"
  We did not include coarser results in addition to the 1024, 128x128 and 128x256 resolutions already presented in the paper. However, it is worth noting that experiments conducted at a 512x512 resolution on datasets used in this paper (incomp. Navier-Stokes 512x512, comp. Navier-Stokes 512x512, Euler 512x512) demonstrate the same improved sample efficiency and accuracy.

• "The baseline models are out of date."
  This important point is addressed in the general comment.

• "The prediction horizon considered in the paper is too short (e.g., 32 steps)"
  As a matter of fact, a trained transformer (e.g. AViT [1]), which is the main baseline for our task, does struggle to predict correctly up to 32 steps already. Improvements in the rollout performance of our model are a step forward toward being able to predict longer horizons.

[1] McCabe M, et al. Multiple physics pretraining for physical surrogate models. Advances in Neural Information Processing Systems, 2023.

We thank the reviewer for their thorough examination of our work and for the useful comments.

1. We will incorporate [1,2] in the new version of the "related work" section. Regarding your last point, our paper does actually provide an experiment (Fig. 3), showing that a learned transformer [3] does not incorporate a translation-equivariant inductive bias. Such inductive bias is however present in the PDE solution as explained in our paper (l.160).
2. The term "inductive bias for physical systems" is indeed vague, and will be clarified in the new version. In particular, we refer to the translation-equivariance of the PDE solution (if the initial condition is translated, then the whole solution should be translated), as well as the fact that the function $g$, defining the PDE: $\partial_t u_t(x) = g(u_t(x),\partial_{x_1}u_t(x),\ldots)$ (l.124), is a pointwise function, in the sense that it operates the same way independently on the spatial position $x$. Our model incorporates such inductive biases by defining an operator $f_\theta$ which is a particular CNN, composed of a first convolution with kernels of size 5 (analogous to discretized differential operators), followed by 1x1 convolutions (effectively learning a pointwise function recombining these operators).
3. This will be precised in the new version.
4. This important point is addressed in the general comment.
5. We thank the reviewer for these suggestions. We indeed intend to showcase more results on the physical meaning of our methods in the next version of the paper.