ConDiff: A Challenging Dataset for Neural Solvers of Partial Differential Equations

There are already a large number of benchmark dataset for scientific machine learning. Lines 90-11 mention about 16 of them. It's not clear how significant another one will be. This benchmark focuses on just a single equation - but this equation is already in other datasets. So the novelty is mainly in the choice of parameters of the equation. This is a rather minor contribution. The existing benchmarks aim to be either: very broad, to compare performance of solvers which can handle many different equations, or very applications specific, e.g. specific and realistic physics problems. The diffusion equation is a foundational equation, but it's is already treated in the benchmarks. If a user want to dive deeper into this particular equation, it's quite easy to generate solutions using the same methodology as proposed in here: use a numerical solver with a parameterized set of coefficients (e.g. GFF) to generate solutions.

There are several weaknesses in this work, summarized in the following points:

1. lack of physical relevance. The authors mention that the class of problems is of "great academic interest" and "shortens the gap with real-world problems". However, they only consider one single simple linear diffusion equation.
2. the authors only consider a simple square domain with rectangular grid. One would typically want to evaluate whether certain architectures can deal with non-periodic boundary condition or non-structured data, which would be closer to real-world applications. This is mentionned in point 3 of the limitations but I do not see why one cannot generate such datasets using a finite element method.
3. In the discussion, the authors state that the "unique feature of the dataset is discontinuous coefficients". However, this has been previously considered in the original FNO paper, and the dataset is publicly available online (containing also more challenging problems), hence the lack of originality of the paper.

First and foremost, I am curious about what differentiates and adds novelty to the dataset proposed in this paper compared to existing datasets created for scientific machine learning (sciML) in previous studies. As cited in the paper, various prior works, such as those by Takamoto et al. (2022), Luo et al. (2023), and Hao et al. (2023), have developed diverse types of datasets as PDE benchmarks. I wonder what major differences exist between this dataset and the ones developed in these studies. The paper mentions that previous datasets lack "a dataset dedicated to the very important class of academic and real-world problems, the class of parametric PDEs with random coefficients." I am interested in understanding more precisely how this differs from previous works. Furthermore, prior datasets include both time-dependent PDEs and steady-state solutions across various resolutions, while this dataset is constructed for a fixed parametric PDE. A more detailed explanation on the novelty of this approach would be helpful.

In relation to the discussion starting on line 159 of page 3, it would be helpful to clarify in more detail how the connection to real-world applications is established. The paper states that Figure 1, showing a dataset created with cubic, exponential, and Gaussian functions using GRF (Gaussian Random Fields), resembles the real-world data shown in Figure 2. However, it would be beneficial to explain more specifically in what ways this dataset offers a unique advantage over other existing datasets. Given the variety of datasets already available, understanding exactly how this proposed dataset provides a novel representation or closer alignment with real-world characteristics would strengthen the paper's argument.

The PDE examined in this paper is relatively straightforward, and existing methods can solve it accurately. Additionally, the models tested on this benchmark are somewhat outdated; for instance, while the authors discuss Physics-Informed Neural Networks (PINNs), they do not include them in the experimental analysis.

The forcing term $f(x)$ is sampled from a standard normal distribution, keeping the focus on coefficient complexity. However, in many real-world scenarios, the complexity of the forcing term itself significantly affects PDE solutions. Investigating the effects of complex or structured forcing terms in the dataset would make ConDiff more representative of practical challenges and allow for a more comprehensive model evaluation.

The dataset is generated on a regular rectangular grid, which may not fully represent the geometrical complexity found in many real-world applications. Expanding ConDiff to include different grid types or irregular geometries, although computationally demanding, would improve its relevance for practical applications involving complex domains.