FNOPE: Simulation-based inference on function spaces with Fourier Neural Operators

We thank the Reviewer for the concrete suggestions. Here, we clarify the key contributions of our work, provide extensive ablation experiments and benchmark against a suggested inverse method. We hope this additional context will allow the Reviewer to rethink their score.

Clarification of Main Contributions

The primary contribution of our work is the development of a simulation-based inference (SBI) approach for function-valued parameters. Importantly, we clarify that Bayesian inference, in general, is not equivalent to ’solving an inverse problem’: In Bayesian inference, we try to estimate the full posterior distribution over parameters, and in simulation-based inference, we aim to do this without access to likelihoods (see [1]). In contrast, inverse problems typically focus on deterministic models, and aim to find a point estimate that optimizes some (regularized) loss, and therefore, e.g., lack uncertainty estimates which are often crucial for scientific applications (see also response to Reviewer 5bMC). Thus, our method is simply solving a different problem to the papers pointed out by the reviewer, which either address deterministic inversion methods, such as Molinaro et al., Wang et al., and Behroozi et al. [R2, R4, R5], or on the other hand, likelihood-based Bayesian inference methods, such as Kaltenbach et al. [R3].

In our manuscript, we discuss related approaches that can be used for simulation-based inference of function-valued parameters, as well as how our work differs from these approaches. However, we agree with the Reviewer that clarifying the distinction between simulation-based inference and deterministic and/or likelihood-based inference in the Neural Operator context will improve the clarity of our work, and we will add this discussion in our revised manuscript.

New baseline: Invertible Fourier Neural Operators (iFNOs)

From the works referenced by the Reviewer, we noticed that only iFNOs [R1] can be framed as an SBI method. We thank the Reviewer for bringing up this work as including a comparison with a Neural Operator-based method will improve our paper. However, we point out that the loss presented in [R1] does not explicitly target the posterior distribution. Instead, it trains a Neural Operator-based conditional probabilistic generative model, but it is unclear whether this model learns the posterior distribution, or a different conditional distribution. To include the iFNO as a baseline in our work, we trained an iFNO on our Darcy flow problem with 1000 training simulations. While the iFNO produces reasonable samples of the parameters in terms of predictive MSE [Table R4.1], its samples lack any uncertainty. This is evident in the simulation-based calibration metric (SBC EoD, lower is better), which is close to 0.25. This value is expected for a point estimate (see discussion at the end). We also point out that this value is considerably higher than the SBC EoD values achieved by any of the methods shown in our manuscript for any task. For this experiment, we considered the low training budget regime as this is what the authors of [R1] used in their work. Furthermore, we use the same settings as in [R1] for the D-CURV experiment. We will include a comparison to iFNO in the manuscript.

Table R4.1: Comparison of FNOPE and iFNO performance (means and standard errors) on Darcy flow task with 1000 training simulations.

Transformer-based Neural Operators

The other two papers mentioned by the Reviewer [R6, R7] introduce new architectures for transformer Neural Operators without investigating their use for inference problems. These works are not directly comparable to our work, but using transformer Neural Operators in our inference framework as opposed to FNOs would make for an interesting approach. Our choice of the FNO architecture was motivated by the inductive bias for smoothly varying functions that FNOs leverage to obtain relatively small network sizes. In the response to Reviewer pNZ3 we provide a full breakdown of the network sizes and training times for our experiments [Table R1.1]. Even for the very large Darcy flow task ($\sim 16$k parameters), we observe that our network converged within just over an hour of training. While this network was comparably large ($\sim 11$M), we repeated the experiment in our ablations (Table R4.3), reducing the number of modes from 32 to 16. Here, we observed comparable performance despite the network size being much smaller ($\sim$3M). Nevertheless, we agree with the Reviewer that exploring transformer Neural Operators could be a beneficial future direction for our work. We will include a discussion of these papers in our manuscript.

NUDFT vs. FFT

We want to highlight that NUDFT is mathematically equivalent to FFT if it operates on a uniform grid, whereas the FFT cannot be applied to non-uniform grids. Thus, FNOPE (fix) can be seen as an ablation of FNOPE where we replace NUDFT with FFT and add no positional noise. Additionally, we ran FNOPE on non-uniform grids, but removed the positional noise to investigate the influence of the non-uniform discretization and the corresponding processing by NUDFT. For the Linear Gaussian task there are essentially no differences of the posterior quality in terms of SWD to the ground truth posterior (Table R4.2). However, we emphasize that without positional noise, we lose the flexibility to evaluate at arbitrary positions.

Table R4.2: SWD (means and standard errors) of the Linear Gaussian task with and without positional jitter.

Ablation Studies

As suggested by the Reviewer as well as Reviewers pNZ3 and u8xK, we have conducted several ablation studies to explore the effect of different hyperparameters on FNOPE. The ablations show that FNOPE performs robustly over a wide range of different hyperparameters, e.g. number of modes, lengthscale and unmasked training points. Below, we refer to tables in response to reviewer u8xK.

Number of modes

To investigate the influence of the number of modes, we adapted the Linear Gaussian task with a new prior, which produces samples varying at higher frequencies. Increasing the number of modes improves performance in terms of SWD as it allows the network to model higher-frequency components of the parameters more reliably (Table R3.1) and we observe a similar trend for the Darcy flow task (Table R3.2). Overall, these results highlight that the number of modes should be chosen depending on the task complexity, but the sensitivity to this hyperparameter may be limited. We refer to the response to Reviewer u8xK for a detailed discussion.

Lengthscale of base distribution

Varying the kernel lengthscale of the base distribution (Sec. 3.1 of paper) has a limited effect on the resulting posterior quality for the Linear Gaussian task (Table R3.3), even in the limit of a white noise base distribution.

Additionally, we conducted an ablation on the Darcy flow task by varying the lengthscale of the base distribution. While the predictive MSE is robust to different lengthscales, calibration performance measured by SBC EoD is getting slightly worse with increasing lengthscale, whereas the posterior log-probability is best for intermediate lengthscales. In summary, a reasonable intermediate lengthscale scaling factor performs well across all metrics, and this is what we used in the main experiment (Table R4.3).

Table R4.3: Darcy flow with 32 modes and different lengthscale scaling factors. Values reported are for 10,000 training simulations.

Number of unmasked training points

Furthermore, we conducted an ablation study to investigate the impact of the number of unmasked training data points on the performance. In summary, FNOPE performs well for different choices of unmasked training data points. For more details on this, we refer to the response to Reviewer u8xK (Table R3.5).

SBC EoD for a point estimate

When training the iFNO baseline on the Darcy flow task, we obtain a Simulation-based Calibration Error of Diagonal (SBC EoD) of close to 0.25 (Table R4.1). This indicates that the distributions yielded by the iFNO are very close to point estimates. Here, we want to refer to our definition of the SBC EoD metric (Appendix S2.2). A one-dimensional marginal of a point mass distribution either always overestimates or underestimates the ground truth value. Hence, the ground truth will have rank $r_{ij}=0$ or $r_{ij}=N_\text{post}$. Supposing that over different observations $x_j^o$, the point mass distribution is equally likely to underestimate or overestimate the ground truth, the cumulative distribution of the ranks will be given by $\text{CDF}(\alpha)=0.5$ for all significance levels $\alpha\in (0,1)$. Given the definition of the SBC EoD, a point mass distribution results in a SBC EoD value of $\int_0^1 |\text{CDF}(\alpha)-\alpha|d\alpha = 0.25$.

[1] Cranmer, Kyle, Johann Brehmer, and Gilles Louppe. "The frontier of simulation-based inference." Proceedings of the National Academy of Sciences (2020).

We thank the Reviewer for their positive evaluation of our work, and for their constructive suggestions. We note that other Reviewers also acknowledge that our method can infer parameters on arbitrary discretizations (all Reviewers) and its empirical performance, especially in the low-simulation regime, (Reviewers pNZ3, 5bMC). To address the Reviewer’s main suggestion, we conducted extensive ablation studies on the hyperparameters of FNOPE. Where appropriate, the information of these tables will be expressed in figures in our revised manuscript.

Investigation of hyperparameters

We conducted new experiments to investigate the impact of various hyperparameters and design choices. Overall, we can show that FNOPE performs robustly over a wide range of different hyperparameters. In detail, we tested how the performance of FNOPE is affected by the number of modes of the FNO architecture, the number of unmasked training data points in training (see Sec. 3.2 of main paper), and the lengthscale of the Gaussian process used for the base distribution.

Number of modes

We investigated how the number of modes used affects the performance of FNOPE for both the Linear Gaussian and Darcy flow experiments. We augmented the Linear Gaussian task by decreasing the lengthscale of the Gaussian process prior over the parameters, resulting in a more challenging inference task with higher frequency components. The Sliced Wasserstein Distance (SWD) of the inferred posterior to the ground truth posterior improves with more modes, saturating at around 128 for both FNOPE and FNOPE (fix) (Table R3.1). This is because the FNOPE networks with fewer modes do not have the capacity to model higher-frequency components of the samples, and so cannot reproduce realistic samples. However, when increasing the number of modes, the network is sufficiently flexible to model the posterior. This is reflected in the power spectrum of the posterior samples: While samples from models with fewer modes have less power compared to the ground truth in higher frequency components, the models with 128 and 256 modes match the power spectrum of the ground truth perfectly.

Table R3.1: SWD means and standard errors for FNOPE and FNOPE (fix) for the Linear Gaussian example with a simulator prior lengthscale of 0.005 and 10k training simulations.

We observe a similar trend for the Darcy flow task, where reducing the number of modes used by FNOPE reduces the posterior log-probability the trained model assigns to data (Table R3.2). Overall, these results highlight that the number of modes should be chosen depending on the task complexity, but the sensitivity to this hyperparameter may be limited. We will provide a discussion in the main paper.

Table R3.2: FNOPE performance (means and standard errors) for Darcy flow task with different numbers of modes and 10k training simulations.

Lengthscale of prior distribution

We assume that when the Reviewer asks to investigate the dependence of the model to the ``prior distribution’’, they refer to the base distribution of the flow-matching formalism, as discussed in Sec 3.1 of our work. In typical inference settings, the prior over the simulator parameters is fixed and encodes any prior knowledge we have about the problem. According to Bayes’ theorem, changing the prior will also change the posterior. In contrast, the base distribution used in the flow-matching approach is a modeling choice, and can be defined freely.

We investigate the impact of the choice of the base distribution on the performance of FNOPE on the Linear Gaussian and SIRD experiments. Concretely, we change the Gaussian process kernel lengthscales (Sec. 3.1), which we set proportionally to the inverse of the number of modes used for FNOPE. To vary the kernel lengthscale, we multiply it with different scaling factors. Both experiments demonstrate that FNOPE performs robustly over a range of different kernel lengthscales (Tables R3.3 and R3.4). We will provide figures illustrating this in the Appendix of the paper.

Furthermore, we conducted the Linear Gaussian and the Darcy flow experiments using white noise as the base distribution instead of the Gaussian process, which would correspond to a Gaussian process with a lengthscale of zero. Both for using white noise and the Gaussian process as the base distribution, the SWD is close to zero for FNOPE and FNOPE (fix) (Table R3.3). Hence, FNOPE performs well over different base distribution choices. In the Darcy flow experiment, however, we observed that using white noise as the base distribution leads to visually noisier posterior samples. We will add figures showing this to the Appendix, but cannot include it in this rebuttal.

Table R3.3: SWD means and standard errors for Linear Gaussian experiment for different kernel lengthscales and white noise as base distribution. The model used 50 modes and 10k training simulations.

Table R3.4: SBC error of diagonal and predictive MSE (means and standard errors) for SIRD experiment with FNOPE for different kernel lengthscales. The model used 32 modes and 10k training simulations.

Number of unmasked datapoints

We tested the performance of FNOPE over a wide range of unmasked training data points in the Linear Gaussian experiment. While the SWD is already small for 128 unmasked datapoints, it decreases as the number of unmasked data points increases (Tab. R3.5). With an increasing number of unmasked data points, the performance improvement starts to saturate.

Table R3.5: SWD means and standard errors of the Linear Gaussian experiment for different numbers of unmasked training data points. The model used 50 modes and 10k training simulations.

Additional clarifications

Sliced Wasserstein Distance (SWD)

We apologize for not including a formal definition of the SWD. The (empirical) sliced Wasserstein(-2) distance between $N$ samples from two probability distributions $p$ and $q$ is defined as

$$\text{SWD}(p,q) = \mathbb{E}\_{u \sim U(\mathbb{S}^{n-1})} \left[ \frac{1}{N} \sum\_{i=1}^N || u^\top x\_i - u^\top y\_i ||_2^2 \right]^{1/2},$$

where $x_i \sim p(x)$, $y_i \sim q(y)$ are samples from the probability distributions and $u$ are uniformly randomly sampled vectors on the unit sphere $\mathbb{S}^{D-1}$. We want to clarify that this metric is only used as an evaluation metric and not as a “loss” for the neural network.

Bochner’s theorem

We state Bochner’s theorem as in Williams and Rasmussen [1]: A complex-valued function $k$ on $\mathbb{R}^{D}$ is the covariance function of a weakly stationary mean continuous complex-valued random process on $\mathbb{R}^{D}$ if and only if it can be represented as $k(\tau) = \int_{\mathbb{R}^{D}}\exp^{2\pi is\cdot \tau}d\mu(s)$ for some positive finite measure $\mu$.

Crucially, in the less general but relevant case that $\mu$ admits a density $S(s)$, the integral is a Fourier transform between the kernel $k(\tau)$ and the spectral density $S(s)$. We apply this result to relate the lengthscale of the square exponential kernel to the spectral density of its Fourier decomposition. The definition of the SWD and the statement of Bocher’s theorem will be added to the Appendix.

[1] Christopher KI Williams and Carl Edward Rasmussen. “Gaussian processes for machine learning” volume 2 (2006).

We thank the Reviewer for their positive and thoughtful feedback. We want to highlight that the other Reviewers agree that our work solves a relevant problem (pNZ3), that the sample efficiency is a particular strength of our approach (u8xK), and noted the clarity of our presentation (pNZ3, u8xK).

To address the Reviewer’s concerns, we provide a discussion on a marginalized approach as suggested by the Reviewer, which we will also add to the revised manuscript. We also provide some clarifications to answer the Reviewer’s remaining questions.

Marginalization

We agree with the Reviewer, that we could in principle learn a marginalized posterior in SBI. However, this is not the setting we are interested in this work. As the Reviewer mentioned correctly, a marginalized posterior is “less than ideal”, especially for function-valued parameters which are (by definition) highly correlated. But even beyond that, a lot of “low” dimensional scientific problems show highly correlated posterior distributions, and this is often what scientists care about. In neuroscience, for example, Gonçalves et al. [1] showed that the parameters of a model of the pyloric network are highly correlated, and posterior draws from the marginalized distribution fail to give reasonable simulation results. Similar properties are observed in gravitational wave detection, where correlations in the posterior are of great interest and e.g. sky position cannot be inferred with marginal inference [2]. However, we are aware that for specific problems the inference of marginal distributions might be sufficient [3]. In such cases other SBI methods might be better suited. The SIRD model pinpoints that even the low dimensional parameters $\eta$ can have strong correlations (Fig. 4a, right) and a mean field approach would suffer from these correlations. Additionally, we want to highlight that a marginalized setup would not easily allow flexible conditioning on arbitrary spatial/temporal observations as is the case for FNOPE.

In terms of performance, we indeed measure the calibration of the inferred posteriors by SBC EoD on the marginals, which is a standard metric in the SBI literature [4]. Similar calibration measures for high dimensional distributions are still an active field of research. However, we want to highlight that all other metrics, e.g. SWD (Linear Gaussian) and posterior log-probability (Darcy flow) are taking the joint posterior into account. Similarly, the MSE metric (see clarification below) is based on posterior predictive samples from the joint posterior and is therefore indirectly also sensitive to correlation structures. Samples from a mean-field approximation often do not lead to predictive simulations which match the observation, as simulators often contain complicated, nonlinear interactions between the parameters. Consequently, our evaluation is not limited to the quality of the posterior marginals.

We thank the Reviewer for bringing the work of Ambrogioni et al. [5] to our attention. Indeed, there are relevant related tasks for which joint posterior estimation may be circumvented via marginalized approaches. For example, if there is a large number of nuisance parameters, which are themselves not of scientific interest, estimating the joint distribution including these parameters can be very challenging. A marginalized approach as in [5] can circumvent this and learn the marginal posterior distribution over the quantities of interest, without explicitly modeling the joint distribution with the nuisance parameters. We will include this discussion on marginalization with the according references in the main paper.

Sample efficiency

As the Reviewer pointed out, one strength of FNOPE is that it requires fewer simulations than existing methods, which could enable inference on computationally expensive simulators. To emphasize this strength, we conducted additional experiments with 100 training simulations. See the response to Reviewer pNZ3 for more details, in particular Tables R1.2 and R1.3 show that FNOPE shows strong performance even when using only 100 training simulations in the Linear Gaussian and Antarctic Ice tasks. We agree with the Reviewer that these are empirical measurements of sample efficiency, rather than theoretical guarantees. However, this is a broad limitation in the field of SBI, and not a particular limitation of this work.

Questions

Using an NPE objective

We are not aware of any NPE approach that additionally conditions on variable positions $l^x$ and $l^\theta$. Standard SBI architectures such as normalizing flows are often restricted to a fixed number of input parameters, and do not provide such flexibility. This of course stands in contrast to more recent methods such as Simformer [6], which is based on transformers. This results in a very flexible framework, but is trained with a different loss function. We also note that NPE often uses embedding nets, which can be flexibly defined without adapting the NPE loss. A Neural Operator-based embedding net could be used with a similar data-augmentation scheme as presented in our work to learn posteriors conditioned on flexibly-discretized observations. However, the discretization of the parameters would still need to be fixed (and comparatively sparse) in this scenario, as the density estimator architecture is constrained and cannot trivially be replaced by a Neural Operator.

Joint posterior and SBC EoD on marginals

As discussed above, we are highly interested in fitting the joint posterior, including correlation structures of the function-valued and vector-valued parameters. This is also what our evaluation metrics measure (except for calibration, where appropriate calibration measures are lacking for high dimensional spaces).

MSE

We apologize for the confusion regarding the MSE metric. Concretely, by MSE we refer to the MSE of the posterior predictive. Concretely, for each test observation $x^o_j$, we sample several posterior samples using the trained model, and for each sample, we simulate a prediction, $x_{ij}$. The MSE is then calculated between each simulation and the corresponding test observation $x^o_j$, defined as $||x_{ij}-x^o_j||^2$. The reported MSE metric is then simply the mean of these individual MSEs over the samples $i$ from the posterior, and also averaged over several different test observations $j$. For example, for SIRD, we use 100 test observations, and 1000 posterior predictive samples for each test observation. We will make this definition clearer in our revisions. The values reported are also normalized by the number of points in the discretization of $x^o$ (full details in the Appendix S2.3).

FNOPE vs FNOPE (fix) on Linear Gaussian

An intuitive interpretation of the slight superiority of FNOPE (fix) in comparison to FNOPE on this task is the effect of the introduced positional noise. The noise injection modifies the likelihood, and as a result, FNOPE learns a posterior under a slightly broader likelihood than what is defined by the model. Therefore, when the true posterior is very constrained as in the Linear Gaussian task, the posterior quality is slightly poorer. However, for more challenging tasks, this is an acceptable trade off, as we gain complete flexibility on the posterior evaluation points, and the influence of the noise injection on the posterior quality is small (e.g. Fig. 5 in main paper).

NPE (spectral) for Darcy flow

Firstly, the Darcy flow simulator presented is a highly smoothing forward process. Even though NPE (spectral) has good performance in the predictive space (e.g. after pushing the posterior samples through the simulator), the statistics of the posterior samples might differ from the statistics of the ground truth parameters, which are drawn from the prior in this case. This is why we additionally included the posterior log-probabilities of the ground truth parameters as an evaluation metric in this task. Unfortunately, we cannot calculate the log-probabilities that the spectral methods (e.g. NPE (spectral) and FMPE (spectral)) assign to the ground truth parameters as they do not model the parameters directly. Hence, these methods are missing in Fig. 5d. We will make this more explicit in the manuscript.

Number of modes for Darcy flow

This is an excellent remark and we are thankful that the Reviewer brought it up. We additionally ran FNOPE with 16 and 24 modes. While there were essentially no differences for MSE and SBC EoD, the posterior log-probability slightly decreased (Tab. R2.1). We do not run the spectral baseline methods with 32 modes, as for this 2-dimensional domain this results in $32\times 32=1024$ basis functions for which these methods would need to infer the coefficients for, which is prohibitively large.

Table R2.1: FNOPE performance (means and standard errors) for Darcy flow task with different numbers of modes and 10k training simulations.

[1] Gonçalves, Pedro J., et al. "Training deep neural density estimators to identify mechanistic models of neural dynamics." elife 9 (2020).

[2] Dax, Maximilian, et al. "Real-time gravitational wave science with neural posterior estimation." Physical review letters 127.24 (2021)

[3] Miller, Benjamin Kurt, et al. “Truncated Marginal Neural Ratio Estimation.” NeurIPS (2021)

[4] Talts, Sean, et al. "Validating Bayesian inference algorithms with simulation-based calibration." arXiv preprint arXiv:1804.06788 (2018).

[5] Ambrogioni et al., “Forward Amortized Inference for Likelihood-Free Variational Marginalization.” AISTATS (2019)

[6]​​ Gloeckler, Manuel, et al. "All-in-one simulation-based inference." arXiv preprint arXiv:2404.09636 (2024).

We thank the Reviewer for their positive and constructive feedback. We note that the other Reviewers also highlighted the applicability of FNOPE to non-uniform discretizations (all Reviewers), and were positive about the simulation-efficiency (Reviewers 5bMC, u8xK), and the clarity of our presentation (Reviewer 5bMC).

To address the Reviewers’ concerns, we clarify the computational complexity which was incorrectly stated to be exponential in the number of discretization points, and report compute times, which are all in a reasonable range. Additionally, we ran extensive hyperparameter experiments to show the robustness of FNOPE across different settings.

We provide tables as supporting information to our answers. Where appropriate, this information will be expressed in figures in our revised manuscript.

Computational complexity and compute time

It is not the case that the computational complexity “scales exponentially … in the number of discretization points”. Instead, in the FNO architecture with $M$ modes and $N$ discretization points, the computational complexity is only $O(N\log N)$ (if using the FFT), and $O(NM)$ (if using the NUDFT). This is one of the main advantages of the FNO architecture—since higher order frequency components are discarded, the main computational complexity of the neural network is independent of the number of points in the discretization, allowing it to be efficiently applied to coarse and fine discretizations alike. It is true that the computational complexity of the FNO scales as $O(M^D)$ in the spatial dimension $D$. However, the focus of this work is on spatial-temporal problems $D=1, 2, 3, 4$ , which cover many scientific applications, and so this exponential complexity is not a critical limitation of the FNO architecture.

We believe the misunderstanding regarding computational complexity could stem from the statement in our limitations section. We apologise for this confusion, and we will clarify the distinction between the dimensionality of the spatial domains and the number of discretization points in our revisions.

Additionally, we provide a full breakdown of our experiments in terms of training time, sampling time, and network sizes of FNOPE and the baseline methods (Table R1.1). Even for the largest inference task considered (Darcy Flow), training was reasonably fast at around 1 hour on a single GPU. Sampling times are larger for FNOPE on the Darcy Flow task compared to the baseline methods, mostly due to the required solution of solving high-dimensional ordinary differential equations. However, even in this case, sampling times are sufficiently fast that this difference is negligible for our applications. For all other experiments, FNOPE is faster than or equivalent to the other methods in terms of training and evaluation times.

We hope that the clarification regarding computational complexities, as well as the empirical validation of the compute times, are enough to alleviate this Reviewer’s main concern regarding the applicability of our approach.

Table R1.1: Network sizes and training times for the main experiments. We report the mean over 3 runs with a training budget of 10k simulations. We omit showing any variation for more readability and since the results are robust across random seeds. Linear Gaussian and SIRD tasks were run on a 2080ti GPU, Darcy flow on an A100 GPU, and Antarctic ice on CPU.

Simulation efficiency

As the Reviewer pointed out, one strength of FNOPE is that it requires fewer simulations than existing methods, which could enable inference on computational expensive simulators. To emphasise this strength, we conducted experiments using only 100 training simulations. For the Linear Gaussian task the SWD is considerably smaller for FNOPE and FNOPE (fix) than for the baseline methods with 100 simulations (Table R1.2). Similarly, on the Antarctic ice task, FNOPE is more robust than the baseline methods (Table R1.3), even though the performance degrades for all methods, .

Table R1.2: SWD means and standard errors of the Linear Gaussian experiment for different numbers of training simulations.

Table R1.3: Predictive MSE means and standard errors of the Antarctic ice experiment for different numbers of training simulations.

Extensive Hyperparameter Ablations

Overall, FNOPE performs robustly over a wide range of different hyperparameters (see response to Reviewer u8xK for more details and result tables). First, we performed an ablation study on the impact of the Gaussian process kernel lengthscale on the performance for the Linear Gaussian and SIRD experiments (Tables R3.3, R3.4), which shows that FNOPE performs well for different lengthscales, even in the limit of white noise. Second, we investigated the dependence on the number of Fourier modes used for the Linear Gaussian and the Darcy flow experiment (Table R3.1, R3.2): FNOPE’s performance increases with the number of modes, but saturates for a higher number of modes. Third, we investigated the effect of the number of unmasked training data points on the performance of our method (Table R3.5). Here, we focus on the Linear Gaussian experiment. FNOPE performs well for all numbers of unmasked training points, but the performance increases slightly with the number of unmasked training points and exhibits saturating behavior with higher numbers of unmasked points. Figures for these ablation studies will be included in the Appendix of the paper.

Different base distributions (e.g. Matérn kernel)

Usually, the base distribution in flow-matching models is the unit Gaussian, $\mathcal{N}(0,I)$. Our main motivation in introducing the Gaussian Process base distribution is to produce base distribution samples which satisfy the smoothness assumptions of our FNO architecture, while still providing a distribution which can be efficiently sampled and evaluated. Thus, the main design choice in the base distribution is the power spectrum of the samples, which can be controlled via the lengthscale (see discussion above on hyperparameter ablations). A Matérn kernel could also present a valid choice of kernel, but we do not expect it to lead to different results.

Simformer as baseline

The computational cost of Simformer is at least $O(N^2)$ in the number of parameters, which is infeasible for a very large number of points as in the Darcy Flow Task ($\sim 16$k).

Our goal in comparing FNOPE to Simformer on the SIRD task is to demonstrate that FNOPE is comparable to an established approach at the challenging task of inferring parameters on non-uniform positions, as well as inferring additional vector-valued parameters. More specifically, for the SIRD experiment, we infer the parameters on much sparser grids than the rest of our experiments (40 gridpoints, compared to e.g. 1000 for the Linear Gaussian). However, FNOPE can be scaled to finer discretizations in contrast to Simformer.