On the Benefits of Active Data Collection in Operator Learning

We thank the reviewer for noting that our work provided rigorous proofs and numerical experiments.

• “I wonder if the authors can discuss what other types of queries they think might be interesting for PDE approximation if they have any in mind.”
  One natural direction would be to explore a pool-based active learning setting, where the learner selects inputs from a large (possibly infinite) unlabeled pool drawn from a fixed distribution $\mu$, rather than querying arbitrary inputs. This may be more appropriate in scenarios where labeling requires running physical experiments, making arbitrary queries impractical. However, in the context of PDE surrogate modeling, where data is generated synthetically, we believe the membership-type query model we adopt that allows arbitrary input queries is reasonable.

We thank the reviewer for their thoughtful feedback. We address their concerns below.

• ``On the assumption of a known covariance kernel $K$:" We agree that assuming knowledge of the kernel $K$ may seem unnatural from the perspective of classical statistical learning. However, in operator learning, this assumption is often more reasonable, as data is typically generated synthetically via simulations, where the input distribution is controlled by the user. For example, common covariance operators like $\alpha(-\nabla^2 + \beta I)^{-\gamma}$ on a periodic domain have known eigenfunctions—specifically, complex exponentials $e^{2\pi i m \cdot x}$, with the parameters $(\alpha, \beta, \gamma)$ only affecting the eigenvalues. Since our estimator relies only on the eigenfunctions, it effectively assumes that inputs are represented in a known basis determined by the domain geometry (e.g., Fourier basis on the torus, spherical harmonics on the sphere), not on the detailed spectral properties of the kernel. That said, this separation breaks down for kernels like the RBF (squared exponential), where the scale influences both eigenvalues and eigenfunctions. In such cases, assuming knowledge of the eigenfunctions is indeed more restrictive.

• In addition, from a learning-theoretic perspective, assuming knowledge of the kernel $K$ is arguably without loss of generality. In active learning, it's common to assume access to an unlimited pool of unlabeled samples $v_1, \ldots, v_m \sim_{\text{iid}} \mu$, where $\mu \in \mathcal{P}(K)$, and focus on minimizing label complexity—the number of labeled samples requested (see Hanneke, 2014). This aligns with our setting, where labeling (e.g., solving a PDE) is the primary cost. Given such unlabeled samples, one can estimate the covariance operator as: $\Sigma\_m = \frac{1}{m-1} \sum_{i=1}^m (v_i - \overline{v}\_m) \otimes (v_i - \overline{v}\_m),$ where $\overline{v}\_m = \frac{1}{m} \sum_{i=1}^m v_i.$ Since $\mathbb{E}[||v_i||^2] < \infty$, Theorem 8.1.2 of Hsing and Eubank (2015) guarantees that $\Sigma_m \to \Sigma$ almost surely in Hilbert-Schmidt norm, where $\Sigma$ is the integral operator associated with $K$. While our work assumes $\Sigma$ has a finite trace norm, this is not required to recover its eigenfunctions: convergence in Hilbert-Schmidt norm suffices for accurate top $n$ eigenfunctions approximation. Thus, assuming access to the eigenfunctions of $K$ is reasonable in theory, even if it may be computationally demanding in practice. We also note that this does not contradict known lower bounds in pool-based active learning, since our approach queries the oracle on estimated eigenfunctions of $\Sigma_m$, which are not i.i.d. samples from $\mu$.

• ``Can one show knowledge of $K$ is necessary to achieve fast rates, at least in some settings?" Yes, some knowledge of the kernel $K$ is necessary to achieve fast rates for certain estimators. Earlier, we showed that if the learner has access to unlabeled samples from a distribution $\mu \in \mathcal{P}(K)$, then $K$ can be estimated from data, making the assumption of known $K$ natural. Here, we consider the more adversarial setting where the learner has no access to such samples. Suppose the learner selects $n$ inputs $v_1, \ldots, v_n$ using any active strategy and receives exact labels $w_i = \mathcal{F}(v_i)$. Let $\widehat{\mathcal{F}}\_n$ be the resulting estimator. As is common with many linear estimators, we assume $\widehat{\mathcal{F}}\_n(v) = 0$ for any $v$ outside the span of $v_1, \ldots, v_n$. This models abstention outside the observed subspace. Since learner had no knowledge of $K$, we can construct a kernel

where the orthonormal functions $\varphi_j$ are chosen to be orthogonal to the span of the learner's queries. This is always possible in infinite-dimensional $L^2(\mathcal{X})$. We then define $\mu \in \mathcal{P}(K)$ as the law of a Gaussian process with kernel $K$. Since the entire support of $\mu$ lies outside the learner’s observed subspace, the estimator cannot generalize. In particular,

If $\mathcal{F}$ is not finite-rank, we can always find some $\varphi_\ell$ with $||\mathcal{F}(\varphi_\ell)|||^2 \geq c > 0$. By setting $\lambda_\ell = 1$ and choosing the rest of the $\lambda_j$ to satisfy a trace constraint, the lower bound remains constant, independent of $n$. In short, if the learner lacks both knowledge of $K$ and access to samples from $\mu \in \mathcal{P}(K)$, there exist kernels that force all probability mass outside the learner’s span, leading to a non-vanishing error regardless of how data is collected.

[1] Hanneke, Steve.``Theory of active learning." Foundations and Trends in Machine Learning 7.2-3 (2014).

We thank the reviewer for their thoughtful comment and noting that our work provides strong theoretical guarantees with empirical evidence. We address reviewer's concern below.

• ``While this is a theoretical study, it would be an interesting next step to see their method on a broader range of PDEs." We agree with the reviewer that generating our approach to handle potentially non-linear solution operators is an important future direction. We view our work as an important first step that lays a theoretical foundation for future works on active data collection in operator learning.

• ``It would be helpful to include an empirical comparison with other active learning methods from the literature, if any exist." Currently, there is no widely accepted active learning baseline for operator learning. Unlike the passive setting, where empirical risk minimization on i.i.d. samples is standard, active learning strategies are typically problem-specific and do not scale well to the infinite-dimensional setting. For example, uncertainty sampling and Bayesian algorithms will both require careful extensions to infinite dimensions.

• "It would also be good for the reader to see some intuition for the lower bound construction in the main paper instead of the appendix." We will include a short proof sketch of our lower bound construction highlighting key ideas in the main text of the final version of the paper.

• ``Have you considered a randomized sampling method, where input functions could be sampled with probabilities inversely proportional to the Christoffel function? Maybe it could be computationally cheaper?" We thank the reviewer for this thoughtful suggestion. Indeed, computing the top eigenfunctions of the kernel requires solving (or approximating) the Fredholm integral equation, which can be computationally intensive for domains with complex geometry. We had not considered Christoffel function-based randomized sampling, but it seems like a promising approach especially when $d$ is large. We appreciate this idea and will keep it in mind for our ongoing and future works.

We thank the reviewer for their thoughtful comments and for noting that the topic and the problem setting in the paper are intriguing. Below we respond to the main concerns and questions.

• On the $L^2$ norm and associated probability measure: We thank the reviewer for pointing this out. As discussed in Section 2.1, the $L^2$ norm is taken with respect to the base measure $\nu$. This is usually Lebesgue but can be more general such as Gaussian weighting. We will make this more explicit in the revision to ensure clarity.

• "Q1: Provide examples where the assumptions in Section 2.3 apply." We believe this is a standard framework commonly adopted in operator learning works [1,2]. While Assumption 2.1 is not always explicitly stated, it is typically implicit in the empirical evaluations of these methods. A more detailed discussion of such kernels can be found in [3]. By making this assumption explicit and combining it with active data collection, we are able to derive stronger guarantees—specifically, uniform bound over all distributions in the family $\mathcal{P}(K)$. In contrast, existing theoretical results usually provide guarantees only for the specific distribution used to generate the training data.

• Q2: Explain why ``we typically have $\epsilon \sim N^{-s}$" above Section 3: if we consider $\epsilon$ as an approximation error, the convergence rate here doesn't suffer from the curse of dimensionality. More important, without assumptions on function spaces (just $L^2$ space), a polynomial rate cannot be achieved in operator learning. The reviewer is correct that achieving polynomial approximation rates is generally non-trivial in operator learning. However, we would like to clarify that $\varepsilon$ in our setting does not denote the approximation error of the operator class. Rather, it corresponds to the error in training data due to the error of the PDE solver (i.e., the oracle $\mathcal{O}$) used to generate training data. That said, if the reviewer is referring to the error from approximating functions in $L^2$ using a truncated basis (e.g., Fourier series), we agree that the commonly stated rate $N^{-s}$ can be somewhat misleading. For functions defined on a $d$-dimensional domain, this rate holds only when all Fourier modes $k \in \mathbb{Z}^d$ satisfying $|k|_{\infty} \leq N$ are included. If instead only the first $N$ modes (in total number) are used, the convergence rate typically becomes $N^{-s/d}$ for functions with $s$-degree smoothness. This rate suffers from a curse of dimensionality as the reviewer notes. In our setting, this smoothness assumption is justified: the PDE solver is only applied to the eigenfunctions of the kernel $K$, which are typically smooth when $K$ is sufficiently regular. We appreciate the reviewer’s sharp observation and will update the discussion in the final version to clarify this point.

[1] Li, Zongyi, et al. ``Fourier Neural Operator for Parametric Partial Differential Equations." International Conference on Learning Representations. 2021.

[2] Kovachki, Nikola, et al. "Neural operator: Learning maps between function spaces with applications to pdes." Journal of Machine Learning Research 24.89 (2023): 1-97.

[3] Boullé, Nicolas, and Alex Townsend. "A mathematical guide to operator learning." arXiv preprint arXiv:2312.14688 (2023).

We thank the reviewer for their encouraging and positive assessment, and for recognizing that our work offers significant insights by establishing minimax-optimal error bounds and clarifying when active learning is beneficial. We address the reviewer’s questions below.

• ``Q1: Could FNO benefit from using the actively selected training data chosen by the proposed linear estimator?" Interestingly, FNO performs poorly when trained on actively chosen data for the linear estimator. We include the error curve of FNO trained on these active samples in Appendix E.

• ``Q2: The number of training samples in the experiments appears to be chosen empirically. Can they be potentially guided by Theorem 3.1 or any theoretical criteria?" Yes, in principle, our bound can guide the choice of sample size. Suppose we want to select $n$ such that the reducible error term
  $|| \mathcal{F} ||\_{\mathrm{op}} \sum_{j > n} \lambda_j$ is at most some small $\delta > 0$. Assume the eigenvalues decay polynomially, i.e., $\lambda_j \lesssim j^{-p}$ for some $p > 1$. Note that $p>1$ is required for the kernel to be Mercer. Then we have

as long as

For example, when $p = 2$, we recover the sample complexity corresponding to the standard fast rates.

• ``Q3: The paper focuses on comparing convergence rates rather than absolute final accuracy. Given that FNO has greater expressive capacity, is it possible that with a large enough dataset, FNO might eventually match or outperform the linear estimator?" As discussed above, when the eigenvalues decay polynomially at the rate $\lambda_j \lesssim j^{-p}$ for some $p > 1$, the error of our estimator scales as

In contrast, with i.i.d. samples, the best convergence rate achievable by FNO is

For FNO model to capture $\mathcal{F}$, the resulting notion of complexity of the model class is generally $\geq || \mathcal{F} ||\_{\text{op}}$. Thus, while both rates decay to zero as $n \to \infty$, our estimator achieves a faster convergence rate when $p > 1$. Therefore, for any given sample size $n$, our estimator should always have a smaller error than FNO in such cases.

• ``Weaknesses: While theoretically strong, the experimental section is limited to relatively simple PDE cases, and it’s unclear how well the approach scales to more complex, real-world scenarios or nonlinear operators. Some practical implications remain unexplored." Since our theoretical guarantees apply only to linear PDEs, we focus on standard linear PDEs in our experiments. That said, we agree with the reviewer that extending active learning approaches to nonlinear operators is essential for addressing complex real-world scenarios. We view our work as an important first step that lays a theoretical foundation for future research in that direction.

We thank the reviewer for their valuable feedback and for recognizing that the overall direction of our work is interesting and potentially impactful. Below, we address the reviewer’s questions and concerns.

• We agree with the reviewer that, in its current form, the scope of our work does not fully capture the complexity of real-world PDE problems. Regarding the comment on the small number of samples needed to recover the operator, we note that this efficiency is possibly due to the linearity of the underlying operator and the use of actively chosen inputs. While the setting is idealized, we emphasize that this is the first work to provide rigorous theoretical evidence establishing the benefit of active data collection in operator learning. Extending these ideas to nonlinear operators is an important future direction to capture real-world settings, and we hope our work lays the foundation for further works in this area.

• Since our theoretical guarantees apply only to linear operators, we focused on standard linear PDEs and did not apply our estimator to nonlinear cases.

• “What are the set of functions the proposed method uses to approximate the original function?” During implementation, our method does not require searching over a pre-specified function class via optimization. That said, as discussed in Appendix A.1, the resulting estimator can be viewed as a solution to a least-squares problem over the space of linear operators, with a specific choice of pseudoinverse.

• "Does the grid size matter in the evaluation?” We observed that our active estimator consistently outperforms the passive baseline across different grid sizes. The choice of $64 \times 64$ was made primarily for computational efficiency, as we trained 16 separate FNO models for different sample sizes to generate the convergence plots.

• “Is there any baseline for comparison in the active learning field that can be used directly for operator learning?” Currently, there is no widely accepted active learning baseline for operator learning. Unlike the passive setting, where empirical risk minimization on i.i.d. samples is standard, active learning strategies are typically problem-specific and do not scale well to the infinite-dimensional setting. For example, uncertainty sampling and Bayesian algorithms will both require careful extensions to infinite dimensions.