Hamiltonian Neural PDE Solvers through Functional Approximation

Thank you for your detailed review and for the suggested related works. We discuss these and some additional points below:

Related Works

Thank you for providing such an extensive list of prior works. Many of these are new to us, so we appreciate the perspective and view they provide. Since the submission, we have also found other works that approximate functionals with an integral kernel [1,2], and other reviewers have also pointed out similar works [3, 4]. We arrived upon the idea independently, which unfortunately accounts for much of the language/framing used in the paper. We are happy to change this, especially since we have a novel perspective on using functionals in Hamiltonian mechanics to evolve PDEs. The related works will be expanded to include suggested prior works on approximating functionals and Section 3 will present intuition on their capabilities from a theoretical standpoint, rather than a derivation. Any language claiming to be the first work on approximating functionals will be changed. Beyond these general revisions, we also discuss some differences to specific works:

Comparison to Miyagawa & Yokota, 2024

Miyagawa and Yokota approximate scalar functionals by a SIREN that takes time and basis coefficients as input, and outputs a functional. In this form, it is only applicable to FDEs. A direct extension to PDEs without introducing Hamiltonian mechanics isn’t obvious, since functionals are usually not involved in solving PDEs. Ours (and prior works) also approximate functionals with an integral kernel to provide another perspective from Neural Operator literature and Riesz representations. Despite these differences, we are able to show that, with an appropriate choice of kernel function, the derivative of an integral kernel can be expressed as the cylindrical approximation of a functional derivative, which ties these two works together. The proof sketch is given above as a response to Reviewer snot.

Comparison to Zhou et. al. and Naming of “Neural Functional”

The similarities to this family of works are likely in just the name “Neural Functional” and the general idea of inputting functions to networks. The authors are mostly interested in principled approaches to learn with the weights of other neural networks, for tasks such as predicting test-time accuracy of models or classifying INRs based on their weights. In our work, we mainly consider functionals as a mathematical object that maps between functions and scalars, and the applications of functional derivatives in PDE prediction.

However, we agree that the overlap in terminology is confusing and will work to remedy this. Specifically, we will change the title of the work to “Hamiltonian Neural PDE Solvers through Functional Approximation” (or something similar) to emphasize the novel perspective of Hamiltonian mechanics, which allows functional approximators to solve PDEs. Since the contribution of our work is no longer centered around introducing neural functionals (the mathematical object), we don’t see a need to introduce a new term and will refer to them more generically as functional approximators, integral kernels, learning the Hamiltonian, etc.

Comparison to Machine Learning for DFT works

These seem to be similar to the introduced method for approximating functionals, as they also employ a kernel integral (where the kernel is an energy density). Surprisingly, these works introduce kernel integration many years before they were developed as a theoretical foundation for neural operators. Beyond architecture, many works also make use of functional derivatives to approximate potentials for downstream use. We will make sure to mention these similarities, however our application of this paradigm to predict PDEs is still previously unexplored, due to the necessary introduction of Hamiltonian mechanics. Hamilton’s equations allow a mapping from scalar functionals to dynamical quantities of interest (du/dt) which differs from the usual Newtonian perspective in PDE works. Furthermore, learnable kernel integration in DFT arises since it resembles known equations and empirical functionals, however in PDEs, it is derived from operator approximation and Riesz representation theorems, which offers a perspective from functional analysis.

Computing Functional Derivatives with AD

Thank you for providing the relevant references. DFT is a field we are not familiar with, however, we recognize there are many similarities and will discuss them in our work. A similar line of work in molecular dynamics (MD) learns an energy functional and uses its spatial derivative (force) to advance a simulation in time. We made note of the MD domain in our related works, but will expand it to create a section about functional approximation/derivatives in other domains.

Overstatements

Thank you for pointing these out, we agree that the quoted lines are overstatements. We will remedy this by removing language that suggests that we introduce functional approximation, but rather use it as a tool to obtain neural PDE solvers that have better abilities to conserve Hamiltonians and extrapolate to new scenarios. Your intuition about the approach taken by Miyagawa & Yokota converging to ours is correct and is something we investigate in the above proof sketch and will discuss in the work.

Clarifying Relationship to Other Neural Surrogates

Thank you for the suggestion. We will expand on the benefits of our method (conserving energies, extrapolation) as well as some of the drawbacks (higher training/inference cost, more effort to derive Hamiltonian structure) with respect to common neural surrogates in the field.

Statistical Significance

Thank you for the suggestion. We have doubled the number of seeds (up to 6 seeds) in our experiments. Here is a comparison of one such table (Table 2 in the paper):

Table 1: Comparison of Mean/Std calculated across 3 or 6 seeds

For brevity we do not present all the updated tables here. While some numbers have changed, the overall findings remain the same. We also add these error bars into Tables 1, 4, 5, 6 (across 6 seeds). Here is one such updated table (Table 1 in the paper):

Table 2: Results for Toy Examples, with Mean/Std calculated across 6 seeds

For chaotic systems (KdV) or nonlinear functionals, spread can be large, which inflates standard deviations.

Example about Future Applications

In design optimization, for example, we are trying to find a set of design variables (i.e. points that define an airfoil spline) that minimize or maximize some scalar functional (i.e., drag of an airfoil). Most design variables are a discrete representation of an underlying object. For example, an airfoil may be defined by 100, 200, …, 1000 spline points, so a direct map from the coordinates to the functional is usually infeasible or at least not generalizable to different discretizations. Parameterizing the functional as a kernel integral is a natural remedy to this problem, and the gradient calculated by AD could be potentially used to adjust initial design variables towards a more desirable state.

Adoption of FiLM

We perform a series of ablation studies to justify the use of FiLM, in Table 5 in the response to Reviewer fiA5. We find that it can improve the performance of HNFs in some problems, but in general this is an empirical choice. Other options for incorporating conditional information into $k(x, u)$ can be investigated in the future.

Extensions to FDEs

This is an interesting direction, and it may be possible. Replacing F[u] with a learnable kernel integral seems possible, but one potential challenge will be using two quadratures to approximate dF[u]/dt: one for integrating the kernel and another for integrating the functional derivatives. An interesting aspect of our approach is it can be used independently of a basis or basis coefficients.

Robustness to Grid Size

We find that HNFs are approximately discretization invariant, shown in Tables 2-4 in our response in Reviewer fiA5. After training on a given resolution, they can be queried at coarser or finer grids without significant changes in performance.

References

1. https://arxiv.org/abs/2408.01362
2. https://arxiv.org/abs/2402.06031
3. https://openreview.net/pdf?id=aQuqw6eVKP
4. https://arxiv.org/abs/2205.03017

Thank you for the review. If there are any additional concerns/questions, we would be happy to discuss them.

Thank you for the review. Apologies for the brevity of tone, since there were many suggestions and questions:

Additional Theory

With the feedback of Reviewer a4TR, we have found additional insight that connects our approach with an approximation of functional derivatives.Consider functionals $F[u]: H \rightarrow \mathbb{R}$, where functions $u(x) \in H$ are in a Hilbert space $H$ with inner product $\langle\cdot, \cdot\rangle_H$. $u(x)$ can be written as $u(x) = \sum_{k=1}^\infty \langle u,\phi_k\rangle\phi_k(x)$, with a basis $\{\phi_1, \phi_2, \ldots\}$. Truncating the sum to a finite number of terms $m$ allows the cylindrical approximation of functionals [1]:

$$f(a_1, a_2, \ldots, a_{m}) := F[\sum_{k=1}^{m}a_k\phi_k(x))]$$

where $a_k$ are basis coefficients $\langle u,\phi_k\rangle$. With the cylindrical approximation, functional derivatives can be written as:

$$\frac{\delta F}{\delta u} \approx \sum_{k=1}^{m} \frac{\partial f}{\partial a_k}\phi_k(x)$$

which converges to the true derivative as $m\rightarrow\infty$. To discretize the derivative, we define a set of $n$ points $\mathbf{x} = [x_1, x_2, \ldots, x_{n}]$, which results in the discrete approximation:

$$\sum_{k=1}^{m} \frac{\partial f}{\partial a_k}\phi_k(x) \approx \sum_{k=1}^{m} \frac{\partial f}{\partial a_k}\phi_k(\mathbf{x}) = \begin{bmatrix} \sum \frac{\partial f}{\partial a_k}\phi_k(x_1) \\\\ \sum\frac{\partial f}{\partial a_k}\phi_k(x_2) \\\\ \vdots \\\\ \sum\frac{\partial f}{\partial a_k}\phi_k(x_{n}) \end{bmatrix}$$

We are interested if the gradient of a kernel integral can represent this discrete object. We skip to the final result, which uses kernel functions of the form $\kappa_\theta(x, \mathbf{u})$, where the kernel depends locally on the coordinate $x$ and globally on function values $\mathbf{u} = [u_1, u_2, \ldots, u_n]$.

The gradient of a nonlinear, global kernel integral can be written as:

$$\nabla_u \int_\Omega \kappa_\theta(x, \mathbf{u})u(x) \text{d}x \approx \mu \nabla_u(\sum_{i=1}^{n} \kappa_\theta(x_i, \mathbf{u})u_i) = \mu\begin{bmatrix} \frac{\partial}{\partial u_1} (\sum\kappa_\theta(x_i, \mathbf{u})u_i)\\\\ \frac{\partial}{\partial u_2} (\sum\kappa_\theta(x_i, \mathbf{u})u_i)\\\\ \vdots \\\\ \frac{\partial}{\partial u_{n}} (\sum\kappa_\theta(x_i, \mathbf{u})u_i) \end{bmatrix}$$

with quadrature weight $\mu$. The expression $\frac{\partial}{\partial u_{j}}(\sum\kappa_\theta(x_i, \mathbf{u})u_i)$ can be expanded:

$$\frac{\partial}{\partial u_{j}}(\sum_{i=1}^n\kappa_\theta(x_i, \mathbf{u})u_i) = u_1\frac{\partial}{\partial u_{j}}\kappa_\theta(x_1, \mathbf{u}) + \ldots + u_j\frac{\partial}{\partial u_{j}}\kappa_\theta(x_j, \mathbf{u}) + \kappa_\theta(x_j, \mathbf{u}) + \ldots + u_n\frac{\partial}{\partial u_{j}}\kappa_\theta(x_n, \mathbf{u})$$

In general, we can construct a kernel $\kappa_\theta$ such that $\frac{\partial}{\partial u_{j}}\kappa_\theta(x_i, \mathbf{u})$ is different for $i=\{1, \ldots, n\}$. An example is provided based on FiLM:

$$\kappa_\theta(x_i, \mathbf{u}) = [W^u\mathbf{u}]^i(W^xx_i +b^x) + [b^u]^i$$

where $W^u \in \mathbb{R}^{n\times n}, W^x \in \mathbb{R}^{1x1}, b^u \in \mathbb{R}^{n},b^x \in \mathbb{R}^1$ and $[\cdot]^i$ is an operation that takes the $i$-th row of a column vector. In this case, $\frac{\partial}{\partial u_{j}}\kappa_\theta(x_i, \mathbf{u})$ is equal to a function $d_{ij}(x_i) = W^q_{ij}W^xx_i$. With deeper kernels, functions $d_{ij}$ can become more expressive, and we can also absorb the basis coefficients into $d_{ij}$.

The $j$-th row of the gradient vector can be written as a sum of functions $\sum_{i=1}^{n} d_{ij}(x_i)$, which reduces to the cylindrical approximation when letting $d_{ij}$ be arbitrary. Letting $n \rightarrow \infty$ increases the number of functions $d_{ij}$ and coordinates $x_i$ to increase the number of bases to approach the true derivative as well as convert from a discrete to continuous representation. The cylindrical approximation makes no assumption of linearity, therefore, using a global, nonlinear kernel integral can represent a functional derivative. The kernel function must have enough inputs; if it restricted to a linear kernel $\kappa(x)$ or a local kernel $\kappa(x, u(x))$, the same calculation only yields a single basis function. Intuitively, the kernel needs to depend on enough inputs so its gradient can be expressive enough.

Discussion of Prior Works

Our related works include a few works from molecular dynamics (MD) that learn energy functionals and use their derivatives to update the simulation. Reviewer a4TR also suggested many works from the DFT field that use functional approximation, which we will include.

Accuracy vs. Conservation of Energy

In general, adding inductive biases to conserve energy can help accuracy, but is not strictly true. For example, learning the identity map conserves energy (since the state does not evolve), but is not accurate. While HNFs are not better than the Unet, the additional inductive biases help it to extrapolate to unseen conditions, which is the axis upon which we seek to improve.

Prediction Speeds

We discuss this in Section 5 and in Appendix B.2 of the paper and in Table 8 of our response to Reviewer fiA5, where we run timing experiments of different models and numerical solvers. The overhead of backprop increases the computational cost with respect to other models and scales with the number of query points, which, as you mention, is a limitation.

J Operator and du/dt Preparation

On real-world systems, prior works sometimes make an ansatz for contributions to the dynamics that conserve energies or satisfy certain properties (for example in climate [2]). If the Hamiltonian structure is not known, we can make a guess for its form based on similar, identifiable systems, and model other dynamics separately. In general, preparing du/dt for training has negligible overhead. It uses finite differences, which is a constant number of operations per sample, and du/dt can be precomputed and cached for training as well.

Impact of Grid Resolution

We perform a set of studies to compare the accuracy of models trained on a given resolution and queried at coarser or finer resolutions, in Tables 2-4 in our response to Reviewer fiA5. At test-time, HNFs can be used at different grids and resolutions without significant changes in accuracy. As for the computational cost, we compare the cost of our model to numerical solvers in Table 8 of our response to Reviewer fiA5. While the cost of our model does scale with the input resolution, the cost of numerical solvers also scales with increasing resolution.

Effect of Coefficients or Sources/Sinks

It is possible to use our work with varying coefficients and source terms with changes to the Hamiltonian structure. We provide examples below by introducing coefficients to 1D Advection as well as varying topography to 2D SWE.

In the presence of coefficients, the Hamiltonian for 1D Advection is:

$$\mathcal{H}[u] = \int_\Omega -\frac{c}{2}(u(x))^2 dx$$

which recovers the equation $-c\frac{\partial u}{\partial x} = \frac{\text{d}u}{\text{d}t}$ To accommodate varying coefficients, we can train HNFs by inputting coefficients to the model: $\mathcal{H}_\theta(u, x, c)$. In cases where the coefficients can be factored out (i.e., $\frac{\delta \mathcal{H}[u, c]}{\delta u} = c\frac{\delta \mathcal{H}[u]}{\delta u}$), once can simply scale the model output $\nabla_u\mathcal{H}(u, x)$ by $c$.

Spatially-varying beds $b(x, y)$ can accelerate or decelerate flows, which acts as a source term on the momentum balance. The shallow water equations become:

$$\partial_th + \nabla\cdot (\mathbf{v}h) = 0, \qquad \partial_t\mathbf{v} + \mathbf{v}\cdot\nabla\mathbf{v}=-g\nabla h -gh \nabla b$$

$h(x,y)$ now represents the height above $b(x,y)$. This source term changes the Hamiltonian to:

$$\mathcal{H}^b[\mathbf{u}] = \mathcal{H}^b[v_x, v_y,h] = \int_\Omega \frac{1}{2}h(v_x^2 + v_y^2 ) + \frac{1}{2}gh^2(1+b) \text{d}A$$

The operator matrix $\mathcal{J}$ is left unchanged. One can check that applying Hamilton's equations recovers the modified shallow-water equations. When comparing this derivation to one with constant bathymetry, one can show that:

$$\frac{\delta \mathcal{H}^b[u]}{\delta u} = \frac{\delta\mathcal{H}[u]}{\delta u} + \begin{bmatrix} 0\\\\ 0\\\\ ghb \end{bmatrix}$$

Therefore, source terms can be potentially captured by adding terms to model outputs. In PDEs where a decomposition like this is not possible, the source term will need to be added to the model input to produce the output $\nabla_u\mathcal{H}_\theta(u, x, b)$.

Extensions to Non-Conservative Systems

Similar to [4], we may be able to model the dissipative dynamics jointly with the Hamiltonian with the Dissipation $[D(u, x), H(u, x)]$. For continuum systems, the theory governing this is not well-understood. We can potentially make an ansatz about how the derivative of the dissipation relates to the dynamics $du/dt$ but this is not well-studied.

Comparison to Other Models

We run more baselines for 2D SWE, in Table 1 of our response to Reviewer fiA5, including a physics-informed baseline.

Training Data

To determine how much training data is needed, we train HNFs on datasets downsampled by a factor f = ½, ¼ or ⅛ . In general, increasing the dataset size consistently improves model performance.

Table 1: HNF Performance vs. Training Set Size

References

1. https://arxiv.org/abs/2410.18153
2. https://arxiv.org/abs/2404.10024
3. https://arxiv.org/abs/2201.10085

Thank you for taking the time to write a review and for your suggestions. We have implemented many of these below:

Related Works

Thank you for providing the reference. Since submitting the paper, we have also found more works that introduce a neural functional parameterized by a kernel integral [1, 2] as well as related works from other reviewers [3, 4] that are similar. Since we arrived upon the idea independently, much of the language in our paper unfortunately suggests that it is a novel idea. We will work to remedy this by changing the title and framing of the work, as well as amending claims of discovering this.

Despite the existence of prior works, there are some innovations that we make from independently developing functionals for Hamiltonian systems. Firstly, we make an effort to verify that kernel integrals can approximate functional derivatives with analytically understood systems. Secondly, we use neural functionals to do a PDE prediction task, which is enabled by Hamiltonian mechanics. Lastly, we investigate implementing the kernel as a conditioned neural field with more advanced architectures (FiLM/SIREN), which is empirically necessary for good performance (see ablation studies below). Specific to the referenced work from Rahman et. al., our approach introduces more complexity into the kernel, since they only work with a linear, local kernel parameterized by an MLP, and the referenced work does not make use of functional derivatives.

Baselines

We have included two additional baselines for 2D SWE, a Transolver [5] and PINO [6] baseline. We decided to compare against Transolver, since it introduces a new architecture (attention-based) and has shown empirically good performance, and PINO, to compare against an established, physics-informed method. The modified results for 2D SWE are:

Table 1: Rollout Error of Models on 2D SWE

We find that adding a physics-informed loss to FNO (i.e., PINO), helps to improve the performance slightly. The loss formulation we employed was: $L = L_{data} + w*L_{PDE}$, where $L_{PDE}$ is the residual of the 2D SWE equations, calculated with finite differences. For a fair comparison, we do not perform test-time finetuning of PINO. We report results with $w = 0.0001$, which was the best-performing model after a hyperparameter sweep.

Surprisingly, Transolver does not perform as well as other methods. We investigated this through an extensive hyperparameter sweep and different model sizes, but was unable to improve its performance beyond what is reported. As a whole, the results are consistent with prior works that suggest Unets, while simple, are a well-performing architecture [7] on regular grid problems (0.01 relative L2 error is also quite small, and is qualitatively indistinguishable from the ground truth).

Discretization Invariance

As suggested, we experiment with querying models at different discretizations than its training resolution. The results are below, with the training resolution in bold. Rollout errors are averaged over the validation set, where validation labels are either downsampled or re-solved at a higher resolution using the same initial conditions. Additionally, for 1D Advection, we run an experiment where the model is queried on an unseen grid (x=[0, 32], n_x=256), after being trained on a grid (x=[0, 16], n_x=128), which requires generating new validation samples.

Table 2: Rollout Error of Models at Different Resolutions, 1D Adv

Table 3: Rollout Error of Models at Different Resolutions, 2D SWE (Sines)

Table 4: Rollout Error at Different Resolutions, 2D SWE (Pulse)

In all cases, HNFs are capable of zero-shot super-resolution, as is possible with Neural Operators. Additionally, the error is approximately constant across discretizations and on a new, unseen grid for 1D Advection. Predictably, FNO models have nearly constant error across discretizations. Interestingly, for an extrapolated grid in 1D Advection, the model struggles to make predictions. For Unet models, the error increases when the grid spacing changes, however for a constant spacing $dx$, it is able to extrapolate to unseen grids (i.e. [0, 16] -> [0, 32] with $dx$ unchanged).

Ablations

We perform ablations to study the effect of different model choices on HNF performance, across all PDEs (1D Adv, 1D KdV, 2D SWE-Sines). We consider the effect of different kernels (linear/nonlinear), architectures (MLP/SIREN), conditioning mechanisms (concat/FiLM) and receptive fields (local/global). Since this design space is large, we consider the effects of incrementally adding model features, where the base model is specified as (Linear, MLP, concat, local), and each successful model is cumulative:

Table 5: Ablation Studies for Different PDEs

In general, using a SIREN vs. MLP architecture improves performance. We can see that using a nonlinear kernel is also important, as all Hamiltonians tested are nonlinear. In some cases, design choices such as using a global kernel or FiLM conditioning are also important.

Numerical Solver Timing

We update Table 6 with the runtime of a single step of numerical solvers, compared to neural solvers. Numerical solvers are run on a AMD Ryzen Threadripper PRO 5975WX 32-Core CPU due to lack of GPU compatibility. In the case of 1D Advection, the analytical solution is used. Furthermore, randomly sampled initial conditions can cause variability in the stiffness of the solution, therefore, solver runtimes are averaged across 10 samples. We note that the 2D SWE solver (PyClaw) uses a Fortran compiler and is optimized for hyperbolic PDEs, while the KdV solver is purely Python-based. Despite the additional overhead of backprop in HNFs, it is still modestly faster than a dedicated numerical solver. For a fairer comparison, the numerical solver would need to be coarsened to match the accuracy of the neural solvers (which are pretty accurate), however, this still serves as a point of reference and gives an understanding of the general timescales involved.

Table 8: Time/Step, in ms

References

1. J. Bunker et. al. Autoencoders in Function Space, https://arxiv.org/abs/2408.01362
2. D. Huang et. al. An operator learning perspective on parameter-to-observable maps, https://arxiv.org/abs/2402.06031
3. Z. Hu et. al. Neural Integral Functionals, https://openreview.net/pdf?id=aQuqw6eVKP
4. T. Miyagawa et. al. Physics-informed Neural Networks for Functional Differential Equations: Cylindrical Approximation and Its Convergence Guarantees, https://arxiv.org/abs/2410.18153
5. H. Wu et. al. Transolver: A Fast Transformer Solver for PDEs on General Geometries, https://arxiv.org/abs/2402.02366
6. Z. Li et. al. Physics-Informed Neural Operator for Learning Partial Differential Equations, https://arxiv.org/abs/2111.03794
7. G. Kohl et. al. Benchmarking Autoregressive Conditional Diffusion Models for Turbulent Flow Simulation, https://arxiv.org/abs/2309.01745v3

If there are any further questions/concerns, we would be happy to address them. Thanks!

Thank you for taking the time to review our work and provide feedback. Please find a response to your questions/concerns below:

Related Works and Framing

Thank you for providing the existing work. Since the submission, we have also found other works that propose functionals approximated by an integral kernel [1,2], and other reviewers have also pointed out similar works [3, 4]. We arrived upon the idea independently, which unfortunately accounts for much of the language/framing used in the title and paper. We are happy to change this, especially since we have a novel perspective on using neural functionals in Hamiltonian mechanics to evolve PDEs. Indeed, this motivates much of our paper, such as verifying the validity of functional derivatives on toy problems and investigating better architectures (SIREN, FiLM). The title will be changed to: “Hamiltonian Neural Solvers through Functional Approximation” (or something similar), and the paper will be revised accordingly. In particular, the related works will be expanded to include prior work on integral kernel functionals and Section 3 will present intuition on their capabilities from a theoretical standpoint, rather than a derivation. Any language suggesting to be the first work on approximating functionals will be changed.

Quadratures and Scalability

This is a limitation, especially in 3D where numerical integration is very expensive. One potential avenue to alleviate this is to train on low-resolution data and only use the desired resolution during inference. We find that HNFs can be discretization invariant, which we verify in experiments suggested by Reviewer fiA5.

Euler Lagrange formula

Thank you for bringing this to our attention, it is a very interesting perspective. We will discuss the relationship of the Euler Lagrange equation to functional derivatives, as well as go through some examples, such as with the 1D Advection equation.

Approximation Guarantees

You are correct that an assumption of compact support would be necessary to approximate a functional with an integral kernel. In PDE problems this is usually a valid assumption, as most practitioners are interested in a physical response in a predetermined domain. Beyond compact support, we will make it clear that more assumptions are needed about H[u], as it must exist in the dual space of functions that are: (1) continuous, (2) have a uniformly continuous derivative, (3) have compact support, (4) have bounded derivatives. These are given in Appendix C.1. Similar restrictions also exist for Neural Operators in order to achieve their approximation guarantees.

Training Differences

This is a great question, and one we will clarify in the work. When constructing the toy problems, we tried to use Algorithm 1 to train the FNO/MLP models, however they did not succeed presumably since their gradients calculated by AD don’t allow them to learn underlying functionals. Therefore, to make things fair in the toy problems, we trained all models to fit the scalar-valued functionals, which has the added benefit of evaluating if models can implicitly learn functional derivatives. For the rest of the work, we also tried fitting the scalar-valued Hamiltonians, however, this had an unintended consequence. Since the Hamiltonian is conserved over time, models would usually learn the identity map for its functional derivative, resulting in the PDE not evolving in time (but still conserving the Hamiltonian). To allow the model to learn dynamical information, fitting it to the functional derivative was necessary.

References

1. J. Bunker et. al. Autoencoders in Function Space, https://arxiv.org/abs/2408.01362
2. D. Huang et. al. An operator learning perspective on parameter-to-observable maps, https://arxiv.org/abs/2402.06031
3. M. Rahman et. al. Generative Adversarial Neural Operators, https://arxiv.org/abs/2205.03017
4. T. Miyagawa et. al. Physics-informed Neural Networks for Functional Differential Equations: Cylindrical Approximation and Its Convergence Guarantees, https://arxiv.org/abs/2410.18153

Looking forward to any future discussions if you have any questions/concerns.