Derivatives Are All You Need For Learning Physical Models

The paper proposes training neural networks by fitting its derivatives to the derivatives of a data set. The authors evaluate this method on physical systems, where the derivatives are obtained from the solution trajectories of differential equations.

I see several fundamental issues with this work:

1. The method is presented as ‘physically consistent’, like physics informed neural networks (PINNs), because it uses partial derivatives. Later, it is argued that the authors’ method is easier to train because the network’s partial derivatives are compared to ‘individual targets instead of being entangled together’. The loss of PINNs incorporates the relationship between partial derivatives; this is what makes them physically consistent. To me, the proposed method appears to be a supervised method through and through.
2. The method is presented as a solution method to physical systems (see for instance in the conclusion: ‘We showed theoretically and experimentally that our method successfully learns the solution to a problem and remains consistent with its physical constraints, …’). What was actually done in the experiments is that the solution was known beforehand, either determined by classical numerical methods or it was a system with a known analytical solution. Then a neural network was fitted to that solution. While it is true that a solution was learned in the literal sense, the statement is misleading, as it indicates their method is a solution method for differential equations, just like PINNs. But more importantly, it raises the question what the method’s intended purpose then is.
3. The role of physics in this method is not really clear. As explained, the proposed technique is a method to fit a neural network to a given curve, not a solution technique. The authors argue that this is a consequence of the uniqueness theorem from the theory of ordinary differential equations, where dx/dt = f(x,t). What the method is actually based on is a simplified case, where the derivatives are known beforehand, corresponding to dx/dt = f(t) and within the scope of the well-known fundamental theorem of calculus. That a curve can be reconstructed from its derivatives is not a specific property of dynamical systems but only a property of a differentiable curve. For this reason, I would also recommend to present this work more related to Sobolev training than to physical systems and PINNs. The connection to Sobolev training should be clear from the beginning due to their strong similarities.
4. In section 2.1, theoretical analysis, the statements of the mathematical theorems are hard to follow as some of the quantities are not declared, or used in a way that does not make sense. For instance, it is not clear what the arrow in theorem 2.1 precisely means as there are no sequences that could somehow be connected to convergence of some sort. Consequentially, the proof also makes no sense to me.

The paper primarily addresses a critical question in the training of neural PDE solvers: what should the learning objective be, or, in other words, how should the loss function be defined. Previous work has commonly utilized either the discrepancy between predicted and true state values or the residual form of the PDE itself as the loss. In contrast, this paper proposes a new loss function based on the derivatives of the state with respect to the time variable $t$ and spatial variable $x$ as supervisory signals. The authors also theoretically demonstrate that gradient-based supervision alone is sufficient and necessary for convergence toward the target solution. Extensive numerical experiments are conducted to validate the proposed method’s effectiveness.

[-] The main content of the paper centers around introducing the authors' perspective and framework—specifically, the IMPORTANCE OF DERIVATIVE LEARNING. However, it lacks a detailed discussion of challenges encountered during this process and the proposed solutions. Thus, the primary contribution lies in the perspective itself, asserting that derivative-based supervisory signals are both important and novel, which is the key selling point of the paper. However, from my viewpoint, this perspective is not entirely novel, as related approaches have been explored in previous work. Here are a few examples:

• [1] Sitzmann, Vincent, et al. "Implicit neural representations with periodic activation functions." Advances in neural information processing systems 33 (2020): 7462-7473.
  • Research has shown that first- or second-order gradients alone can be effective for image supervision and has examined the associated challenges, such as activation function selection.
• [2] Li, Chongchong, et al. "Gradient information matters in policy optimization by back-propagating through model." International Conference on Learning Representations. 2022.
  • In reinforcement learning model training (closely related to the ODE scenarios in this paper), gradient information has been shown to be crucial for downstream control tasks, with solutions also proposed to address this.
• [3] D'Oro, Pierluca, and Wojciech Jaśkowski. "How to learn a useful critic? model-based action-gradient-estimator policy optimization." Advances in Neural Information Processing Systems 33 (2020): 313-324.
  • In the learning of value functions in reinforcement learning, since downstream policy learning relies on the gradient of the value function (similar to how PDE or ODE gradients are essential for model evolution in this paper), gradient-based supervision is incorporated into value function learning.

These are merely indicative examples, suggesting that this topic has been explored in deep learning literature. The authors should demonstrate awareness of such research to avoid reinventing the wheel.

[-] Moving to the specific research content, Definition 2.1 attempts to prove the sufficiency of the proposed loss, but it does not clarify whether methods like PINN, OTL, or SOB are also sufficient, nor does it establish any clear superiority of the proposed loss over these methods.

[-] If we are given a PDE with an analytical form along with initial and boundary conditions, how would the proposed method obtain supervisory signals? Would it still require numerical methods to generate data $u(x, y, t)$, followed by gradient estimation via interpolation?

[-] For a system with a 3D input (x, y, t) and a 2D output (u, v), first-order gradient information actually forms a Jacobian matrix (2x3). Would the second-order Hessian matrix then be a higher-dimensional matrix? What computational resources would be required to estimate such derivative information?

[-] Regarding the experiments, could the authors clarify why the metrics in Tables 2, 3, 4, 5, and 6 differ so greatly ( Perhaps Table 3 includes the three more intuitively understandable metrics)?

[-] The proposed method does not perform well in Tables 4 and 5, and its performance in Table 7 is comparable to that of SOB (which aligns with my earlier concerns about the theoretical part). I reasonably suspect that, if given appropriate weight adjustments, the SOB method would be at least as effective as the proposed loss, given that the proposed loss is a subset of the SOB method’s loss function. The paper provides no evidence (or intuitive argument) to suggest that adding a reasonable constraint to a loss would have adverse effects.

[-] The paper's presentation could be improved. Too many important details are placed in the appendix, leaving the main text with almost no information about the algorithm and experimental specifics—such as how the loss function is implemented. There is nearly one page of unused space in the main text, which could be used to provide these crucial details.

This paper proposes a method for learning solutions to ODEs/PDEs by training with information of the derivatives of the solution, called DERL. The method using a loss term consisting of the error in the derivatives and the error in the initial/boundary conditions. DERL is compared to PINN, OUTL, and SOB on several differential equation problems. Additional, the method is applied to learning the solution from a "teacher" model.

This paper shows that it is possible to "learn" the solution to a particular PDE from the boundary/initial conditions as well as labeled data for the partial derivatives at collocations points in the space-time domain. In essence, this is an alternative approach to supervised learning of the solution, but using the gradients of the solution field rather than the solution itself.

We thank the reviewers for their comments and suggestions and hope to engage in a fruitful discussion during this next phase. Here we provide answers to the most important points raised by the reviewers. We also reply to each reviewer individually.

First, we would like to point out that the transfer of physical knowledge between models through derivative distillation, one of the key contributions of our work, was largely disregarded in the reviews. We highlighted the novelty and relevance of derivative distillation in the introduction and methodology. The related experiments in Section 4.5 led to very promising results. We believe derivative distillation to be both a novel and high-impact contribution to future research, as it allows for an incremental definition and training of physical models, improving efficiency and the learnability of constraints by composing more than one model. Unfortunately, the reviews did not comment on this point and the scores did not seem to take derivative distillation into consideration.

From the reviews, it seems that the supervised nature of our method is a disadvantage. We do not understand why this should be the case. Although the comparison with PINN may have been misleading for some reviewers, we never claimed our method to be unsupervised. Furthermore, there are many methods in the literature, even recently published, that define and create new supervised methods to learn physical systems: NeuralODEs [1], Hamiltonian Neural Networks [2], Fourier Neural Operators [3] are just some examples of supervised methods with different architectures published in recent years, as we also report in our related works section (3). Our approach adopts the supervised learning paradigm and outperforms other supervised methods. And even PINN, in many cases.

We also would like to stress that our method is not specific to a PDE but it is rather of general application (as shown by our experiments on different PDEs). It only requires automatic differentiation to be employed. We tested it against PINNs, as they are the state-of-the-art for physical consistency, with an MLP backbone. However, DERL works also on other architectures, as we showed by using it with the Neural Conservation Laws architecture [4] in section 4.5.2.

[1] Chen, Tian Qi et al. “Neural Ordinary Differential Equations.” Neural Information Processing Systems (2018).

[2] Greydanus, Sam et al. “Hamiltonian Neural Networks.” Neural Information Processing Systems (2019).

[3] Li, Zong-Yi et al. “Fourier Neural Operator for Parametric Partial Differential Equations.” ArXiv abs/2010.08895 (2020).

[4] Richter-Powell, Jack et al. “Neural Conservation Laws: A Divergence-Free Perspective.” ArXiv abs/2210.01741 (2022).

The paper proposes a supervised learning approach for training neural networks by fitting their derivatives provided in a dataset. This method is evaluated on physical systems and compared with other approaches that utilize different forms of data. While the numerical results appear to favor the proposed method, all reviewers raised concerns about the fairness of these comparisons. For example, PINNs is an unsupervised approach that does not require labeled data, unlike the proposed method. This fundamental methodological difference makes direct comparisons challenging.

To improve the paper, the authors could focus on scenarios where different methods are equally applicable (e.g., OUTL and PINNs). One promising scenario mentioned is model distillation from a pre-trained model, where diverse types of data are available. However, the motivation and structure of the paper need clearer organization to enhance readability and impact. Additionally, comparisons with Sobolev learning (SOB)/gradient-augmented learning methods would provide significant value. The proposed method closely relates to SOB but uses fewer supervision signals while reportedly achieving better performance. The reason for this phenomenon is only vaguely explained. As noted by reviewers, a more systematic investigation into this phenomenon, including robustness analysis, would significantly enhance the paper’s scientific contribution.

Overall, the proposed method shows promise, but the paper in its current form is not ready for publication due to the aforementioned concerns. Addressing these points could help strengthen the work for future submissions.

Reject