Efficiently Parameterized Neural Metriplectic Systems

Thank you for your thoughtful review. Below you will find responses to your comments and questions. Please let us know if we can do anything further to aid you in your final decision.

Weaknesses

While the authors improve the scaling from cubic to quadratic in the number of state variables, the total complexity (quadratic) still scales poorly with size of the problem (number of state variables / dimensions). Further, the current experiments and comparisons were performed on small benchmarks. However, since this paper is the first to point out that the cubic scaling can be improved by reflecting constraints due to degeneracy, I think the experimental component of the contribution is not the most important.

We acknowledge that the experimental component of the paper is limited to small-scale problems, although it is true that the main contribution of our work is not experimental (c.f. the contributions paragraph at the end of Section 1). We also agree that quadratic complexity in the state dimension $n$ is somewhat disappointing, but this is the minimum necessary for expressing an arbitrary nondegenerate metriplectic system, which can be seen based on the fact that $L$ depends on an arbitrary skew-symmetric matrix $A\in\mathbb{R}^{n\times n}$. Future work will have to sacrifice some degree of generality to obtain a better scaling rate.

Questions

Lemma 3.2 involves projection matrices to construct the $L,M$ matrices. How are these chosen in the experiments?

The projection matrices here are determined canonically from the gradients of the learnable energy and entropy $E,S$, and are never formed in practice (although it would be easy to do so). Instead, the products $L\nabla E$ and $M\nabla S$ are formed directly following Lemma 3.2 and the rules of the exterior algebra. This ensures that the necessary computations are done as compactly and efficiently as possible.

I am not an expert in this area of ML for physical modeling. Could the authors provide some insight into the ultimate goal system they would eventually wish to model with metriplectic neural systems? Are there real world physical data which could be well captured by these kinds of models?

The metriplectic formalism is especially useful for capturing dissipative perturbations of conservative systems. An example are the Navier-Stokes equations thought of as a perturbation away from the Stokes equations. Particularly, metriplecticity gives a useful mechanism for generating interpretable, energy-conserving and entropy-generating machine-learned models given only observed (or simulated) data. For example, with more assumptions on the system in question (to enable better scaling), it may eventually be possible to reliably learn the metriplectic evolution of a climate system given observations at various locations around the world. This would enable a cheap and useful surrogate for such systems which behaves in a thermodynamically realistic manner, potentially enabling better predictive capabilities.

Thank you for your thoughtful review. We are glad to hear that you found our paper well-written and could use it to study the current literature in this interesting area of ML.

Below you will find responses to your comments and questions. Please let us know if we can do anything further to aid you in your final decision.

Weaknesses

Unfortunately, the writing ends up being quite dense and technical with minimal outside applications.

Sure, emulating these physical systems in the experiments is quite nice, but what types of applications does this lead to? This is more of a writing based thing and the paper could be refactored around one of these applications if possible.

The primary application of this work lies in producing more interpretable and physically realistic models from data. Because the models produced by our NMS method are provably metriplectic, the conservative and dissipative contributions to the learned dynamics can be clearly identified, the dynamical evolution is (provably) stable, and the generalization error in the model is (provably) bounded. This contributes to the utility of the metriplectic formalism in phenomenological modeling, since any observed or simulated data can be fit to a metriplectic system in a way that "energy" is conserved and dissipative effects are captured through the generation of "entropy". This allows for more physically realistic surrogate models which are seen to train easier and generalize better than structure-uninformed approaches like NODE.

Questions

L138: Why do GFINNs require cubic scaling if they only need to learn O(rn^2) functions?

This is because the $r$ could be almost as large as $n$. More precisely, we do not know the rank $r$ of the dissipation matrix $M$ ahead of time, except that $1\leq r\leq n-1$. So, the scaling is potentially cubic in $n$.

In general, what is the lower theoretical bound for learning metripletic systems?

As illustrated by our parameterizations in Lemma 3.2, the lower bound on learnable functions for a general nondegenerate metriplectic system is quadratic in both $n$ and $r$. This is because $L$ depends on a general skew-symmetric matrix field $A$ described by $n(n-1)/2$ functions and similarly $M$ depends on a symmetric Cholesky factor containing $r(r+1)/2$ functions. To do better than this, we must give up on full generality and assume more knowledge about the metriplectic system in question, which is an interesting avenue for future work.

L363: what are relevant 3-d problems of the size that you’d like to tackle?

It would be highly interesting to apply an NMS-like approach to large-scale fluid problems such as the Navier-Stokes equations, or to tackle observational data about the climate or atmosphere. On the other hand, the method in this paper cannot apply directly to these scenarios due to its quadratic scaling in the state dimension $n$, and so more than just metriplectic structure will have to be assumed in order to improve this rate.

Fig 3b: is model parameters measured in thousands here? Don’t think we can have 4.3 parameters…

Yes, thanks for the catch. We have corrected this in the revised manuscript.

Could the authors generate plots like Fig 2 for the other two experiments? I think it may be useful to understand why the double pendulum example shows less relative improvement in that system?

We have done this, and included the relevant plots in the Appendix to the revised manuscript. We observe that the double pendulum is more difficult to train for all methods due to its inherent complexity, which is likely why there is less relative improvement in this case. Please refer to the PDF file in the global response.

Are the two main text experiments fully observed? Could the authors run these without full observations (e.g. with the diffusion for the unobserved states) if that makes sense in this setting?

Good question. The main experiments are not fully observed, which in particular makes them "harder": the network has to figure out the evolution of entropy density on its own. Assuming access to the full state data improves results substantially, as can be seen in Table 3 in Appendix C.

L226: why tanh activation for this task?

We choose tanh because of its smoothness. Our theoretical results assume a sufficient degree of regularity in $L,M,E,S$ (at least $C^1$ in most cases), and tanh ensures that this holds in our architectures. On the other hand, many different activations could be used instead, and it is possible (maybe even likely) that infinite regularity is not the best option in all cases.

Writing comments

• I would personally suggest moving a bit more of appendix E into the main text, as well as Fig 3. These are both interesting and pretty compelling.
• I would also suggest then moving section 3.1 into the appendix as I don’t think you really need much of that material. Pieces of Section 2 are much too dense and could be moved into the appendix as well; it should probably be half a page at most.

Thank you for the suggestions. In view of your comments, in the revised version we have moved some of the details in Section 2 into the Appendix in order to move some more of Appendix E (plus Fig 3) into the main text. We have elected to keep Section 3.1 as it is, since this material is essential for understanding the NMS method.

Thanks for responding to my questions. I now have a bit better understanding of this paper, and still think it should be accepted.

L138

Thanks, that makes sense, although I believe it could reduce the computational gains compared to the other methods.

Thank you for also attaching the figures for double pendulum. I see that exploiting the structure as your approach does tends to improve the qualitative performance as well.

Thank you for your thoughtful review. We are glad to hear that you found our paper well-written and reflective of a valuable contribution.

Below you will find responses to your comments/questions, including your main concern that the work may not be well-suited for a machine learning audience. For space requirements, we have occasionally truncated your original (italicized) comments with "...". Please let us know if we can do anything further to aid you in your final decision.

Overall, as a non-expert, my main concern is with the suitability of this work for a machine learning conference - I defer to the AC on this point. It seems that the machine learning techniques used within NMS are fairly straightforward, while it is the parametrization in Theorem 3.4 that seemingly constitutes the crux of the method. However, the statement and proof of Theorem 3.4 would be more accessible to a physics or math audience, than an ML audience.

It is true that the parameterizations in Theorem 3.4 and the associated theoretical results (Proposition 3.7 and Theorem 3.9) are a primary contribution of this work. However, NMS is fundamentally a machine learning method: the goal is to learn a nondegenerate metriplectic system from data. We think that the use of mathematically rigorous machinery in pursuit of this goal enhances the value of the work and does not disqualify us from high-quality machine learning venues such as NeurIPS. Indeed, the presented mathematics have allowed us to decouple metriplectic structure-preservation from any optimization error incurred during training, leading to a provably structure-preserving model while simultaneously guaranteeing universal approximation and meaningful estimates of the error in the trajectories. Note that several related works have been published in NeurIPS. For example:

• Greydanus et al. Hamiltonian neural networks. NeurIPS 2019
• Finzi et al. Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints. NeurIPS 2020
• Chen et al. Neural symplectic form: Learning Hamiltonian equations on general coordinate systems. NeurIPS 2021
• Lee et al. Machine learning structure preserving brackets for forecasting irreversible processes. NeurIPS 2021
• Gruber et al. Reversible and irreversible bracket-based dynamics for deep graph neural networks. NeurIPS 2023

Quality: ...it seems like other methods (e.g. which preserve energy but do not increase entropy, or even those which are not physics-informed at all), should be included too.

We agree that additional experiments are always helpful. Note that the case of networks which are not physics-informed has already been handled by the NODE architecture (see Tables in Section 5), and it is clear that this approach is not as performant as NMS. The case of purely energy-conserving networks (e.g., Hamiltonian NNs) is being included in the revision, and is also notably less performant than NMS. This is expected, since the underlying dynamics are not Hamiltonian.

Significance: I am not sure how widely applicable metriplectic learning systems are...

The metriplectic formalism has been widely adopted in nonequilibrium thermodynamics, and a large variety of interesting physical systems can be understood through this lens. Besides the references mentioned in the Introduction, a representative monograph with many examples is [1]. While we are limited by space requirements, we have added a bit more background information to the revised Section 1.

[1] Öttinger, Hans Christian. Beyond equilibrium thermodynamics. Vol. 5. Hoboken: Wiley-Interscience, 2005.

Weaknesses

1. Although well-written, the paper is not accessible to most machine learning audiences...

It is true that a background in the relevant mathematics is useful for fully understanding our work. However, we do not assume that all readers possess this knowledge; references are left for those seeking more background, and the paper has been written so that non-experts can follow the main ideas at a high level.

2. Mathematical terms such as algebraic brackets, Poisson brackets, degenerate metric brackets, etc. should be defined in the beginning of the paper, or with a reference to a textbook or other paper defining them...

Note that the first set of requested terms are defined in the second paragraph of page 2, and the second in Section 3.1, albeit in a terse way because of the space requirements. Since this was not immediately clear, we have attempted to include a few more details and citations in the revised manuscript.

Questions

1. NMS is a more efficient parametrization than GNODE and GFINN. Is it also more efficient computationally (end-to-end)?

This is a good question. While NMS is not epoch-to-epoch competitive with the structure-agnostic NODE in cost, we observe that it does perform better than GNODE and GFINN for the same number of parameters. This is illustrated in Figure 3b in Appendix D.

2. Are there non-metriplectic methods that are suitable for comparison in the experiments?

As mentioned above, we plan to include a comparison to Hamiltonian NNs in the revised manuscript.

3. What is the motivation for metriplectic methods overall — do they have drawbacks relative to methods that do not preserve energy/increase entropy? It would be helpful to include these in related work.

Metriplectic methods are designed to encode the first two laws of thermodynamics in a way that is suitable for modeling physical systems with dissipation. By capturing dissipative phenomena through entropy gain, metriplectic models are interpretable and thermodynamically closed. Their primary drawback is theoretical: it is difficult to write physical systems in metriplectic form, and there is no general algorithm for doing so. This is what motivates machine learning approaches like NMS, which remove the need for this difficult step. For clarity, we have added a few more details about this in the revised manucript.