Efficiently Parameterized Neural Metriplectic Systems

Thank you for your review. We are glad to hear that you found our paper well-structured and clearly written. Below you will find responses to your "Weaknesses" and "Questions" sections.

Response to weaknesses:

We believe there has been a significant misunderstanding which can be easily clarified. The gradients $\nabla E,\nabla S$ do not vanish anywhere in any of our examples, and indeed they should not. To guarantee that energy conservation and entropy generation hold along trajectories, it is only necessary that $\dot{x}\cdot\nabla E=0$ and $\dot{x}\cdot\nabla S\geq 0$, which is proven to occur for NMS in the paper. It is certainly not the case that $\nabla E=0$ is necessary for energy conservation, since this would imply that the energy function is globally constant over the state space, which is un-physical. Moreover, $S_1,S_2$ are state variables in the TGC example, meaning $S=S_1+S_2$ implies that $\nabla S = (0,0,1,1)^\intercal$ and the entropy is not constant (note that $x=(q,p,S_1,S_2)^\intercal$). We reiterate that the nondegeneracy assumption of $\nabla E,\nabla S \neq 0$ is quite mild and the minimum necessary for a metriplectic system to include both conservative and dissipative effects.

Response to questions:

1. It is true that a smaller learning problem does not directly imply superior performance. On the other hand, the optimization problems underpinning metriplectic methods such as GNODE, GFINN, and NMS are highly non-convex with formidable loss landscapes and (in the case of the former two methods) built-in redundancy. This creates a more challenging learning problem for GNODE and GFINN as their optimization takes place in a larger parameter space than is necessary. For instance, there is nothing stopping gradient descent on these architectures from moving their weights along fibers in the direction of this redundancy, thereby making no progress and hindering learning. Conversely, NMS eliminates (most of) this redundancy, leading to an easier learning problem which is empirically shown to produce solutions which are more accurate and generalize better than previous methods.

2. Please see Figure 4 in Appendix E for a computational study in this direction. Additionally, please see the new Appendix F which demonstrates the performance of NMS on a larger system with 201 degrees of freedom.

3. The metrics used are mean squared error (MSE) and mean absolute error (MAE) evaluated over the test dataset. As requested, we have included formulae for these in the revised Appendix.

4. While this is an interesting question, we believe it is out of scope for the present work. Note that there is inherent freedom in the representations for $E,S$, since only their gradients enter the metriplectic equations of motion. Therefore, there is no reason to expect that the energy and entropy learned by NMS directly approximate the known counterparts unless additional supervision is provided. This is no issue in practice, as it is guaranteed by Theorem 3.9 that the error will not exceed a threshold which is computable from the trained model.

Thanks again for your continued interest. We are happy to provide further clarification.

Thank you for your reply. I realize now that I misunderstood the set of state variables mentioned in the paper. However, I feel the response regarding energy still does not address the issue of energy having steady states, where $\nabla E=0$. It might also be helpful to include a calculation to confirm whether the examples presented satisfy this assumption (I verified that the energy in the second example does).

We reiterate that this nondegeneracy is a fundamental property of metriplectic physics, as the total energy $E$ is always in flux between its "kinetic" and "potential" components. We have calculated the gradients of energy and entropy for all examples and left them in the Appendix, where you can verify that they are never zero.

It is not entirely clear to me why optimization in a larger parameter space than necessary would yield worse results. In addition, from the reply, it seems the claim is that the proposed NMS has fewer parameters—am I understanding this correctly? It is evident that the proposed parameterization requires fewer learnable scalar functions, which I think does imply that the NMS involves fewer parameters. According to the hyperparameter section in pages 17 and 18, the number of parameters across the different models appears to be similar.

To clarify, there are two uses of the word "parameter" here which should be distinguished (as we have attempted to do in the paper). We have proven that NMS scales optimally in the number of learnable scalar functions required to express general metriplectic dynamics, which is a notable improvement over previous methods, e.g., GFINNs. Conversely, each scalar function can have as many "learnable parameters" (in the machine learning sense) as desired, and therefore one can build NMS architectures with a larger parameter count than GFINNs. It is the first notion of parameter (i.e., the number of learnable functions) which causes NMS to perform better, since this is where we have ensured that all such functions contribute nontrivially to the learned metriplectic operators. This cannot be said for GFINNs, which has inherent redundancy in its parameterization: many configurations of learnable functions lead to the same metriplectic system.

Additionally, regarding Figure 4, it is described as showing "the ground-truth and predicted trajectories..." but there seems to be no results presented about computational time or resource usage. Is this a typo?

Yes, we apologize. Please see Appendix G and Figure 7, as well as the new Appendix F.

Furthermore, it is apparent that the learned energy and entropy do not need to approximate their counterparts well. However, if the goal is to learn the correct differential equation, the learned governing functions (i.e., the right-hand side of the ODE) should approximate the true ones closely.

This is certainly true. We believe this approximation is evident from the numerical results, which show correct long-time generalization behavior of NMS. This could not occur if the RHS is not approximated well, as the error in the solution is directly bounded by the error in the RHS through the fundamental theorem of calculus in concert with the Cauchy-Schwarz inequality. In fact, the solution error (which is seen to be small) is precisely the integral of the error in the RHS. However, at the reviewer's request, we have conducted an additional experiment to this effect and included it in Appendix C.

Dear reviewer QHfC,

Thank you again for your feedback. Based on your comments, we have included analytic expressions for the metriplectic data $L,M,\nabla E, \nabla S$ in Appendix B (along with formulas for the MSE/MAE metrics), as well as a new experiment in Appendix C showing the error in the approximation of the metriplectic vector field by our NMS method. Besides the scaling study in Appendix G, we have also added a new experiment in Appendix F demonstrating the performance of NMS on a larger-scale example problem.

As the submission deadline for the revised paper is November 27, we kindly remind you to review our responses. We have made every effort to address your concerns thoroughly. We would greatly appreciate it if you could confirm whether our revisions have sufficiently resolved your questions and consider revisiting your score, or let us know if there are any further issues which we can help with.

Best regards,
authors

Thank you for your review. We are glad to hear that you found our work theoretically convincing and generally well written. Below are responses to your questions.

Response to questions:

0. This approach is novel in the field of metriplectic learning. However, the reviewer is correct that exterior algebra has been used in other numerical methods in order to enforce hard structural constraints. The work Gruber et al. (2023a) mentioned in the paper has used similar ideas to construct metriplectic reduced-order models, and the mentioned textbook Dorst et al. (2007) contains numerous examples of using the so-called geometric algebra (which extends the exterior algebra) in computer science applications. In a related but different direction, there is a long line of work applying ideas from exterior calculus to numerical methods, including the finite element exterior calculus (e.g., [2]) and the discrete exterior calculus (e.g., [3]). We have added a bit more discussion of this in the revised manuscript.

• 1+2) Yes, the reviewer has understood correctly. By "properly contain", we mean in the set theory sense: the set of matrix fields $L,M$ produced by NMS is strictly larger than the set of $L,M$ which define valid nondegenerate metriplectic systems. This discrepancy is precisely due to the failure to enforce the Jacobi identity for $L$ in NMS (which technically violates the metriplectic formalism), as well as the reviewer's observation that these fields are defined in terms of orthogonal projection. Since this was a point of confusion, we have clarified the language in the revised manuscript.

3. The reason for this is visible from the proof of Lemma 3.2, where it is shown that the exterior algebraic expressions derived are equivalent to the claimed matrix expressions after "adding zero". We have clarified this in the revision.

4. The reviewer is correct that this choice is not fully general. In the absence of other information, we have chosen to assume a normalized linear increase in entropy (or entropy density) over time, as consistent with the governing thermodynamics.

5. We thank the reviewer for catching this typo, which has since been corrected.

6. The use of "Lemmata" here is intentional, as this is the plural of "lemma".

7. The reviewer brings up a good point. As the construction of general metriplectic time integrators is an area of active development, we have focused mostly on standard, structure-agnostic integrators for the purposes of this work. In particular, the mentioned experiments are carried out using either the implicit midpoint method or the fourth-order explicit Runge-Kutta method, both of which have demonstrated reasonable performance in practice. More sophisticated integration methods based on splitting the conservative and dissipative dynamics are potentially possible, although this becomes quite delicate in the present case due to the full coupling of state variables in $L$ and $M$. Due to this, the issue of time integration is left for future work.

[2] Arnold, Douglas et al. Finite element exterior calculus: from Hodge theory to numerical stability. Bulletin of the AMS. 2010

[3] Hirani, Anil Nirmal. Discrete exterior calculus. California Institute of Technology, 2003.

Thanks for your continued interest. We are happy to provide further clarification.

What we mean is the following: in the definition of a metriplectic system, $L$ must generate a Poisson bracket, and therefore $L$ should satisfy the Jacobi identity given in Remark 3.5. However, the parameterizations leading to our NMS method do not directly enforce this condition, which contributes to the model class optimized by NMS being larger than the space of "admissible" metriplectic systems under this definition. As the reviewer has noticed, this issue is somewhat distinct from the nonuniqueness caused by the use of orthogonal projection. We will clarify this further in the paper to avoid confusion.

The technical issue preventing both energy conservation and the Jacobi identity in the present case is similar to the "usual" case of symplectic integrators outlined in the cited paper Zhong and Marsden (1988). Precisely, the chosen NMS parameterizations are constructed to exactly conserve energy pointwise in time, and also after the application of certain special integrators (say, conservative-dissipative splitting). Therefore, the $L$ parameterized by NMS cannot be symplectic, and hence cannot satisfy Jacobi, or else the discrete integration performed would correspond to the exact evolution of the dynamical system. Since hard enforcement of energy conservation is just a particular choice that has been made for this work, it could also be interesting to consider future learning methods which place symplecticity of the conservative dynamics at the forefront.

Thank you for your review. We are glad to hear that you found our work well-motivated and clear overall. Below you will find responses to your "Weaknesses" section.

Response to weaknesses:

Physics background and relation to Hamiltonian systems. We appreciate that the criticism that the background on metriplectic systems is brief. However, we are quite limited by space constraints. Note that we have made efforts to "signpost" these ideas by including numerous citations in Sections 1 and 2 where more exposition can be found.

The reviewer is correct that, in canonical Hamiltonian systems, $L(x)=J$ is the canonical symplectic matrix and there is no state dependence. At your request, we have left descriptions of the $L,M$ matrix fields corresponding to our examples in the Appendix.

On the decoupled block-wise structure. No, the considered examples are not trivializable in a way that admits their comparison with the previous methods mentioned in Section 2. Other examples include Navier-Stokes-Fourier, collisional kinetic plasmas, and resistive magnetohydrodynamics, to name a few. See, e.g., [1], for more examples.

Comparison with Hamiltonian learning. While Hamiltonian learning methods could conceivably be applied to the examples in Section 5 if entropy is ignored, this would be the wrong inductive bias for these systems. In particular, it is impossible for a Hamiltonian system to equilibrate in the way that is observed for the TGC example, as this requires a decrease in the Hamiltonian along the solution trajectory. To illustrate this point, please see the new Appendix E where we have compared NMS with a standard Hamiltonian neural network.

Choices of the initial unobserved states. All experiments use the 0-to-1 strategy unless otherwise specified (e.g., Appendix D). The justification for this is as follows: in the absence of other information, we have assumed a normalized linear increase in entropy density over time. Of course, this is not true in general, and there may exist other strategies which lead to better model performance. Whether entropy states and their dynamics are uniquely determined by the observable states is a good question. We suspect that this is not the case, although we do not have a proof of this. We thank the reviewer for pointing out the strategy in Chen et al. 2020, which would be interesting to consider in the future.

Test systems are low-dimensional. The reviewer is correct that the examples considered are relatively low-dimensional. On the other hand, it is clear from the numerical results that this is enough to expose noteworthy differences between NMS and previous state-of-the-art methods. Moreover, we believe the provided theoretical results, which are more extensive than those included in previous work, is of value to the field of structure-preserving machine learning. To further strengthen this argument, please see the new Appendix F, where we have included an additional example of state dimension 201.

A minor issue: misplaced parentheses for citations on Page 1 (probably due to mixing up \citep with \citet). Thank you for pointing this typo out to us. It has been addressed in the revision.

[1]  P. J. Morrison and M. H. Updike, “Inclusive curvaturelike framework for describing dissipation: Metriplectic 4-bracket dynamics,” Physical Review E, vol. 109, no. 4, p. 045 202, 2024.

Dear reviewer gp5d,

Thank you again for your feedback. Based on your comments, we have included new examples in Appendices E and F demonstrating both the necessity of metriplectic methods over Hamiltonian ones, as well as the performance of the proposed NMS method on a higher-dimensional example.

As the submission deadline for the revised paper is November 27, we kindly remind you to review our response. We have made every effort to address your concerns thoroughly. We would greatly appreciate it if you could confirm whether our revisions have sufficiently resolved your questions and consider revisiting your score, or let us know if there are any further issues which we can help with.

Best regards,
authors

Thank you for your review. We are glad that you have appreciated the comprehensiveness of our work and the mildness of the required assumptions. Below you will find responses to your "Weaknesses" and "Questions" sections.

Response to weakesses:

We appreciate the criticism that parts of the paper may be difficult to read for non-experts. Due to space constraints, we are unable to give a complete introduction to each topic, but we have stressed intuition throughout and made efforts to "signpost" the necessary ideas by including citations in Sections 1 and 2 where more exposition can be found.

To be clear, the key piece of intuition necessary for understanding the complexity reduction inherent in NMS is the following: while previous methods lift the problem of enforcing degeneracy in matrix fields $L,M$ to a problem of enforcing symmetry in tensor fields $\zeta,\xi$, this creates redundancy where multiple tensor fields can correspond to the same matrix field. Conversely, our NMS method makes use of the fact that the degeneracy conditions $L\nabla S = M\nabla E = 0$ factor the tensors, removing this unnecessary redundancy and producing parameterizations which exhibit superior scaling with no loss in representative power.

Response to questions:

This is an important point. You are correct that GNODE in particular does not work as well because of its restriction to constant tensors $\zeta,\xi$, but this is not the only reason for the lower performance of other metriplectic methods. The primary issue is that the optimization problems underpinning metriplectic methods such as GNODE, GFINN, and NMS are highly non-convex with formidable loss landscapes and (in the case of the former two methods) built-in redundancy. This creates a more challenging learning problem for GNODE and GFINN as their optimization takes place in a larger parameter space than is necessary. For instance, there is nothing stopping gradient descent on these architectures from moving their weights along fibers in the direction of this redundancy, thereby making no progress and hindering learning. Conversely, NMS eliminates (most of) this redundancy, leading to an easier learning problem which is empirically shown to produce solutions which are more accurate and generalize better than previous methods.