Noether's Razor: Learning Conserved Quantities

Thank you for your feedback and help to improve the paper. We thank the reviewer for finding the paper well-structured and self-contained with a clear goal. The reviewer appreciated the empirical validation and experiments that demonstrate improved generalization of the proposed approach.

Q1. Novelty

Firstly, most works on symmetry learning focus on relatively simple groups on static domains (e.g. affine invariances on images), whereas we tackle the more novel and open problem of learning symmetries in dynamical systems using train data. The Noether's theorem parameterization of symmetries as conserved quantities in Sec. 3.1 is indeed an important contribution to achieve this. Further, we took considerable steps to actually scale model selection with VI to DNNs, through non-mean-field posteriors and optimized prior/output variances (details in App. D). This differs greatly from typical mean-field and fixed variances commonly used in Bayesian NNs, which have not been shown to capable of performing model selection. Most importantly, we demonstrate successfully learning symmetries of dynamical systems, which are demonstrably correct, through conservation laws in dynamical systems. We deem this a very novel contribution.

Q2.a. Metrics

We thank the reviewer for pointing out potential nuances in measuring the learned symmetries. First, we’d like to point out that due to the assumption of quadratic conserved quantities, their dimensionality is only roughly quadratic in the dimensionality of the phase space, which is $(2 \times 5)^2=100$ for the 5-harmonic oscillator, and less for the other experiments. Thus the spaces are not very high dimensional. There is no standard way of evaluating learned symmetries. Our symmetry correctness measurement reduces to: measure the degree by which two linear subspaces, each generated by non-independent basis vectors, differ. The approach we proposed seemed like a simple way of measuring this, but welcome suggestions for alternatives.

Q2.b. Baselines

Most recent work on Bayesian model selection that scales to deep learning has relied on the KFAC-Laplace approximation. Laplace is often easier than VI because the posterior covariance does not need to be optimized but can be obtained by estimating the local curvature through gradients. In our particular case, however, Laplace can not be used straightforwardly (we tried this first!). The technical reason for this is that KFAC requires gradients through our forward-pass which in our model already required gradients through the symplectic form. The required 2nd order gradients are not readily supported efficiently in most deep learning framework, not even in JAX/diffrax which is tailored for higher-order derivatives and differentiable solvers. We did take inspiration from Laplace-based approaches in our use of Kronecker-structured posteriors and optimizing variances (see App. D). We thank reviewers by bringing this up and will include this in the main text.

Q3. Seeds

We will run all experiments with multiple seeds and provide mean and standard error in the final version.

Q4. Generalization

In principle, generalisation properties are determined by the prior over functions (which includes choices in the neural network architecture). To obtain better performance regions far away from the data we therefore either need data in these regions, or improve the prior over Hamiltonians. This work provides important insights in automatic model selection to DNNs, and therefore provides a pathway to making this process automatic, without the need for retraining and manual cross-validation.

Limitations

Thank you for your suggestions, we will make sure these limitations are more clearly highlighted in the final version of the paper.

First successful model selection in DNNs

We follow what is common in Bayesian statistics and refer to the 'model' as the joint distribution (likelihood + prior). Hence, the variational parameters (\mu, \Sigma) are not considered to be part of the model and we therefore do not consider this to be a form of model selection. The hyperparameter \eta parameterizes symmetries/conserved quantities and is part of the model, impacting the effective prior over Hamiltonians H. We show that we can successfully optimize \eta with the ELBO, which demonstrates model selection in DNNs. Although VI can indeed be scaled to DNNs for uncertainty estimates [1], its use for model selection has not been successfully demonstrated yet, as also evidenced by the following citation from [1]: “Empirically we found optimising the parameters of a prior (by taking derivatives of Eq.1) to not be useful, and yield worse results.”. We took considerable efforts to improve the tightness of the lower bound to make model selection work, using non-mean-field posterior and optimized variances (see App. D for details). So far, we are not aware of other works that have managed to do model selection in DNNs with VI, and therefore consider this to be significant. We hope this clarifies our claim, and will make sure that sufficient context is provided in the final version.

[1] Blundell, et al. "Weight uncertainty in neural network." ICML 2025

Thank you for raising the score. We are glad that our rebuttal resolved the most pressing concerns, and the impact of Noether's razor is more clear now. To address your follow-up questions,

VI for model selection

The main topic of the paper is learning symmetries by optimizing learnable conserved quantities by optimizing the marginal likelihood. The main text therefore focuses on the proposed use of Noether's theorem to parameterize symmetries as conserved quantities, and describes how marginal likelihood optimization yields a Noether's razor effect in the prior that enables symmetry learning. We did put quite some effort in scaling VI to model selection, but view our use of VI ultimately as a means to an end to approximate the marginal likelihood in deep networks. For this reason, we include details on VI in the appendix, even though some aspects could also be regarded as contributions. Nevertheless, we highlight being able to perform model selection with VI as a significant achievement in the main text, since this has not yet been demonstrated succesfully for deep neural networks before. We expect the details of our VI training scheme to be of independent interest to some VI practitioners, even though we view these as implementation details in the context of this paper, which is symmetry learning through Noether’s razor. Another reason for discussing this in an appendix is that some components are not direct contributions. Nevertheless, combining them together and applying them to perform model selection is. For instance, our use of structured covariances is not novel on its own (matrix normal posteriors are also proposed in [3]), but using them to obtain a tight enough lower bound to perform model selection is novel. Moreover, this particular choice of structured posterior was based on the insight that the matrix normal posteriors of [3] is equivalent to the covariance of the KFAC approximation, which had been shown capable of performing model selection in deep networks using Laplace. We believe that this connection is not widely recognized by the community. Similarly, optimizing prior variance and output variance is not new. However, we note that a large majority of Bayesian deep learning papers relies on downweighting the KL term by a beta factor, which we argue primarily fixes misspecification in the likelihood variance and can better be resolved by optimising this variance empirically (as we do in closed-form). We hope this clarifies why we share details on VI in an appendix, but do highlight the ability to do model selection in deep neural networks with VI as a significant achievement in the main text.

Closeness to [4]

Our work indeed builds upon [4] and is similar in that both works propose to learn symmetry through a lower bound on the marginal likelihood. We do feel that our work is a big leap forward compared to this prior work in both a) the parameterization and b) the objective function. Firstly, [4] only learns simple affine transformations of pixel coordinates of the input images (like rotations and translations) in supervised learning context. We consider symmetry learning of phase space in the more complex task of modelling dynamical systems, which requires learning the Hamiltonian and predicting trajectories over time. To do so, we use Noether's theorem to parameterize symmetries in terms of conserved quantities, one of our key contributions (see also our overall response). Secondly, [4] has only successfully demonstrated symmetry learning in single hidden layer neural networks with a bound that only integrates out parameters of the last layer. As such, the bound in this paper is not tight enough to go deeper and [4] was not able to successfully scale to deep neural networks. We use a structured and layer-factored posterior and end up with a significantly different lower bound that does scale. Unlike [4], we are able to demonstrate VI-based model selection in deep neural networks.

Metrics

We're learning a linear subspace of symmetry generators. Via SVD, we find an orthonormal basis of unit vectors of this space. For each of these basis vectors $v_i$, we compute how much overlap there is with the linear subspace of ground-truth symmetries, with orthonormal basis $w_j$. The spaces fully overlap if for each $i$,

$$\lVert v_i^\parallel\rVert^2=\sum_j \langle v_i, w_j \rangle^2 = 1$$

Note that we’re not comparing the overlap between a single vector $v_i$ to and a vector $w_j$, but instead we measure the degree by which $v_i$ lies in the span of all the $w_j$’s. The inner product in this expression is between two vectors of unit norm, so is equivalent to the cosine similarity.

While we are unaware of standard-practice ways of measuring subspace overlap, we deem our proposed metric using singular vectors to be a natural choice. We welcome suggestions for alternatives. We will emphasise the fact that we need to compare linear subspaces, not vectors, better in the paper.

Thank you for your feedback and help to improve the paper. We thank the reviewer for finding the paper 'intriguing' and 'conceptually very interesting' and that the empirical evaluation shows that the ideas can be promising

Difference with prior work

We thank the reviewer for pointing this out and will clearly state differences with prior work.

The reviewer is correct that [Alet et al., 2021] is closely related. There are, however, important differences. The most important difference is that we explicitly use the learned conserved quantities to infer the symmetries of the phase space. These symmetries then are used to symmetrize the Hamiltonian. In doing so, our work uses the essence of Noether’s theorem. In contrast, their work only learns conserved quantities, without inferring the resulting symmetries. Secondly, [Alet et al, 2021] models images with low dimensional phase spaces, whereas we consider complex dynamical systems with higher dimensional phase spaces. Lastly, while both works appreciate that maximum likelihood training will not allow learning of symmetries, we use approximate Bayesian model selection using VI whereas they rely on a meta-learning objective. Meta-learning can not readily be combined with an ODE solver (as that would require a 2nd order solver) without considerable memory cost, and requires additional hold-out validation data.

The works [van der Ouderaa and van der Wilk, 2021, Schwöbel et al., 2022, Nabarro et al., 2022] use a similar way of lower bounding learnable symmetries via the ELBO, but only consider simple affine invariances in image classification tasks, whereas we learn more complex symmetry groups in dynamical systems. A second difference is that we use Noether’s theorem to parameterize learnable symmetries in terms of their associated conserved quantities. Lastly, prior works that use VI for symmetry learning have so far only demonstrated this successfully using single-layer approaches (e.g. single layer / last-layer VI), whereas we manage to scale our work to deep neural networks (details in App. D.)

Connection between Noether's theorem and Occam's razor

While it may not be necessary to learn Hamiltonians symmetrized by conserved quantities via Occam’s razor, we argue that it is very compelling to do so. Learning symmetries naively is plagued by learning trivial symmetries, or only some of the symmetries repeatedly. Occam’s razor via Bayesian model selection provides a single ELBO objective that is simple to optimize. For example, we have shown that our method is able to discover a 25-dimensional symmetry group in the 5-harmonic oscillator. Also, we found all 7 conserved quantities of a 2-body system, generating a much larger group than the 3-dimensional group SE(2) we (naively) expected to find. This shows our method is very successful in avoiding these trivial solutions and instead finds all the symmetries in a system.

Q1. Baselines

To be clear, we train both the HNN baseline and HNN + symmetry adaptation using the same variational inference scheme (so are both `Bayesian' in that sense) to allow fair comparison. As such, the improvements in test performance can therefore be attributed to the proposed symmetry learning through Noether's razor. We will make sure this is very clear from the main text. All works on symmetry learning that we know of acknowledge the fact that learning symmetries can not reliably be done through maximum likelihood only, and therefore rely on some form of validation data, regularisation, or marginal likelihood estimates to perform model selection. Regardless, we will also run a ('non-Bayesian') maximum likelihood baselines of both HNN and HNN + symmetry adaptation in the final version.

Q2. Bayes instead of cross-validation

This is a great question. The main benefits of Bayes compared to CV is that the objective is 1) differentiable and we can thus optimize hyperparameters with gradients and 2) we have a single training procedure that does not require any retraining. With CV, we can evaluate only a finite number of settings for the conserved quantities that would have to be specified in advance. Further, it requires retraining the entire model for each possible configuration to be evaluated on hold-out validation data. We thank the reviewer for raising this, as it is an important motivation behind our method which we will make more clear from the text.

Q3. Wall clock time

We will report the memory and computational cost of our method and baseline in the final version.

Thank you for your feedback and help to improve the paper. We thank the reviewer for finding the paper clear, well-written, easy to follow, and highly relevant to both the AI + Science and geometric machine learning communities. Further, we appreciate the reviewers deemed the paper very principled and 'worthy of publication'.

Quadratic form

We indeed consider conserved quantities of the quadratic form, but feel this is not an overly strong assumption. It restricts us to the symmetry group having an affine (linear+translation) action on the phase space, but it does not limit the structure of the symmetry group itself. As far as we know, basically any work on in/equivariant deep learning uses groups with an affine action. The model can in principle still learn any dynamical system, also those with non-quadratic symmetries, given a sufficiently flexible network. Compared to regular HNN models, which do not have any learnable symmetries built-in, we deem our model as an improvement, even in those cases, as it can improve generalisation by picking up on some quadratic conserved quantities which may be present. Please see our overall rebuttal for more details.

High-dimensional data

In our experiments, we focus on n-harmonic oscillators in higher-dimensions and chaotic n-body systems, both directly on phase space. We did find that our method remains to work well when increasing dimensionality. We discover a 25-dimensional symmetry group for n=5, which for groups is very high-dimensional.

Our focus is demonstrating the principle of learning symmetrisation through conserved quantities. We therefore do not also consider learning models from images or only observing positional data. These tasks would imply a latent variable model which would significantly complexity the model description and distract from our primary research interest of demonstrating the Noether's razor effect. Regardless, we see no a-priori reason why Noether’s razor could not be applied to such extensions since they have been shown to work for regular HNNs in prior work. We consider this an interesting follow-up research direction, but out of scope for this paper.

Regularizing objective

This is an interesting suggestion. Instead of averaging the neural network along the group orbit, we indeed could also have considered regularising it in the same direction. In principle, this would also be a valid way of incorporating symmetry in the prior and is common for some other hyperparameters, such as the prior variance on weight magnitudes. We hypothesise that averaging generalises better as it smoothes the function globally, not only around train data, but refrain from making hard statements on this, since we did not try this yet. Although regularization could offer an interesting alternative to averaging to perform symmetrization, our use of averaging should not impact the main conclusions of our work, which was demonstrating Noether's razor. Nevertheless, we agree with the reviewer that it would be interesting to try this and intend to add it as an alternative baseline in the final version.

Typos will be fixed, thanks for spotting them.

Thank you for your feedback and help to improve the paper. We thank the reviewer for finding the paper 'intriguing' and 'conceptually very interesting' and that the empirical evaluation shows that the ideas can be promising

Q1. Non-quadratic conserved quantity

First, we believe that the quadratic conserved quantity is not an overly strong assumption. It restricts us to the symmetry group having an affine (linear+translation) action on the phase space, but it does not limit the structure of the symmetry group itself. As far as we know, basically any work on in/equivariant deep learning uses groups with an affine action. Please see our overall rebuttal for more details.

If one wishes, our method can be generalized to non-quadratic conserved quantities whose flows have no explicit solution! To optimise our objective we require differentiable samples from our flow. In case of a non-quadratic conserved quantity, we can use the reparameterization-trick by sampling a particular symmetry time and solving the flow through a differentiable ODE solver. Our code, in fact, also includes an implementation of this in JAX (using the diffrax library), which we will release upon acceptance. We chose not to describe this in-depth in the paper, because we relied on quadratic conserved quantities for our main results.

Q2. Direct optimization of conserved quantities

As described in Sec. 4, we can not directly learn conserved quantities with data fit loss because no symmetry will always make fitting easier (even though it can hurt generalization). This is acknowledged by virtually every symmetry learning paper, which is why most methods rely on validation data or other forms of model selection. Unlike cross-validation, which requires validation data and can only evaluate certain settings of hyperparameters through expensive model retraining, approximate Bayesian model selection allows for gradient-based optimization of hyperparameters.

Q3. Relevance of distribution over \tau

We only considered a zero-centred Gaussian distribution over \tau with learning the variance and found this to work well across experiments. Although we are free to choose any distributional family, we expect that unimodal distributions around the origin are often the most sensible and least difficult to optimize. Further, we can argue that a Gaussian subsumes a regular HNN, which would be equivalent to having a \delta peak (zero variance Gaussian) at the origin. Taken together, we think a Gaussian \tau is sufficient in most cases, especially since the variance is learned, and will add some of these considerations in the main text.

Q4. Can we use this to learn the specific conservation law?

Yes! We can directly inspect the learned conservation laws as well as the associated symmetries, which was not possible in prior work. We find that we learn the correct conservation laws for our n-harmonic oscillator and n-body system experiments. With the quadratic conserved quantity, we can compute the symmetry action with a matrix exponential. With a free-form conserved quantity, this would require solving an ODE. This is a clear advantage of our method compared to prior work, and we will make this more clear from the main text.

Description of our methodology

The reviewer points out that the current methodology is not always clear from the main text. We agree that we might have focused too much on the conceptual idea of Noether's razor and will move more practical details and contributions from the appendices to the main text (e.g. App. D). This should give a more explicit description of how the forward-pass and objective calculation are implemented in practice.

Comments

We agree that the difference between parameters \theta and symmetry parameters \eta can be made more explicit early on. In our work, we have a neural network for F and a quadratic conserved quantity for C, so \theta are all neural network parameters and \eta are the matrices A and vectors b. With learning symmetries, we mean learning \eta, which parameterise learnable conserved quantities and thus directly imply learnable symmetries by Noether's theorem. We hope this resolves the issue.

Typos will be fixed, thanks for spotting them.

Thank you for your feedback and help to improve the paper. We appreciate that the reviewer rated our paper with a "9: Very Strong Accept", found our approach a novel and significant contribution, well presented and recognized the new perspective on symmetry learning as well as convincing experimental results.

We agree with the authors that including computational analysis can further strengthen the paper and will report time measurements in the final version. In terms of the choice of hyperparameter K, the singular values described in Sec. 5.2 can be used to validate a chosen setting of K. If there are many (close to) zero singular values, K can be decreased, and if there are none such values, K should be increased. Setting K too high will not hurt performance, but add some computational overhead. In our experiments, we typically set K=10 or K=20.