Thank you for your feedback.

Q1. Necessity of phrase in line 174. Yes. For all $f \in \mathcal{D}$ and for all $s \in [-\sigma, \sigma]$.

Q2. Clarification in Section 4.1. To be more rigorous, our formulation leaves us a set of constraints $S(\vartheta^*_s,f) <C$ for all $f \in \mathcal{D}, s \in [-\sigma,\sigma]$, which is intractable to solve. Instead, with a mild assumption that non-symmetries have comparably higher values of the validity score than the true symmetries, we turn it into the proposed optimization problem. We will make this clearer in the revised version of our paper.

Q3. Typo in line 217. Yes. This is a typo. it should be $(\vartheta_\mathcal{X}(f))(x) = f(\vartheta_\mathcal{X}^{-1}(x))$. We apologize for the critical typo and thank you for pointing it out.

Q4. The number of symmetries in line 226. In our framework, we assume knowledge on a rough upper bound $N_{\text{sym}}$ on the true number $N_{\text{sym}}^*$. The stop-gradient operation in the orthonormality loss turns the optimization problem into a sequence of optimization problems with hierarchical constraints; the first slot is optimized without being affected by other slots, the second slot is only affected by the first slot, and so on. Consequently, the true symmetries are learned in the first $N_{\text{sym}}^*$ slots in the MLP. We believe having a rough estimate on an upper bound of the number of symmetries is reasonable.

Q6. Orthogonality assumption in line 230. A set of symmetry generators form a vector space. For example, a sum of the unit $x_1$-axis translation generator and the unit $x_2$-axis translation generator gives a translation in the direction of the $45^\cdot$ vector. Symmetry generators listed in Table 1 and Table 4 are in fact a basis of the space of symmetry generators. Hence, if we set an inner product in the space of vector fields, there is always an orthonormal basis of symmetry generators.

Q9. Further details about the most invariant symmetries in line 280. As discussed in the answer to Q4, the stop-gradient operation gives hierarchical constraints in the optimization problem, causing it to learn the earlier slots first and then the latter slots. Since the translations are found in the first and second slots, we conclude that they are “the most” invariant ones. This means that the outputs of the neural network are altered by the smallest magnitude along those symmetries.

Q11. True symmetries in cKdV in line 322. As we elaborated in the answer to Q6, symmetry generators form a vector space. In the cKdV case, the exact and approximate symmetries form a vector space of dimension 3. We identified three orthonormal basis components of that vector space. Although these components do not match the closed-form expressions exactly, they span the same vector space as the closed-form symmetries. Therefore, we conclude that the three symmetries have been correctly learned.

Q12. Evidence for learning actual symmetries. Again, symmetry generators form a vector space, so learning linear combinations can be as well regarded as learning the true symmetries. Moreover, we compare the distributions of validity scores and report them, both with images (top) and PDEs (bottom) in the tables below, where the non-symmetries are modeled by neural networks with random initialization. The values of quantile 0.95 shows that the true symmetries indeed have lower validity scores than non-symmetries.

Q13. Necessity of equation (8). The equation (8) is not explicitly used in implementation, but it states how transformations on the spaces $\mathcal{X}$ and $\mathcal{Y}$ induce a transformation of the function $f:\mathcal{X} \rightarrow \mathcal{Y}$. For example, an image can be viewed as a function $f:\mathbb{R}^2 \rightarrow \mathbb{R}^3$, which is then discretized as $\\{ f(x\_i) \\}$ on pixels $\mathcal{X}\_{\text{grid}} = \\{ x_i \\}$. Transformation of pixels $x\_i \mapsto \vartheta_\mathcal{X}(x\_i)$ results in a transformed function $\tilde{f}:\vartheta\_{\mathcal{X}}(x\_i) \mapsto f(x\_i)$, which corresponds to the function described in Equation (8).

Thank you for your feedback.

Weakness 1, 2. Reliance on the validity score. We see the requirement of a validity score as a trade-off for not requiring the predefined set of symmetry generators, and searching for symmetries across the entire class of continuous transformations, which increases the degree of freedom of search space by an infinite amount. In general, a validity score is a quantification of some criterion that transformed data should meet. If one aims to find symmetries of a given dataset, they should first question “what do symmetries of the dataset truly mean”, and then should come up with a criterion that the symmetries should meet. Our validation score is a quantification of that criterion.

For PDEs, the proposed validity score comes out quite naturally. For image data, while it might seem ad-hoc at a first glance, we argue that the proposed validity score is still based on the principle shared with PDE case – the invariance measure of the targets we wish to learn (PDE solutions for PDE data and features extracted from the encoder for image data). That is, the generic principle behind the validity score is to measure invariance of a target function with respect to transformations. For PDEs, we use the L1 error to measure this invariance. For images, we adopt the cosine similarity, which is a popular metric for measuring feature distances. We also note that other methods learning symmetries from images also have similar counterparts, such as the discriminator in LieGAN [2] and the classifier in Augerino [1] and LieGG [3].

Weakness 3. Range of the parametrization. Our parametrization can model any continuous (or more rigorously, differentiable) groups acting on Euclidean space, given that neural networks, according to the universal approximation theorem, can approximate any continuous vector fields. The only additional inductive bias imposed on our model is Lipschitz continuity of the vector field to be learned. Below is a proof sketch of this argument. Note that we consider connected Lie groups, so discrete groups such as reflections are not considered.

Let’s say a connected Lie group $G$ acts on a Euclidean domain $\mathbb{R}^n$ as $g:x \in \mathbb{R}^n \mapsto g\cdot x$, with corresponding Lie algebra $\mathfrak{g}$ and the exponential map $\exp:\mathfrak{g} \rightarrow G$. By the properties of connected Lie groups, any group element $g \in G$ can be expressed as $g = \exp(\epsilon_1\alpha_1) \cdots \exp(\epsilon_k\alpha_k)$ for $\alpha_1,\cdots,\alpha_k \in \mathfrak{g}$ and $\epsilon_1,\cdots,\epsilon_k \in \mathbb{R}$. Each component $\exp(\epsilon_i\alpha_i)$ for $i=1,\cdots,k$ can be seen as an action of a one-parameter group defined by $(t,x) \mapsto \exp(t\alpha_i)\cdot x$, with the infinitesimal generator given by $V_i(x) = \frac{d}{dt}|_{t=0} (\exp(t\alpha_i)\cdot x)$.

Weakness 4. Discrete groups. Unfortunately, our method cannot learn finite groups. As in the image task, if there exists a known finite group symmetry, it can be incorporated as augmentation.

Related work/Comparisons and Experiments 1, 2. Baselines. Please refer to paragraph “Comparison with baselines” in the general comment.

Experiment 3. Augmentation performances. Since we parametrize the symmetry generators by MLPs, they may exhibit numerical instability. So the closed-form symmetries are expected to perform slightly better.

Question 1. importance of the weight function. As described in Appendix A.1., the choice of weight function is crucial in the image task since each pixel has a different level of importance when put in a learned neural network. For example, if we use a constant weight $\omega \equiv 1$, then we may end up learning transformations that only move the pixels near the boundaries by a large amount. We compute a pixel sensitivity function using jacobian-vector products and use it as a weight function.

We conducted additional experiments ablating different weight functions. We tested two handcrafted weight functions: the “plain weight” defined by $\omega(x) = 1$ for all $x\in [-1,1]^2$, and the “step weight” defined by $\omega(x) = 5$ if $||x||<0.25$, $\omega(x) = 4$ if $0.25<||x||<0.5$, and so on, decreasing in steps until $\omega(x) = 1$ if $||x||>1$. The weight functions and the results are depicted in Figure 3 and Figure 4 in the general comment. Indeed, the correct affine symmetries are not found with these weight functions.

Question 2. Role of stop-gradient operation. In our framework, we assume knowledge on a rough upper bound $N_{\text{sym}}$ on the true number $N_{\text{sym}}^*$. The stop-gradient operation in the orthonormality loss turns the optimization problem into a sequence of optimization problems with hierarchical constraints; the first slot is optimized without being affected by other slots, the second slot is only affected by the first slot, and so on. Consequently, the true symmetries are learned in the first $N_{\text{sym}}^*$ slots in the MLP. Stop-gradient operation reduces dependence of our algorithm on the hyperparameter $N_{\text{sym}}$, and without the stop-gradient operation, we would learn mixtures of true symmetries and non-symmetries when $N_{\text{sym}}$ is larger than $N_{\text{sym}}^*$.

Question 3. One-param group in the negative direction. Solutions of ODEs $\frac{d}{ds}\vartheta_s^V(x) = V(x)$ and $\frac{d}{ds}\vartheta_s^{-V}(x) = -V(x)$ with initial conditions $\vartheta_0^V(x) = \vartheta_0^{-V}(x) = x$ are related by $\vartheta_s^{V} \equiv \vartheta_{-s}^{-V}$ due to uniqueness of ODE solutions.

Presentation/Exposition. Thank you for your feedback. We will incorporate your suggestions in the revised version of our paper.

Thank you for your comment.

• Q1. We indeed acknowledge that choosing the validity score is critical and not direct. However, we clarify our stance: "symmetry" is not a fixed mathematical concept but a task-dependent jargon. For example, isometries of a metric space are transformations that preserve the metric, while symmetries of a physical system preserve the Hamiltonian or governing equations. Image symmetries are typically not rigorously defined but refer to transformations that maintain invariance to human perception; we replace "human eyes" with "neural network" in this context.

• Q2. The two experiments we have done are exactly that. We replaced the neural ODE with an MLP that does not involve ODE integration, which now requires taking Lie derivatives (i.e. derivative along the vector field modeled by an MLP) to the validity score. We compared these setups using both image data and a toy experiment searching for isometries in Minkowski space. While using Lie derivatives (i.e., a plain MLP) fails with the image task, our method succeeds in both scenarios.

Thank you for your feedback.

Question 1. Validity score in equivariant task. We take a two-step approach: first, we learn symmetries based on validity scores, and second, we use these learned symmetries as augmentation and solve machine learning tasks. The validity score is designed to measure invariance of a certain function – the neural network in the image task, and the PDE residual values in the PDE task. We do not take equivariant tasks into account in this paper. However, if one were to design a validity score using equivariant neural networks, it could be achieved by modeling the transformation in both the input and output spaces and establishing an invariance criterion that corresponds to that equivariance.

Question 2. Cosine similarity and instability. Cosine similarity values lie in $[0,1]$, and if arccos is applied, the angle values lie in $[0,\pi/2]$. The arccos function has an infinite gradient at 1, so the cosine similarity values must be clipped to a maximum $1-\epsilon$ for some small $\epsilon$. Except for this, we did not experience any instabilities coming from cosine similarity.

Question 3. Validity score and augmentation. The validity score is only used when learning symmetries. Once the symmetries are found, we use the learned symmetries to augment the dataset for the target task – image classification task in this case. Table 2 shows comparison of augmentation performance with other commonly used augmentation methods.

Question 4. Scale of losses. Sorry, but we didn’t fully understand your question. We will try to answer based on our understanding, but if you find it insufficient, please let us know.. The phrase “normalized inner product” in the paper refers to the following process: two vectors $v_1,v_2$ are normalized as $\hat{v_i} = \frac{v_i}{||v_i||}$ for $i=1,2$, and then the inner product $\langle \hat{v_1},\hat{v_2}\rangle$ is computed. Then we put arccos to make it lie in $[0,\pi/2]$. Similar process is applied for all terms, to make sure all losses are on a similar scale.

Clarifications 2, 3. Meaning of validity. We say transformed data are valid for the given task when they are beneficial to the learning task. For example, in an image classification task, images rotated by small angles may help the task, but transformed images that are distorted too much may harm training. To approximate this validity, we design a function known as the validity score within our framework.

Clarification 4. Interpolation method. We use bilinear interpolation for images, and Whittaker-Shannon interpolation for PDEs. Due to page limits, discussion on that is in Appendix A. and Appendix E.2., including discussion on why bilinear interpolation does not work for PDE data but Whittaker-Shannon works.

Limitations 1. Baselines. Please refer to paragraph “Comparison with baselines” in the general comment.

Limitations 2. Computational costs. We use a fairly small MLP, with around 150k parameters. Direct comparison of computational complexity with other methods is not feasible since all symmetry learning approaches have varying assumptions and goals. For example, L-conv [9] learns rotation by comparing rotated 7x7 random pixel images with the original ones.

We provide some analysis on computational costs of our algorithm. The major computational burden of our algorithm arises from the ODE integration. Since we use the adjoint method for backpropagating [6], although the time complexity is $O(N_{\text{step}})$ where $N_{\text{step}}$ is the number of ODE steps, the memory requirement is $O(1)$. The choice of validity score also affects the complexity. However, the effect of dimension $n$ of the space is negligible, since it would only require changing the input and output dimension of the MLP by $\mathbb{R}^n \mapsto \mathbb{R}^n$.

[4] Dehmamy et al., “Automatic symmetry discovery with lie algebra convolutional network.” NeurIPS 2021.

[6] Chen et al., “Neural ordinary differential equations.” NeurIPS 2018.