Explicit Discovery of Nonlinear Symmetries from Dynamic Data

Thank you for your careful reading and valuable feedback! Below we will address each of your concerns point by point.

Theoretical Claims

Taking the discovery of linear symmetries in the Heat equation $u_t=u_{xx}$ as an example. We specify the function library as $\Theta=[t,x,u]^T\in R^{3\times1}$. Then, the infinitesimal group action takes the form

where $W=[W_1,W_2,W_3]\in R^{3\times 3}$. Our ultimate goal is to solve for W.

The algorithm first constructs $\Theta_n$. From Equations (7) and (8), we obtain $pr^{(2)}v=\phi^t\frac{\partial}{\partial u_t}+\phi^{xx}\frac{\partial}{\partial u_{xx}}+\dots$ (here, for simplicity, we omit irrelevant terms), where $\phi^t=-u_tD_t\Theta^T W_1-u_xD_t\Theta^T W_2+D_t\Theta^T W_3,\phi^{xx}=-(u_tD_{xx}\Theta^T+2u_{tx}D_x\Theta^T)W_1-(u_xD_{xx}\Theta^T+2u_{xx}D_x\Theta^T)W_2+D_{xx}\Theta^T W_3$. We rewrite it as

Therefore, we get $\Theta_2$ in Equation (9). Substituting $D_t\Theta=[1,0,u_t]^T,D_x\Theta=[0,1,u_x]^T,D_{xx}\Theta=[0,0,u_{xx}]^T,u_t=u_{xx}$, we have

The PDE can be expressed as $F=u_{xx}-u_t=0$, so its Jacobian matrix with respect to $u_t,u_{xx}$ is $J_F=[-1,1]$. Substituting into Equation (12), we have $[u_{xx},-2u_{tx},-2u_{tx}u_x,u_x,-2u_{xx},-2u_xu_{xx},-1,0,0]vec(W)=0$.

Comparing the coefficients, we obtain the basis of vec(W) as $[2,0,0,0,1,0,0,0,0]^T,[0,0,0,0,0,0,0,1,0]^T,[0,0,0,0,0,0,0,0,1]^T$. Substituting into $v=W\Theta\cdot\nabla$, we get the infinitesimal generators $v_1=2t\partial_t+x\partial_x,v_2=x\partial_u,v_3=u\partial_u$. These correspond to the linear generators $v_2,v_4,v_5$ shown in Table 6 (in Appendix I).

Relation To Broader Scientific Literature

Thank you for your addition! We will include the following content in the Related Work section.

A series of recent works have made outstanding contributions to extending diffusion models from Euclidean space to Lie groups [1,2,3]. Since our method can directly discover the infinitesimal generators of Lie groups from data, we look forward to future integration with these works, eliminating the need for prior knowledge of the group to guide their processes.

Questions For Authors

Both of these scenarios involve discovering linear symmetries from static systems. By setting the prolongation order n=0 and specifying the function library $\Theta(x)=x$ as linear terms, LieNLSD can cover them (see Appendix F for details). The core issue is how to define F in Algorithm 1 and estimate $J_F$, which we will discuss in detail below.

(1) For a two-dimensional manifold in $R^3$, its implicit equation can be written as $F(x)=0,x\in R^3$, but it may not have an explicit equation (e.g., S2). Therefore, we use the $k$-NN to estimate the Jacobian matrix $J_F$. Specifically, we fit a tangent plane $J_F(x-x_0)=0$ to the manifold using the k neighboring points around the given point $x_0$, and assign a Gaussian kernel weight $w_i=\exp(-\\|x_i-x_0\\|^2/2)$ to each point to reduce the influence of more distant points.

In practice, we set the number of neighboring points to k=5. The experimental results are shown below. We successfully discover the generators of SO(3).

Singular value: $[57.4,36.9,36.6,36.5,36.2,36.1,5.1,4.6,4.2]$

The Lie algebra basis (W) corresponding to the 3 smallest singular values:

(2) In this case, the group transformation acts on the coordinates $x\in R^2$, so we need to compute the derivative of the classifier with respect to the coordinates $D_xF$. We train a CNN on MNIST, which takes grayscale images I(x) as input. Automatic differentiation yields $D_IF$. We then estimate $D_xI$ using central differences on the image and apply the chain rule to obtain $D_xF=D_IF\cdot D_xI$. The experimental results are shown below, which demonstrate that we successfully discover the SO(2) symmetry.

Singular value: $[48.8,46.8,41.2,22.0]$

The Lie algebra basis (W) corresponding to the smallest singular value:

Reference

[1] arxiv.org/abs/2502.02513

[2] arxiv.org/abs/2405.16381

[3] arxiv.org/abs/2205.13699

Thank you for your careful reading and valuable feedback! Below we will address each of your concerns point by point.

Methods And Evaluation Criteria

(1) See point (1) in Methods And Evaluation Criteria section of Rebuttal to Reviewer 1vcx.

(2) According to the procedure in Section 2.4 of the textbook [1], given the expression of the PDE, we can mathematically derive its true infinitesimal generators:

Burgers: $\partial_x,\partial_t,\partial_u,x\partial_x+2t\partial_t,2t\partial_x-x\partial_u,4tx\partial_x+4t^2\partial_t-(x^2+2t)\partial_u,\alpha(x,t)e^{-u}\partial_u$, where $\alpha_t=\alpha_{xx}$

Wave: v_1=\partial_x,v_2=\partial_y,v_3=\partial_t,v_4=-y\partial_x+x\partial_y,v_5=t\partial_x+x\partial_t,v_6=t\partial_y+y\partial_t,v_7=x\partial_x+y\partial_y+t\partial_t,v_8=(x^2-y^2+t^2)\partial_x+2xy\partial_y+2xt\partial_t-xu\partial_u,$$v_9=2xy\partial_x+(y^2-x^2+t^2)\partial_y+2yt\partial_t-yu\partial_u,v_{10}=2xt\partial_x+2yt\partial_y+(x^2+y^2+t^2)\partial_t-tu\partial_u,v_{11}=u\partial_u,v_\alpha=\alpha(x,y,t)\partial_u, where $\alpha_{tt}=\alpha_{xx}+\alpha_{yy}$

Schrodinger: $\partial_x,\partial_y,\partial_t,-y\partial_x+x\partial_y,-v\partial_u+u\partial_v,-2t\partial_t-x\partial_x-y\partial_y+u\partial_u+v\partial_v$

Then, it is easy to verify that the infinitesimal generators discovered by LieNLSD, as shown in Table 3, are all correct.

(3) See point (2) in Methods And Evaluation Criteria section of Rebuttal to Reviewer pziR.

(4) See point (2) in Methods And Evaluation Criteria section of Rebuttal to Reviewer 1vcx.

Relation To Broader Scientific Literature

Thank you for your addition! We will include the following content in Related Work section:

Some recent works have introduced symmetry into PINNs [2,3,4,7,8,9] or performed data augmentation based on PDE symmetries [5,6], significantly improving the performance of neural PDE solvers. We expect to combine our data-driven symmetry discovery approach with these works in the future to explore methods for PDE solving without requiring prior knowledge of symmetries.

Questions For Authors

(1) True symmetries of the wave equation are provided in point (2) in Methods And Evaluation Criteria section of the Rebuttal (calculation process see [1]). As mentioned in Related Work section (Lines 128-130, Column 2), existing works on linear symmetry discovery use Lie algebra representations to characterize symmetries. Then, according to Eqn (2), they can only discover linear generators of the form $\phi(v)=d\rho_X(v)x\cdot\nabla$. For the wave equation, $v_1,v_2,v_3$ (translation symmetry) and $v_8,v_9,v_{10}$ (special conformal symmetry) do not conform to this form, meaning these symmetries cannot, in principle, be discovered by such methods.

(2) We will revise the sentence to "Then the infinitesimal group action can be expressed in the form of $W\Theta$. Our ultimate goal ...".

(3) We will revise the sentence to "... with a basis $\\{v_1,v_2,\dots,v_r\\}$, in the neighborhood of the identity element, we have ...".

(4) In Chapter 1.4 of the textbook [1], Eqn (1.46) defines the infinitesimal group action as $\psi(v)=\frac{d}{d\epsilon}|_{\epsilon=0}\Psi(\exp(\epsilon v),x)$. However, $\psi(v)$ is usually written in the form $-y\partial_x+x\partial_y=(-y,x)\cdot\nabla$ rather than (-y,x). In Chapter 1.3 of the textbook, the paragraph immediately following Eqn (1.4) also mentions that readers can regard $\partial_x$ as "place holders" or a special "basis". To avoid confusion, we add the partial differential operator $\nabla$ to the infinitesimal group action, as shown in Eqns (1) and (2) of our paper.

(5) We will make the revision. Thank you!

(6) For a 1st-order dynamic system, the PDE can be written as $F(x,u^{(n)})=f(u')-u_t=0$ (Line 238, column 1). By treating u' as the input and $u_t$ as the output, we can train a neural network to approximate f, thereby obtaining F.

(7) Due to page limitations, the experimental results of the KdV equation are provided in Table 6 of Appendix I rather than in the main text.

(8) As mentioned in the response to Question 6, we indirectly obtain F by training a neural network to fit f. The training details are provided in Appendix H.

(9) From methods, the original paper [6] only uses the 1D KdV equation as a worked example to discuss the implementation process of data augmentation. The trigonometric interpolation they employ becomes more complex in higher-dimensional cases. From experiments, they only test on 1D evolution equations as mentioned in Section 4.1 of their paper. Moreover, their code implementation is also limited to 1D cases.

Reference

[1] Olver, Peter J. Applications of Lie groups to differential equations.

[2] arxiv.org/abs/2310.17053

[3] www.mdpi.com/2673-9984/5/1/13

[4] arxiv.org/pdf/2410.02698

[5] arxiv.org/pdf/2212.04100v1

[6] proceedings.mlr.press/v162/brandstetter22a.html

[7] arxiv.org/abs/2206.09299

[8] arxiv.org/pdf/2502.00373

[9] arxiv.org/abs/2311.04293

Thank you for your careful reading and valuable feedback! Below we will address each of your concerns point by point.

Claims And Evidence

Note that in our paper, the symmetry is defined on $X\times U$ (see the "Symmetries of differential equations" paragraph in Section 2, lines 82-88, column 2). Therefore, the symmetries we find can be nonlinear with respect to both the coordinates $X$ and the field $U$.

However, from an implementation perspective, the types of PDE symmetries discovered by LieNLSD (ours) and LaLiGAN differ (perhaps this is what you are concerned about?). Taking $X=R^2,U=R$ as an example (e.g. $u(x,y)$ represents a planar image), the symmetries found by LieNLSD act pointwise on $X\times U=R^3$. Such symmetries are commonly referred to in the literature as "Lie point symmetries." On the other hand, the symmetries found by LaLiGAN are defined over the entire discretized field. Specifically, if $u$ is a field on a $100\times100$ grid, then LaLiGAN's symmetries act on $R^{100\times100}$ (see the Reaction-diffusion dataset in the original paper of LaLiGAN for details). For PDEs, the setting of Lie point symmetries (which act pointwise on both coordinates and the field) is more common, as the search space is significantly reduced compared to symmetries defined over the entire discretized field, and the physical interpretation is more intuitive.

We will add the above comparison of the two types of symmetries to the related work section. Thank you!

Methods And Evaluation Criteria

(1) Noisy samples

We add multiplicative noise $u'=u(1+\epsilon)$ to the variables in the training dataset to evaluate the robustness of LieNLSD against noise, where $\epsilon\sim N(0,\sigma^2)$. The quantitative comparison results (Grassmann distance) between LieNLSD and LieGAN for different noise levels $\sigma$ are as follows. Although the performance of LieNLSD declines as the noise level increases, its accuracy remains higher than that of LieGAN, which shows robustness to noise.

(2) Insufficient samples

To evaluate the error impact of estimating high-order derivatives using central differences on low-resolution grid points (insufficient samples), we downsample the discrete points in the training dataset with a sampling interval of $s$. For the low-resolution dataset, we employ higher-precision central differences to estimate the derivatives. For example, we replace the first-order derivative calculation $u_x=[u(x+h)-u(x-h)]/(2h)$ (with a truncation error of $O(h^2)$) with $u_x=[-u(x+2h)+8u(x+h)-8u(x-h)+u(x-2h)]/(12h)$ (with a truncation error of $O(h^4)$). The results of the ablation study (Grassmann distance) are presented below.

As the resolution decreases, LieNLSD still performs better than LieGAN. In fact, works on PDE symmetry discovery can hardly bypass the limitation that numerical derivatives on low-resolution data tend to have large errors. For example, when calculating the validity score, Ko et al. [1] also needed to compute numerical derivatives for the transformed data to evaluate whether it satisfies the PDE (see Section 4.2 of the original paper [1] for details). However, higher-precision difference methods can be introduced to mitigate the impact of low resolution.

Reference

[1] Ko et al. "Learning Infinitesimal Generators of Continuous Symmetries from Data."

Thank you for your careful reading and valuable feedback! Below we will address each of your concerns point by point.

Methods And Evaluation Criteria

(1) See the Methods And Evaluation Criteria section of the Rebuttal to Reviewer 1vcx.

(2) We conduct ablation experiments using a limited $\Theta$ (including only linear and constant terms) and a large $\Theta$ (additionally including sine and exponential terms) to evaluate the robustness of LieNLSD with respect to the choice of function library. The quantitative comparison results (Grassmann distance) with LieGAN are presented below.

For large $\Theta$, although we misspecify several terms, LieNLSD can still obtain accurate results within a large search space. For limited $\Theta$, LieNLSD may indeed miss some generators, but for all generators whose terms are fully included in $\Theta$, it can still accurately discover them (as shown in the table above, LieNLSD with $\Theta$ containing only linear and constant terms can correctly identify all linear symmetries).

Experimental Designs Or Analyses

Although the Grassmann distance is not applicable to the method of implicit symmetry discovery, we additionally compare the long rollout test NMSE of FNO with LPSDA based on LieNLSD and FNO with LPSDA based on Ko et al. [1]. For Ko et al., while they cannot obtain explicit expressions for the infinitesimal generators, feeding a given point into their trained MLP still yields the specific values of the infinitesimal generators at that point, thereby guiding data augmentation. Note that for PDEs, the types of symmetries found by LaLiGAN and our method differ (details can be found in the Claims and Evidence section of the Rebuttal to Reviewer 1vcx), making it impossible to establish a unified metric. The experimental results are presented below. Ko et al. can improve the accuracy of FNO, but not as significantly as our method. In addition to quantitative advantages, Ko et al. require the explicit expression of the PDE to compute the validity score (see Section 4.2 of their original paper), whereas our LieNLSD does not rely on this prior knowledge.

Other Comments Or Suggestions

$\Theta(x,u)\in\mathbb{R}^{r\times 1}$, then $\Theta(x,u)^\top\in\mathbb{R}^{1\times r}$. By repeating it $(p+q)$ times and arranging them diagonally, we obtain $diag[\Theta(x,u)^\top,\dots,\Theta(x,u)^\top]\in \mathbb{R}^{(p+q)\times((p+q) \cdot r)}$. We have double-checked that it is not a typo, but thank you for your reminder!

Questions For Authors

As mentioned in the Experimental Designs Or Analyses section of the Rebuttal, the accuracy of FNO with LPSDA can serve as one metric. In fact, the accuracy of many recent symmetry-informed neural PDE solvers (detailed in the Relation To Broader Scientific Literature section of the Rebuttal to Reviewer LaJV) can also be used to measure the quality of symmetry. Essentially, these use the performance on downstream tasks after applying symmetry as a metric.

Additionally, following the approach of LieGAN and LaLiGAN, a discriminator can be employed to quantify the distributional differences between the original and transformed data. These metrics do require some training overhead, but we believe this is nearly unavoidable in scenarios where the PDE is entirely unknown, as evaluation can only be conducted based on the dataset.

Reference

[1] Ko, Gyeonghoon, Hyunsu Kim, and Juho Lee. "Learning Infinitesimal Generators of Continuous Symmetries from Data." arXiv preprint arXiv:2410.21853 (2024).