Neural Manifold Operators for Learning the Evolution of Physical Dynamics

Dear reviewer, we very appreciate for your careful reading and kind help in reviewing our work. We are largely encouraged and have learned your suggestions carefully.

(1) Reply for the "central point is lacking enough illustrate".

At first, we recognized that this part is not our original mathematical theory and omit some contents. In order for the completeness of mathematics, we rewrite this part to illustrate the local neighborhood of these points by KNN algorithm and manifold learning methods.

(2) Reply for the "lacking enough illustrate for figure caption".

Thank you for pointing out these issues. We have revised the captions accordingly, adding more details that were originally present in the main text and appendix into the figure captions. For instance, in Figure 4, we have added more information about prediction length and input length.

(3) Reply for "grammar error in Method part".

We have revised the Method part. On the one hand, we have reduced overly verbal and hand-wavy expressions, and on the other hand, we have augmented the proof of operator equivalence.

(4) Is the intrinsic dimension differentiable? Is it computed in pre-training and then frozen?

The intrinsic dimension is calculated upon completion of the pre-training. Because after the pre-training, the parameters of encoder and decoder are frozen and the latent tensor after the encoder are enable to be used for calculating the intrinsic dimension.

(5)How do we choose "k" (Eqs 9-10) and what is the effect of "k" on the results?

"k" is a hyper-parameter of our manifold algorithm. In plain language, manifold method to calculate submanifold by KNN method and define a neighborhood, where points outside this neighborhood are considered to have infinite distance, for the purpose of employing the KNN algorithm for clustering. If k is excessively small, it might lead to the "short-circuiting" of submanifolds that it should not be connected. However, according to our experiments, "k" is not so sensitive to the result and it is accustomed to calculate by the square root of the total number of points.

(6) Eq (5) --> can you clearly mark what is trainable? $Q$, $P$ are frozen, does theta parameterizes both $\mathcal{L}$ and $\mathcal{K}$?

In Eq(5), $Q$, $P$ are frozen. $\mathcal{L}$ is non-trainable linear projection. In train process, $\mathcal{K}_{\text{id}}$ is trainable by prediction loss.

(7) Section 3.4. a notation suggestion --> K_id and R^id should probably be called K_m and R^m as this is an estimation of the intrinsic dimension but no guarantee to always converge to that.

Yes, it is. We have revised the paper and change the notation system. $\mathcal{K}_{\text{id}}$ is ideal situation. $\mathcal{K}_m$ is obtained by the optimal estimation.

(8) About ablation study.

Due to the intrinsic dimension of complex dynamics and real-world dynamics cannot be analytically calculated, we have got to experimentally prove that when we set the dimension is $m$, NMO get the better performance independent network implementation. In our ablation study, enough experiments have had this effect and I think it is not a coincidence. If it is statistical variations, it should not appear that all ablation experiments show $m$ is the optimal dimension.

Finally, thank you for your review for our paper. According to your suggestion, we have improved our paper furtherly.

We greatly appreciate your time spent for reading our paper and providing valuable criticism for our Method part. We have revised our notation system and added a description of all variables in the appendix part. We rewrite the notation of the operator mapping $G$ as

$G_{\theta}: R^{n} \times P(T) \rightarrow R^{n}\times P(T), \theta \in \Theta$

where $P(T)$ is the power set of $T$. In plain language, $P(T)$ contents all possible subsets of $T$. Based on the operator definition, we rewrite the whole mathematics prove. Furthermore, we provide the proof that the operator for submanifold representation is equivalent to the original high-dimensional underlying operator. Using the manifold method to do the intrinsic dimension representation for the complex physical dynamics and get the state-of-the-art performance in both accuracy and efficiency is our novelty. We supplemented the proof of operator equivalence, which is another theoretical novelty.

Hopefully, our revised version can address your concerns and make our idea accessible to all readers. Apart from the above responses, you can also read our response to other reviewers for more information.

We greatly appreciate your time spent for reading our paper and providing valuable review. According to your review, we have revised our paper.

(1) Reply for "Mathematical inconsistencies and overly informal presentation of the equations"

Thanks for your comments about the unclear mathematical symbols. We have to admit that our mathematical descriptions are not perfect. At first, in order to make the article easy to follow, some non-original mathematical theories were omitted. Based on your review, we have revised the notation system in the paper and added explanations for all mathematical notation in the appendix.

1. $G$ is the operator mapping from the physical variable $X$ at time $t$ as input to the variable at time $t+\varepsilon$ output while $\Theta$ is the finite-dimensional parameter space. The notation is followed by the previous related articles about neural operator[1,2]. Because our model design for sequence-to-sequence prediction, which means we learn the map from a time series(i.e. t:0-9) to a time series(i.e. t:10-100). We didn't have the autoregression strategy. So our paradigm are applied in autonomous and non-autonomous system. In order to be more clear, we rewrite the notation of the operator mapping $G$ as $$G_{\theta}: R^{n} \times P(T) \rightarrow R^{n}\times P(T), \theta \in \Theta$$

where $P(T)$ is the power set of $T$. In plain language, $P(T)$ contents all possible subsets of $T$.

2. Our paradigm is not \varepsilon\-ahead evolution. More details are discussed above.
3. The method of calculating the dimensionality of data tensor by maximum likelihood estimation[3] is not our original work, . Our novelty is that calculating the optimal representation of operators by the method. We will supplement the proof of the equivalence of the original operator and the operator with the submanifold representation. According to "no notion of randomness", we have mentioned that in our paper "The number of the observation points falling into the k-nearest ball $B$ can be approximated by a Poisson process" to introduce the random variable[3]. In our design, in plain language, $G$ are the mapping with objective paramter $\Theta$ in general form, $G_\theta$ denotes the operator mapping are parameterized by a neural network. We have introduced new notation $\theta^{\dagger} \in \Theta$ to describe the optimal parameters. As you said, we are trying to learn a map $G_{\theta}$ to get the time evolution, but it is not exactly equivalent to $\mathbf{F}$ in Eq. (2).
4. In order to improve the performance furtherly, we design a relatively complicated time evolution module, it is the reason why we introduce a mathematical description for operator decomposition in Eq. (11).

[1] Neural operator: Learning maps between function spaces. Journal of Machine Learning Research, 2022.

[2] Fourier neural operator for parametric partial differential equations. ICLR 2021.

[3] Maximum likelihood estimation of intrinsic dimension. Advances in neural information processing systems, 17, .

(2) About ETO and Koopman operator

Our theory follows the Green function integral theory (such as FNO). We have revised our Method part to add the mathematics illustration. It seems plausible to use Koopman theory to illustrate our paradigm becasue we use auto-encoder structure to reconstruct the physical variables, which can be regarded as learning the observation function and its inverse function. However, linear representation for the Koopman operator seems more intuitive but our ETO is nonlinear. If using the theory to illustrate, maybe we need to prove that it is invariant subspace of the Koopman operator after submanifold representation. I'm not sure if it's mathematically easy to prove, but we think it's a good idea for the further work.

(3) Reply for "Minor issues"

Thanks for pointing out some typos of our paper. We have revised our paper.

When $\mathcal{L}$ is not squared, $\mathcal{L}^{-1}$ denotes pseudoinverse.

$E$ is expectation.

Eq. (11) should be composition.

Finally, we appreciate your valuable review for our paper and helping us to improve our work.

Dear reviewer, we express our sincere gratitude for your review and we have learned your suggestions carefully.

(1) Reply for "It is not clear why the proposed method is a valid operator mapping between infinite dimensional spaces".

We follow the existed neural operator theory, which derive from Green function kernel integration [1,2]. This theory transforms the operator learning problem into the parameterization problem of kernel integration. Our method focuses on how to choose the appropriate dimension to represent the kernel integral problem by a theory-based method. Thanks for the suggestion, we have incorporated the relevant mathematical proofs into the revised version of our paper.

Through the Navier-Stokes equation with high Reynolds number, we prove that our model can learn operators accurately. In this scenario, our model can generalize different initial condition functions, which means our model can generalize in the Banach space.

(2) Reply for "the novelty of our intrinsic dimension algorithm".

The differences between intrinsic dimension calculation and the traditional KNN based dimensionality reduction algorithm can be illustrated by comparing the following formulas:

1. Our model (NMO).

   • The latent variables $$V = {V_1, \ldots, V_n}$$ are regarded as n independently and identically distributed vectors in the latent space $$R^{d_L}$$.
   • The potential variables are bounded on an m-dimensional Riemannian manifold M$ $ in R^{d_L}$$, where m$ $ is smaller than d_L$$.
   • Use probabilistic methods to estimate the density of distributions of latent variables on the subfluidic form and compute the local intrinsic dimension by maximum likelihood estimation.
   • Example formula: $$\log g(f(v))\mu(dv)$$ Integrates over a ball $$B(v_0, r)$$ centered at $$v_0$$ with radius $$r$$.

2. Traditional KNN algorithm.

   • The function $KNN(Vt, n neighbors, n jobs) is used to compute the nearest neighbor$.
   • Convert $$V_t$$ to a Numpy array and move to CPU.
   • Initialize a nearest neighbor object with the given neighbors and jobs.
   • Return the distance and index of the nearest neighbor.

We take into account the complex structure of the data on potential Riemannian manifolds when dealing with intrinsic dimensions, which is significantly different from traditional KNN methods based on distance computation. While traditional KNN algorithms focus on calculating distances between data points and finding nearest neighbors, our approach focuses more on understanding the intrinsic structure of data through probability distributions and Riemannian geometry. This approach provides a more effective means of dealing with high-dimensional and complex data, especially when considering that the data may exist in low-dimensional manifolds.

(3) Reply for "The Method part of our paper is verbal."

Thanks for your review. We've added more mathematical description to method includes proof of learning operator, equivalence between the operator represented by the submanifold and the original operator and optimal dimension.

In sum, we would like to take this opportunity to appreciate your timely help in improving the readability of our paper. We have carefully learned your comments and will improve our paper accordingly. Hopefully, our revised version can address your concerns and make our idea accessible to all readers. Apart from the above responses, you can also read our response to other reviewers for more information.

[1] Neural operator: Learning maps between function spaces. Journal of Machine Learning Research, 2022.

[2] Fourier neural operator for parametric partial differential equations. ICLR 2021.

(4) Reply for "Although I see none of the manifold learning methods used as a baseline. So the comparative analysis is poor."

Manifold learning methods are designed to reduce the dimensionality of data and cannot be directly applied to learn the physical system. Therefore, this is the reason why we do not choose it as the baseline model. Our baseline model has been carefully selected. For instance, our baseline model includes some representative models in time series prediction and state-of-the-art models in the neural operator, such as the Fourier Neural Operator, as well as the model with strong physical conductive bias named TF-Net. We provide a detailed illustration of baseline model selection in the Experiments section.

Dear reviewer, we very appreciate for your careful reading and kind help in reviewing our work. Knowing your efforts and time devoted to comprehensive checking our paper, we have learned your suggestions carefully.

(1) Why do we separate the autoencoder and the sub-manifold mapping?

Our paradigm mainly focus on computing the optimal dimensional representation of the underlying operator in the latent space, which requires that we need to train the encoder-decoder structure to learn the mapping function $\mathcal{P}$ and its inverse function $\mathcal{Q}$. This process is trainned by the self-supervised reconstruction strategy. Having trainned the encoder-decoder structure, we can compute the optimal dimension representations for various dynamics by the tensor in the latent space, which process is solving for the intrinsic dimension of the dynamics.

In order to avoid retraining the encoder-decoder, we introduce a non-trainable linear dimension projection $\mathcal{L}$ to map the original high-dimensional space to the intrinsic dimension. Therefore, we can train the time evolution opeartor module with intrinsic dimension $m$ separately, instead of retraining an encoder-decoder for mapping dimensions to the intrinsic dimension. Otherwise, retraining such an encoder-decoder would result in a wasteful consumption of training resources.

(2) Why do we set the high dimension mapping for our encoder?

The core of our intrinsic dimension (ID) algorithm is that computing the minimal dimensional representation for the underlying operator, aiming to prevent dimension redundancy. The process is referred to as describing the degrees of freedom of a physical system in physical dynamics. Employing the minimal degrees of freedom to describe the physical system contributes to the model to better learn the underlying operator[1]. However, before calculating the intrinsic dimensional representation of various physical systems in the latent space, we cannot ascertain the specific value of these minimal degrees of freedom. Therefore, we need to map the physical space to a dimension higher than the intrinsic dimension to render our paradigm feasible.

(3) Why we need to find $m$?

We need to determine the value of $m$ as the dimension for our time evolution operator module to learn the underlying operator representation. If we do not calculate $m$, we will lack a theory for setting the dimension of the evolution operator module. We will further revise the mathematical description of our paper to be more clearly. In our ablation study, we have demonstrated that $m$ is the optimal dimension experimentally.

Finally, we would like to thank you for your review.

[1] Automated discovery of fundamental variables hidden in experimental data. Nature Computational Science, 2(7), 433-442, 2022.