Physics-informed Dynamics Representation Learning for Parametric PDEs

Main concern

• Table 2 and Figure 5: I am a little bit surprised about how bad the L2 relative errors are for the baselines on certain benchmarks, especially on the training horizon. It makes it hard to believe that these are the state-of-the-art existing approaches. Could you specify the hyperparameter tuning strategies used for the different approaches? In particular, could you confirm that you have spent as much effort tuning the hyperparameters of the baselines and of your approach? There might also be other existing better-performing baselines to compare to in addition to the ones presented in the paper. Any additional information that could help understand why it is not surprising that the baselines are performing so badly would be welcome

Minor weaknesses or suggestions/questions:

• Figure 1 is not very clear. I think it needs to be rethought to propose a clearer description of the proposed approach
• Figure 2: I am not sure I understand exactly the green and red markings. Is it supposed to show that training is done at the integer times {0,1,2,3,...} but that you also predicted at the in-between value 0.5? If that's the case, I think this could be presented differently in a cleaner way
• Section 3.4.1, 2nd paragraph: I think this needs to be investigated a little bit more. I don't think the results in Figure 2 are sufficient to make these strong conclusions. That is only for one example of dynamical system where the evolution is very slow over the prediction time interval. I am not convinced this applies more systematically, that the model learns excessively fluctuating latent dynamics, and would be curious to see the same experiments carried for faster-varying dynamics. I think this connects as well to the comment made in Appendix D that the smoothing regularization used for stability training can potentially penalize high-frequency physics information. While the latent dynamics smoothing helps for the slower-varying systems, it would be interesting to discuss it in the context of faster-varying systems.
• Section 3.4.2, 1st paragraph: "by analyzing the evolution of latent embeddings over extended time horizons, we trace this issue to the phenomenon of latent embedding drift". Can you provide more details here about the latent embedding drift, and in particular about the analysis you carried that allowed you to trace the issue back to that phenomenon?
• Figure 4 is not clear. I think the paper needs a better figure to explain the latent space smoothing and the latent space alignment.
• Section 4.2, Generalizing across initial conditions: where are these percentages coming from? I don't see any such results in the tables and figures. Also, it should be made clearer what the percentages mean: what does 63% means, does it mean a l2 relative error going down from 70% to 7% for instance?
• Appendix C.3: The short comment about the computational efficiency should be made in the main text (while table 11 can be kept in the appendix). The results are only provided for the CE1 setting. Could you confirm that similar discrepancies in computational times were observed with the other problems and strategies tested? How does it scale up compared to the other strategies?
• Appendix D about limitations and future work should be in the main conclusion, not in the appendix.

Minor concern:

1. Some shorthands should be explained before use. E.g., in Fig. 1, you may need to explain DEC, ENC, and DYN.

2. Some typos exist. For example, in Line 175, “n this section”. Line 259, “This provide us”. Please do a complete proofreading.

3. Why do you conclude that Fig. 2a is a trivial solution caused by large training steps?

4. What is the performance after the author applies $R_A$ in Fig. 3?

Major concern:

1. Some notations in Fig. 1 are not explained in the previous pages. So, it’s hard to understand Fig. 1 when reading the caption and descriptions in the Introduction.

2. The key employed technologies, e.g., auto-decoding and EDM, are not well motivated and explained. Why should we build a latent space? Why should we use auto-decoding?

3. It seems that smoothing regularization will hurt the expressivity of the model. I suggest that the author conducts a sensitivity analysis with respect to regularization penalty terms and evaluates the performance.

4. As the author claims that including physics-informed loss is important with limited numbers, the author should give strong numerical support. For example, in Table 3, PIDO and DINO are compared. However, it’s not clear what’s the performance of PIDO under different percentages of datasets. Although there is a sensitivity analysis in Fig. 6 in the Appendix, I don’t see the result of DINO.

5. In Section 3.3, the author motivates physics-informed loss by saying that the data-driven method requires prohibitive data, which implies that the author’s design can handle data-insufficient cases. However, in Section 3.4.2, the proposed regularization aligns the latent dynamics with the dynamics obtained by data. If data is scarce, the effectiveness of the regularization is doubtful. The author should give some analysis or numerical support.

6. There is no discussion about the weakness of the model and future work.
   

Some description of the training procedure is unclear or misleading.

• How to train encoder and decoder is not clearly explained. The decoder is trained through the equation (13) and this formulation includes $c_{0}^{i}$ which is obtained through $\mathcal{E}$. But this encoder is obtained through equation (6), which assumes one to have a learned decoder. It would be very helpful to provide a step-by-step description of the training procedure, clarifying how the encoder and decoder are initialized and trained iteratively.
• Arguments of the loss functions such as (9) and (10) include both trainable and fixed parameters, which makes it a bit difficult to understand the overall training procedure. Maybe it would be helpful to provide a clear distinction between trainable and fixed parameters in the loss functions, perhaps using different notation or explicitly stating which parameters are optimized in each step of the training process.

Some statistical parameters are missing from the results. While the metric used for experiments is clearly explained, the numbers reported in tables and figures do not have error bars or standard deviations, which makes the results sound less reliable. I encourage the authors to include error bars or standard deviations for the reported results in tables and figures. This would provide a more complete picture of the method's performance and variability.

Minor:

• L.175 ‘n this section'
• Numbers in some tables are not in bold fond while they are the best. For example Table 7 and 8.

The paper is positioned at the intersection of EDMs, NOs and PINNs. However, the targeted task seems to be solving parametric PDEs with a data-free approach. The paper should be compared to meta-learning frameworks such as [1, 2]. Instead, the authors both compare it to a data-driven method (Dino) and model-based (PI-DeepONet, PINODE, MAD); each method seems instead to tackle a different task here.

For auto-decoding, if my understanding is correct, only one gradient descent step is needed in order to estimate new latents $c_0^i$. I am a bit concerned with the approximation made. How many steps do you actually need to find the optimal latent? How do you explain that such approximation still lead to very good perf? It could be nice to add some ablation studies comparing the impact of the number of gradient steps for learning parametric PDEs.

The visualisation in Figure 2 is not very well described. What do (green) training and (red) test steps refer to here for a same trajectory? Each NS frame corresponds to a different time-step? It should be made clearer at least in the figure and its title to better justify the necessity of adding a regularization term.

Cho et al., Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks, NeurIPS, 2023.

Belbute-Peres et al, HyperPINN: Learning parameterized differential equations with physics-informed hypernetworks, NeurIPS, 2021.

• The datasets used to demonstrate the performance of the model are too simple. More complex examples, e.g., 3D reaction-diffusion system, 3D NS dataset, should be tested. In addition, the 2D NS example in the paper is rather simple, where the flow patterns are simple since the domain $[0, 1]^2$ with periodic BC is too small. The authors should consider a larger domain, e.g., $[0, 2\pi]^2$ (please see the JAX-CFD paper).
• The comparison with baseline models is weak, where the considered baselines (e.g., PI-DeepONet) are outdated. There exist many physics-informed neural operators in the literature that need to be compared.
• The paper claims in Table 1 that the proposed model is flexible on a given grid choice. However, the example didn’t show any evidence. Testing the generalization of the model over different mesh grids (including unstructured) should be conducted.
• The model should also be tested to solve PDEs with zero training data, when the complete information of PDE, IC, and BC is given.

1、The number of experiments is not sufficient, and the selected equation types are also not sufficient. 2、Some spelling in the article needs to be carefully checked, such as in line 175

1. I think the paper could use a better explanation of how network and training parameters were chosen, especially when comparing different methods:

• Was any hyperparameter optimization done for any of the methods discussed in the paper? If not, it seems difficult to compare their performance.
• When comparing to data-driven approaches, my understanding is that those methods use a different training set (one containing actual full solutions of the PDE as opposed to physics-informed training used by PIDO). How do the authors then ensure a fair comparison between the two methods?

2. Figure 1 is hard to understand, I was only able to decipher it after reading the paper.

• I would move the "DYN" box over the unrolling arrow that goes from $c_0$ to the entire sequence $c_{0:N}$.
• I would split the unrolling arrow into two, one for coefficients $\alpha$ and one for $\alpha'$.
• The decoding of the solution curves should yield different results for $\alpha$ and for $\alpha'$.
• It's confusing that in the figure, for the initial conditions the decoder takes only the latent state $c_0$, while later it also takes a position $x$. Perhaps it can be indicated somehow that in the former case, we're evaluating the decoder at multiple positions.

3. Limitations are currently found in appendix D. Ideally these would be discussed in the paper itself.
4. Minor:

• in Fig. 5, the x-axis should be labeled.
• on line 175: missing "I": "[I]n this section"