PIORF: Physics-Informed Ollivier-Ricci Flow for Long–Range Interactions in Mesh Graph Neural Networks

Q1. Whether over-squashing exist in or significantly affect fluid dynamic learning?

Edges with high negative curvature[1] and high degree nodes[2] cause graph bottlenecks and over-squshing. The mesh graph also contains negative curvature, which can potentially lead to over-squashing. The change in graph topology before and after applying PIORF shows that high negative curvature is removed and the overall ratio of negative curvature is reduced. Furthermore, existing rewiring methods do not remove this negative curvature, indicating that they are not suitable for mesh graphs. The results of graph topology changes are added to Appendix G.1

In fluid dynamics, which primarily uses triangular meshes, the regions excluding those near the boundary conditions are composed of hexagonal meshes. However, in boundary condition areas such as walls, nodes with high degrees and edges with negative curvature are created, which can lead to potential bottlenecks. In addition, since the nodes around these boundary conditions are areas where the fluid flow changes, efficient message passing to these nodes is crucial, as they are highly important from both a learning and domain knowledge perspective.

[1] Jake Topping, et al. "Understanding over-squashing and bottlenecks on graphs via curvature". ICLR, 2022.
[2] Andreea Deac, et al. "Expander graph propagation". ICML, 2022.

Q2. How does PIORF alleviate over-squashing? more insights into model designs.

We design the model considering the following points.

• Learning dynamics models generate the accumulated error through iterative predictions. Existing methods are ineffective because only a single graph is constructed within each trajectory.
• The area around the boundary conditions can cause potential over-squashing.

In fluid dynamics, the gradient of velocity is an important physical quantity and is equally crucial in the learning process. We empirically find that the gradient of the velocity is related to the accumulated error (Please refer to Appdendix G.3). Therefore, by considering the physical context and adding edges with boundary condition nodes, over-squashing can be alleviated and even long-range interaction effects can be considered.

Q3. Does the performance improvement result from that PIORF alleviate over-squashing

In the added discussion (Appdendix G), we explain the alleviation of over-squashing through changes in graph topology and total resistance after applying PIORF. The mesh graph suggests that a rewiring method which considers physical context is more effective in several models, such as MGN, BSMS, GT, and HMT.

We appreciate Reviewer SBAf for the positive feedback and insightful comment, highlighting our strengths in

• The paper first studies the over-squashing problem in fluid simulation.
• The proposed methods consider both the graph topology and physical quantity

Below, we carefully address your concerns. We uploaded the revised paper, where changes are highlighted in red.

W1. Most graph rewriting methods study the classification problem in graphs like citation networks and social networks. The significance of over-squashing in fluid simulation is underexplored. Whether and how does it affect learning performance?. One observation that raises such a question is that in Table 1, existing graph rewriting methods enhance errors in most cases, indicating that over-squashing might not be the key issue in learning fluid dynamics

In the field of dynamics learning simulations, such as MGN, the model iteratively predicts the next step. Since a single trajectory consists of multiple steps, existing baseline methods, which form a single graph for the entire trajectory, are not suitable for this field. In contrast, in PIORF, physical quantities change at each step within a single trajectory, resulting in a different graph being constructed each time.

W2. For the method, connecting the nodes with the highest velocity gradient is straightforward. How does the author avoid connecting nodes with low influence but high-velocity gradients?

In fluid mechanics, governing equations such as Navier Stokes are helpful in predicting fluid flow because they depend on gradients of velocity[1]. Therefore, high-velocity gradients are an important element in dynamics and have an equally important influence on learning. We confirmed that the accumulated error is related to the velocity gradient, and the result of discussion is added to Appendix G.3.

[1] Charles Meneveau. "Lagrangian dynamics and models of the velocity gradient tensor in turbulent flows." Annual Review of Fluid Mechanics, 43(1):219–245, 2011.

W3. What is the ratio of negative ORC? Do you consider it to determine the pooling ratio?

The ratio of negative ORC (<0) for CylinderFlow and Airfoil are 21.88% and 14.71%, respectively. We do not consider the ratio of negative ORC. The pooling ratio is the only hyperparameter of PIORF and is adjusted to find the optimal value. The ORC distribution for each dataset is added to Appendix G.1.

W4. The authors do not provide time complexity.

The complexity of PIORF is $\mathcal{O}(|\mathcal{E}|d^3_{\mathrm{max}})$, where $|\mathcal{E}|$ is the number of edges and $d_\mathrm{max}$ is the maximal degree. The complexity is added to Appendix A and highlighted in red.

W5. From Table 4, Random and Only Removal also significantly reduces model errors. Do they alleviate the over-squashing problem?

In the case of "Random", this ablation study evaluates the influence of physical context. While shortcut edges are added dynamically at each step, which helps alleviate over-squashing from a topological perspective, the results suggest that physical phenomena must also be considered, as evidenced by the comparison with PIORF.

In the case of "Remove Only", edges with high positive curvature are removed, which does not indicate over-squashing problems. Additionally, except for the velocity RMSE in CylinderFlow, all other errors increase compared to MGN. The experimental conditions for "Only Removal" are missing, so they are added and marked in red.

Dear Reviewer SBAF,

We appreciate your comments on helping us improve our paper in many aspects. Since we have only a few days until the discussion period ends, we wanted to remind you of our responses to your valuable feedback.
We hope that we have clarified your concerns and questions. If we have satisfactorily addressed your concerns and questions, we kindly ask you to reconsider your score or let us know if you have any follow-up questions.

Best,

Authors

Dear Reviewer SBAf,

We are writing to follow up on our recent responses and remind you that we have just a few days remaining until the discussion period concludes on December 2nd.

We would like to restate that we have made comprehensive efforts in our responses to address your remaining concerns. We believe our responses address your concerns regarding our method and its significance in fluid mechanics applications.

If you have had the opportunity to review these responses, we greatly appreciate your consideration in potentially revising your score. We remain open to further discussion if you have any remaining questions.

Thank you for your ongoing engagement and considerate review process.

Best regards,

Authors

We appreciate Reviewer isvf for the positive feedback and insightful comment, highlighting our strengths in

• The paper is well written and the proposed approach is easy to follow
• The motivation is very clear with an elegant solution based on ORC

Below, we carefully address your concerns. We uploaded the revised paper, where changes are highlighted in red.

W1. The proposed approach is mainly developed upon mesh-based representations. Adding more discussions to whether this could be generalized to ubiquitous graph network simulators (e.g., with particle representations or rigid bodies) would enhance the contribution.

Thanks for the nice suggestion. Although we have looked into only the fluid dynamics problems in this paper, as pointed out by the reviewer, we expect our approach can be further extended into other types of physical processes. As shown in MGN paper, there are diverse sets of physical processes that can be simulated over mesh-graph-net (MGN) framework (including rigid/soft-body dyanmics) and these simulations could be benefited by the proposed approach in terms of improved computational efficiency.

W2. Some metrics are not clear to readers, e.g., Pressure, Density. It would be helpful to provide some explanations on the actual physical interpretation of these quantities, in the context of the considered simulation environments.

The Navies Stokes equation, which is the governing equation in fluid mechanics, depends on physical quantities such as velocity, pressure, and density. The velocity refers to the speed at which a fluid moves at a specific point in space. The pressure is the force exerted by a fluid per unit area on the surfaces. The density is a measure of how much mass is contained within a given volume of the substance. Additional descriptions of physical quantities are added in Appendix B.

W3. All datasets seem to be artificial and adding more experiments (if applicable) on some real-world datasets would significantly enhance the paper.

This is a good suggestion, however, Unfortunately, we do not have access to real-world datasets. In industry, such datasets are typically not made public due to security concerns. We also want to evaluate how effective PIORF will be on real-world datasets. However, we attempted to include well-established dataset into our benchmark problem (for example, large-scale benchmark dataset in EAGLE, which consists of 270 GB of Navier--Stokes equations solution snapshots).

Q1. Is the approach also applicable to particle-based simulations?

We expect that PIORF can be applied to particle-based simulations, although some technical or implementational modifications may be required. In particle-based simulations, the distance between nodes is calculated, and edges are created if the nodes are within a predefined radius (a hyperparameter). Therefore, PIORF can be applied considering curvatures and physical contexts.

Q2. Could the approach be combined with other physical inductive biases like equivariance? It would be interesting to have some of these discussions in the paper or as related literature.

Thanks for the nice suggestion. The advantage of rewiring is that it can be implemented by constructing a new graph through prior data processing without modifying the model itself. Therefore, combining the EGNN [1] model with PIORF appears feasible and presents an intriguing direction for future research. The future research are added to Section 7.

[1] Satorras et al. "E(n)-equivariant Graph Neural Networks." ICML, 2021.

Q3. Could the approach be applied together with constraint-aware graph simulators like C-GNS?

Our model basically uses the core framework of GNS[1] and MGN[2]. C-GNS[3] is used in combination with these frameworks. Since PIORF can be applied without changing the model, it appears that it can be applied together.

Since GNS, the core framework used in C-GNS, is the same as the framework used in MGN, it seems feasible to apply PIORF alongside them without modifying the model. However, it may be necessary to define a constraint function that is specifically suited for fluid dynamics

[1] Alvaro Sanchez-Gonzalez, et al. "Learning to simulate complex physics with graph networks." ICML, 2020
[2] Tobias Pfaff, et al. "Learning Mesh-Based Simulation with Graph Networks." ICLR, 2020.
[3] Yulia Rubanova, et al. "Constraint-based graph network simulator". ICML, 2022

Q4. Are there any real-world dataset on which the model can be evaluated? Practitioners would be more exciting to see how the method performs in actual real-world scenarios.

Please refer to W3.

Dear Reviewer isvf,

We appreciate your positive ratings and constructive feedback, which have helped us strengthen our paper. We are giving you a reminder as there are only a few days left until the discussion period ends. We have carefully considered your valuable suggestions to further improve our paper. We will appreciate your continued support if you are satisfied that our responses and revisions have addressed your concerns and comments. If you have any additional suggestions for improvement, please let us know.

Best,

Authors

We totally agree with you and have added the relevant content to the 'Physical interpretation' paragraph in Section 4, highlighting it in red.

We appreciate Reviewer Jfxr for the positive feedback and insightful comment, highlighting our strengths in

• Combining topological and physical features.
• the manuscript is well written and has many ablations/baselines.

Below, we carefully address your concerns. We uploaded the revised paper, where changes are highlighted in red.

W1. Why ORC? I get the point of wanting to connect clusters (according to the used library for computing the ORC, low ORC reveals "bridges within clusters":), but I'm not convinced that this is the best way. Note that I don't say that it is not the right way, I just say that there is no discussion of alternatives.

The reason we use ORC is that it not only captures the nodes and edges near the boundary conditions that alter fluid flow from a domain knowledge, but also identifies potential bottlenecks created by these nodes and edges. Metrics that can be presented as alternatives with a similar role to ORC are Forman curvature [1], Ricci Flow [2], and Betweenness Centrality [3]. The discussion of alternatives are added to Appendix A.

The problem of considering inter-cluster connections can be seen as an important metric in fields involving multiple objects, such as multibody dynamics, a hierarchical model [4] that message passing at two different resolutions (fine and coarse). The rewiring method that considers connectivity between clusters is added in Section 7 as a future research topic.

[1] RPS reejith et al. "Forman curvature for complex networks" Journal of Statistical Mechanics. 2016.
[2] Chien-Chun Ni. "Community Detection on Networks with Ricci Flow" Scientific Reports 2019.
[3] M. Barthélemy. "Betweenness centrality in large complex networks" The European Physical Journal B.
[4] Meire Fortunato et al. "MultiScale MeshGraphNets." ICML 2022 2nd AI for Science Workshop. 2022.

W2~W4 Additional Ablation Study.

We address W2 to W4 by integrating them into a single response. Our proposed rewiring method includes node selection steps that depend on factors such as degree, ORC, and physical context. To evaluate performance across different factor selections, we conducted additional ablation studies.

Table shows performance based on ingredient selection in CylinderFlow. The first step is to select nodes based on curvature("Former", Algorithm 1 lines 3-4), and the second is to select nodes based on physical context("Latter", Algorithm 1 lines 5-6).

"Ablation 1: effect of high degree" shows results with high-degree selection, achieving improved performance compared to MGN. However, it underperforms relative to PIORF, as it exhibits varying curvature values for the same degree. "Ablation 2: physical features" and "Ablation 4: graph topology" show performance based on the choice of physical context and ORC, respectively. Both outperform MGN, and Physical Context provides slightly better performance than ORC. "Ablation 3: randomly" is the result of randomly selecting both the former and the latter and adding edges, and is similar to the performance of MGN.

The results of additional experiments are added to Appendix F.

Minor/Typos

Thank you for your valuable feedback. Errors and log scale were changed in the uploaded paper

Dear Reviewer Jfxr,

Thank you for your thoughtful and detailed comments, which have helped us enhance our paper. As we have only a few days until the discussion period ends, we wanted to remind you of our responses to your valuable feedback.

We have carefully implemented and analyzed "ablation studies of W2-4" as you recommended, and the results and discussions are in our response. Given that we have addressed this feedback, we hope you will consider increasing your score as you indicated. If you have any additional thoughts about our ablation studies or other aspects, we would be happy to address them.

Best regards,

Authors

Dear Reviewer Jfxr,

Regarding your question about W1, we would like to check if our explanation has adequately addressed your concerns. As our discussion period has been extended *until December 2nd, we remain open to further discussion if you have any additional questions or points you would like to explore. We appreciate our discussion so far and are happy to continue clarifying any aspects of our work.

Best regards,

Authors

Q1. (W1). Inadequate theory compared to previous work(BORF). And authors complain that 'BORF works in batches and calculates the curvature with a minimum and maximum in each batch. Then, connections are added to the set with the minimum edge value to uniformly weaken the graph bottleneck. To save computation time, BORF does not recalculate the graph curvature within each batch, but rather reuses the already computed optimal transfer plan between sets to determine which edges should be added.', but there is no indication in this paper that there is an increase in efficiency compared to BORF

Our paper focuses on computational aspects of the MGN method; the main goal is to develop novel rewiring mechanism, which considers physical quantities of complex physical phenomena, for improved computational efficiency. To this end, our experimental design is oriented toward large-scale and complex physical simulations. We make comparisons against other rewiring algorithms in extensive sets of experiments. Although the proposed PIORF is less equipped with theories, we attempted to support the effectiveness of the methods by providing strong empirical evidences.

To iterate the difference between BORF and the proposed PIORF, BORF uses rewiring batches, which increases computational cost with each batch iteration. Figure 6 in Section 6.3 shows the computation times for several baselines and confirms that BORF is the slowest. In addition, BORF has three hyperparameters: the number of rewiring batches, edge addition, and edge removal, which can be a disadvantage when trying to find the optimal values. In contrast, PIORF has only one hyperparameter: the pooling ratio.

Q2. (W1). The addition of edges also changes the topology, which may lead to negative consequences that are not addressed in the paper.

We thank your concern about the potential negative consequences of topology changes. Our PIORF is carefully designed to add only meaningful connections that enhance information flow while preserving physical relationships in the mesh. Let us explain this more precisely:

Our edge addition follows two principles:

• We identify topological bottleneck regions using Ollivier-Ricci curvature
• We connect nodes with significant velocity differences to capture important physical interactions

To theoretically show that these additional edges improve rather than harm the ability to propagate information, we analyze the effective resistance between nodes. Following [1], effective resistance measures how well information can flow between nodes $u$ and $v$ in a graph: $R_{u,v} = \sum_{i=0}^{\infty} (\frac{1}{d_u} \hat{A}^i_{uu} + \frac{1}{d_v} \hat{A}^i_{vv} - \frac{2}{\sqrt{d_u d_v}} \hat{A}^i_{uv} ),$ where $(\hat{A}^{i})_{u,v}$ represents the number of paths of length $i$ between nodes $u$ and $v$.

This equation shows that effective resistance decreases with more and shorter paths between nodes. Our empirical analysis supprots that PIORF reduces the total effective resistance ($R_{tot} = \frac{1}{2} \sum_{u,v} R_{u,v}$). We randomly sampled 100 nodes from each dataset and measured the total effective resistance before and after applying PIORF:

• CylinderFlow: Reduction from 2,433,015 to 1,606,768
• Airfoil: Reduction from 15,644,891 to 10,143,689

These reductions (34% and 35%, respectively) show that our method effectively alleviates topological bottlenecks by adding physically meaningful connections. We believe that we address the reviewer's concern about potential negative consequences.

[1] Mitchell Black et al. "Understanding Oversquashing in GNNs through the Lens of Effective Resistance". ICML, 2023.

Q3. (W1). Computational efficiency is demonstrated only through experimental illustrations of efficiency gains, with no accompanying theoretical analysis

The complexity of PIORF is $\mathcal{O}(|\mathcal{E}|d^3_{\mathrm{max}})$, where $|\mathcal{E}|$ is the number of edges and $d_\mathrm{max}$ is the maximal degree. Since PIORF does not rely on batch or iterative computations, it is particularly effective for datasets with a large number of nodes, such as those in fluid dynamics or mechanics. The complexity is added to Appendix A and highlighted in red.

Q4. (W1). Contrary to the existing theory. Topping's work suggests that a fully connected graph does not have over-squashing (maximum curvature). However, as shown in Figure 3(a), the authors claim that the degree of a node is negatively correlated with curvature. These two theories are clearly contradictory. The authors should clarify whether this discrepancy arises from specific conditions discussed in the context of fluid dynamics. The degree may not be correlated to the level of curvature.

The ratio of negative edge curvature in CylinderFlow is 21.88%. These negative values can potentially cause bottlenecks. We converted this edge curvature to node curvature by averaging with neighboring nodes and analyzed the relationship between degrees, confirming that higher degrees correspond to higher negative curvature.

According to Deac et al., 2022[1], nodes with high degrees can cause bottlenecks, thereby exacerbating the issue of over-squashing. In fluid mechanics, which primarily uses triangular meshes, non-uniform degree distributions form near boundary conditions such as walls, resulting in nodes with high degrees.

[1] Andreea Deac et al. "Expander graph propagation". NeurIPS Workshop, 2022

Q5. (W2). There is no clear rationale for why adding edges between $s$ and $r$ would address the underlying physical phenomena. The authors only provide some intuitive insights from physics and experimental simulations, but these edges could potentially have a negative impact.

In fluid dynamics, governing equations such as Navier Stokes are helpful in predicting fluid flow because they depend on gradients of velocity[1]. the distinction between laminar and turbulent flows, as quantified by velocity and the Reynolds numbers, is important for understanding system behavior.

Velocity gradients are an important element in dynamics and have an equally important influence on learning. We confirmed that the accumulated error is related to the velocity gradient, and the result of discussion is added to Appendix G.3.

PIORF effectively integrates this physical context into graph rewiring by adding edges between nodes with significant velocity differences. This allows the model to help with long-range interactions to better simulate real-world phenomena such as fluid turbulence. To confirm the effect of physical phenomena, we conduct additional experiments by randomly selecting nodes and applying physical context.

Table is the result before and after rewiring and shows better performance after applying the physical context in CylinderFlow. The results of additional experiments are added to Appendix F.

[1] Charles Meneveau. "Lagrangian dynamics and models of the velocity gradient tensor in turbulent flows." Annual Review of Fluid Mechanics, 43(1):219–245, 2011.

Q6. (W3). Legends should be provided, and the meanings of the colors should be clarified to help readers better understand the impact of color changes in the figures (Figure 1, 4, 7, 8, 9,10 11).

Thank you for your valuable comments, we have added legends to all figures in the uploaded paper.

Dear Reviewer 5Uy1,

We appreciate your comments on helping us improve our paper in many aspects. This is a gentle reminder since we have only a few days until the discussion period ends. We have tried our best to address your questions. We would appreciate it if you could either confirm that our responses have satisfactorily addressed your concerns and consider a more favorable score or let us know if you have any follow-up questions.

Best,

Authors

Thank you very much for the explanation and effort in the rebuttal period.

Q4. I didn't find that conclusion in paper [1]. In paper [2], the only relevant statement I encountered was that 'High-degree nodes are particularly prone to over-squashing'. This does not necessarily imply that the improved results are due to alleviating over-squashing by removing the heigh degree nodes. It’s possible that removing the node with the high degree has led to other effects that enhance performance.

Dear Reviewer 5Uy1,

We noticed that you raised your initial rating without comment and we appreciate you recognizing our efforts. However, we are writing to follow up on our response to Q4 and note that the discussion period has been extended until December 2nd.

If you have had the chance to review our response to Q4, we believe our explanation provides important technical insights about the geometric properties of mesh graphs in fluid dynamics applications. We would like to understand if there are any remaining concerns preventing an acceptance score and what the key reasons are for your score remaining below the acceptance bar.

Additionally, in our response to Q2, we provided a theory-based intuitive explanation of how PIORF's appropriate edge addition can mitigate over-squashing from an effective resistance perspective while also addressing your concerns about the potential negative impacts of edge addition. In Q3, we discussed computational efficiency with theoretical analysis, and in Q4, we explained why we designed our rewiring algorithm using physical context.

Thanks to your valuable feedback, we improved our paper significantly. We remain open to further discussion if you have any remaining questions. Thank you for your time and thoughtful engagement in this review process.

Best regards,

The Authors