On the Completeness of Invariant Geometric Deep Learning Models

Thank you very much for your thoughtful comments and questions! Below, we address each of your points in detail, and we hope our responses provide clarity and insight.

---

The work is based on global connectivity assumption, and this assumption significantly limits this work.

Response:

We acknowledge your concern regarding the Fully-Connected Condition, and we have explicitly and extensively discussed this limitation in Section 5.4. Here, we would like to further emphasize the reasonableness of this assumption in the context of our study:

1. Global connectivity is a standard assumption in expressiveness research for invariant models.

   As discussed in Section 5.4, plenty of previous works in the literature adopt this assumption, given the challenges faced by invariant models in preserving local patterns during message passing. Unlike equivariant methods, which can rely on equivariant features to retain local information even in sparse graphs, invariant methods inherently lose such patterns.

Additionally, we respectfully refer you to our response to Reviewer jQxs (for question “Several studies [1], [2], [3], [4] have explored….”), where we further justify the reasonableness of this assumption for invariant methods.

Also, the experimental results seem to be quite weak.

Response:

We appreciate your concern on the experimental results! Actually, experiments presented in the main text are primarily intended to support the theoretical claims of the paper (e.g., the near-completeness of DisGNN and the completeness of other methods). For additional empirical evidence, we would like to respectfully draw your attention to the extensive real-world experiments on GeoNGNN provided in Appendix D.

We demonstrate that GeoNGNN achieves promising performance despite its simple design. For instance, it surpasses advanced methods such as DimeNet, GemNet, and PaiNN on MD17; MACE and Equiformer on MD22; and ComENet and SphereNet on QM9.

Furthermore, we outline potential directions for improving GeoNGNN to further enhance its empirical performance. While we acknowledge that there is room for improvement, we view these experiments as a foundational step for future work, given the theoretical emphasis of the current study.

Do you have any insight on achieving E(3)-completeness for frame-based approaches?

Response:

Thank you for this insightful question! Some frame-based methods (e.g., [1], [2]) indeed face challenges with symmetric structures where global-level frames may degenerate.

A potential way forward is breaking the overall symmetry through node marking, as demonstrated by GeoNGNN. By marking a node, frames can be calculated on the symmetry-broken point cloud (e.g., using PCA while ignoring the marked node). To ensure permutation invariance, the results can be averaged across different marked nodes. We believe this strategy could be a promising direction for extending frame-based approaches toward E(3)-completeness.

Can you comment on achieving E(3)-completeness by using node features beyond pairwise distances, e.g., dihedral angles?

Response:

We greatly appreciate the reviewer’s comment on this topic. Actually, complete models we establish like DimeNet and GemNet provide good examples of incorporating angles and dihedral angles, and their completeness is formally established in Theorem 5.5. Below, we briefly highlight some key points.

1. Achieving completeness using angle information.

   To prove this, we show that models like DimeNet can effectively implement GeoNGNN. For example, with angle information $\theta_{aij}$ and distances $(d_{ai}, d_{ij})$, the remaining distance $d_{aj}$ can be calculated. This allows us to express DimeNet equivalently using only distance information, aligning it with our proof framework. Details are provided in Appendix H.5.1.

2. Dihedral angles as redundant for completeness.

   While angle information alone is sufficient for achieving theoretical completeness when properly integrated like DimeNet, dihedral angles (as used in GemNet) are redundant in fully connected scenarios, as noted in lines 398–399. However, these features may enhance generalization in practical applications, as demonstrated in empirical results.

---

We hope these responses could address your concerns and provide clarity on the limitations, insights, and future directions of our work. Thank you again for your thoughtful feedback and the opportunity to further explain our contributions, and we would greatly appreciate it if you could raise your score accordingly.

[1] FAENet: Frame Averaging Equivariant GNN for Materials Modeling

[2] Frame Averaging for Invariant and Equivariant Network Design

We sincerely thank you for taking the time to review our work and for providing thoughtful feedback and constructive suggestions! Below, we address your comments in detail.

---

Some of the definitions, while they might be customary in the machine learning literature, are somewhat unfortunately choses from a mathematical perspective. Completeness of a space in the mathematical sense implies that each Cauchy sequence has a limit within that space

Response:

We greatly appreciate your expert remarks on the use of terms like completeness and isomorphism!

To clarify, we closely adhere to conventions established in prior works in this area ([1-3]). In these contexts, (in)completeness typically refers to the (in)ability of geometric models to distinguish point clouds, differing from its mathematical sense related to Cauchy sequences. Similarly, the term isomorphism aligns with recent literature ([4]), where the aim has been to bridge the expressiveness research in geometric deep learning with that of traditional graph learning, particularly in relation to the graph isomorphism problem.

That said, we greatly value the reviewer’s perspective on these terms and will carefully consider modifying the terminology if necessary to avoid ambiguity and ensure clarity from both mathematical and machine learning perspectives.

How does this method for expressivity generalize to different types of architectures? To the reviewer it seems that this method for showing is very specific and would be difficult to generalize to other types of equivariant architectures acting on point clouds.

Response:

Thank you for raising this excellent question. Our approach to establishing expressiveness is primarily focused on invariant models working with distance graphs, as supported by the reconstruction proofs (see Appendix H.2 and H.3). However, we offer the following thoughts on generalizing this framework:

1. Generalizing to invariant models using other invariant features such as angles:

   Models such as DimeNet that incorporate higher-order geometric features like angles can still be analyzed within our framework by transforming these features into distances. For instance, a function using $(\theta_{AC}, \theta_{AB}, d_{AC})$ as input can be equivalently expressed in terms of $(d_{AC}, d_{AB}, d_{BC})$. This equivalence allows us to evaluate whether the provided geometric information is sufficient for completeness. Indeed, we demonstrate this in Appendix H.5, where we prove the completeness of models like DimeNet, SphereNet, and GemNet under our framework.

2. Generalizing to equivariant models:

   Equivariant models typically learn higher-order geometry using vector or tensor products. These operations often implicitly capture invariant features, for example, by runtime geometry calculation methods in [5]. For a more simple example, the vector product $\vec{r}\_{ij} \cdot \vec{r}\_{ik}$ captures angular information, such as the angle $\angle jki$. A promising direction for extending our proof framework to equivariant models would involve demonstrating that these operations can extract the necessary invariant features required for geometric reconstruction in our proof, thus satisfying the completeness criteria established in our work.

---

We hope this response provides clarity and addresses your concerns. Once again, we are very grateful for your thoughtful questions and constructive feedback, which have inspired us to further refine and expand upon our work. We are looking forward to further discussion, and would greatly appreciate it if you could raise your score when you feel your concerns are addressed.

[1] Is Distance Matrix Enough for Geometric Deep Learning? NIPS 2023

[2] Complete Neural Networks for Complete Euclidean Graphs. AAAI 2023

[3] Incompleteness of graph neural networks for points clouds in three dimensions.

[4] On the Expressive Power of Geometric Graph Neural Networks.

[5] ViSNet: an equivariant geometry-enhanced graph neural network with vector-scalar interactive message passing for molecules

2. GeoNGNN is a promising foundation despite not being fully optimized for SOTA:

While GeoNGNN does not consistently achieve SOTA results, we believe it represents a highly promising approach. Its current design is deliberately simple, serving as a nested version of the basic invariant model, DisGNN, for our theoretical purpose. As discussed in lines 1017–1055, its architecture has significant potential for refinement and extension to enhance performance further. However, given the theoretical focus of this work, we prioritized simplicity to underscore the model’s completeness rather than optimizing it for empirical SOTA performance.

In my view, the primary contribution is the proposal of the geometric counterpart of NGNN and the proof that this approach effectively resolves the limitation of DisGNN in identifying symmetric point clouds when the graphs are even fully connected. The authors should also consider relevant experiments in this direction to emphasize the novelty and significance of this work.

Response:

We are grateful for your thoughtful observation regarding the primary contribution of our work.

1. Main focus of our work

We would like to clarify that the central focus of our study is to rigorously establish the expressiveness of a broad class of models, including DisGNN and complete models, thereby advancing the theoretical understanding of their capabilities.

While GeoNGNN is introduced as a pivotal proof-of-concept, it is not the exclusive focus of our work. Instead, as the simplest complete model, GeoNGNN provides a foundational framework for demonstrating the completeness of other models. Our experiments are designed to align with this theoretical emphasis rather than highlighting the novelty of GeoNGNN alone.

2. Existing Evidence Supporting the Effectiveness of Complete Models

In Section 6.2, we assess all established complete models on tasks specifically designed to distinguish pairs of point clouds that DisGNN cannot differentiate. The results consistently demonstrate that complete models effectively address these limitations, providing strong evidence of their ability to resolve symmetric structures.

---

In conclusion, we are deeply thankful for your thoughtful and constructive feedback. Your insights have greatly helped us improve the presentation and focus of our work. We hope our clarification address your concerns and we would greatly appreciate it if you could raise your score accordingly.

We sincerely thank you for taking the time to provide detailed and constructive feedback on our submission. Your comments are highly valuable and have significantly helped us identify areas for improvement in our work. Below, we address your suggestions and concerns.

I recommend the authors integrate key aspects of NGNN ... into the main text. Additionally, including a comparison with the original DisGNN would be helpful—highlighting the differences and explaining what enables NGNN (intuitively) to overcome the limitations of DisGNN.

Response:

Thank you for this insightful suggestion! We appreciate your recommendation to include key aspects of NGNN in the main text for enhanced clarity.

1. Further elaboration of NGNN

While we recognize the value of integrating NGNN’s core equations or an architectural diagram directly into the main text, the constraints imposed by the extensive technical results and page limits make this challenging.

To address this, we have expanded the relevant discussion in the Appendix, highlighting these additions in green in the revised version. We hope this approach strikes a balance between clarity and adherence to formatting constraints.

1. Intuitive explanation of GeoNGNN’s advancements over DisGNN

In the main text, we have provided an intuitive explanation for how GeoNGNN resolves previously unidentifiable cases in DisGNN. Specifically, GeoNGNN leverages symmetry-breaking through node marking (see lines 272–278 and 302–308). Since all unidentifiable cases are symmetric (Theorem 4.2), this symmetry-breaking approach suffices to address all these corner cases effectively.

That said, we understand that the dense technical details may still pose challenges to accessibility. We sincerely value your feedback and are committed to further refining our explanations to enhance the paper’s clarity and make it more approachable for a broader audience

Several studies [1], [2], [3], [4] have explored scenarios where the graph is not fully connected, underscoring the need to evaluate the performance of invariant neural networks in sparse graph settings.

Response:

Thank you for raising this important point! We acknowledge that our work primarily focuses on fully connected graphs and that this limitation is discussed in Sections 5.4 and 7 of the paper. Regarding the references you provided, we discuss them as follows:

1. ComENet [1] also establish theoretical conclusions assuming strongly connected 3D graphs.

   We respectfully note that while ComENet [1] establishes important theoretical results, these conclusions are still derived under the assumption of strongly connected graphs, as outlined in Section 3.1 of [1]. Additionally, it does not address cases involving multiple nearest neighbors.

2. Invariant methods we study cannot trivially extend to sparse settings compared to previous equivariant methods [2, 3, 4].

   We would illustrate this through an example: Consider a point cloud with two clusters connected by a single edge. Such sparsity is common, where the graph has only local connectivity but remains overall connected. In these cases, invariant methods struggle to detect the relevant orientations of the two clusters (e.g., by swinging the edge connecting the two clusters), highlighting a fundamental limitation of invariant models in sparse graphs. We extensively discuss this challenge in lines 406–408, alongside relevant works.

   We appreciate your suggestion to consider sparse graph settings in future work and will strive to address this in subsequent research endeavors.

The authors should aim to demonstrate the significance of their approach by clarifying in which specific cases their method outperforms existing methods. Providing examples or scenarios where GeoNGNN has a clear advantage would strengthen the empirical contributions.

Response:

Thank you for this excellent suggestion! We appreciate the opportunity to clarify the empirical contributions of GeoNGNN and would like to direct your attention to Appendix D, where we provide extensive experimental evidence. Below, we highlight some key points:

1. GeoNGNN demonstrates strong performance, surpassing advanced geometric models:

Despite its straightforward design, GeoNGNN achieves very promising results. For instance: It surpasses DimeNet, GemNet, and PaiNN on MD17; It outperforms MACE and Equiformer on MD22; It exceeds ComENet and SphereNet on QM9. These results underscore GeoNGNN’s capability to deliver competitive performance across diverse benchmarks.

Again, we deeply appreciate your continued engagement and valuable feedback on improving our paper! We deeply acknowledge your efforts, and we remain committed to refining and updating the manuscript.

We recognize that the presentation of GeoNGNN may have caused some confusion, particularly for broader audiences who are less familiar with subgraph GNNs. To address this, we have prepared a preliminary revision that provides both an intuitive and formal explanation of GeoNGNN in the main text. Additionally, we have highlighted its key advantages over DisGNN, as discussed with you (e.g., subgraph versus subtree). These modifications are highlighted with $\textcolor{orange}{yellow}$ in Section 5.1. We believe these updates will help align readers’ understanding with our intended message.

We hope this revision addresses your concerns to some extent. We are eager to engage in further discussions with you to continue improving the paper. Once again, thank you so much for your valuable guidance and support!

Q8: The selection of ModelNet40 does not seem to rigorously test the theoretical claims of the paper, which focus on nearly isomorphic point clouds. Could the authors provide more rigorous testing on datasets that better align with their theoretical focus?

Response:

We thank the reviewer for their insightful observation and for raising this important point. Below, we clarify how our evaluation aligns with the theoretical focus of the paper:

1. Theoretical claims of DisGNN focus on “Near E(3)-completeness.”

   Datasets like QM9 and ModelNet40 are selected as real-world examples sampled from natural distributions, providing practical demonstrations of our theoretical claims. The relevance of these datasets has been clarified in the discussion, particularly in “Why we evaluate DisGNN…on QM9.”

2. Corner cases theoretically unidentifiable by DisGNN are not “Nearly isomorphic point clouds.”

   While “Nearly isomorphic point clouds” can result in similar graph embeddings (e.g., two molecular conformations differing by a small disturbance in one atom’s position), in theoretical view, they are still distinguishable by DisGNN since precision is assumed large enough. Theoretical works such as [1-3] (including ours), as well as a broader range of studies in traditional graph learning, focus on inherently indistinguishable point cloud (or graph) pairs—those that result in identical embeddings even under models with infinite precision. These cases represent the intrinsic limitations of the model, rather than practical challenges related to noise or perturbations.

   Consequently, experiments that align with our theoretical focus should evaluate the distinguishability of models on such challenging pairs. We address this explicitly in Section 6.2, where our experiments demonstrate the model’s limitations in such cases while showcasing the capabilities of complete models in handling them.

Q9: It is unclear from the text and appendix what each structure in the synthetic dataset represents, how these structures were constructed, and why they are significant. Could the authors provide more detailed explanations on the construction and relevance of these synthetic structures?

Response:

We thank the reviewer for their valuable suggestion. We have now included detailed explanations about these synthetic structures, including their construction and significance, in the appendix, highlighted with green text.

---

Thank you once again for your detailed and thoughtful feedback. We recognize that some of the questions raised may have arisen from misunderstandings due to shortcomings in our initial presentation. We have clarified them in our responses and are continuing to improve them in the next revision.

We sincerely hope that our responses resolve your concerns and offer deeper insights into our work. We are looking forward to further discussion with you. If there are no further technical concerns, we would be delighted if you could consider raising your score accordingly.

[1] Is Distance Matrix Enough for Geometric Deep Learning?

[2] Complete Neural Networks for Complete Euclidean Graphs.

[3] Incompleteness of graph neural networks for points clouds in three dimensions

Thank you for taking the time to provide a very detailed and constructive review of our submission! We sincerely appreciate your thoughtful feedback and valuable suggestions, which have helped us identify areas for improvement and refinement in our work.

Below, we address your comments and questions individually, incorporating clarifications and additional insights where relevant.

---

Q1: The excessive use of bold text and the absence of a clear outline make the paper’s contributions difficult to discern. Could the authors consider restructuring the introduction for better clarity?

Response:

Thank you for your valuable feedback! In the revised version, we have reduced the excessive use of bold text to enhance readability. We are committed to further restructure the introduction to provide a clearer outline of the paper’s contributions in the furture.

Q2: I find that the statements in the section Theoretical Characterization vs Practical Use rely on the example C.2. How increasing the sparsity beyond this simple example is understudied despite the authors strong claims that relaxing the fully-connected condition leads to better expressiveness of GeoNGNN compared to DisGNN.

Response:

Thank you for pointing out the reliance on Example C.2 in the discussion of sparsity and expressiveness! Here are further clarifications:

1. Broader Examples Supporting Sparse GeoNGNN’s Expressiveness

Beyond Example C.2, one can easily check that sparse GeoNGNN can distinguish all symmetric point clouds in [1]’s Appendix A that DisGNN cannot. We give some more illustrations:

• 6-node pair in Appendix A.1 of [1]: Sparse GeoNGNN embeds isosceles triangles with different base lengths into node features, distinguishing the two graphs, even when subgraphs cover only the two nearest neighbors.
• The second 10-node pair in Appendix A.2 of [1]: Sparse GeoNGNN captures regular pentagons in one graph and partial rings in another, again with a two-nearest-neighbor subgraph radius.

2. General Rule of Sparse GeoNGNN’s Advancements

These examples illustrates a key property of GeoNGNN in sparse settings: it can capture local subgraph patterns (e.g., triangle structures in Example C.2) in its inner GNN and embed these into node features for the outer GNN. In contrast, DisGNN only captures subtree structures, which leads to confusion in symmetric cases (e.g., distinguishing triangles from rings in Example C.2). This aligns with broader findings in graph learning, showing the superiority of subgraph patterns over tree patterns.

Q3: There is no supporting evidence for GeoNGNN over existing architectures in the primary paper. Additionally, there is no comparative analysis involving node feature information generated by a complete invariant function. Could the authors address this gap?

Response:

We greatly appreciate your observation about the need for comparative evidence. We provide further clarifications:

1. Extensive Experimental Evidence for GeoNGNN in Appendix D

The experiments in the main paper are primarily designed to support our theoretical conclusions. However, to provide a broader perspective and deepen the experimental analysis, we have included real-world evaluations in Appendix D. In these evaluations, GeoNGNN demonstrates competitive performance, surpassing models such as DimeNet, GemNet, and PaiNN (on MD17), as well as MACE, Equiformer (on MD22), and ComENet and SphereNet (on QM9) across multiple targets and on average.

2. GeoNGNN as a Promising Model

Though not achieving SOTA universally, GeoNGNN demonstrates potential as a simple nested extension of DisGNN. We deliberately keep its design minimal to focus on theoretical insights, but it can be further tuned for enhanced performance (see lines 1017–1055).

3. Node Features via Complete Invariant Functions

The paper emphasizes global expressiveness. However, as shown in our proof (lines 1816–1817), complete methods generate powerful node features capable of solely reconstructing entire geometries, thus ensuring global-level completeness. Node-level geometric expressiveness is an intricate topic, which, however, now lacks a commonly adopted formalization like E(3)-completeness in relevant field, thus we defer to future work.