Simplicial Representation Learning with Neural $k$-Forms

We thank the reviewer for their constructive comments. Judging from their comments, it is clear that the reviewer made a concerted effort to understand the mathematical concepts and provided some excellent questions that helped us clarify the presentation of the math.

Answer primary weaknesses

1. We agree with the reviewers comments that the paper could be made more accessible. We appreciate the reviewer’s suggestion of cutting out some of the basic ML definitions, since these should be background for the audience. We have decided in the revision to globally put more emphasis into explaining the ML aspects of the work in the body of the paper. The trade-off is that we were unable to give a more detailed explanation of the mathematical concepts which we leave for the appendix.

   We have done this in two major revisions. Firstly, we have added a new subsection Architectures and Parameters which explains in detail how one can generally construct integration matrices from MLPs, and what the parameters and hyper-parameters of the algorithm are. Judging from the majority of the reviews, the pipeline through which the k-forms are actually used in an ML pipeline was a core concept that was poorly conveyed. We hope that this additional section goes some way to remedy this problem.

2. Secondly, as suggested by the reviewer, we have greatly expanded the descriptions of the experiments section in the body to include more details of the implementation. Further, we have tried to provide a more guided curation of how to interpret the results. Our hope is that for readers who are put-off by the technical definitions in the body will now have more chance of grasping the natural and meaning of the synthetic experiments and core ideas.

Answer Minor Weaknesses

1. Thank you for pointing this out - there were a number of unfortunate typos in the text which we have now clarified.

2. We think that the graph-learning tasks - nine in total - constitute a large enough sample to argue that our method can be used in practice. For us, the main goal of the paper is to introduce the core concepts, which are quite different to the usual message-passing architectures used in these kinds of tasks and require the majority of the space in the paper. We agree that the method should be tested on a wider and more rigorous framework. However, we propose that this is beyond the scope of this initial, introductory paper.

We thank the reviewer for their extremely in-depth and constructive review. In particular, the reviewer’s feedback about the equivariances and embeddings prompted to better clarify that our model explicitly ignores node/simplice embedding information which is equivariant with respect to actions on the ambient space. We realise we did a poor job of conveying this in the original version and have since amended this in the revision.

Answer to Weaknesses

1. We note that this was indeed a common criticism amongst the reviewers. We have taken the advice and decided to greatly expand the details of the experiments section.

2. Our basic premise is that a simplicial complex is itself only a topological object and has no intrinsic geometry. Indeed, for a combinatorial simplicial complex, the standard ‘geometric realisation’ coming from algebraic topology without an embedding is only a topological space. We argue that in simplicial and graph neural networks the geometry of the simplicial complex is determined by how it is embedded in $\mathbb{R}^n$: a space that has a Riemannian structure and notions of distance, angle and geometry.

   Hence our model is dependent on both the particular geometric embedding and the intrinsic combinatorial structure of the graphs and complexes. Indeed the integration matrices, and therefore the cochains and the output as well, are determined by the connections (edges, k-simplices) between the nodes since these connections tell us where exactly to integrate the k-forms. The same set of node positions but with very different connections will yield different cochains

   Most graph message-passing schemes have some non-equivariant component of the node features - for example the atomic weight, one hot encoded atom type, valence in the molecular datasets. In this work, we are interested in only the geometry of these types of features. For example, even when we ignore the ‘positions’ of the atoms in the molecular graph, we claim that there is still interesting geometry in the node feature space. In the message-passing paradigm, the topology and combinatorics of the complex are then incorporated into how messages are passed. For us, the difference is that the topology and combinatorics are represented as determining the regions of integration (k-simplices) in the embedding space over which learnable k-forms are integrated.

   The question of how to adapt our model to deal with node/complex embeddings that are equivariant in the ambient space - such as the ‘position’ of atoms in a molecule - is related. For this we ask the reviewer to see our detailed response to Question 2 and the added example on ‘convolutional 1-form networks’ in the revised appendix Additional Experiments.

3. At least for us the TUDataset for experiments in Table 1 and MoleculeNet datasets in Table seem relatively common in the literature and are recommended in the (Dwivedi et al., 2023) reference. We would be open to suggestions of further datasets on which to evaluate our model given we have an implementation of the architecture ready.

   We have decided to follow the reviewer’s advice on the point of adding an extra point of comparison with a Geometric GNN; namely, the equivariant GNN of (Santorras et. al 2021). As we had poorly conveyed and now stressed in the revised version, our method the equivariant component of node coordinates (such as the positional information) is always omitted from our model.

   The TU datasets in Table 1 have both positional and non-equivariant features from the 1-hot encoding of the atom type. For the GCT, GAT, GIN and our model, the position features were suppressed; the EGNN was provided with the additional equivariant information. We found that the equivariant model did not perform significantly better than the models without the positional information. Curiously the out-of-the-box baseline EGNN model also had significantly more parameters than the others.

   For the larger MoleculeNet dataset, we selected only datasets without equivariant, positional node embeddings so that we could apply our model. While we ran out of time to do this before the revision, we would be happy to run further experiments on larger datasets in MoleculeNet against EGNN that have positional coordinates in addition to non-equivariant coordinates if the reviewer thinks it would significantly improve the paper. The main thrust of this experiment was to show that our method scales drastically better than the standard (non-geometric) GAT, GIN and GCN models.

Answer weaknesses:

Thanks for the remarks concerning the experimental comparison. We added EGNN, a state-of-the-art equivariant GNN, in the revision and observed that, despite only using a fraction of the number of parameters, our method is still competitive and often outperforms it. We did not compare to any simplicial neural networks because these networks are not compatible and applicable to our tests— notice that, for the sake of better comparability, we only use graph classification tasks. Since there is a lack of strong ‘simplicial’ data sets, we found it best to first remain in the realm of graphs.

Answer questions:

The learnable parameters are the coefficients in the k-forms modelled as MLPs as well as the layers post integration that can act on the integration matrices. The hyper-parameters are the number of hidden layers and the number of cochains to be learned. The model is composed of multiple steps that come together in the following way: The neural $k$-form as an MLP provides a function that can be evaluated point-wise on the embedding space of the graph, however point-wise evaluation of the output is not interesting as it does not tell us anything about the graph or simplicial complex structure. Thus through integration of the k-form on the region of integration determined by the $k$-simplices (a deterministic process without learnable parameters), we can transform the point-wise information into cochains, yielding the integration matrix. The integration matrix can then be processed by further layers that will provide the output prediction. This prediction is used to update the model. The integration part can be understood as an operation that “aggregates” the information of the neural k-form with respect to the structure of the graph.

Given the concerns you raised we have taken two steps to ensure that the details are clear to the reader. Firstly, we have provided a detailed subsection regarding the general architecture and parameters in Section 3. Secondly, we have expanded the explanation of experiments in the body of the paper which we believe gives useful examples of how the basic architecture can be implemented in practice.

We thank the reviewer for the helpful comments about our manuscript. We have made some changes according to these comments. A detailed answer to the weaknesses and questions can be found below as well as a description of the changes made to the paper accordingly. In the manuscript the main changes have been added in blue for more clarity.  

Answer weaknesses:

1. After the helpful comments from yourself and the other reviewers, we realise that we could have done a better job of stressing how the graph/simplicial structure determines its vector representation. The graph information is incorporated as follows: the nodes are embedded in the ambient feature space and the edges (1-simplices) serve as regions of integration for 1-forms. Thus, if the edge structure changes, the 1-forms will be integrated over a different region, and produce a different vector representation of the graph. For example, two graphs with the same embedded nodes but different edges can have dramatically different integrals against the same form.

It is true that if the node embeddings are relatively uninteresting geometrically - for example, one hot encoding of atom types etc. - the simplicial regions of integration are a bit less intuitive. However, the integral can still be calculated in the same manner, and despite the ‘discreteness’ of the node embeddings the method performs well. When moving to higher-dimensional, real-valued scalar node features the method indeed performs much better, and overall we agree with your intuition that real-valued, node embeddings are preferable as inputs for our algorithm.  

2. Indeed the computational cost of integrating $k$-forms with $k>2$ becomes quite expensive. Given the results in this paper we believe that an important next step to this paper would be to better develop the computational aspect of this method in higher dimensions. One possible trick that we could use is the isomorphism between 2-forms and 1-forms in $\mathbb{R}^3$, which in theory should speed up computing surface integrals for surfaces in $\mathbb{R}^3$ considerably. We plan to investigate this in future work.  

3. For the moment, we have decided to focus on classification only due to space constraints in the paper. However, the method could easily be adapted for node prediction and we put this as future work. For example, one could integrate a set of 1-forms across the local 1-neighbourhood of each node to produce a vectorization that could serve as basis for node-based learning tasks.  

Answers to questions  

1. We opted to restrict benchmarking to graph classification, since it has a more established benchmarking framework. This was the approach taken by one of the main papers on simplicial neural networks, Bodnar et. al [1]. We did not directly compare to this framework, but based on the reported results in that paper it is likely our algorithm is competitive with them.  

[1] Bodnar, C., Frasca, F., Wang, Y., Otter, N., Montufar, G.F., Lió, P. &, Bronstein, M.. (2021). Weisfeiler and Lehman Go Topological: Message Passing Simplicial Networks. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:1026-1037 Available from https://proceedings.mlr.press/v139/bodnar21a.html.  

2. Yes! This is entirely feasible and in our view an exciting future direction. Due to questions raised by this reviewer and others we have decided to include an example of convolutional 1-form network in the Additional Experiments section in the appendix and how this addresses the reviewer’s question.

We appreciate your positive assessment of your paper. May we kindly ask what prompted the change in your score from an initial 8 to a 6? We believe that we not only addressed all reviewer concerns but also that our paper benefitted from these changes with respect to clarity.

Since the deadline for the rebuttal is coming up soon, please let us know if there's anything else we can do to address your concerns. Such a score change without any comments puts us at a fundamental numerical disadvantage with respect to more applied papers. We would love to answer any questions you might have!

We thank the reviewer for the helpful comments and the careful reading of our manuscript. Similar comments about the architecture of the method have been made by other reviewers. We have addressed these comments in the new version of the manuscript. The main changes have been incorporated in blue for an easier overview. Below we answer the specific comments and questions detailing the changes we have made.
 

Answer to weaknesses:

1. The fact that our algorithm produces a representation without message-passing is a basic property of the architecture. This seems to have been lost in our presentation of the paper, since many of the reviewers seemed confused about this point. We have added a new section called Architecture and Parameters to clarify how the procedure works in practice.

   Simplicial message-passing (see Bodnar et al. [1]) layers take a matrix of cochains as input, and produce another matrix of cochains as output before readout and downstream analysis. The representations of each simplex correspond to rows of this matrix when it is in the standard basis of simplicial chains. The point is that our algorithm replaces message passing by generating a matrix of cochains via integration matrices independently of any message passing. We have put this in the paper just above Remark 8 at the end of the new section to clarify, and hope that it addresses the reviewer’s concerns.   We agree with the reviewers comment regarding the last line in Remark 5 and have removed it from the paper.

[1] Bodnar, C., Frasca, F., Wang, Y., Otter, N., Montufar, G.F., Lió, P. & Bronstein, M.. (2021). Weisfeiler and Lehman Go Topological: Message Passing Simplicial Networks. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:1026-1037 Available from https://proceedings.mlr.press/v139/bodnar21a.html.    

2. Again, since many reviewers found it difficult to understand the general architecture, we have expanded on the figure by adding a whole sub-section Architecture and Parameters in Section 3.

3. It is slightly unclear whether the reviewer is referring to the dimensionality of the node embedding or the dimensionality of the form. For high dimensional node embeddings, 1-form computations are still efficient as shown in our real-world graph case.

   We understand the general concerns about showing higher dimensional form examples. We have decided to benchmark the method on well-known graph data sets as is usually done in the graph learning community. We show an example as a proof of concept with $2$-forms and $2$-dimensional complexes, to show how the method works in the higher dimensional case. Unfortunately data sets with higher dimensional structure are not widely available and rarely used as benchmarks hence the decision of having only an synthetic example for the higher dimensional case. In addition we believe that the difficult step in understanding the theory behind our model was going from one dimension to any dimensions higher than 1. Understanding and visualization of 2-dimensional objects being easier to communicate about, we decided to keep the proof of concept to two dimensions only, as a way to show that one can go from simple 1-dimensional case to more complicated multidimensional cases.

   About the performance, we believe that the strength of our method is due to its geometric foundations. In the experiments shown in Table 1 the point is not only that our model performs better than or similar to other models, but also that the number of parameters needed is smaller, this aspect is even more prevalent in the second set of experiments (Table 2) where not only the number of parameters is significantly smaller than other standard models but our method performs the tasks much better.