MANTRA: The Manifold Triangulations Assemblage

Dear reviewer, thank you for your positive review. Below we answer your weaknesses and questions.

Weaknesses

1. We view the dataset as an entry point for models that aim to leverage high-order features in data. We would be glad if the dataset quickly loses its relevance, as this would indicate that topological models are becoming robust in predicting topology. However, if this occurs, we can expand the dataset to include triangulations with up to 11 vertices, as specified in our response to reviewer fkvZ. This would make the dataset significantly larger. In fact, we can make the dataset arbitrarily large through subdivisions of triangulations, posing new challenges in computational efficiency for topological models.

2. We are rerunning the experiments with some more models to address this concern. We expect to publish the first preeliminar results on surfaces by tomorrow on three more models, as specified in other reviewers' answers.

Questions

1. Our dataset is a complete enumeration of triangulations of closed, connected two- and three-dimensional manifolds up to 10 vertices. This number of vertices is similar to the average number of vertices in some datasets like molecular datasets (see OGB molecular datasets
   $1$ or ZINC $2$ ), which usually contain between 20 and 30 vertices, but is significantly smaller than the number of vertices in some other datasets, such as fine-grained mesh datasets. It might seem that transferring observations from experiments executed in MANTRA to other datasets would be challenging due to the difference in size between our dataset and some other ones. However, subdivisions can be applied to obtain simplicial complexes with an arbitrarily large number of vertices, allowing the graph algorithms to work on more similar domains.

   Moreover, there is still room to add more triangulations. Specifically, using the same source we used for the original version of MANTRA, we could extract some of the triangulations up to $n=14$, although we would face significant storage and enumeration challenges since the number of triangulations grows rapidly with $n$ and, to the best of our knowledge, there are no complete enumerations of triangulations for $n\geq 13$ available. See answer to Reviewer fkvZ for more details.

   Regarding the dimension, we believe that our dataset compiles triangulations for the most relevant higher dimensions, namely 2 and 3, as most real-world data are one-, two-, or three-dimensional (e.g., molecular graphs are 1D, meshes are 2D, cad reps can be up to 3D...). Furthermore, to the best of our knowledge, enumerations of triangulations in dimensions greater than 3 (and even in dimension 3) are still active areas of research.

   In general, our code implementations and data are flexible enough to develop additional variants of the dataset: the large-scale version proposed to Reviewer fkvZ, or a dataset containing only minimal triangulations, that is, triangulations that are not subdivisions of other triangulations. Thus, we firmly believe that our dataset, although limited on the number of vertices and dimensions, is a good starting benchmark dataset to test high-order models, covering topological invariants of the most-abundant type of high-order structured data, namely surfaces and volumes. We have included a paragraph on the limitations, summarizing our response.

2. You are right and we will rephrase the sentence. To be completely fair, we should perform a grid search of several hyperparameters, what we did not do due to lack of resources and lack of computational efficiency of existing implementations.

$1$ Hu, Weihua, et al. "Open graph benchmark: Datasets for machine learning on graphs." Advances in neural information processing systems 33 (2020): 22118-22133.

$2$ Sterling and Irwin, J. Chem. Inf. Model, 2015 http://pubs.acs.org/doi/abs/10.1021/acs.jcim.5b00559

I thank the authors for their detailed response. I do not have further questions or remarks, apart from a minor comment:

While I understand this might be important for testing TDL in very controlled environments, the complete enumeration of triangulations has less significance from the perspective of many applications. For geometric meshes, for example, handling much larger triangulations is of much greater importance instead. While I understand this can be mimicked with subdivisions, this is not the primary focus of the submitted work. While I appreciate the rigor, overfocusing on this synthetic aspect poses the danger of creating a gap to relevant applications that the TDL methods should ultimately tackle.

Regarding the minor suggestions, we will try to change in some way the presentation of the tables, as we agree that they are hard to read being rotated. Regarding the use of degraded, we have rephrased the whole sentence avoiding its use.

These are the references used in our answer(s).

$1$ Telyatnikov, Lev, et al. "TopoBenchmarkX: A Framework for Benchmarking Topological Deep Learning." arXiv preprint arXiv:2406.06642 (2024).

$2$ Benson, Austin R., et al. "Simplicial closure and higher-order link prediction." Proceedings of the National Academy of Sciences 115.48 (2018): E11221-E11230.

$3$ Papamarkou, Theodore, et al. "Position: Topological Deep Learning is the New Frontier for Relational Learning." Forty-first International Conference on Machine Learning. 2024.

$4$ Papillon, Mathilde, et al. "Architectures of Topological Deep Learning: A Survey of Message-Passing Topological Neural Networks." arXiv preprint arXiv:2304.10031 (2023).

$5$ Papamarkou, Theodore, et al. "Position: Topological Deep Learning is the New Frontier for Relational Learning." Forty-first International Conference on Machine Learning. 2024.

$6$ Röell, Ernst, and Bastian Rieck. "Differentiable Euler Characteristic Transforms for Shape Classification." The Twelfth International Conference on Learning Representations.

Dear reviewer, thank you very much for your thorough review. Let us address the weaknesses you pointed out below.

1. We are grateful for the references you provided here. We are going to add them to our revised version of the paper, as they are indeed important for the problem at hand. However, we want to mention that the approaches of both papers are fundamentally different because:

   1. in the first case, deep learning is applied to point cloud samples of manifolds, which forget the topological structure of the manifolds, in contrast to our dataset where the full topological structure of the input data is preserved by the triangulation;

   2. in the second case, a classical approximation algorithm is used to approximate Betti numbers, that is fundamentally different to study if neural networks can predict these quantities robustly. In fact, our results suggest that, as for now, classical approximation algorithms are better-suited to analyze the topology of simplicial complexes, at least when it comes to standard tasks such as Betti number calculations.

2. Due to space constraints, we added hyperparameter details of the architectures in Appendix B, and we cited the relevant architecture papers in the main text. However, we have expanded Appendix B to add more details about the different architectures. Regarding the suitability of the architectures used, graph architectures were all published in high-impact conferences and have been extensively tested in the literature and two of three simplicial models were already published in important venues, i.e. PMLR and IEEE PAMI and also used in a TDL benchmark on graph datasets before $1$ . Regarding the length of the random node features, we modeled it after the maximum degree of the vertices, which for the $2$ manifolds is $8$. One hot encoding of the degree yields an $8$-dimensional vector, and thus we took the same length for random features. This is an arbitrary decision, because the idea of random features is to let the model learn that features carry no meaning on the input data.

3. We tried to moderate our tone when stating our conclusions and claims, and to justify our conclusions in the best way we could think of when writing the paper. Regarding graph vs high-order models, we observed what was accepted as a fact in the community: simplicial models were better than graph models for simplicial datasets. This was further justified in the paper by the theoretical fact that different surfaces can have the same graphs and thus graph neural networks are unable to tell the difference between the triangulations because you need the triangle information to discern between both manifolds. Regarding overall performance of simplicial models, we found that they (or rather their current implementations) are not yet robust to remeshing invariance nor topological property prediction, which was unexpected due to that, to the best of our knowledge, the position paper $5$ claimed that simplicial models were remeshing invariant. We tried to check this claim, but, surprisingly, it did not hold. Still, we are completely open to discuss specific claims and to rewrite them according to the discussion. The main objective of this paper is not to categorically claim properties of graph or simplicial models, but to provide a new dataset that allows researchers to go deeper on the questions we posed in our paper, such as the remeshing invariance property. This means that we are completely open to adjust our claims after a proper discussion of them.

Dear reviewer,

1. We think there is a misunderstanding between us. We are not saying that there is no high-order information in complex networks or other applications aside algebraic topology. In fact, we believe that the whole point of topological deep learning is to be useful in scenarios like science, complex networks, topology, etc. What we are arguing is that, although the encoding of the high-order information as simplices/cells is useful, it is not clear that the representations of the current datasets, which contain labels, cannot be learned and handled by graph models without the need of high-order models. In your example, a non-high-order deterministic algorithm could retrieve the high-order information by simply looking at the cliques of the graph and then filtering them by weights. This is probably not the most efficient way of learning from the complex network data, but it is certain a way of retrieving information using only the graph information. This is not possible in our dataset, as several simplicial complexes share the graph part, and thus successful models in MANTRA need to use the high-order information of the simplicial complexes. Regarding the complex hull dataset, the complex hull is inferred from a sample of points, so you do not need high-order information. In contrast, manifolds are naturally occurring, non-synthetic, structures in topology materialized in real-world phenomena (like the manifold hypothesis) with a lot of relevance in (modern) pure and applied (low-dimensional) topology; see for example, the series of papers on manifold triangulations cited [1-7]. With this, we want to stress out that triangulations are important and meaningful in its own field, although they are not part of the community of network science. Since we deeply care about the quality of our work and the scholarship aspects, we are adding all the dataset references in complex networks you cited in your answers in the introduction.
2. We actually used our version of degrees of simplices, that contains structural information about the simplicial complexes.
3. This is a standard approach in the graph and simplicial learning community. For instance, refer to the "Preliminaries" section of [8] (equation 2.4). More involved readout methods, such as those listed in [9], or more sophisticated pooling techniques beyond our mean pooling, as in [10], can lead to better results, but we think a further exploration of the dataset is out of scope and must be performed in follow-up papers.

[1] Gunnar Brinkmann and Brendan D. McKay. Fast generation of planar graphs. MATCH Commun. Math. Comput. Chem. 58, 323-357 (2007).

[2] Frank H. Lutz. Enumeration and random realization of triangulated surfaces. Discrete Differential Geometry (A. I. Bobenko, P. Schröder, J. M. Sullivan, and G. M. Ziegler, eds.). Oberwolfach Seminars 38, 235-253. Birkhäuser, Basel, 2008.

[3] Thom Sulanke and Frank H. Lutz. Isomorphism-free lexicographic enumeration of triangulated surfaces and 3-manifolds. Eur. J. Comb. 30, 1965-1979 (2009).

[4] Mark N. Ellingham and Chris Stephens. Triangular embeddings of complete graphs (neighborly maps) with 12 and 13 vertices. J. Combin. Des. 13, 336-344 (2005).

[5] Frank H. Lutz. Triangulated Manifolds with Few Vertices: Geometric 3-Manifolds. Preprint, 48 pages, 2003; arXiv:math/0311116.

[6] Frank H. Lutz. Triangulated surfaces and higher-dimensional manifolds. Oberwolfach Reports 3, No. 1, 706-707 (2006).

[7] Frank H. Lutz. Periodic foams and simplicial manifolds with small valence. Oberwolfach Reports 4, No. 1, 228-230 (2007).

[8] Xu, Keyulu, et al. "How powerful are graph neural networks?." arXiv preprint arXiv:1810.00826 (2018).

[9] Navarin, Nicolò, Dinh Van Tran, and Alessandro Sperduti. "Universal readout for graph convolutional neural networks." 2019 international joint conference on neural networks (IJCNN). IEEE, 2019

[10] Ying, Zhitao, et al. "Hierarchical graph representation learning with differentiable pooling." Advances in neural information processing systems 31 (2018)

Dear reviewer, thank you for your review. Below, we answer to the weaknesses of the paper.

Weaknesses

1. Regarding topological diversity, we took all possible triangulations up to ten vertices of surfaces and 3-manifolds. This means that the diversity of labels on the surfaces and $3$-manifolds cannot be improved unless we add more triangulations, as the label distribution is the one occurring naturally in these topological objects. Regarding the size of the dataset, we decided to stop at $n=10$ because for $n=11$ there are $11,590,894$ and $172,638,650$ different triangulations for surfaces (from which we can gather $219,321$ triangulations) and $3$-manifolds, rendering the dataset too large (however, it is not impossible to retrieve triangulations of $n=11$, see $1$ and https://www3.math.tu-berlin.de/IfM/Nachrufe/Frank_Lutz/stellar/surfaces.html). We will be happy to add a large-scale version of the dataset if you consider it necessary to the current paper before the camera-ready revision.

2. Initially, we chose only simplicial models from TopoModelX because the cellular models can't leverage properties of cell complexes that are not being leveraged by simplicial models, as there are no cells that are not simplices in our dataset. Regarding equivariant models, we avoided geometric equivariant models because MANTRA is purely combinatorial, meaning that there are no geometric features on the simplices such as coordinates. However, we have rerun the experiments for the cellular models you referenced as $1$ for a more comprehensive benchmark. We expect to publish the first preliminar results by tomorrow. Please, let us know if you consider that there are more models that we should add to our benchmarks.

3. We noticed that most graph-based models converged rather quickly after only two or three epochs and that simplicial-based models stopped converging after the same number of epochs. Hence, training for more epochs is not likely to improve the results. Also, we noticed a strong tendency for the models to predict constant outputs, suggesting that hyperparameter tuning will most likely not result in increased performance.

Questions

1. We extracted the triangulations from previous works of Frank H. Lutz, that are gathered in the following webpage: https://www3.math.tu-berlin.de/IfM/Nachrufe/Frank_Lutz/stellar/. We have explicitly specified this in the revised version of the manuscript, concretely, at the beggining of Section 2.

2. They represent the variance across the entire set of models.

3. Unfortunately, computational complexity does not originate from our dataset, but from a combination of the simplicial model implementations of the TopoModelX library, that are currently not optimal, and the large sets of experiments performed (five random seeds for each pair of dataset and model). Regarding model efficiency, the library is under continuous development, and we expect simplicial models to get faster with new versions of the library. For a precise measurement of the slow-down, please refer to Table 7 in our Appendix.

$1$ Sulanke, Thom, and Frank H. Lutz. "Isomorphism-free lexicographic enumeration of triangulated surfaces and 3-manifolds." European Journal of Combinatorics 30.8 (2009): 1965-1979.

$2$ Hajij, Mustafa, et al. "TopoX: a suite of Python packages for machine learning on topological domains." arXiv preprint arXiv:2402.02441 (2024).

Dear reviewer,

Thank you for your review. We answer your two main concerns below:

1. Regarding focusing on MP networks, this is mostly because current simplicial models are of this nature. To the best of our knowledge, there are only two papers on transformers for high-order domains: one centered on cliques $1$ not specifically tailored for simplicial complexes (although they define how to work with them) and another paper, still in review, studying general topological transformers paper $2$ . However, we are executing the experiments with the model of $2$ (that already incorporates concepts from

$1$ ). We expect to publish the first preliminary results by tomorrow. Regarding equivariant neural networks, we decided not to include them because MANTRA is purely combinatorial, meaning that there are no natural geometric features. More specifically, equivariant methods are useful if the simplicial complexes are embedded in space, i.e. the vertices have coordinates. This is not the case in MANTRA. However, if you have (a) specific network(s) in mind, please, let us know and we will do our best to execute the experiments before the discussion stage and camera-ready version.

2. Regarding variance, there is hardly any variance in the results per network and dataset. We acknowledge that current tables may lead to confusion, as we use the variance across models and transforms in them. We are working on a revision of our table presentation to make this clear.

Questions:

1. (Q1 and Q2) The triangulations without barycentric subdivisions are a complete enumeration of triangulations up to 10 vertices of closed and connected two- and three-manifolds. Initially, these triangulations and their labels were discovered, gathered and published online by Frank H. Lutz in his websites: https://www3.math.tu-berlin.de/IfM/Nachrufe/Frank_Lutz/stellar/surfaces.html and https://www3.math.tu-berlin.de/IfM/Nachrufe/Frank_Lutz/stellar/3-manifolds.html. Labels can also be computed using deterministic algorithms from the field of computational topology, specifically using algorithms to compute homology groups that are $\mathcal{O}(m^3)$ with $m$ being the number of simplices for their most basic implementation. However, the computational complexity can greatly vary depending on the algorithm and the boundary matrices, even achieving sublinear complexity in some cases.

2. We expect MANTRA to be useful as the first dataset on topological reconstruction, as you can generate a new dataset by eliminating certain simplices from either the original triangulations or from the subdivision ones. Thanks to the code provided, it is easy to generate this dataset. Also, due to the high quantity of triangulations, we firmly believe that MANTRA will be a strong proof of concept for papers tackling topological reconstruction.

$1$ Zhou, Cai, Rose Yu, and Yusu Wang. "On the Theoretical Expressive Power and the Design Space of Higher-Order Graph Transformers." International Conference on Artificial Intelligence and Statistics. PMLR, 2024.

$2$ Ballester, Rubén, et al. "Attending to Topological Spaces: The Cellular Transformer." arXiv preprint arXiv:2405.14094 (2024).