TopoTune: A Framework for Generalized Combinatorial Complex Neural Networks

W3. Democratization of TDL. Indeed, we believe that our ‘democratization’ claim can be better contextualized. Here are the reasons why TopoTune democratizes TDL.

• A plug-in approach that removes the need of expert TDL knowledge. To date, the design of new TDL methods has required a great expertise of the field, as no standardized, base method has been made available to practitioners wishing to apply TDL on their own data. However, by turning these technical steps into a simple component of the architecture –as you noted under “Strengths”-- TopoTune enables both experienced TDL practitioners and newcomers the design of new TDL architectures and message passing pipelines.

• A plug-in approach that improves upon a given GNN performance. TopoTune also democratizes TDL because it relies on GNNs, which are commonly used by deep learning practitioners. The typical use case is as follows. A deep learning researcher is using a GNN model for a given application. This researcher can plug their GNN into TopoTune, which automatically makes it “higher-order” and can potentially improve its performance. You are right that this increases the hyperparameter search space by considering previously fixed traits (topological domain, message function, etc.) as variables. However, we stress that these results with GCCNs were obtained with very minimal “traditional” hyperparameter tuning, keeping parameters like learning rate and embedded dimensions largely fixed across experiments, while the hyperparameters used with CCNNs were the result of an extensive benchmarking study (https://arxiv.org/abs/2406.06642). We now specify this at Section 6.1.

• A framework that opens the door to a systematic exploration of the TDL landscape. We emphasize that, going beyond a search tool, TopoTune provides an integrated and standardized framework that significantly reduces the implementation overhead associated with TDL. Rather than manually engineering architectures from scratch for each task, users can leverage TopoTune to explore architectural choices within a unified, expressive framework that encapsulates and generalizes previous work. By organizing the vast design space of TDL into a manageable and interpretable framework, TopoTune shifts the challenge from ad-hoc design to systematic exploration, which we believe is critical for advancing the field. We have further clarified this claim in lines 403-405.

W4. Experimental questions on how GCNNs outperform CCNNs.

Addressing the different subpoints as follows:

• “GCCNs outperform CCNNs: why?”: This performance is due to a myriad of reasons. First, GCCNs allow for more flexible neighborhood structures (“per-rank” neighborhoods) than CCNNs (see response to W1, Bullet 1). For example, on MUTAG, a cellular GCCN using a per-rank neighborhood significantly outperforms CCNNs. Secondly, GCCNs benefit from the use of $\omega_\mathcal{N}$ modules based on existing, well-studied architectures, rather than relying on the custom, ad hoc message-passing schemes typically implemented in CCNNs. By using existing GNNs in PyTorch Geometric as base $\omega_\mathcal{N}$, for example, GCCNs leverage a much larger body of literature and much more established software. Finally, GCCNs are now shown to be strictly more expressive than CCNNs (see response to W2).
• “GCCNs are smaller than CCNNs: why?”: After testing many combinations of GCCNs, we found the best performing models to have, in general, small parameter budgets. We found this empirically; it was not intended by design. However, we can hypothesize that this happens because i) GCCNs allow for smaller, more focused neighborhood structures, which we included in our search space and ii) two of the message functions (GCN, GAT) we consider are particularly lightweight, and could introduce less parameters than the ad-hoc message functions of CCNNs.
• “GCCNs improve over existing CCNNs: a matter of hyperparameter search?”: You are right that GCCNs introduce additional search space by making previously fixed parameters like topological domain and message function variable. However, as discussed in W3 Bullet 2, we stress that all the other potential critical parameters that were optimized while benchmarking CCNNs (learning rate, hidden channels, dropout values, batch size, readout function, etc) were totally fixed in our experiments. We now specify this at lines 420-423. In this way, despite increasing the overall combinatorics, we actually performed a more limited hyperparameter search than TopoBenchmark –yet we obtained better performance. This underlines the methodological contribution of GCCNs as an architecture.
• “Performance-cost trade-off and running times.” Thank you for bringing up this excellent point. We now provide time complexity analysis in Appendix C as well as training runtimes in Appendix G. Please see our response to all reviewers, points 1A and 1B.

See next reply.

Thank you for your detailed review and valuable feedback. We believe it has significantly helped to improve our paper. We address your main concerns (weaknesses (W) and questions (Q)) below:

W1. How GCCNs would unlock new operations and patterns.

Bullet 1: This is an important point. Comparing Equations 3 and 8: The collection of neighborhoods considered in CCNNs implicitly exclude per-rank neighborhoods. Specifically, in the definition of CCNNs by Hajij et al 2023 (Eq. 3), the notation "N" refers to the neighborhood N across all ranks. This means that, for example, if the incidence neighborhood is considered, all possible 1-hop rank incidences are considered (edges to nodes, faces to edges). It is not possible, for example, to only consider incidence between edges and nodes. GCCNs, on the contrary, are not constrained by this, hence unlocking new possible message-passing paths. This increased flexibility of GCCNs over CCNNs is precisely what allows them to outperform previous architectures. We realize the original text describing Eq. 3 was misleading, thank you for pointing this out.

While $\psi$s in Eq. 3 could potentially be neighborhood- and rank-dependent, by default the vast majority of CCNNs consider all of them to be of the same nature (Papillon et al. 2023). We have clarified the text to reflect this. Moreover, by definition, CCNNs are message-passing architectures, as the function $\psi$ can at most define the message between cells. Contrary to this, nothing forces the $\omega$s of Eq. 8 to be message passing based (e.g., by using a Transformer or MLP architecture). This introduces a completely new landscape of possible TDL models, which have until now been very focused on message passing (see response to Q3 for more). We have updated the descriptions of CCNNs (Eq. 3, Section 2) and GCCNs (Eq. 8, Section 4) to further clarify these points.

Bullet 2: Regarding the possibility of ‘marking’ node and edge features to specify rank and neighborhood information, it is indeed a valid approach. In fact, it is precisely what simulation works like [1] do (see Section 3, paragraph Retaining expressivity, but not generality). However, the goal of our work is to design a model whose architecture (as opposed to features) naturally incorporates such inductive biases. Indeed, this fulfills TDL’s goal of preserving the topological symmetry of the domain, and requires an inherently different model. To better explain why this idea of incorporating inductive biases goes beyond marking features, we consider an example (now at lines 231-238).

Consider a molecule, represented as a combinatorial complex: bonds are modeled as edges (1-cells) and rings such as carbon rings are modeled as faces (2-cells). Two bonds can simultaneously share multiple neighborhoods. For instance, they could be lower adjacent because they have a common atom (0-cell) and, at the same time, also be upper adjacent because they are part of the same molecular ring (2-cell). Despite their different chemical meaning, the whole Hasse graph (i.e., the approach of [1]) would collapse these two relations (upper adjacent, lower adjacent) into one. Moreover, the resulting GNN would not be able to distinguish anymore which node of the Hasse graph was an atom or a bond or a ring in the original molecule, and would process all the connections with the same set of weights.

Therefore, even if a GNN on the whole augmented Hasse graph of a combinatorial complex is as expressive in a WL sense as a CCNN on the CC, expressivity itself is not enough to employ a GNN rather than a TNN, as the resulting learning models are still inherently very different. In this sense, GCCNs are the first class of models to retain all the properties of [1] while being proper TDL models. In other words, GCCNs still preserve the topological symmetry of the domain.

[1] Jogl et al. “Expressivity-preserving GNN simulation.”

W2. Stronger expressivity proof. We agree that Proposition 3 as it stood was not very interesting. We took your feedback to heart and now provide and prove a stronger proposition about expressivity. We show that GCCNs are strictly more expressive than CCNNs, using a higher-order analog to the k-WL test. The full proof is provided in Appendix B3. We provide a brief outline here:

• We define WL-tests related to CCNNs and GCCNs, called CCWL and GCCWL.

• The CCWL test is equivalent to a WL test on a strictly augmented Hasse graph.

• The GCCWL test is at least as powerful as the k-WL test on a strictly augmented Hasse graph, for any k.

• Using the fact that k-WL is more powerful than WL on graphs, we find a pair of two strictly augmented Hasse graphs that cannot be distinguished by WL but can be distinguished by k-WL. This yields two combinatorial complexes that cannot be distinguished by CCWL, but can be distinguished by GCCWL.

• As such, GCCNs are strictly more expressive than CCNNs.

See next reply.

Zooming out a little: I hope my previous comments and questions do not sound gratuitously combative. As an official reviewer, I felt it important to convey my opinion that some claims are somewhat overemphasised, and some arguments and responses have been given in a way I found, in some cases, slightly deceptive.

Other than this, in the above response I had additional comments to share and question to raise, which I invite the authors to concisely respond to.

Thank you very much for your time and your very helpful feedback. We address your points about weaknesses (W) and questions (Q) below.

W1. Justifying the summary statement “GCCNs outperform CCNNs” and contextualizing TopoTune’s benefits. It is true that GCCNs only outperform outside 1\sigma on 2 datasets across all domains. However, when considering inter-domain performance, as has been traditionally the case in TDL, GCCNs outperform (beyond 1\sigma) the counterpart best CCNN on 11 out of 16 of the domain/dataset combinations tested. We now specify this at Section 6.2, paragraph GCCNs outperform CCNNs.

Moreover, we stress that these results with GCCNs were obtained with very minimal “traditional” hyperparameter tuning, keeping params like learning rate and embedded dimensions largely fixed across experiments, while the hyperparameters used with CCNNs were the result of an extensive benchmarking study (https://arxiv.org/abs/2406.06642). We have clarified this point in Section 6.1. More broadly, we would like to emphasize how powerful these results are in the context of how much easier and simpler TopoTune makes TDL. Each of the CCNNs we benchmark against is the product of a painstaking, one-at-a-time generalization of a specific choice of model to a specific choice of domain, always with a “from scratch” message-passing scheme. In contrast, each GCCN that achieves comparable performance is the result of a structured and systematic exploration of domain, neighborhood, GNN, and related parameters facilitated by TopoTune. While it is true that selecting the best GNN sub-modules or parameter combinations requires some effort, TopoTune provides an integrated and standardized framework that significantly reduces the overhead associated with this process. Rather than manually engineering architectures from scratch for each task, users can leverage TopoTune to explore architectural choices within a unified, expressive framework that encapsulates and generalizes previous work. This approach not only accelerates the design process but also ensures consistency and comparability across models. As shown in Fig. 5, TopoTune also provides insight into how design choices—such as the number of parameters, neighborhood configurations, or GNN submodules—affect performance, offering a more principled way to navigate and refine architectural decisions. TLDR: By organizing the vast design space of TDL into a manageable and interpretable framework, TopoTune shifts the challenge from ad-hoc design to systematic exploration, which we believe is critical for advancing the field.

W2. Comparison in runtimes and memory usage. For runtime, we point to bullet point “Training times” in our global answer to all reviewers. Due to the large amount of experiments and necessity for distributed training, reporting used memory beyond the number of parameters was not possible in this timeframe.

W3. Comparison with higher-order GNNs. This is a very interesting point, thank you for bringing this work to our attention. Indeed, these models perform very well on graph benchmark tasks.

First, let us directly answer your question about how works like PPGN++ are different from TDL, and specifically GCCNs. The main difference between higher-order GNNs and Topological Neural Networks (TNNs) lies at a very fundamental level.

GNNs run on graphs, which are simple topological spaces. GNNs have often been studied through the lens of Geometric Deep Learning (GDL). GDL (in the sense of [1]) is built on group-theoretic arguments along with the frequent usage of Hilbert Spaces (strictly related to manifold learning and, in general, to metric spaces).

TNNs run on combinatorial topological spaces that are inherently higher-order. TNNs have been studied through the lens of Topological Deep Learning (TDL). TDL is solely built on the modeling assumption of data living on the neighborhoods of a combinatorial topological space and having a relational structure induced by the neighborhoods’ overlap. Further insights can be gained from the thesis in [2]-[3].

This said, higher-order GNNs still run on graphs. As such, they usually use higher-order information to update node embeddings. In contrast, in TNNs (GCCNs included), all the cell embeddings are updated, as each cell is an element of the underlying space. This is not a detail. Indeed, while higher-order information is usually used in GNNs to achieve some desirable property (e.g., improved expressivity) but without any other additional theoretical argument, TNN usually results in improved expressivity while being supported by the sophisticated and powerful machinery of algebraic topology. This fact allows us to leverage additional structure (e.g., spectral theory, homology theory, homotopy theory,...) when combinatorial complexes are particularized to simplicial, cell, or path complexes [4][5][6][7].

See next reply.

We are happy to hear we were able to address many of your questions! Thank you for engaging with our response. We respond to your additional points below:

W1. Benefits of GCCNs vs CCNNs. Great question! We chose our hyperparameters using TopoBenchmark defaults, and for those in which no default was provided (e.g. hidden state dimension) we just selected the lower value that was considered in TopoBenchmark’s original grid search, which is recorded in [1]. We now specify this in the paper (line 423); thank you for your feedback.

[1] https://github.com/geometric-intelligence/TopoBenchmark/blob/main/scripts/reproduce.sh

W2. Time complexity and runtimes. Yes, that’s exactly right. We currently project combinatorial complexes onto strictly augmented Hasse graphs directly inside the forward function of the model, instead of pre-saving the expanded dataset. This introduces higher runtime grossly proportional to dataset size, which explains why we are comparable/faster on smaller datasets and slower on larger datasets.

W3. Comparison between higher-order GNNs and TNNs.

• We begin by addressing your questions specific to ZINC. In terms of the literature, the best reported pure GNN performance on ZINC without edge features is 0.320 MAE using PNA [1]. With edge features (representing bond types), this improves to 0.188 MAE. The best TDL model on ZINC, CIN [2], achieves 0.079 MAE, leveraging edge features. From what we can see, PPGN++ does not include ZINC results in its evaluation. In terms of our results, we remark that the current TopoBenchmark implementation disregards edge features. As highlighted in [1], edge features significantly influence performance. This omission likely accounts for the observed performance gap in our results compared to state-of-the-art models. You are correct that TDL models generally outperform GNNs on ZINC. We appreciate your observation and will explore incorporating edge features into TopoBenchmark.

• We now address your point about higher-order GNNs. We agree that a direct comparison between higher-order GNNs and TDL methods would be valuable, both for highlighting the benefits of TDL and evaluating its potential for real-world problems. Unfortunately, such a comparison belongs outside the scope of this paper, which aims to first and foremost generalize and make it easy for TDL methods to incorporate arbitrary layers. That being said, we absolutely plan to incorporate higher-order GNNs as baselines in future work and help expand the TopoBenchmark framework accordingly.

• To answer your question about choice of dataset: The datasets used in this and other TDL works are largely inherited from the GNN literature. Molecular datasets, in particular, are appealing for TDL due to the significance of cycles and hyperedges in representing chemical rings and functional groups. TDL is not inherently limited to these datasets. However, the lifting procedures used to generate higher-order cells impose computational constraints, specifically on memory. Current implementations (e.g., finding cycles or cliques) do not scale well with large, densely connected graphs. There is ongoing research to design scalable lifting procedures that would enable TDL methods to generalize to broader datasets. For instance, Bernardez et al. [3] propose innovative approaches to extend TDL beyond the graph domain.

All of these aspects have been clarified in the revised manuscript: we provide further details on dataset selection (line 425), excluding edge features(line 430), higher-order GNNs as future work (line 539), and dataset selection + lifting limitations (Appendix E2).

References

[1] Corso, Gabriele, et al. "Principal neighbourhood aggregation for graph nets." Advances in Neural Information Processing Systems 33 (2020): 13260-13271.

[2] Bodnar, Cristian, et al. "Weisfeiler and lehman go cellular: Cw networks." Advances in neural information processing systems 34 (2021): 2625-2640.

[3] Bernardez et al. “ICML Topological Deep Learning Challenge 2024: Beyond the Graph Domain.” Proceedings of the Geometry-grounded Representation Learning and Generative Modeling Workshop (GRaM) at ICML 2024 https://arxiv.org/abs/2409.05211

Hello! We are following up to ask if our previous response has addressed your remaining questions about the selected training hyperparameters and the nature of the ZINC dataset. We would appreciate being able to incorporate any additional feedback into the manuscript before the deadline tomorrow. In the event you have found our responses to be satisfactory, we would really appreciate an updated rating. We believe your feedback has significantly helped improve the manuscript. Thank you!

We wish to express deep gratitude for your time and your thoughtful review. We are happy to read that the figures were helpful. We address the raised points about weaknesses and questions below.

W1 Contextualizing the contribution for TDL newcomers. (Response 2B in main reply to all reviewers above). We completely understand that TDL might be an unfamiliar field, and appreciate your feedback about better contextualizing TopoTune’s contributions with respect to the field’s open problems. As such, we have baked into the paper’s stated contributions and conclusions a stronger connection with a position paper authored by many of the field’s leaders (https://arxiv.org/abs/2106.04051) which defines 11 open problems. This work fully or partially tackles 7 of these problems, which we elaborate upon in our second main reply to all reviewers above.

W2 Clarifying the simplicial vs. cellular domains. We have incorporated in Appendix A a brief overview of the different topological domains of TDL. This addition is now referenced in Section 2 Background, and helps make this work more self-contained.

Math typos. Thank you for pointing these out –we have fixed both.

Questions

1. Since a GCCN is made up of message-passing blocks, each individually assigned to a given neighborhood, modifying the amount of neighborhoods necessarily modifies the amount of message-passing blocks, which in turn modifies parameter size. You are correct that each GNN independently does not change in size (whether it be assigned to node-level adjacency, or face-to-edge incidence, for example), but the amount of GNNs used certainly does affect total size. We have also made sure to better clarify this in the paper in Section 6.2.

2. Each GCCN model is parametrized by choice of neighborhood structure, choice of topological domain, choice of message function (i.e. choice of GNN to use as building blocks), and choice of graph expansion method (we consider both ensembles of strictly augmented graphs and one augmented Hasse graph). For the purposes of this work, we only consider one set of intra- and inter- neighborhood aggregations, just to reduce the scope of comparisons already introduced by the previously mentioned parameters. Specifically, intra-neighborhood aggregation is left up to the choice of GNN, and inter-neighborhood aggregation is a sum. Considering and comparing various aggregators would be an interesting future research direction.

3. Architecture wise, this opens the door to newer advances in the graph learning field, going beyond “standard” GNNs, such as graph-based MLP models (https://arxiv.org/abs/2106.04051), and diffusion models (https://arxiv.org/abs/2106.10934). Application wise, we envision TopoTune to be a tool for newcomers from the various fields which have already been shown to benefit from learning on higher order spaces, as outlined in Appendix B of TDL’s most recent position paper (https://arxiv.org/abs/2106.04051). These include: data compression, natural language processing, computer vision, chemistry, virus evolution, and more. Much of this interdisciplinary work is early and its future success is in part deeply tied to the accessibility of TDL, something we hope TopoTune will accelerate.

We are happy to hear that our answers were helpful! Addressing the additional question about GNN module parameterization:

• Small clarification 1. On the ZINC dataset we in fact compared every single possible combination of GNN and neighborhood structure listed in lines 415–418. Figure 5 only shows three of these combinations because of the plot’s range, which only shows models that performed within 10% of the best model and within 40–100% of the best model’s parameter size. On ZINC, there are only 3 models, all GIN-based, who belong in this range.
• Small clarification 2. The neighborhood notation in Fig. 5 is not an accurate reflection of neighborhood size. The key here is that while the orange neighborhood is written in terms of per-rank neighborhoods (notation includes rank superscripts, see example at line 283), the green and gray neighborhoods are written in terms of regular, rank-independent neighborhoods (notation does not include rank superscript, see example at line 303). We were hoping to write out these neighborhoods in terms of their per-rank notation, but for some reason Open Review is having a very hard time displaying the math -- so sorry about this. We write them as clearly as possible in text:

{N_{A, \uparrow}, N_{A, \downarrow}} = {N_{A, \uparrow}^0, N_{A, \uparrow}^1, N_{A, \downarrow}^1, N_{A, \downarrow}^2}

N_{A, \uparrow}, N_{A, \downarrow}, N_{I, \downarrow}} = N_{A, \uparrow}^0, N_{A, \uparrow}^1, N_{A, \downarrow}^1, N_{A, \downarrow}^2, N_{I, \downarrow}^1, N_{I, \downarrow}^2

We completely understand this could be a confusing choice of notation. To better clarify this distinction but still avoid having space issues, we have added to Fig. 5’s legend a symbol indicating which neighborhoods can only be expressed as per-rank, and mentioned it in the caption (line 526).

Thank you again for your continued feedback. We appreciate engaging with you and improving the paper through your input.

Thank you for taking the time to provide such valuable feedback, we truly appreciate it. Let us address each of the points you raised in Weaknesses:

W1 Test larger datasets on node-level tasks. We now provide additional experiments on 4 larger node-level benchmark datasets (Amazon Ratings, Roman Empire, Minesweeper, Questions) that our machine can support memory-wise –please see results in Table 8 (Appendix I), as well as below. Specifically, all TDL models are constrained by the currently available topological liftings, as large graph-based datasets significantly increase in size in the process. (Note: developing better, more lightweight liftings is a current field of research, see https://arxiv.org/pdf/2305.16174). As an example, Tolokers dataset (11,758 nodes, 519,000 edges) raises OOM issues when storing either cliques (simplicial) or cycles (cell) due to its extremely dense connectivity, as was reported in Table 1 of (https://arxiv.org/pdf/2406.06642). We have added Questions dataset (48,921 nodes, 153,540 edges), not evaluated in TopoBenchmark, to show that our GCCNs are applicable as long as the lifting procedures are feasible. Regarding the results on the other three datasets, we observe that GCCNs achieve similar performance than regular CCNNs, outperforming them by a significant margin on Minesweeper.

W2 Give time complexity and training times. We refer to response 1B in the main reply to all reviewers above.

W3 Analyze performance vs size. (2A in the main reply). We have made sure to better clarify this contribution in Section 6.2 by separating into two renamed subsections: “Lottery Ticket GCCNs” and “Impactfulness of GNN choice is dataset specific.” There is indeed large variance in performance (and size) between choices of GNN. Such a variation in performance between GNNs is to be expected, as some message-passing functions are better suited to certain tasks/datasets, as has been studied in the GNN field through extensive benchmarking (see for example https://arxiv.org/abs/2003.00982). In the context of Topological Deep Learning, we are interested in understanding how these differences in performance couple with choice of neighborhood, and how we can optimize these hyperparameters for maximal performance at minimal cost. We also kindly refer to Figure 7 which includes performance versus size results on all tested datasets.

Figure 5 specifically aims to show “lottery-ticket” models with high performance and low parameter cost (ie, size). This figure only considers models performing within 10% of the best model. In the case of ZINC, both the best performing model and the within-10% models use GraphSAGE. On the other hand, PROTEINS and Citeseer achieve within-10% performance with models using GAT, GCN, and GraphSAGE. This indicates that for some choices of dataset/task, the choice of message-passing function is much more consequential than in other cases. Both sets of results in PROTEINS and Citeseer show that cutting down on parameter complexity can be as simple as choosing a lighter message-passing function (such as GAT or GCN) while keeping choice of neighborhood constant. For example, both in the PROTEINS and Citeseer cases, choosing GAT over GIN or GraphSAGE leads to equivalent performance for a given choice of neighborhood structure (purple).

If you have additional ideas on how to deepen this analysis, we would be happy to hear them during the rest of the discussion period.

Questions For Q1, we refer to point 1 above. For Q2 we refer to point 2 above. For Q3 we refer to point 3 above.

Hello Reviewer TxXe. We are following up a fourth time to ask if our rebuttal has addressed your concerns. Your input has informed many new improvements to the paper including adding new experiments, runtimes, time complexity, and clarifications to the main text. We would really, really appreciate hearing back from you about your thoughts on this. That way, if there are remaining issues, we’d be happy to clarify before the deadline.

Otherwise, if you’re satisfied, we kindly ask that you reconsider the rating of our updated submission.

2.B. Contextualizing the contribution (continued) Below, we outline the 7 open problems of TDL we partially or fully address in this work. These problems were defined by many of the field’s leading authors in a recent position paper (https://arxiv.org/pdf/2402.08871).

• Open problem 1: Need for adaptation of TDL in practice and making TDL more accessible for relation learning in real-world applications. TopoTune removes much of the software-based barrier to entry and makes the choice of base model flexible. Hence, TopoTune makes TDL a more attractive choice for practitioners from the several fields that stand to benefit from relational learning, such as neuroscience, protein engineering, chip design, and so on (see https://arxiv.org/abs/2106.04051 at Appendix B).

• Open problem 3: Need for benchmarking. As TDL evolves quickly across an increasingly wide landscape of models, it becomes challenging to understand which model is best suited for a particular application. Through its integration in TopoBenchmark and its structure, TopoTune makes basic components of TDL such as choice of topological domain, neighborhood structure, and message-passing function easily comparable across many benchmarking tasks. This is a fundamental change of perspective compared to previous literature, which always considered one new model with one set of such choices at a time.

• Open problem 4: Limited availability of software. On one hand, TopoTune directly integrates with the fully open-source TopoBenchmark platform, which itself leverages other open source TDL tools from packages like TopoModelX and TopoNetX. On the other hand, TopoTune makes the much wider and more well-established platform of software tools for GNNs (such as PyTorch Geometric) directly applicable to TDL.

• Open problem 5: Need for a cost-benefit analysis framework. By exploring a large model space systematically, TopoTune makes it much easier to assess cost (model size, training time) in comparison to performance. We show an example of such an analysis in Figure 5.

• Open problem 6: Building scalable TDL models that work across domains. To our knowledge, GCCNs are the first models empirically tested across many higher-order domains. This feature is enabled by baking the implementation into TopoBenchmark, which already contains the data liftings for many topological domains.

• Open problem 9: Consolidating relative advantages of TDL through theoretical foundations. Beyond proving that GCCNs generalize and subsume CCNNs, we also provide expressivity and permutation equivariance proofs that are cross-domain applicable. To our knowledge, the WL-tests and related definitions of expressivity that we define on combinatorial complexes (in the updated manuscript) are the first definitions existing on combinatorial complexes. Further, we leverage these definitions to prove that GCCNs are strictly more expressive than CCNNs (in the updated manuscript), which is the first proof of expressivity for combinatorial complexes.

• Open problem 11. Developing a transformer architecture for TDL models that “lays a unified foundation for TDL across different higher-order domains.” As presented in Table 1, TopoTune offers the possibility of using a vanilla transformer as a “fully-connected” message-passing function. However, TopoTune offers a more general answer to this problem, in that it provides a foundation, both theoretical and practical, for building models across topological domains.