Directed graph transformers meet metabolic networks

We thank the reviewer for evaluating our work and providing insightful comments. Below we address the concerns raised by the reviewer:

---

Weakness 1 Answer:

Thank you. It is true that we have not been clear enough with the motivation for our work. To clarify this, we have added the following line in the manuscript:

“In this work, a model is proposed that is able to generalise essentiality reaction prediction across different microorganisms by pre-training on synthetic knock-out data. It is shown that the model can serve for effective transfer learning to real metabolic data, in this case, gene essentiality obtained experimentally in the lab. This is in contrast with existing LP methods that cannot generalize outside the modelled problem. To the best of our knowledge, this is the first model to successfully demonstrate transfer learning on metabolic data.”

In summary, we don't want to make essential computation more efficient, but rather to enable transfer learning on metabolism, motivated by the scarcity of the data and the difficulty in modelling complex phenomena. Regarding processing reactions in parallel, this is just to show computationally the complexity that our problem involves. With LP-based methods, essential computation involves individual knockout over all reactions (solving one LP for each reaction), while here it is performed through a single-pass of the model. While one approach is not necessarily more efficient than the other, it gives a good insight into the complexity of the problem.

---

Question 1 Answer:

We believe that part of the question was answered in the previous question.

Regarding the advantages of our method against LP-based, our method enables data-driven prediction and transfer learning without explicitly modelling the biological phenomena (which is indeed impossible to perform with LP-based optimization).

Regarding the application to other metabolic network problems, we believe that the metabolic transformer trained with our objective should potentially capture enough information about the metabolic network to serve as a useful backbone for other different tasks. Our work shows good performance approximating essential reactions (which is indeed a convex optimization problem). Other works have proven theoretically that GNNs are powerful enough to approximate convex optimization solutions (see [1]), however, these are bounded by the number of message-passing layers. From our work, we believe that:

• The metabolic transformer can potentially serve as a backbone for transfer learning on other metabolic-related tasks.
• From our results, we believe that the architecture GPS+LapPE+Dir-Gated could serve as an adequate architecture also for other metabolic network modelling tasks.

Regarding sampling feasible flux vectors, we recall that LP can have from zero to an infinite number of solutions. Therefore, a deterministic model is not able to sample different fluxes. We believe, however, that the model studied here can help towards this goal, nevertheless, different training and inference schemes are needed for such task. We thank the reviewer for their question.

References:

• [1] Qian et al. Exploring the Power of Graph Neural Networks in Solving Linear Optimization Problems

---

We hope we have addressed clearly the concerns raised by the reviewer and that we have helped in the clarification and evaluation of the manuscript. We are open for further discussion if required.

Question 4 Answer: We agree that more baselines can be provided with non-dedicated models. However, the ablation studies in Appendix D, and theoretical results from Appendix B already show that message-passing alone or undirected graph transformers are limited in their application to metabolic modelling tasks. Moreover, the work from FlowGAT already showed that their model exceeded the performance from GAT.

---

Question 5 Answer: Thank you, an answer is provided to this in Question 3

---

We hope we have adressed clearly the concerns raised by the reviewer and that we have helped in the clarification and evaluation of the manuscript. We are open for further discussion if required.

We thank the reviewer for evaluating our work and providing insightful comments. Below we address the concerns raised by the reviewer:

---

Weakness 1 Answer: Thank you for identifying this weakness. We are sorry that this was not clear enough. There is a substantial difference between FlowGAT method and ours. FlowGAT is an ad-hoc method optimized particularly for transductive learning on only 1 microorganism, with around 50 test samples. Our method is the first contribution to generalize essential prediction among 100s of microorganisms with 1000s of samples for each node in an inductive manner. We believe that this enables the first step towards the construction of large-pretrained metabolic network models.

---

Weakness 2 Answer: Thank you for remarking on this. It is true that works exist that propose directed graph transformers. However, as far as we know, this is the first work that combines the directional message-passing from [2] with existing graph transformer architectures. Additionally, this work also shows experimentally, that undirected positional encodings + directed message passing exceeds directional positional encodings, which seems at first as a totally counterintuitive idea. This is a very valuable insight that we have not seen in any other work.

In addition to this, we want to note that, the ultimate goal of this work is not to propose a novel “directed graph transformer”, but to answer the question “What is the best neural network model for metabolic modelling?”. Our claims that the proposed model is the best one for our case are supported both theoretically and experimentally in our work.

Refernces:

• [1] Rossi et al. Edge directionality improves learning on heterophilic graphs, 2024.

---

Weakness 3 Answer: Thank you for noting this which is indeed interesting. There is however no issue with the model evaluation. The values reported are correct. We answer to this in the questions below.

---

Question 1 Answer: The architecture shown in “Transformers Meet Directed Graphs” is the equivalent of SAT+MagLapPE+GCN in our work. In our work, besides that, we also investigated, the GPS architecture (not only the SAT), we studied both laplacian (LapPE) and magnetic laplacian (MagLapPE) as positional encoding (not only MagLapPE), and we included directional message passing (DirGCN, DirGAT, etc) in the graph transformer. This differs with the work from “Transformers Meet Directed Graphs”, in that directionality information is provided in message-passing, not only in positional encodings.

---

Question 2 Answer: Thank you for your question. In Sec 5, we used the best-performing model according to the results from Table 1, this is, GPS+LapPE+Dir-Gated. We have included a clarification sentence in the paper to indicate this:

“As metabolic transformer, here we used the best-performing model according to Table 1, this is, GPS+LapPE+Dir-Gated.”

---

Question 3 Answer:

We have further investigated this. To provide an insight into this, we have obtained this confusion matrices from the best F1-score models in each case. Below we show the confusion matrix from FlowGAT on synthetic reaction essentiality:

For comparison, here we include the confusion matrix from the metabolic transformer on the same task:

From this, we can see that FlowGAT (almost) always predicts samples as positive class. Since this is the majority class the F1-score obtained is considerably larger. However, the discriminative power of FlowGAT is very poor regarding the negative class, which is why the ROC AUC shows indeed non-learning. Our model, on the other hand, has a lower rate of True Positives, but the performance with the negative class is more balanced which explains the lower F1-score but larger ROC AUC.

To further clarify this in our work, we have included confusion matrices for our experiments in Figures 6 and 7, and we have repeated the experiments computing the opposite problem, this is, predicting non-essentiality (the minority class instead of the majority class). The results, reported in Tables 2 and 3 show a substantial difference between our model's performance and FlowGAT.

Question 1 Answer: Here we provide the PRAUC values.

The table below contains the results from "real essential gene prediction" as explained in the previous comment, and as denoted in the paper:

The table below contains results on "synthetic essential reactions" prediction with similar format:

We believe that both tables are in agreement with the results from F1 and AUC ROC and the discussion provided in the previous answer, this is, that the metabolic transformer discriminative performance is considerably better for both problems.

---

We hope we have adressed clearly the concerns raised by the reviewer and that we have helped in the clarification and evaluation of the manuscript. We are open for further discussion if required.

We thank the reviewer for evaluating our work and providing insightful comments. Below we address the concerns raised by the reviewer:

---

Weakness 1 Answer: While it is true that the base Transformer used in this work is based on prior works (Chen et al., 2022, and Rampasek et al, 2022, as well noted by the reviewer), it is also important to mention that both of those approaches are based on results on undirected graphs. While some works have been proposed to address directed graphs, see [1, 2], there is no agreement in literature on how to model directed graphs, as both approaches are completely different:

• [1] Introduces directionality information with magnetic laplacian.
• [2] Introduces directionality in message-passing.

Since no clear agreement exists on GNN applied to directed graphs, we had to conduct our reported experimentations, combining different message-passing schemes, positional encodings and transformers architectures. We argue that the combination of undirected-graph positional encoding + directed-message passing, which shows the best results in our work, is indeed a novel contribution, since no previous work exists where such architecture is used. We believe that this insight would be especially useful for the graph-learning community and would help towards scaling graph transformers in directed datasets.

Additionally, we want to highlight that the goal of this work is not to contribute with a novel graph transformer architecture, but to answer the question: "what is the best architecture for modelling metabolic networks?". Through the work, we provide theoretical and experimental results to base our claims towards this goal.

References:

• [1] Geisler et al., Transformers Meet Directed Graphs, 2023.
• [2] Rossi et al. Edge directionality improves learning on heterophilic graphs, 2024.

---

Weakness 2 Answer: Thank you for your comment. We are sorry that the contribution of the work is not clear enough. One of the results presented in this work is the finetuning of gene essentiality from a model pre-trained on reaction essentiality. In other words, this is the first work that shows successful transfer learning on metabolic networks. The experimental validation provided is from the data from Escherichia coli. While we would like to extend our experimental results, metabolic data is very scarce and holistic (in vitro) essentiality information is only available at the moment for Escherichia coli.

---

Weakness 3 Answer: Thank you for pointing out that the F1 and ROC results are inconclusive. We have further investigated this. To clarify this matter, below we show the confusion matrix from FlowGAT on synthetic reaction essentiality:

For comparison, here we include the confusion matrix from the metabolic transformer on the same task:

From the above matrices, we can see that FlowGAT (almost) always classifies all reactions as essential. Since the positive class is the majority class here, the F1 score obtained is high. However, the confusion matrix shows that indeed the discriminative power regarding the negative class is very poor (or nonexistent). This result is in agreement with the AUC ROC conclusion, this is, that FlowGAT has poor discriminative performance.

To further clarify this, we have run additional experiments, reported in Tables 2 and 3 in the updated manuscript, where we compute the opposite problem, this is, predicting non-essentiality (the minority class). These further results strongly support that FlowGAT shows very poor discriminative performance and that our model performs considerably better.

See for instance, the Table 2, where (+) denotes predicting the positive class (essentiality), and (-) denoted predicting the negative class (non-essentiality):