LLM4GRN: Discovering Causal Gene Regulatory Networks with LLMs - Evaluation through Synthetic Data Generation

We are grateful to reviewer ftKG for their detailed review of our work.

Question 1: Could the authors elaborate on how the performance of the LLM-based GRN inference might be affected by the quality and representativeness of the data used to train the LLMs?

We thank the reviewer for the relevant observation and point to the "Discussion and Limitations" section. We have also included the relevant information from the "Discussion and Limitations" to answer Q5 to the second reviewer, who raised a similar question.

Question 2: It would be helpful if the authors could discuss potential future work on refining the GRN inference process, especially in addressing the noise and specificity issues mentioned in the biological plausibility analysis.

We appreciate the reviewer’s suggestion and agree that further refinements are needed.W However, we plan to generate cell type-specific TFs to cover all important TFs for each cell type in each dataset and generate more biological driven GRNs.One of the directions we envision is learning causal graphs for each cell type separately, as the graphs can be cell-type specific. However, we emphasize the biological plausibility of our proposed method for the PBMC data set is better than the state of the art.

Question 3: The paper mentions the use of GPT-4 and Llama-3.1-70B models. Are there any plans to compare the approach with other LLMs or to investigate the impact of model size on the performance of GRN inference?

We appreciate the suggestion. Our results demonstrate strong performance with open-source model, which encourages further exploration of even smaller models. We used Llama-3.1-8b to generate potential TFs and observed that there existed 82.7% overlap for PBMC, 94.2%overlap for BM, and 77.6% overlap for BM. Additionally compared to Llama-3.1-70b model, we observed Llama-3.1-8b struggled with instruction following. In contrast, Llama-3.1-70B demonstrated robust performance while maintaining a good balance between computational cost and accuracy, making it a favorable choice for our GRN inference tasks.

We hope our reply clarifies the reviewer's concerns and would be happy to answer any further question the reviewer may have.

We appreciate reviewer h4j7’s detailed review of our work. We provide in-line responses to your questions below:

Question 1&2:

• How might the LLM-based approach handle incomplete pathway information or limited TF-gene knowledge in certain datasets?
• How does pathway information specifically improve GRN inference compared to simpler approaches that omit it?

We appreciate the reviewer’s question and would first like to clarify our understanding of “pathway information” to ensure we address the concerns accurately. We suppose the reviewer is referring to curated biological knowledge, such as well-established interactions between transcription factors (TFs) and target genes. In our approach, pathway information enhances GRN inference by providing biological context that guides the statistical algorithm toward more context-specific and biologically relevant network predictions. Our results demonstrate that this approach produces synthetic data that is not only more faithful to biological constrains but also more biologically plausible compared to methods that omit it (see below figure for Stage 1 non-causal synthetic dataset comparison). Moreover, unlike traditional methods that depend on manually curated, static knowledge bases -- a process that is both labor-intensive and prone to human error -- our LLM prior-based approach dynamically integrates pathway data by consulting multiple knowledge sources.

Additionally, we propose a hybrid strategy that leverages LLM to gather information about potential TFs and combines this with data-driven statistical inference to address gaps in pathway information.

Weakness 1 & Question 3:

• Limited Dataset Variety: Only three datasets (PBMC-All, PBMC-CTL, and Bone Marrow) are used, which limits generalizability to broader biological contexts.
• What criteria were used to select the three datasets, and how might the results generalize to datasets with different biological characteristics?

We would like to clarify that while our study focuses on three datasets, they were chosen to represent sufficient diversity across key biological contexts. Specifically, the PBMC-ALL dataset is a human dataset, the Bone Marrow dataset represents mouse data, and PBMC-CTL focuses on a specific cell type. This selection allows us to capture variations across species (human vs. mouse) as well as across different biological and cellular granularity, providing meaningful insights and demonstrating the versatility of our approach. Additionally, the datasets were selected to enable direct comparison with the GRouNdGAN paper since we considered the approach proposed by Zinati et al. as a baseline (current state of the art).

Question 4: How does the complexity of LLM4GRN impact reproducibility, and are there ways to simplify the model without losing performance?

To maintain simplicity and reproducibility, we make sure that all the experiments have the fixed seed, the code is packaged and well documented. We described the experimental setup is described in detail in Appendix A. We also deliberately evaluated Llama so it can be run locally without the need for external cost associated with Open-AI GPT-4 models to ensure full reproducibility. Additionally, we utilize a straightforward process where the LLM is prompted to generate a list of transcription factors (TFs), which are then seamlessly incorporated into the GRouNdGAN workflow and will release the code after publication. We also deliberately evaluated Llama so it can be run locally without the need for external cost associated with Open-AI GPT-4 models to ensure full reproducibility.

Question 5: Can you discuss how robust the model is to potential biases in the LLM-based knowledge base, especially given the dependency on prior information?

We agree that the LLMs can be prone to biases in the data and have provided the discussion in the paper. Due to the lack of space, we point the reviewer to the Section 5.4 “Discussion And Limitations”.

Question 6: Why do you believe Llama performs better than GPT-4 in certain settings, and what does this imply for LLM-based GRN inference?

We thank the reviewer for raising this question. To address this point, we conducted an additional ablation study to investigate differences between Llama and GPT-4 (It can now be accessed in the supplementary materials). Our findings indicate that Llama’s superior performance in certain settings is primarily due to its ability to propose a more comprehensive initial list of transcription factors (TFs). This larger TF list provides the statistical algorithm with a broader foundation, enabling it to better infer relevant relationships from the data. However, we also note that Llama underperforms in settings where the statistical algorithm is not included in the pipeline. This suggests that Llam’s strength lies in its capacity to complement the statistical inference process, rather than functioning as a standalone tool.

We thank reviewer HajQ for their detailed review of our work.

Question 1&2:

• Please consider adding other orthogonal direct ways of validating GRNs, such as the one suggested above by testing the exact same probing strategy on whether it can identify context specific variations of a well known and validated TF GRN
• Please include an orthogonal method to validate the context-specific TF list, such as distinguishing TFs based on uncorrelated expression patterns across distinct cell types, as mentioned in the Weaknesses section.

We appreciate the reviewer’s insightful suggestion to include additional orthogonal validation strategies for GRNs, such as testing our probing on context-specific variations of a well-known and validated transcription factor GRN. However, we acknowledge that the lack of reliable ground truth GRNs presents a significant challenge in such validation efforts. Our approach is specifically designed to address this limitation by focusing on practical utility and demonstrating robust performance in the absence of established ground truth. Furthermore, our results are supported by both statistical and biological evaluations. They demonstrate the effectiveness of our method, combining GPT-4 knowledge base with a statistical GRN inference approach, compared to the baseline.

Weakness 1: The primary limitation is the lack of objective metrics for validating the inferred GRNs. While the authors use synthetic data generation as an indirect evaluation method, results vary significantly across datasets, with the GPT-based GRNs performing well on PBMC_ALL but showing weaker, near-random performance on other datasets (as measured by MMD and RF AUROC). These results indicate low systematic performance on the author's suggested standards.

Thank you for pointing out this concern. We appreciate the opportunity to clarify. We would like to highlight that the performance of our approach remains consistent across most datasets analyzed. Specifically, both PBMC-ALL and PBMC-CTL demonstrate that GPT+GRNB2 is the top-performing method. While there is a minor variation in performance for the Bone Marrow dataset—specifically in terms of Cosine, Euclidean, and MMD distances between Human+GRNB2 and GPT+GRNB2—these differences are not substantial enough to indicate a systemic issue. For example, when examining the Euclidean distance score, Human+GRNB2 achieved 7+- 5 compared to 78 +- 4 for GPT+GRNB2, demonstrating a marginal 1-point difference in favor of the existing method. In contrast, for the PBMC-CTL dataset, GPT+GRNB2 outperformed with a more notable margin, achieving 57 +- 6 versus 65 +-6 for Human+GRNB2, reflecting an 8-point difference in our method’s favor. Additionally, as noted in the paper (Line 484), the observed lower performance of large language models (LLMs) on mouse data aligns with the limited availability of encoded information for such data, a factor that we discussed. We hope this context helps clarify the observed results and highlights the robustness of our approach across different datasets.

We hope our reply clarifies the reviewer's concerns and would be happy to answer any further question the reviewer may have.