InfoSEM: A Deep Generative Model with Informative Priors for Gene Regulatory Network Inference

We appreciate the reviewer’s insightful comments. Below, we address their questions.

1. Do Functionally Similar Genes Share Causal Relationships?: The assumption that functionally similar genes share causal relationships is well supported by biological literature. Evolutionary principles suggest that genes used for related functions often maintain similar regulatory architectures [1,2]. Mendelian randomization studies show that genes affecting related traits frequently share causal variants in lipid metabolism and coronary heart disease [3,4]. Pleiotropy analyses further demonstrate that cell-type-specific genes influence multiple neuropsychiatric traits via overlapping pathways, reinforcing the notion of shared causal roles among functionally related genes [5]. We will include this discussion and relevant citations in the camera-ready version.

2. Evaluation of the Output GRN: We use the established BEELINE GRN evaluation dataset [7], which only evaluates interactions with recorded ground-truth labels, consistent with previous works [8,9].

3. Reliability of BioBERT-Derived Prior: BioBERT has been shown to enhance scRNA-seq information for GRN inference in prior work [10]. In our InfoSEM framework, we treat BioBERT embeddings as an additional prior, with their influence adjustable through a cross-validated hyperparameter that allows the model to reduce their importance if deemed unreliable. Still, to further validate the reliability of BioBERT embeddings in response to the reviewer's question, we performed Gene Ontology enrichment analysis on gene clusters derived from hierarchical clustering of the similarity matrix of BioBERT embeddings, controlling for false discovery rate (FDR). Analysis of 908 genes revealed several significant functional enrichments post-FDR correction (FDR < 0.05), notably in lipid transport (FDR=6.80e-12), DNA replication initiation (FDR=6.95e-07), and RNA polymerase II transcription (FDR=3.49e-07). The enrichment patterns displayed clear biological specificity; for instance, one cluster included APOA1, APOA2, APOA4, and APOB, enriched for cholesterol efflux (FDR=4.12e-14), while another cluster encompassed multiple MED genes related to transcription regulation. These robust enrichments strongly suggest that BioBERT embeddings capture meaningful biological relationships between genes.

4. Training Single Model for Different Cell Types: This appears to be a misunderstanding. We do not train a single model for multiple cell types. Like DeepSEM [8] and other baselines, we train separate models for each cell type to capture cell-type-specific regulatory interactions instead of assuming a universal gene regulatory network.

5. Handling Latent Variables & Network Asymmetry: The impact of latent variables on inferred direction of GRNs is an important research area, as shown in studies like [6]. We will include a discussion of this in our paper and cite [6]/other relevant works. However, our study relies on established datasets with experimentally validated interactions from prior works [7], which we believe minimizes the impact of latent confounders. Addressing confounders comprehensively will be a non-trivial task that warrants a separate paper, beyond the scope of our work.

6. Alignment of Test Set Split and ChIP-seq Ground Truth: Like question 2's response, we adhere to established literature by considering only known interactions from ChIP-seq databases [7,8,9] and splitting them into training and test sets.

7. Choice of 1000 Genes Instead of 5000: We follow BEELINE's established preprocessing method [7], selecting the top 1000 highly expressed genes to enhance the quality of inferred networks and ensure comparability with prior methods [8,9].

In light of additional experiments & clarifications provided in response to reviewer's comments, we kindly request consideration on raising the score. We believe these clarifications address all the comments!

Reference

[1] Evaluating the potential role of pleiotropy in Mendelian randomization studies, 2018
[2] Inferring causality and functional significance of human coding DNA variants, 2012
[3] Diagnostics for Pleiotropy in Mendelian Randomization Studies: Global and Individual Tests for Direct Effects, 2018
[4] Pleiotropy-robust Mendelian randomization, 2018
[5] Selecting causal genes from genome-wide association studies via functionally-coherent subnetworks, 2016
[6] Gene Regulatory Network Inference in the Presence of Selection Bias and Latent Confounders, 2025
[7] Benchmarking algorithms for gene regulatory network inference from single-cell transcriptomic data, 2020
[8] Modeling gene regulatory networks using neural network architectures, 2021
[9] Graph attention network for link prediction of gene regulations from single-cell RNA-sequencing data, 2022
[10] scGREAT: Transformer-based deep-language model for gene regulatory network inference from single-cell transcriptomics, 2024

We thank the reviewer for their thoughtful and constructive comments. We appreciate the recognition of our new benchmark and the value of integrating external priors for GRN inference. We address the reviewer’s questions below and will include them in the camera-ready version.

1. External Priors in Section 2.2: Yes, the external priors refer to gene embeddings and gene relationships. These priors can be derived from various sources, including textual embeddings (e.g., BioBERT), knowledge graphs (e.g., Gene Ontology), or known gene interactions, all of which original DeepSEM cannot use directly.

2. Comparison with SCENIC: We have now included a comparison with SCENIC, which demonstrates similar performance as GRNBoost2. Our InfoSEM-B and InfoSEM-BC outperform SCENIC across all cell lines for AUPRC and across all but hHEP for Hit@1% metric for the cell-type specific datasets.

3. Reason for InfoSEM generalization to D_{unseen}: InfoSEM's main training objective is the gene expression reconstruction rather than the interaction prediction used in the supervised learning framework. This encourages the model to learn the gene-gene relationships from the gene expression data, instead of utilizing the gene-specific bias from the labels, hence contributing to its ability to generalize to the unseen gene set, D_{unseen}. Additionally, using gene embeddings as priors aids our model's generalization to the unseen gene set by leveraging those priors to learn gene relationships.

4. Use of Gene Embeddings from Single-cell Foundation Models: We initially focused on gene embeddings from pretrained language models like BioBERT due to their effectiveness in providing semantic context for various models and tasks (see [1,2]). Furthermore, BioBERT is easier to use; it requires only the gene names already available, unlike single-cell models that necessitate specific preprocessing of scRNA-Seq data.

Following the reviewer's suggestion, we experimented with embeddings from scBERT [3], a foundation model designed for single-cell RNA data, but did not observe significant improvements in performance over InfoSEM-B working with a one-hot baseline instead. We hypothesize that this may result from scBERT's binning of scRNA-Seq data, which could lead to a loss of important information. Additionally, scBERT's embeddings, derived from binned scRNA-Seq data, are similar to those used in InfoSEM already and may not provide complementary insights for the adjacency matrix compared to embeddings like Gene Ontology or BioBERT, which are derived independent of scRNA-Seq data. We believe, though, that exploring other scRNA-Seq foundation model derived gene embeddings presents an interesting avenue for future exploration.

In response to a similar query from reviewer bvmZ regarding the use of external priors from knowledge graphs, we conducted additional experiments using Gene Ontology (GO) embeddings as priors. The AUPRC results, shown below on the unseen-genes benchmark, demonstrate that this further enhances our InfoSEM results on human cell lines, reinforcing the importance of incorporating external priors from independent sources.

5. Data Leakage and Validation Measures: Thank you for the important remark! Unlike seen-gene benchmarks, we ensured there was no data leakage between training and test sets in our proposed unseen benchmarking setup, as the genes between training and testing are different.

We hope our clarifications and additional experiments answer all your valuable comments. In light of the additional experiments and results presented, we kindly ask if the reviewer would consider raising their score. Thank you!

References

[1] scGREAT: Transformer-based deep-language model for gene regulatory network inference from single-cell transcriptomics, 2024
[2] GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT, 2024
[3] scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data, 2022

We appreciate the reviewer’s feedback and acknowledgment of our real-world benchmark as a key advancement in GRN inference, along with their praise for our theoretical clarity. Below, we address their questions and provide additional results, which we will include in the camera-ready version.

1. Ablation studies (BioBERT embeddings+LR, AE loss): Thank you for the suggestion! We have now tested BioBERT embeddings in LR on unseen genes benchmark. Results below show that BioBERT+LR performs similarly or slightly better than the 1-hot LR baseline in paper. This is expected, as in the unseen-gene setting, both LR baselines (with 1-hot or BioBERT) can't leverage gene-specific biases for GRN inference. The slight improvement with BioBERT embeddings highlights the informativeness of the embeddings over 1-hot encoding. However, both these baselines perform considerably worse compared to our InfoSEM-B/InfoSEM-BC models, underscoring the benefits of our proposed framework as a whole beyond simply incorporating BioBERT as prior.

We explored the AE loss function (DeepSEM-AE) suggested by the reviewer, but it performed significantly worse than DeepSEM and our InfoSEM-B and InfoSEM-BC models due to DeepSEM's assumption of a Gaussian distribution for Z [1, 2], which requires regularization in the variational Bayes loss. Our InfoSEM-B and InfoSEM-BC naturally serve as ablation studies on the baseline DeepSEM. InfoSEM-B serves as an ablation study by replacing DeepSEM's Laplace prior over A with our informative prior, while InfoSEM-BC adds Aˡ and its informative prior to InfoSEM-B. We are open to any further ablation studies if the reviewer suggests.

2. Evaluation on Original Benchmarks: Thank you for your comment! We have now included InfoSEM on the original seen-gene benchmarks in our response to reviewer bvmZ, with results presented in the accompanying table. Our results show that for traditional seen-gene benchmarks, simple supervised models (One-hot LR & MatComp) perform best by leveraging gene-specific biases without using scRNA-seq data, a fact which our work highlights for the first time and is a key contribution of our work. Even in this context, InfoSEM-BC maintains a competitive balance, avoiding overfitting gene-specific biases while performing better than unsupervised baselines. However, the key takeaway is that for seen-gene benchmarks, simpler models that exploit gene-specific biases are sufficient, and hence, we propose to evaluate advanced machine learning models on unseen-gene benchmarks.

3. Limited architectural innovation over DeepSEM: We emphasize that our contribution is not on architectural innovation over DeepSEM, but in extending it within a principled variational Bayes framework to leverage informative priors (e.g., gene embeddings from language models) or available ground truth information without overfitting to gene-specific biases, an aspect not explored so far. Additionally, we show for the first time that existing supervised learning models tend to leverage gene-specific biases in the dataset, and hence, we introduce a new unseen genes benchmark to provide a reliable testbed for evaluating generalizability of GRN inference methods. As the reviewer themselves noted, our benchmark for evaluating generalizability to unseen gene relationships is a crucial advancement for GRN inference, and we are happy that it has been well received.

4. Including citations to proposed references: We thank the reviewer for additional references. We have already cited the first and third references in the paper and will include the second one in the camera-ready version too. These citations will enhance the discussion and provide further context to our work.

We hope that our clarifications and additional results address your concerns! We kindly request the reviewer to consider raising their score, taking these into account.

References

[1] DAG Structure Learning with Graph Neural Networks, 2019
[2] Modeling gene regulatory networks using neural network architectures, 2021

We appreciate the reviewer’s valuable feedback and acknowledgment of our work’s novelty and rigor. We are happy that our analysis of biases in supervised GRN benchmarks and InfoSEM’s performance on the unseen-gene benchmark were well received. Below, we address the reviewer’s questions and provide additional results, which we will include in the camera-ready version of the paper.

1.Evaluation in Traditional Seen-Gene Setting: Thank you for the important point! We now show InfoSEM's performance in this setting below for all datasets (cell-type-specific ChIP-seq targets).

The results show that for traditional seen-gene benchmarks, simple supervised models (One-hot LR, MatComp) without utilizing scRNA-seq perform best by leveraging gene-specific dataset biases, a fact which our work highlights for the first time and is a key contribution of our work. Even in this context, InfoSEM-BC maintains a competitive balance, avoiding overfitting to the gene-specific biases while performing better across multiple datasets compared to other unsupervised methods (GRNBoost2, DeepSEM). However, we again emphasize that the key takeaway is for seen genes benchmark, simple supervised models (One-hot LR, MatComp) that exploit gene-specific biases from known interactions are sufficient even without scRNA-seq. Hence, we propose to evaluate more advanced machine learning models on unseen-gene benchmarks where simple models entirely fail as shown in paper.

2.Gene embeddings from knowledge graphs: Thank you for the suggestion! Our flexible framework can incorporate embeddings from other sources, such as knowledge graphs. To showcase this, we now did additional experiments using Gene Ontology (GO) knowledge graph embeddings as priors (AUPRC shown below on unseen-genes benchmark), which improve our InfoSEM-B (BioBERT) results even further on human cell lines but lag behind our original InfoSEM-B (BioBERT) on mouse cell lines. This is because GO knowledge graph used is specific to human cells. Unlike BioBERT, which is more generic, knowledge graphs have to be chosen carefully depending on the context.

3.Compute Efficiency: We have now included comparison of run times (in seconds) for different configurations of (num_cell, num_gene) for InfoSEM and other baselines. Our results demonstrate that InfoSEM is faster than popular supervised learning baselines and only slightly slower than DeepSEM, while consistently outperforming them in terms of performance in unseen genes benchmark.

4.Rationale for Aᵉ and Aˡ splitting of interaction matrix: By splitting the interaction matrix into two components—Aᵉ for the magnitude and Aˡ for the logit—we create a setup where both components can incorporate prior biological knowledge independently so that any possible misspecification of one prior will not affect the other. The prior on Aᵉ, informed by embeddings, acts as a starting point, but the model can adjust its MAP estimate based on data. Similarly, the prior on Aˡ allows for a probabilistic treatment of known interactions, but leaves room for learning from the data where these interactions might not be fully observed.

We hope these clarifications and new results address the reviewer’s concerns and enhance our paper's contributions. We kindly request consideration for a score increase based on this. We are happy to address any further suggestions!