The OMG dataset: An Open MetaGenomic corpus for mixed-modality genomic language modeling

Thank you for your thoughtful review. We are glad that you find OMG as "a major contribution to the field" and that gLM2 protein-protein coevolution marks a "novel and compelling contribution to biomolecular modeling".

Insufficient information is provided about the evaluation methodology and associated metrics.

Thank you for this suggestion. We have added the following sentences to Section 4.1: "We benchmark gLM2 on the Diverse Genomic Embedding Benchmark (DGEB) (West-Roberts, 2024). DGEB is a comprehensive benchmark that evaluates model representations across diverse taxa and 18 tasks representing multiple axes of biological function, such as evolutionary distance similarity, remote homology prediction, enzyme classification, and retrieval sensitivity."

gLM2 can not match ProGen2 on amino acid (AA) tasks (Appendix J), which is not reported in the current manuscript. The authors should either include these results or justify their omission.

The primary focus of this paper is to motivate OMG as a pre-training dataset. We focus our comparison with the ESM2 series to minimize conflating other factors such as Progen2's autoregressive architecture and causal language modeling task. We design gLM2 to closely match ESM2 (same number of layers, hidden dimensions, pre-training objective). In order to include Progen2 as an appropriate baseline, we would need to train an autoregressive model on OMG, which is beyond the scope of this study.

The authors should include evaluations on standard benchmarks such as FLIP or ProteinGym.

Thank you for the suggestion. We have added ProteinGym results in Appendix M. Because the DMS task is strictly a single-protein task (without context), we benchmark gLM2_650M after fine-tuning for one epoch of OMG_prot50 dataset (Table 1) and we find that gLM2 performs slightly worse than ESM2 on Avg Spearman score (0.384 vs 0.414). We note that the ProteinGym benchmark includes eukaryotic sequences, which are poorly represented in the OMG dataset, and likely impacts gLM2 performance.

For the experiment with 2ONK, the comparison would benefit from including NT2 and ProGen2.

We choose ESM2 and Evo as pLM and gLM baselines because the Categorical Jacobian relies on per amino-acid/nucleotide resolution, which NT models do not have due to k-mer tokenization. We expect Progen2 to perform similarly to ESM2 as neither has been trained on multi-gene contexts.

While sections 4.4 and 4.5 present well-analyzed individual cases, for contact prediction in particular, more examples are needed to demonstrate that gLM2 consistently outperforms alternative methods.

To address the reviewer's feedback of presenting individual cases on Section 4.4, and 4.5, we added supplementary information expanding on these two analyses in Appendix P, Additional File 5 and Additional File 6.

Is gLM2 trained on contig sequences containing both nucleotide sequences (IGS) and protein sequences?

Yes, gLM2 is trained on multi-modal contig sequences with up to 4096 tokens. This limits modeling between proteins in the same context (in the case of microbial systems, interactions often occur with co-located genes).

Can gLM2 predict contacts for Protein hetero-multimer interactions (also RNA-Protein interactions and DNA-Protein interactions)?

The 2ONK example from Figure 4 is an example of hetero-multimer prediction using gLM2. For homo-multimers, we were not able to decouple inter-chain interactions from intra-chain interactions in Categorical Jacobian analysis of a single sequence. This could be an interesting avenue for future work. For RNA/DNA-protein interactions, we showcase a putative RNA-protein interaction in Appendix O, however there currently exists no structural groundtruth for the L10 leader RNA and RplJ complex. In our future work, we hope to follow up on RNA/DNA-protein interactions.

How ESM2 FLOPs were estimated?

ESM2 pretraining flops were computed in xTrimoPGLM (https://arxiv.org/pdf/2401.06199) Table 4 (page 83).

Questions regarding model hyperparameter design

As mentioned above, we choose architecture parameters to match the ESM2 series.

Thank you to the authors for your responses. I appreciate the effort in curating such the dataset for the community, which I anticipate will be useful. However, while the paper claims its primary objective is to introduce the OMG dataset, over half of the content discusses the gLM2 model. This seems contradictory to me. Given the current structure of the paper, I believe the gLM2 model requires more rigorous evaluation and should significantly influence the review scores. If the focus is intended to be on the dataset, I suggest reformatting the paper.

Thank you for your thorough review and feedback. We appreciate that you find OMG and gLM2 provides "important and novel contributions to biologic sequence modeling".

Authors should be more explicit about the application for eukaryotic sequence modeling.

Thank you for the suggestion, we have added the following sentence as a limitation to metagenomic datasets (Section 1): "While pretraining on metagenomic datasets allows models to leverage rich molecular diversity and genomic context, these models are most suitable for microbial genomes and may result in out-of-distribution effects on eukaryotic sequences."

It would be interesting to see performance of gLM2 in areas where more and diverse training data is of importance, such as remote homology, or where the multi-modal training strategy is helpful, such as contig-level prediction of taxonomy.

We agree that other applications of gLM2 could be explored, in addition to the applications we introduce for protein-protein interaction (Section 4.4) and regulatory syntax (Section 4.5). We note that the DGEB benchmark includes a diverse set of tasks; notably operon prediction, paralog matching, and phylogenetic distance prediction tasks, where training with genomic context boosts performance (Appendix K). We agree that taxonomic classification is an interesting application, in Fig 1A we show that gLM2 indeed learns distinct clustering of taxonomy via mean-pooled example-level representations without supervision, which motivates future exploration of this application.

Authors claim "more efficient scaling", however, they only show two training flops.

We agree that the scaling efficiency claim is not sufficiently strong with 2 model scales, and we have removed this claim from Section 4.3.

Minor comments...

1. We intended to say "start a new contig" as element refers to a protein (CDS) or intergenic sequence (IGS). When an invalid sequence is found, we remove it and start a new contig.
2. Thank you for pointing this out. We added a definition of DGEB in Section 4.1
3. The caption defines each "n" as the number of ground-truth contacts for each example.

Thank you for your review and feedback. We appreciate that you find OMG to be a "very useful resource for the community" and that we "demonstrate its utility in training a novel mixed-modality genomic language model".

gLM is poorly described and not validated

In section 4.3, we validate gLM2's performance on the DGEB benchmark (Section 4.3, Figure 3). As other reviewers mention, DGEB is not well described in the paper, and we have added the following Section 4.1: "We benchmark gLM2 on the Diverse Genomic Embedding Benchmark (DGEB) (West-Roberts, 2024). DGEB is a comprehensive benchmark that evaluates model representations across diverse taxa and 18 tasks representing multiple axes of biological function, such as evolutionary distance similarity, remote homology prediction, enzyme classification, and retrieval sensitivity." Additionally, we added results on the ProteinGym benchmark in Appendix M. We note that the ProteinGym benchmark includes eukaryotic sequences, which are poorly represented in the OMG dataset, and likely impacts gLM2 performance.

Can you give us some sense of how you'd go about doing the validation for [protein-protein interaction]? Where it struggles, why do you think it struggles?

We conducted the same PPI analysis on 32 evolutionarily conserved complexes with PDB structures previously characterized in [1] and [2]. We include all results in Appendix P in Additional File 5. We find that for 16 complexes we detect at least 10 true positive co-evolutionary signals that match inter-chain contacts in PDB structure. However, not all co-evolutionary signals align well with physical contacts. We find that when there are distinct and localized patterns, gLM2 is more likely to capture such patterns, whereas, when these patterns are diffuse and contacts are spread out across the interface, gLM2 covariance metric is less accurate in detecting exact contacts. Additionally, previous studies [3, 4] have shown that covariance signal does not always align with PDB structures due to the dynamic nature of protein structure and protein-protein interactions, and the averaging effects categorical jacobian analysis results in for multiple stable conformations that may be present across distant taxons. Additionally, for structures that assemble in homo-multimers, PDB contacts do not correspond well with the combined covariance signal compressed in a single sequence. We refer to this additional analysis in section 4.4.

Can you give us some sense of how you'd go about doing the validation for [regulation]? Where it struggles, why do you think it struggles?

We have repeated the same regulatory element analysis in Figure 6 in E.coli K-12 MG1655 (U00096) genome, for which terminator and promoter annotations exist on EcoCyc. We identified 26 intergenic regions in the genome containing at least 1 promoter region and 1 promoter (sigma factor binding region) that is at least 500 base pairs long. We include all results in the zip file, where the terminator (green) and promoter (red) regions correspond with covariance signals directly. We include this zip file as Appendix P in Additional File 6. We demonstrate that the pattern observed in Figure 6 is recapitulated across the E.coli K-12 genome. We refer to this additional analysis in Section 4. 5.

[1] https://openseq.org/cplx.php?sort=prob&order=DESC&mode=pdb

[2] https://pmc.ncbi.nlm.nih.gov/articles/PMC4034769

[3] https://www.pnas.org/doi/10.1073/pnas.2406285121

[4] https://elifesciences.org/articles/02030)

Thank you for taking the time to review our paper. We are glad that you find our paper provides background on the importance of releasing publicly-available metagenomic data for pretraining, as well as the challenges of balancing and filtering large-scale, unstructured metagenomic datasets.

We appreciate your questions regarding data filtering, and we have updated our manuscript to clarify filtering steps, and provide supporting figures for specific thresholds used.

How many elements are removed for the edge-element removal?

Thank you for this question. We have updated Section 3.1 with this information: "This filtering step removes ~1.4B genomic elements likely to be poor quality, partial sequences with high likelihood of assembly errors."

In the assembly filtering, a histogram of the number of invalid characters, why a threshold of 20% was chosen?

Thank you for raising this point. This filter is designed to remove genomic elements with poor assembly. To clarify the impact, we add a histogram to Appendix D, which shows distribution of invalid characters prior to filtering. Appendix D includes the impact of filtering: "The assembly quality filter removes elements containing more than 20% invalid characters, resulting in 0.004% of CDS and 0.2% of IGS being filtered from OMG"

For the element length-based filtering section, what is the impact of why the specific threshold was chosen?

This filter is designed to exclude regions with poor gene-calls. To clarify the impact, we added to Section 3.1 " This filtering step removes 2.5e-5% of CDS , and 1e-4% of IGS elements from OMG". Appendix C shows the length distributions of CDS and IGS after filtering, and because only a small fraction of long tail data is removed, the overall data distribution is not affected. The threshold of 4000 bp for IGS sequences was determined to be able to include bonafide long IGS elements (e.g. 23S rRNAs that are ~3000bp long in microbial genomes). We used the threshold of 15,000 AAs for CDS sequences because there is a drop off in the frequency of proteins at 15,000 AAs in microbial genomes as previously identified in West-Roberts et al, 2023 [1] Fig 1A .

What is the meaning of "(>3 CDS >7...?

This is referring to the filtering step in Section 3.1 "Contig length-based filtering and preprocessing", where we remove contigs lacking multiple genes. For N/X filtering, because we discard the invalid element and start a new contig, this may result in contigs which do not meet the length filter. In this case the contig is discarded.

The importance of the results is not explained.

Our results from training gLM2 on the OMG dataset highlight the importance of (1) OMG's multi-gene context (unlike protein only datasets like UniRef50 used by ESM2) and (2) OMG's scale and diversity (currently the largest and most diverse publicly available dataset for pretraining).

(1) Multi-gene context allows zero-shot gLM2 prediction of protein-protein interactions. Understanding protein-protein interactions is critical for biological systems, because it defines molecular function. Existing sequence-based PPI prediction tools require supervision (e.g. PIPR [2], D-SCRIPT [3], Topsy-Turvy [4]), meaning they do not perform well for novel types of PPIs; this is a significant gap in the field because PPI data are extremely sparse. Furthermore, these tools cannot resolve interacting residues between proteins, an important aspect of PPI prediction task. Structure-based tools (e.g. Alphafold-multimer) typically assume that input proteins interact and also rely on supervision of known PDB interfaces. Therefore these tools have lower utility in predicting the presence of PPI or discovery novel PPIs in less characterized protein complexes. In our paper, we show gLM2 is capable of detecting covariance signal corresponding to PPI interfaces, a capability that existing foundation models lack.

(2) OMG's scale and diversity is shown by clustering at 50% sequence identity, resulting in >3x more clusters than UniRef50 (OMG_prot50 described in Section 3), as well as gLM2's strong performance on DGEB (Figure 3). We believe that OMG, the largest and most diverse publicly available genomic dataset to date, will be a valuable resource to the community.

[1] https://www.biorxiv.org/content/10.1101/2023.11.21.568195v1

[2] https://academic.oup.com/bioinformatics/article/35/14/i305/5529260

[3] https://www.cell.com/cell-systems/fulltext/S2405-4712(21)00333-1

[4] https://academic.oup.com/bioinformatics/article/38/Supplement_1/i264/6617505