PhyloGen: Language Model-Enhanced Phylogenetic Inference via Graph Structure Generation

Weakness:

W1: Table 5 Clarification: Tab. 5 presents results for the DS1 dataset, not an average across eight datasets. The impact of removing KL or S has already been shown in this table.

Figure 6 Explanation: Fig. 6 shows ablation results for the DS1 dataset, chosen for its representativeness, as many recent studies also use this dataset. This is a common practice in the field. Due to time constraints, we have added ablation results for some other datasets in the attached file. We can continue to add results if you feel it is necessary.

Performance Explanations: The complete ablation experiments are shown in Tab. R1 indicates a significant performance decline when KL or S is removed, underscoring their importance in model regularization and overfitting prevention. Replacing our distance matrix with Euclidean or cosine distances significantly lowers MLL, indicating these conventional distance methods fail to capture complex evolutionary patterns. Using one-hot encoding exhibited the lowest MLL, highlighting the superiority of our feature extraction module and the pre-trained language model in representing complex biological data.

W2: Aligned vs. Unaligned Sequences

Thank you for your feedback. There might be some misunderstanding regarding the concept of "aligned sequences". In our manuscript, we refer to equal-length sequences, not multiple sequence alignments (MSA). This definition is consistent with recent methods like PhyloGFN and GeoPhy, which refer to "pre-aligned sequences" as uniform-length sequences, as seen in statements like "These datasets feature pre-aligned sequences" or "Let Y be a set of aligned sequences with length M from the species.". Our method analyzes raw sequences directly, avoiding biases introduced by alignment algorithms, thus preserving the original sequence diversity and more accurately reflecting sequence variation and evolutionary relationships.

W3: Writing Issues

Thank you for highlighting the writing issues. We will thoroughly review and improve the manuscript for clarity.

Questions:

Q1: MLL:

Thank you for your question. Let me clarify these concepts:

PhyloGFN:

1. Evolutionary Model: PhyloGFN uses an evolutionary model to compute the marginal log likelihood (mll) through trajectory sampling and importance sampling. This involves generating trajectory data, calculating the forward and backward path log probabilities, and computing the log scores for each generated tree.
2. Aligned Sequences: PhyloGFN does not rely on traditional multiple sequence alignment (MSA). Instead, it uses sequences of equal length, which do not require MSA for alignment.

Our Method:

1. Evolutionary Model: Our mll computation does not involve PhyloGFN's evolutionary model. We use a mutation model based on mutation probabilities, not the trajectory and importance sampling methods used in PhyloGFN.
2. Aligned Sequences: Our method does not require either MSA or equal-length sequences. PhylGen directly analyzes raw sequence data, avoiding the computational overhead and potential errors introduced by sequence alignment. Thus, it more accurately reflects sequence variations and evolutionary relationships.
3. Consistency with Claims: Our method aligns with the statement that sequence alignment is unnecessary. By handling unaligned, variable-length raw sequences, PhylGen more accurately reflects sequence variation and evolutionary relationships. This makes our method distinct from others that rely on equal-length sequences, like PhyloGFN and GeoPhy, and consistent with the claims made in our article.

Q2: Memory Usage

Thank you for your question. We used an NVIDIA A100-SXM4-80GB GPU for our experiments. However, the choice of hardware is not crucial for running our training algorithm. Detailed memory usage can be found in the General Response and the Tab. R2.

Dear Reviewer,

We sincerely appreciate your efforts and valuable feedback. If you are satisfied with our responses and our improvements, please consider updating your score.

If you need further clarification, please don't hesitate to contact us. We are grateful for your time and look forward to your response!

Weakness: Formulation Clarify:

Thank you for your valuable feedback. Regarding the introduction of the R term in Eq. 7, we state the following:

Introduction of R: R represents the posterior probability in the second part of the variational network. Its inclusion aims to enhance model expressiveness by capturing the underlying structure and parameter dependencies more effectively.

Expression of $q(\tau(z))$: Detailed derivations in Appendix, Equations 19-31 clarify that $q(\tau(z))$ is indeed Q(z), indicating that our model includes the KL divergence between R and Q, which is a crucial component. We will improve the presentation and explanation of these concepts in the revised manuscript to make them clearer.

We will also enhance the overall presentation of the paper to reduce any confusion.

Question：

Q1: Overlooked Sequence Features

Thank you for your question. The key sequence features we refer to involve the semantic aspects of genetic information, which are often overlooked by traditional distance-based methods. These methods typically rely on simplified one-hot encoding, which fails to capture the complex interactions and biological functions within sequences. In contrast, our approach utilizes pre-trained language models to extract richer representations from the sequence context, effectively capturing complex relationships and semantic information. This enhances the accuracy and flexibility of phylogenetic inference.

Q2: Differentiability of NJ Algorithm

Thank you for your question. You are correct that the Neighbor-Joining (NJ) algorithm is not differentiable. However, in our approach, NJ is only used to generate the initial tree topology and does not require differentiation. The overall model is trained end-to-end, with differentiation occurring in other parts of the model, not within the NJ algorithm.

Q3: Clarification on Section C.1:

Thank you for pointing out this issue. The notation $p(\tau(z))$ in the main text is incorrect and should be corrected to match the accurate representation in the appendix, specifically on line 540. Additionally, $q(\tau(z), B_\tau)$ corresponds to the distribution detailed in Eq. 19. We will ensure these notations are consistent and clear in the revised manuscript.

Q4: ELBO Metric:

Thank you for your insightful comment. You are correct that ELBO alone may not fully capture the model’s inference ability, which is why we introduced the Scoring function. Additionally, it is important to note that our baselines and recent methods also use ELBO and MLL, so we must use these metrics for fair comparison.

Q5: LLM Embeddings and Tree Construction:

Thank you for your question. The embeddings generated by the LLM are primarily for feature extraction, not for direct tree construction. The initial topology and the subsequent adjustments made by the first module are essential to ensure the embeddings are constrained to a Gaussian distribution via the encoder, as assumed in our model. Without this step, the embeddings might not converge properly, and the phylogenetic tree construction could be less accurate.

Q6: Scoring Function:

Thank you for your question. The scoring function maps each leaf node in the latent space to a score, representing their contribution to the model's overall performance. The function is jointly optimized with the ELBO objective to ensure the directionality of parameter updates during training. We minimize the scoring function to enforce low-rank representations in the matrix, which helps construct a well-structured initial tree, leading to faster convergence and improved accuracy.

Limitations:

L1: Computational Resources

1.1 Monte Carlo: PhyloGen does not use traditional MCMC to sample from posterior distributions of tree topologies and branch lengths. Instead, we employ a Variational Bayesian approach [1], which significantly reduces computational demands by optimizing the variational lower bound to approximate the posterior, avoiding the inefficiencies and high sample requirements of MCMC in high-dimensional tree spaces.

1.2 NJ Algorithm: NJ is used to construct the initial tree structure and requires some computational resources. It is executed during each iteration's initialization phase. While it adds to the computational load, this step ensures accuracy and stability, providing benefits that outweigh the computational costs and are not achieved by other methods.

1.3 DNABERT2: This step is highly efficient. For instance, processing 100 sequences takes only 1.76s, and our benchmark datasets are much smaller, ensuring that embedding computation isn’t a bottleneck.

[1] Zhang C, Matsen IV F A. Variational Bayesian Phylogenetic Inference[C]//ICLR (Poster). 2019.

L2: Training and Inference Performance:

You are correct in noting that we trained one model per dataset, similar to recent methods in the field. During inference, the model's performance is limited to the current dataset on which it was trained. We are exploring fully end-to-end approaches to address these limitations and improve generalization across different species and varying numbers of species.

Weaknesses：

W1: Clarity in Methods:

We acknowledge the reviewers' concerns about the presentation of the methods section. In response, we have made a thorough revision:

1. $f_i$ enriches the node feature $i$ by integrating contributions from its children and its inherent traits. In Eq. 2, $f_j$ represents the features of child nodes, and for multiple children, we aggregate their contributions directly in Eq. 1.
2. And 3. h Clarification: We acknowledge the confusion caused by using the same symbol h for two conceptually different functions in the manuscript. The first $h(x_i, x_j)$ is an abstract representation used to describe a distance measure between two nodes. The second $h(x_i, x_i - x_j, d_{ij}^2)$ is a specific implementation of the first, calculating differences in the latent space between nodes x_i and x_j, evaluated using the formula $\sum (\| x_i - x_j \|^2 - d_{uv})^2$.
3. Max Aggregation selects the most significant features from parent and child nodes, ensuring the most prominent features are retained during propagation.
4. Scoring Function S maps each leaf node in the latent space to a score reflecting its contribution to the model's overall performance. Joint optimization with the ELBO objective ensures that parameter updates during training align with the intended direction.

W2: Ablation:

We chose DS1 because it is representative and commonly used in the field, ensuring comparability with other recent studies. Due to time constraints, we have included ablation results for some datasets in the attached file. We can add more if you feel it’s necessary. For a complete comparison of runtime, please refer to Tab. R2.

W3: Code:

Please see the General Response.

M1, M2, M3, M4:

Thank you for your correction. We have corrected the caption and subsection numbers accordingly. We will also explicitly reference RQ7 in the experimental section to better connect it with the appendix details.

Questions:

Q1: Superior Performance Over MrBayes:

We appreciate your insightful inquiry. PhyloGen outperforms MrBayes and other methods on MLL and ELBO due to enhanced genetic sequence representations via a pre-trained genomic language model. This approach allows for a deeper utilization of genetic information, improving the accuracy of phylogenetic tree construction. For topology comparisons, we use metrics such as Simpson's Diversity Index and Frequency of the Most Frequent Topology. Results across multiple datasets, detailed in Tab. R1 shows PhyloGen's ability to generate diverse and balanced topologies, revealing subtle evolutionary relationships that traditional methods may miss while maintaining biological accuracy.

Q2: Low Stds:

Tab. 2's low stds demonstrate PhyloGen's robustness against random seed variations, maintaining low variance despite supporting diverse topologies. Our variance levels are comparable compared to recent methods like ARTree and PhyloGFN. In contrast, MrBayes, which uses a sampling-based approach, exhibits higher variance across varied topologies but does not match PhyloGen's performance even when considering maximum variance. Furthermore, MrBayes employs MCMC algorithms for estimating posterior distributions of phylogenetic trees, necessitating lengthy MCMC chains to achieve convergence. While parallel computing and evolutionary model optimization can mitigate some computational demands, they do not overcome the inherent efficiency and variance control limitations.

Q3: Bipartition Frequency Distribution:

Thank you for your inquiry. We have included a comparison of bipartition frequency distributions for GeoPhy, as detailed in Fig. R2. However, direct comparisons with ARTree are not feasible since its autoregressive generation method does not produce Newick files.

Q4: Runtime Comparison:

Thank you for your suggestion. We detailed this in Tab. R2.

Q5: Efficacy of a Simple FC Layer:

The simple FC layer's effectiveness can be attributed to the high-quality features provided by DNABERT2, which allow the basic architectures to handle complex tasks efficiently. Additionally, minimal parameters and appropriate regularization ensure model stability and convergence.

Q6: XOR in Distance Matrix:

Using XOR to construct the distance matrix effectively captures discrete nucleotide mismatches without requiring sequence alignment, directly measuring genetic variations. This approach aligns with recent methods like GeoPhy and VaiPhy. Additionally, we explored the UPGMA algorithm on DS1, and the results are summarized in Fig. R1 and the following table:

UPGMA assumes uniform evolution rates and is straightforward but limited to consistent-rate scenarios. In contrast, NJ doesn’t assume constant rates, offering more accurate reflections of evolutionary branching in uneven-rate organisms. Overall, NJ proves more effective in handling complex and uneven evolutionary data.

Q7: Suggestion:

Thank you for your suggestions; we will change them in the revised version.

Limitations:

Thank you for your inquiry. Our first use of pre-trained language models to extract embeddings in this domain is validated by our experimental results and not contested in the literature, and we expect that this could spark further discussion in the field. Regarding the dependence on tree topology and branch lengths, our assumptions align with recent methods such as ARTree, GeoPhy, and PhyloGFN, which are widely accepted. A detailed discussion of limitations can be found in the General Response.

Thank you for answering my questions in detail and for providing a discussion on the limitations of your method. In light of this, I am slightly raising my score.

I cannot comment on the standard practice of using only DS1 for the experiments, but given the various somewhat ad-hoc design choices in your method, I would expect to see at least the experiments of Section 4.5 performed using more data sets in a final revision. I also hope the authors pay careful attention to the multiple presentation issues raised by me and the other reviewers.

Dear reviewer XKPL,

We have provided the complete results of the ablation study for the requested dataset (DS1-DS8) based on your suggestions.

We sincerely appreciate all your valuable feedback to help us improve our work. We believe that the issues raised by the reviewers have been addressed in the revision. We hope that you will reconsider our submission of this response. If you are satisfied with our response and efforts, please consider updating your rating. If you need any clarification, please feel free to contact us. We are very pleased and look forward to hearing from you!

Best regards,

Authors

Weakness:

W1: Limitations:

Please see the General Response.

W2: Comparison with Recent Methods:

We appreciate the reviewer's concern regarding the comparison with recent methods. We believe that our manuscript already includes comparisons with several of the latest approaches, specifically GeoPhy (NIPS2023), PHyloGFN (ICLR 2024), ARTree (NIPS2023), and VBPI-GNN (ICLR2023), which have been influential in recent literature. However, we acknowledge the rapidly evolving nature of the field and invite the reviewer to suggest any additional methods that should be considered for comparison.

W3: Code:

Please see the General Response.

Questions:

Q1: Performance on Larger Datasets:

Thank you for your question. We want to emphasize that PhyloGen has been tested on the same datasets as all recent methods, ensuring a fair evaluation. PhyloGen is designed to scale, yet there are currently no datasets available that contain hundreds to thousands of species. This is primarily because most phylogenetic analyses don’t require simultaneous processing of such extensive species counts; instead, research focuses on constructing subtrees for specific subsets. Phylogenetic methods fall into two categories: distance-based (UPGMA and NJ) and character-based(Maximum Parsimony (MP), Maximum Likelihood (ML), and Bayesian methods). The former is fast but limited, while the latter, although computationally intensive, offers greater precision. Review studies [1] suggest Bayesian is the most accurate, followed by ML and MP. Although PhyloGen offers significant speed improvements while maintaining accuracy, its performance on large datasets is still constrained by the inherent time complexity of these methods.

[1] Hall, Barry G. "Comparison of the accuracies of several phylogenetic methods using protein and DNA sequences." Molecular biology and evolution 22.3 (2005): 792-802.

Q2 and L4: Pre-trained Language Models:

Thank you for your question. We selected the genome-specific foundation model DNABERT2 for our phylogenetic inference research. Although models like HyenaDNA and Nucleotide Transformer (NT) excel in long-sequence modeling, they are less apt for our specific needs. Upon your suggestion, we tested these models:

DNABERT2 outperformed others, likely due to its specific optimization for genomic data. Our revised manuscript will detail these findings to justify our choice and showcase our method's flexibility.

Q3: Hyperparameter Sensitivity:

Thank you for your question. Our Experiment Analysis (see Section E.5 in Appendix A, Table 9) demonstrates PhyloGen's robustness across various hyperparameter settings, including Output Dimension, Hidden Dimension, and Layer Normalization. We also evaluated the impact of changing hidden dimensions with and without layer regularization. The results show that PhyloGen maintains robustness to hyperparameter choices.

Q4: Computational Bottlenecks:

We've detailed this in General Response's Limitations.

Q5: Simulated Datasets Tests:

Thank you for your insightful question. In phylogenetic analysis, "true phylogeny" often relies on theoretical assumptions derived from widely accepted biological software. As discussed in our introduction, these traditional methods require sequence alignment, which is time-consuming and computationally intensive. Additionally, our manuscript includes "PhyloTree Case Studies" (RQ6) and visualizations in Sec. E.6 demonstrates PhyloGen's ability to cluster biologically relevant species and highlight evolutionary relationships effectively. While defining true phylogenies in simulated datasets presents challenges, natural tree structures offer a more valid basis for inference. Furthermore, this setup is rare in standard references and baselines. We appreciate your interest, as it helps clarify our method's scope and potential enhancements. Future work will explore this area more thoroughly to validate PhyloGen's effectiveness.

Limitations:

L1: Robustness:

Thank you for your comments. We've extended our tests to include scenarios such as genetic mutations. Here are the results from dataset DS1 under different settings:

The results show that the ELBO metric for Setting 3 is similar to the original result while Setting 4’s result is closer to the original than Setting 5. This indicates that our model maintains robustness under different types of data noise and perturbations, particularly in handling genetic mutations.

L2: Interpretability:

Thank you for your feedback. Our manuscript includes case studies in RQ6 and visualizations in Sec. E.6, demonstrating PhyloGen's application to real-world genetic data. These provide intuitive understanding and help enhance overall interpretability. Specifically, Fig. 7 illustrates the clustering and phylogenetic tree structure, effectively showcasing the representation results. Compared to similar studies, our approach provides a more extensive and comprehensive analysis, setting a high standard for interpretability within the field. We appreciate your feedback and will continue to refine this aspect.

L3: Data Generalizability:

Thank you for your question. Indeed, the benchmark datasets used by PhyloGen include a wide variety of organisms covering marine animals, plants, bacteria, fungi and eukaryotes. This ensures that our model has been tested in various biological environments, demonstrating its ability to generalize to different types of genetic data and organisms not specifically represented in the test dataset.

L4:Pre-trained Models:

Please see our response to Q2.

Dear Reviewer QAeZ,

We highly value your feedback and suggestions, which are crucial for us to review further and enhance our work. We have revised our manuscript and addressed all your concerns. We sincerely invite you to review our responses, hoping that it will help dispel any misconceptions about our work. Below is a summary of our response:

• Performance on Larger Datasets: We clarified PhyloGen's scalability and ensured a fair comparison by testing it on the same datasets as other state-of-the-art methods.
• Pre-trained Models and Hyperparameter Sensitivity: We justify the selection of DNABERT2 and demonstrate PhyloGen's robustness across different hyperparameter settings.
• Interpretability and Generalizability: We highlight several case studies and visualizations that demonstrate PhyloGen's interpretability and generalizability in various biological contexts. We also expanded the robustness tests and addressed concerns regarding computational bottlenecks.

In this revised version, we have sincerely and diligently responded to your valuable comments. We trust that our responses have clearly addressed your questions and invite you to review them.

We are pleased to inform you that several reviewers have given high scores, with one even raising their score after reviewing our responses. Notably, the reviewer with high confidence recognized the strength and robustness of our approach. In light of this, we hope you reconsider your assessment and increase your score. If you have any further questions, we would be happy to address them.

Best regards,

Authors