Benchmarking Structural Inference Methods for Interacting Dynamical Systems with Synthetic Data

Q2. Are there any other datasets that could be used that have a more defined structure? For example, some of the gene regulatory network datasets are popular applications of this methodology, and I would like to see if there are any that could be used for the purposes of evaluation.

Thank you for your question about incorporating more defined datasets, such as gene regulatory network (GRN) datasets (the ones used in [1]). Your suggestion aligns perfectly with our objective to enhance the applicability and relevance of our benchmarking approach. Apart from GRN datasets, we may also use following datasets for evaluation: road maps in California [3], and chemical reaction networks [4].

1. Gene Regulatory Network (GRN) Datasets: We recognize the potential of GRN datasets in providing a more concrete structure for evaluation. While we find the idea promising, we need to assess the reliability and representativeness of these datasets before inclusion. Notable examples under consideration are datasets from the DREAM challenges [2] and other widely-recognized, publicly available resources that have been used in benchmarking studies.
2. Road Maps in California (PEMS Dataset): The PEMS dataset [3], with its traffic flow data from numerous sensors across California's roads, offers an intriguing possibility for reconstructing road maps. However, challenges such as missing data and dynamic changes in graph structures due to road construction need to be addressed, as these factors could significantly affect the accuracy of our inferences.
3. Chemical Reaction Networks [4]: The dynamic nature of chemical reaction networks presents a unique opportunity for evaluation. A critical factor here is determining the appropriate observation intervals to accurately capture the dynamics involved, which is crucial for effective structural inference in such complex systems.

We acknowledge that there are other datasets potentially suitable for evaluating structural inference methods. Our team is actively exploring these options and aims to include more diverse datasets in our future research. We believe that, incorporating datasets like GRNs not only enhances the robustness of our study but also paves the way for applying our methodologies to real-world problems in fields such as biology , geography and chemistry.

We would like to thank Reviewer gd54 for the positive and inspiring comments. We hope our answers have successfully addressed the concerns. To facilitate easy identification, sections and paragraphs that have been revised in the manuscript are highlighted in blue.

References

[1] A. Pratapa, A.P. Jalihal, J.N. Law, A. Bharadwaj, T.M. Murali. Benchmarking algorithms for gene regulatory network inference from single-cell transcriptomic data. Nat Methods 17, 147–154 (2020).

[2] K. Sun, D. Yu, J. Chen, D. Yu, Y. Choi, C. Cardie. Dream: A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics. 2019 Apr 1;7:217-31.

[3] C. Chen, K. Petty, A. Skabardonis, P. Varaiya and Z. Jia. Freeway performance measurement system: mining loop detector data. Transportation Research Record 1748(1):96–102.

[4] W. Poole, A. Pandey, A. Shur, Z.A. Tuza, R.M. Murray. BioCRNpyler: Compiling chemical reaction networks from biomolecular parts in diverse contexts. PLOS Computational Biology. 2022 Apr 20;18(4):e1009987.

We would like to thank Reviewer gd54 for the thoughtful comments. We are happy that the reviewer thought our benchmarking study is equipped with both review of the investigated methods and reliable benchmarking datasets. We are glad that the reviewer agreed with the challenges in obtaining real-world data for structural inference. Here are our answers to the concerns raised by the reviewer:

W1. The proposed datasets seem a bit random. For example, the authors state that the miRNA dataset is too specific, but the springs dataset is a reasonable benchmark. The springs dataset seems very specific and maybe a bit contrived for most purposes.

In response to your concerns about our selection of datasets, we would like to further clarify our rationale. The miRNA dataset, while valuable in its specific biological context, was deemed less suitable for our study due to its lack of chaotic dynamics. This is a critical aspect we are focusing on. In miRNA simulations, the infinitesimal changes in RNA concentration levels do not typically result in significant long-term differences in gene expression. This is due to the relatively stable inter-regulatory relationships within the miRNA network, which limits its representation of chaotic systems. In essence, the miRNA dataset does not sufficiently capture the unpredictability and sensitivity to initial conditions that are characteristic of chaotic systems.

On the other hand, the Springs dataset, despite appearing specific, is actually a more relevant and illustrative example for our study. This dataset effectively models the N-body problem, a classic example of chaotic dynamics. It employs second-order differential equations to describe the interactions between objects, accurately capturing the essence of chaotic behavior. In such systems, minor initial variations can lead to significantly divergent outcomes over time, making it a compelling choice for illustrating the complex and unpredictable nature of chaotic dynamics.

Therefore, while the Springs dataset might initially seem specific or contrived, it is, in fact, a more universally applicable and illustrative example for the study of chaotic dynamics, which is a central focus of our research. Our decision to favor the Springs simulation over the miRNA dataset was driven by this desire to explore and model the unpredictability and sensitivity to initial conditions inherent in chaotic systems, elements that are less pronounced in the miRNA simulations.

We are grateful to Reviewer MpHi for the insightful comments and are pleased that the reviewer recognize the originality and comprehensive nature of our benchmarking study, as well as its value to the research community. We have carefully considered the concerns raised and provide the following responses:

W1. The experiments rely on some synthetic datasets. However, it is unclear if the synthetic datasets are representative enough for real-world dynamic systems. It is also unclear how reliable it is to benchmark these synthetic data, i.e., whether the observations are reliable.

We appreciate your concern regarding the representativeness and reliability of our synthetic datasets in benchmarking real-world dynamic systems. As detailed in Section 4, our synthetic data generation process has been meticulously designed to emulate real-world scenarios closely. We construct interaction graphs that reflect the network properties of real-world graphs, as outlined in Table 1, to ensure our models accurately capture real-world system characteristics. Additionally, the dynamics within these graphs are simulated using NetSims and Springs methods, chosen for their widespread adoption in structure inference research. These methods offer a robust benchmark aligned with field practices, presenting a comprehensive challenge to structure inference methods and allowing us to rigorously assess their performance in complex and chaotic dynamics.

W2. The package is mainly based on Python and R. It is unclear whether the implementation is efficient.

We thank you for highlighting the efficiency of our Python and R-based packages. We have adhered closely to the original scripts provided by authors with minimal modifications. This approach allows our benchmark to include methods with diverse computational requirements and implementations across various programming languages, including Python, R, Julia, and C++ (with Python or R wrappers). Given the variety of methods optimized for GPU and CPU, with varying degrees of parallelizability, a direct comparison of implementation efficiency is not feasible within our study's scope. Our primary goal is to evaluate these methods' applicability, with efficiency optimization being a secondary concern. For large-scale applications, we provide the flexibility to optimize or rewrite the code in more efficient languages like C++ or CUDA, catering to diverse research needs.

Q. How to ensure the synthetic datasets align with the real-world dynamic systems?

Your question about aligning our synthetic datasets with real-world dynamic systems is crucial for our study's validity and applicability. To ensure this alignment:

1. Realistic Parameter Selection: Parameters for generating underlying interaction graphs are based on real-world graphs from eleven disciplines, incorporating realistic variable ranges and special structure biases like self-loops from empirical studies.
2. Incorporation of Real-World Characteristics: Our datasets are designed to include key characteristics of real-world systems, such as trajectories with noise, variability in initialization, and different dynamics types. We model these dynamics using models of first and second order ODE and models with quadratic dependencies, capturing real-world complexity and unpredictability.
3. Transparency and Limitations: We have documented our dataset creation methods and assumptions, allowing other researchers to understand the limitations and potential biases, ensuring informed application of our datasets.

In conclusion, while achieving perfect real-world alignment is challenging, our steps ensure that our synthetic datasets are as close a representation as possible, making them valuable for dynamic system modeling research.

We thank Reviewer MpHi once again for their thorough review and hope our responses fully address the concerns.

Dear Reviewer MpHi,

We hope this message finds you well. As the deadline for the author-reviewer discussion phase is approaching, we wanted to respectfully inquire whether our rebuttal to your review of our paper has successfully addressed the concerns you raised.

We deeply appreciate the insights and feedback you provided, on the synthetic datasets and the implementation of the baseline methods, which have been instrumental in enhancing the quality of our work. In our rebuttal, we endeavored to thoroughly address each of the points you mentioned, and we are keen to know if our responses meet your expectations and clarify the aspects you highlighted.

We understand that you have a busy schedule, and we greatly appreciate any time you can spare to provide us with your feedback on our rebuttal. Your insights are not only important for the review process but also invaluable for our continued learning and development in this area.

Thank you once again for your time and expertise. We look forward to your response.

Best regards, Authors

Dear Reviewer MpHi,

Thank you for your feedback on our manuscript. We would like to clarify some points regarding your response, with the aim of enhancing the scientific discourse around our work.

Firstly, we acknowledge the uniqueness of our study as the first to benchmark structural inference methods. The utilization of simulation data and original implementations in our benchmarking process, we argue, is not a limitation but rather a foundational step that paves the way for future research. This approach provides a controlled environment to evaluate and compare different methods systematically.

In response to your first question (Q1) in the response, we value your perspective on the challenges associated with collecting real-world data that possess a reliable underlying graph structure, which is indeed a critical concern in our field of research. We recognize the importance of including real-world datasets in our study and welcome any recommendations for datasets that meet specific criteria. These include datasets with a trustworthy ground truth for interaction graphs, those that contain both one-dimensional and multi-dimensional data, and those featuring graphs of varying sizes. We are also open to any suggestions or methodologies you could provide for collecting such data in a manner that would allow us to benchmark these datasets effectively, evaluating their accuracy, scalability, robustness, and sensitivity to graph properties.

We must acknowledge, however, that gathering reliable datasets can be a time-consuming and costly endeavor. Therefore, any specific instructions or insights from you in this regard would be immensely valuable to our study. We concur that real-world data typically present a host of challenges, including data occlusion, measurement errors, and the inherent limitations in capturing a comprehensive scope of all nodes. These challenges highlight the complexity and intricacy involved in evaluating structural inference methods in a way that is both unified and objective, as well as reproducible. Any guidance you can provide on enhancing the reliability and representativeness of these observations would be greatly appreciated and would significantly contribute to the robustness and relevance of our research.

In response to your second question (Q2), we have adopted the original implementations of these methods directly from their respective literature to ensure accuracy and consistency in our benchmarks. In our revised submission, we have added Appendix D.6, which details the running time for each method. Our study aims to provide comprehensive benchmarking results and practical use cases for these methods, enabling researchers to select and implement the most suitable approach for their needs, which is also confirmed by your review "3. The benchmark results could save the efforts for future research in this domain". We have provided detailed instructions and resources on our anonymous GitHub repository for each method, facilitating their installation and application. We believe that the ease of installing Python, R, Julia, and C++, as is common in our field, should not pose a significant barrier.

We are committed to contributing meaningful insights to this domain and appreciate the opportunity to discuss and refine our work based on your valuable feedback.

Thank you once again for your time and review.

Best regards, Authors

W2. Complexity/running time is missing.

We appreciate your insightful comment on the necessity of including running time information for the structural inference methods we investigated. In response, we have added a detailed table summarizing the average running times, calculated over ten runs. The times are reported in minutes unless otherwise specified:

These running times were recorded using BN_NS trajectories, chosen for their one-dimensional feature representation. This approach allows us to evaluate both VAE-based methods and other methods under uniform conditions. As indicated in the table, VAE-based methods generally require more time due to the necessity of initial training. In contrast, other methods can directly infer structure without this training phase, making them more efficient in terms of computation time. However, it's important to note that VAE-based methods offer greater versatility, as they are applicable to both multi-dimensional and one-dimensional trajectories. This broader application scope might justify the longer running times for certain use cases.

This comprehensive evaluation of running times has been included in Appendix D.6 of our revised manuscript.

W3. I wonder can the authors consider robust testing for this paper?

We thank you for the insightful suggestion regarding the implementation of robust testing in our research. Recognizing the critical nature of validating our findings across diverse conditions, we have undertaken the following measures:

1. Testing Under Varied Conditions: Our synthetic dataset, comprising over 213,000 trajectories and 231 distinct underlying interaction graphs, provides a broad and diverse testing ground. This diversity in data inherently encompasses a range of conditions, thereby addressing robustness through varied input parameters within the scope of our benchmarking study.
2. Sensitivity Analysis: The influence of varying input data on our model's performance is a vital aspect of our research. While a dedicated sensitivity analysis is beneficial, we believe our study partially addresses this through an examination of how different graph properties can impact the effectiveness of structural inference methods. This investigation indirectly contributes to our understanding of model sensitivity to input variations.
3. Error and Exception Handling: We have rigorously evaluated the response of the structural inference methods to varying levels of Gaussian noise. This approach is instrumental in assessing how our methods handle common anomalies and errors in input data, thereby providing insights into their reliability under less-than-ideal conditions.
4. Replicability and Generalizability: To foster a culture of transparency and replicability in our field, we have made available the implementations of the structural inference methods, along with their hyper-parameter configurations and datasets. This initiative is aimed at enabling fellow researchers to replicate our study seamlessly, further reinforcing the robustness and applicability of our findings.

We are confident that these measures collectively fortify the robustness of our research. They contribute significantly to a comprehensive understanding of the applicability and reliability of our findings in real-world scenarios. Furthermore, we remain open to and welcome any additional suggestions for enhancing the robust testing of our paper. Your guidance in this regard is highly valued and will be considered earnestly in our ongoing and future research endeavors.

We extend our sincere gratitude to Reviewer fGzL for the thorough review and insightful feedback on our paper. We are delighted that the accessibility of our datasets and the comprehensiveness of our website were well-received. Additionally, we appreciate your recognition of our extensive experiments, insightful findings, and detailed implementations. Here are our answers to the concerns of the reviewer:

W1. Why Gaussian noise. Can the authors consider other types of noises? Also, can the authors test the models' performance under Gaussian noise with different conditions.

We thank you for raising the question about our choice of Gaussian noise and the exploration of alternative noise models in our study. Our initial use of Gaussian noise was guided by its fundamental role in scientific modeling, given its well-characterized bell-shaped distribution and its definition by two simple parameters: mean and variance. And most measurement errors can be modeled with Gaussian noise. This choice aligns with the standard practices in structural inference research, as evidenced by its frequent application in seminal works [1, 2]. The linear and additive nature of Gaussian noise simplifies both analytical and computational modeling, making it an ideal candidate for initial model assessments.

However, we fully acknowledge the diversity of noise types encountered in real-world data, which often deviate from the idealized Gaussian model. Variants such as Poisson, uniform, and salt-and-pepper noise present unique challenges and characteristics that are critical to consider for a comprehensive evaluation of our methods.

In line with your valuable suggestion, we plan to extend our study to include these various noise types. This broader approach will allow us to assess the robustness and adaptability of our models under a wider range of conditions, enhancing the applicability and relevance of our findings.

Regarding the specific request to evaluate our models under differing Gaussian noise conditions, we aim to conduct additional experiments varying both the mean and variance of the noise. This will enable us to better understand the resilience and precision of our models under a spectrum of noise intensities and distributions. We believe that these further analyses will greatly enrich our comprehension of the practical limits and strengths of our approaches in real-world scenarios, where noise characteristics can be highly unpredictable.

We regret to inform that due to resource constraints, it might not be feasible to complete these additional experiments before the rebuttal deadline. However, we are committed to pursuing this extended analysis and will update our findings on our website and in future publications as soon as they are available. We encourage interested readers and fellow researchers to stay connected for these upcoming developments.

Dear Reviewer fGzL,

We hope this message finds you well. As the deadline for the author-reviewer discussion phase is approaching, we wanted to respectfully inquire whether our rebuttal to your review of our paper has successfully addressed the concerns you raised.

We deeply appreciate the insights and feedback you provided, on the Gaussian noise and the running time of the structural inference methods, which have been instrumental in enhancing the quality of our work. In our rebuttal, we endeavored to thoroughly address each of the points you mentioned, and we are keen to know if our responses and the revisions meet your expectations and clarify the aspects you highlighted.

We understand that you have a busy schedule, and we greatly appreciate any time you can spare to provide us with your feedback on our rebuttal. Your insights are not only important for the review process but also invaluable for our continued learning and development in this area.

Thank you once again for your time and expertise. We look forward to your response.

Best regards, Authors

We sincerely appreciate the encouraging and constructive feedback from Reviewer HMKQ. It's gratifying to know our paper's strengths in originality, quality, clarity, and significance are recognized. We also value the reviewer’s perception of the potential impact of our work within the research community.

Here are our answers to the concerns raised by the reviewer:

W1 & Q7. Lack of Real-World Applications / Generalizability to Other Domains : The paper primarily focuses on benchmarking structural inference methods with synthetic data. However, it does not provide concrete examples or case studies demonstrating the practical application of these methods in real-world scenarios. Including real-world use cases and applications would make the paper more relevant to practitioners who are interested in applying these methods in their work.

We are grateful for the reviewer’s insightful comment regarding the emphasis on real-world applications. The primary objective of our study has been to benchmark structural inference methods across various disciplines, aiming to establish a unified, objective, and reproducible standard for evaluating these methods. Utilizing simulation data, which allows controlled properties and complete observations, seemed an ideal approach for developing this initial benchmark in structural inference.

However, we acknowledge and concur with the reviewer’s point that discussing real-world applications would substantially increase the paper’s appeal to practitioners. To address this, we have expanded Section 1 in our revised manuscript to include the widespread adoption of structural inference methods across diverse fields. These include, but are not limited to, the inference of gene regulatory networks [1, 2, 3], the deduction of gene co-expression networks [4, 5], chemical reaction network reconstruction [6, 7], road map reconstruction [8, 9, 10], and financial network inference [11, 12]. This addition highlights the broad applicability and relevance of these methods beyond theoretical settings, underscoring their practical utility in various real-world domains.

We believe that these enhancements in our manuscript will more effectively bridge the gap between theoretical benchmarking and practical application, thereby fulfilling the interests and needs of both researchers and practitioners in the field.

W2 & Q2. Incomplete Hyperparameter Exploration / Hyperparameter Tuning Details: While the paper mentions a hyperparameter search for some methods, it lacks a comprehensive discussion of the specific hyperparameters explored, the range of values considered, and the impact of hyperparameter tuning on the results. Providing more detail on hyperparameter exploration would help researchers understand the sensitivity of the methods to parameter settings.

We appreciate your insightful comments regarding our hyperparameter exploration. Detailed information about our tuning process, including the hyperparameter range, is indeed provided in Appendix C of our original submission. We employed a grid search strategy, but due to computational limitations, this tuning was focused on a specific graph type. We recognize that variations in network properties can impact model sensitivity. To prevent misleading generalizations, we reported our findings with a conservative approach. While this strategy offers initial insights, we acknowledge, as you highlighted, the importance of a more comprehensive hyperparameter analysis across varied networks in future studies.

W3. Limited Discussion of Algorithm Mechanisms: The paper briefly describes the structural inference methods but does not delve deeply into the underlying mechanisms of each method. A more detailed explanation of how each method works, its assumptions, and the computational complexity involved would provide a better understanding of the methods for readers who may be less familiar with the specific techniques.

We thank you for your feedback on the description of algorithm mechanisms. Our aim was to provide a high-level overview of the methods in Sections 3.1 - 3.5, facilitating a clear understanding of their basic mechanisms. We endeavored to describe these methods succinctly, drawing on our in-depth understanding of their workings and grouping them based on their fundamental mechanisms for ease of reading. However, given the current length of the paper (exceeding 50 pages), a more detailed exposition of all 12 methods would significantly expand the manuscript. To avoid overwhelming readers, we recommend the readers consulting the original papers for in-depth information on these methods.

Dear Reviewer HMKQ,

We hope this message finds you well. As the deadline for the author-reviewer discussion phase is approaching, we wanted to respectfully inquire whether our rebuttal to your review of our paper has successfully addressed the concerns you raised.

We deeply appreciate the numerous insights and feedback you provided, which have been instrumental in enhancing the quality of our work. In our rebuttal, we endeavored to thoroughly address each of the points you mentioned with several clusters of answers to similar questions, and we are keen to know if our responses and the revisions meet your expectations and clarify the aspects you highlighted.

We understand that you have a busy schedule, and we greatly appreciate any time you can spare to provide us with your feedback on our rebuttal. Your insights are not only important for the review process but also invaluable for our continued learning and development in this area.

Thank you once again for your time and expertise. We look forward to your response.

Best regards, Authors