PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models

Reviewer ckR1, Thank you very much for the time dedicated to reading the paper and providing your review. Addressing your points below:

"The paper describes a useful system. The key weakness is a lack of scientific novelty, which makes it more suited to a systems-focused conference or workshop." and "The paper describes a system which looks to be of general use, however I question whether this is the right venue. It would be more suited to a systems-focused workshop, e.g., MLSys."

-> The ICML 2025 Call for papers lists the following as of interest for the main conference: Machine Learning Systems (improved implementation and scalability, hardware, libraries, distributed methods, etc.). Our paper is inline with this interest.

-> The ICML 2025 Call for papers lists the following as of interest for the main conference: Application-Driven Machine Learning (innovative techniques, problems, and datasets that are of interest to the machine learning community and driven by the needs of end-users in applications such as healthcare, physical sciences, biosciences, social sciences, sustainability and climate, etc.). Our paper is inline with this interest.

-> The paper establishes novelty on the fronts of model evaluation criterion, datasets, task definitions, and machine learning system design.

-> As an example of precedent for ICML publication, our paper cites the ICML paper (Kahn et al., 2022) . It is also an ML systems paper and is a good example we'll cite for some of your other expressed concerns : https://proceedings.mlr.press/v162/kahn22a/kahn22a.pdf

--

"This paper is largely descriptive, rather than making specific scientific claims and providing evidence for them. The stated characteristics and utility of the system is evidenced through high-level diagrams and an example application."

-> This is inline with MLSys publications at ICML such as (Kahn et al., 2022) https://proceedings.mlr.press/v162/kahn22a/kahn22a.pdf

-> Papers presenting new software (models, algorithms, systems, etc) are necessarily descriptive, including those published at ICML. To illustrate this, an ICML paper chosen at random from last year is also largely descriptive with results being showed starting on page 6. https://openreview.net/pdf?id=HsseRq2FAx

--

"This work does not present novel methods or evaluation criteria, rather it describes a system for evaluating existing methods under existing evaluation criteria."

-> The comparisons in Table 1 demonstrate the significant novel contribution of PyTDC. -> The novel dataset collection, benchmark organization, metric design, and API architecture are well-executed and demonstrate the platform's utility. -> The proposed methods and evaluation criteria seem appropriate for the problem domain. The datasets are well-curated across diverse modalities, and the evaluation metrics appear aligned with both machine learning performance assessment and domain-specific biological relevance, such as out-of-distribution generalizability across cell types.

--

Overall: This review is correct in stating we don't emphasize novel theoretical claims. However, it appears to over-generalize this and argue the work does not present novelty aligned with ICML interests. In this rebuttal, we re-emphasize novelty and how it aligns with ICML Call for papers.

--

In light of this, we kindly request the reviewer re-evaluate their claims and recommendation, and share any further reservations. A revision of the claims made in the review, given inconsistency with ICML criteria and precedent as well as with the content of the paper, and, consequently, the overall recommendation, should be considered.

However, the necessity for adopting the specific 'API-first' architecture and Model-View-Controller (MVC) design pattern are underexplained. While these approaches are well-established in software engineering, the paper does not clearly justify why they are uniquely advantageous for this use case compared to alternative architectures, including comparison with the previous versions. Further explanation is needed to clarify how this design improves the implementation of the pyTDC server, in perspectives including modularity, scalability, or reproducibility.

We ensure up-to-date retrieval by invoking external APIs rather than storing static data. The chosen MVC design pattern and the resulting “data view” allow us to perform data manipulation per dataset definition and, as such, obtain an ML-ready dataset fitting the task or resource definition. The single-cell gene expression data retrieval API and the shown DSL-based dataset are examples of data views leveraging this pattern, which we can more clearly emphasize in the paper.

Minor comments: There are several technical issues with references throughout the manuscript, particularly in the Appendix. Compilation errors resulting in ?? citations appear in multiple locations, including: Page 3 line 129 (right column), Page 21, Page 24 lines 1316~1318, Page 38 ...

Thank you. We will address all of the minor comments raised in the camera-ready version of the paper.

While the authors highlight the novelty of implementing the ‘API-first’ architecture via MVC design as one of the main contributions, there are limited demonstration or explanation to support the statement throughout the paper. How does this architecture differ from other versions or databases? In which way might the users benefit from such design? A section for supporting such claim, either quantitatively or qualitatively, may clarify the readers in the technical contribution of pyTDC architecture.

We contrast this architecture difference with other benchmarks in B.1, noting that alternatives, including the previous version (which we can note as well), provide access to “file dumps.” We can reword and expand this to more thoroughly distinguish between static datasets provided by alternatives and our “API-first dataset.” The MVC allows us to implement data views (https://www.w3schools.com/mysql/mysql_view.asp), our chosen implementation of the “API-first dataset” abstraction.

In B.1.3 we state “[PyTDC] develops an Application-Embedded Domain-Specific Data Definition Programming Language facilitating the integration of multiple modalities by generating data views from a mapping of multiple datasets and functions for transformations, integration, and multimodal enhancements, while…”.

We agree that while all components for answering your query are present in the paper, and shared above, a more distinguishable narrative would be helpful and will write an addition to our discussion of the architecture more thoroughly detailing: The API-first architecture and the benefits of invoking external APIs, the “API-first dataset” abstraction, the choice of MVC design pattern, how that implements data views and the overarching “API-first dataset”, and illustrating via existing data views, would be helpful. We cite works demonstrating individual benefits and our contribution is composing these into an innovative design.

Another area of concern relates to the redistribution and usage of pre-trained models and datasets through the PyTDC platform. Given the diverse licensing terms associated with pre-trained models (some under open-source licenses, others with academic or non-commercial restrictions), how does PyTDC plan to: - Clearly surface licensing terms and permissible use when users retrieve models especially through python editors? - Ensure that models with restrictive licensing (e.g., non-commercial, academic use only) are not inadvertently used in ways that violate their original licenses? - Including a transparent licensing and provenance management layer would greatly enhance trust, reproducibility, and adoption in both academic and commercial settings, particularly for users in regulated industries like pharmaceutical research.

We surface license terms on the website for all datasets and will do so for models on our model hub pages as well. We can explore the possibility of augmenting classes to expose licenses more easily. Beyond exposing licenses, it is likely impractical for PyTDC to prevent license violations in its current state as a PyPI package (and Harvard dataverse repository). However, deploying a transparent licensing and provenance layer is certainly a requirement of interest for deploying a distributed service based on the current PyTDC platform. This has been added to our product backlog for future releases.

Dear Reviewer ATdV, Thank you very much for your time dedicated to reviewing our work. Including our responses to your queries below.

PyTDC doesn't include text data, yet natural language is a key modality in biomedicine. Is it appropriate to claim comprehensive support for biomedical foundation models without this modality?

This is fair. Supporting biomedical foundation models is an aspirational goal in this case. We focus this release on single-cell foundation models and are working to add ESM3 to incorporate the protein sequence modality. We could more strongly emphasize this point, or adjust our claim/headline to match “foundation models in therapeutic discovery”.

As a non-commercial platform, how does the group implement the continuous update of new data?

This is addressed via an API-first approach. By invoking external APIs rather than storing static data, we assert up-to-date retrieval. The chosen MVC design pattern and the resulting “data view” allow us to perform data manipulation per dataset definition and, as such, obtain an ML-ready dataset fitting the task or resource definition.

As an open-source, API-based platform, the user experience of PyTDC will require more feedback from future users to fully evaluate its effectiveness and usability.

This is true. It is hard to fully and rigorously assess the success of this release. However, we have some very positive indicators, including a sharp increase in Github stars (now 1000+), a 2x increase in the Pypi package's MAU, and ~5k invocations of the model server APIs in the last month alone.

What managing mechanism is adopted for continuous data updates in PyTDC?

Our platform emphasizes an API-first approach rather than storing static data, thus ensuring up-to-date retrieval. However, for static datasets, we cannot yet guarantee continuous data updates.

How does PyTDC manage the use of data with various licenses? For example, if a foundation model is trained using data accessed through PyTDC from sources with different licenses, how does the platform supervise and ensure compliance with these licenses?

We don’t yet have rigorous licensing enforcement. However, we expose all licenses for datasets on our website and urge users to review permits for the datasets they are using.

Dear Reviewer exTK, thank you very much for your time dedicated to reviewing our work. Including responses to your queries below.

The only concern for me is that the target of PyTDC is supporting "biomedical foundation models", which may be somewhat overclaiming. Biomedicine encompasses a vast range of disciplines, for which various tasks, datasets, and benchmarks have been proposed. There have also been many models claimed to be "biomedical foundation models", which actually focus on very different applications, such as clinical NLP, medical image processing, and drug discovery. Therefore, as an AI platform, PyTDC can hardly cover the whole field of "biomedical foundation models".

This is true. Supporting biomedical foundation models is an aspirational goal in this case. We focus this release on single-cell foundation models and are working to add ESM3 to incorporate the protein sequence, structure, and function modalities. We can either make a stronger note of this limitation or rephrase our claim to be “foundation models in therapeutic discovery”.

For the structure-based drug design component of PyTDC, beyond the scoring functions already included, there are two recent popular aspects to be considered in evaluating the molecules: The physical plausibility of 3D structures, such as PoseCheck [1]. The diversity of generated molecules, new metrics including NCircles [2] and HamDiv [3]. The SBDD component may not be the main contribution of PyTDC, but as a comprehensive benchmarking platform, I believe these metrics would help.

We agree. We are still working on the final release of the codebase, which could be released simultaneously to a camera-ready version. This suggestion has been added to the list of tasks for that release.

Tiny improvements in writing could be made: In Section 4.3, "PPI" and "ppi" are mixedly used. Some citations in-line act as components of sentences, where \citet should be used instead of \citep. For example, the last line of page 7. "Table" vs. "table", "Figure" vs. "figure", "Section" vs. "section" are mixedly used in many places, which should be unified.

This will be done for any camera-ready version.

How many single-cell data items are there in the dataset? Is this amount enough for training a robust model?

Our website and appendix show the total number of items for all datasets introduced in the paper. For the case study on the single-cell drug target nomination task, we have 394760 (cell, gene) pairs in the dataset. We will mention this explicitly when we release the documentation for the pinnacle resource. This dataset was large enough to train PINNACLE and Node2Vec to satisfactory results. It is possible a larger dataset would have yielded better performance from the tested transformer models. However, we note the dataset presents what is realistically feasible to curate for these diseases based on the current literature.