KinDEL: DNA-Encoded Library Dataset for Kinase Inhibitors

Thank you for your positive comments and valuable feedback. We have addressed your comments in the responses below.

W1. “Why RF method performs the best on on-DNA set (Line 335)”, “why there are different performance rankings on on-DNA and off-DNA settings”, and “why DEL-Compose(M) and DEL-Compose(S) performs differently on on-DNA and off-DNA settings?”

For discussion on the RF and DEL-Compose results, refer to our responses to Reviewer 6jAS (W4 and W5). Additionally, there are differences between on- and off-DNA data because off-DNA effects are unobserved in the DEL data (that is one of the tradeoffs for generating data at scale). However, it is good to see that some models do observe good prediction correlation to off-DNA data, though this will vary from target to target (and library to library). The difference between two variants of DEL-Compose will greatly vary between datasets because it depends on whether the important binding structures are localized to particular synthons or are larger molecular scaffolds. In the former case, we expect DEL-Compose$^{(S)}$ to work better, whereas in the latter case DEL-Compose$^{(M)}$ might be better because it explicitly represents the entire molecule.

W2. More views and new settings are required, such as case study of top-ranking candidates

Thank you for this suggestion. We believe that the core of this benchmark should be testing model performance under different data splits to examine models’ ability to generalize, with different levels of task difficulty. This is exactly what KinDEL provides with random, disynthon, and similarity-based splits evaluated on on- and off-DNA testing compounds (see the information about the new similarity-based split in the general response above). However, we like the idea of expanding benchmarks to new setups like the ones presented in the DEL-Dock paper. The subset ranking in KinDEL would not provide much meaningful information because the validation compounds are already close to the library distribution (see Figure 5b). Other evaluations in the DEL-Dock paper assumed the existence of a strongly binding motif being benzenesulfonamides, which do not have a clear correspondence to our targets. However, to enable creation of new benchmarks similar to those present in DEL-Dock, we publish an additional BCA dataset using the same compound library. For more details, refer to the general response above.

Q1. Not enough comprehensive view, more metrics/comparisons to DEL-Dock

DEL-Dock uses a very small library (100k) for training purposes which gives a much smaller training set size and the validation set is a set of curated public datapoints of unknown quality. BCA (the protein used in DEL-Dock) is known to have a chemical structure with known binding affinity (benzenesulfonamide). As discussed on pages 13 and 14 of DEL-Dock, there is a discrepancy between the presence of this common potency conferring motif in the training set and validation set for DEL-Dock that may substantially impact performance. This is likely the source of divergence in performance for methods between these papers. In KinDEL we have multiple distinct potency conferring scaffolds and thus suffer less from this problem.

Q2. Is it possible to provide an advanced version of KinDEL dataset with machine-aided of molecule docking poses?

Please refer to the general response. We are planning to release 3D poses for purposes of consistent model development. The first batch of the poses for the top molecules has been already uploaded. Thank you for the suggestion.

---

We hope that we have addressed your concerns, and the inclusion of 3D poses will make our benchmark more valuable. Please, let us know if you have any further questions or concerns we can resolve to make your review even more positive. Thank you again for your time and valuable feedback.

Would you please provide a subset Spearman metric evaluation just as mentioned in [1]? It is because a benchmark paper should cover most existing metrics. Thank you.

[1] Shmilovich, K., Chen, B., Karaletsos, T., & Sultan, M. M. (2023). DEL-Dock: Molecular Docking-Enabled Modeling of DNA-Encoded Libraries. Journal of Chemical Information and Modeling, 63(9), 2719-2727.

We appreciate your thoughtful feedback and have addressed each of your points below.

W1. I am not sure about the importance of this paper for the ICLR community.

We believe that DEL represents an emerging data modality with significant potential as a resource for addressing complex chemistry problems. In particular, one of the shortcomings of current foundational models trained on public data is that many models fail to generalize to particular targets. Because DEL data can generate a large amount of chemical data for any protein target, we can leverage this data modality to finetune these models. Another open problem in the field is how to correlate binding poses with actual binding affinity, and DEL data provides the supervision necessary to tackle this problem. To aid these efforts, we are updating the dataset to provide docking poses for the entire dataset. We are excited about the possibilities of this paired 3D pose and binding score dataset. This is why we believe that this data is a good contribution to ICLR and the machine learning community.

W2. The dataset has only two targets and both are kinase. The biophysical assay validation set is relatively small.

For the discussion about our targets, please refer to the general response above (we have added one non-kinase target). Regarding the size of the validation set, we believe that this number of validation compounds is reasonable for assessing model performance. To optimize the costs of compound resynthesis and biophysical assays, we selected a diverse set of molecules that cover the chemical space of the whole library (see the UMAP plot in Figure 5b). The validation set selection is discussed in Appendix C.

W3. The authors did not mention the library size or sequence depth of the DEL dataset. Does it have an effect on the dataset?

Library size is ~81M (which was described in section 2.2), the sequencing depth (number of read counts/sequence counts) is 514.5M/65.8M for MAPK14, 296.5M/49.8M for DDR1, and 440.7M/52M for BCA in all three replicates. We expect a difference between the two numbers as we amplify to install indices and the sequencing primers. Sequencing depth always has an effect on the data quality, as higher sequencing depth usually leads to better data quality (since there are more samples). However, this is often limited by resources, and we observe good correlation between replicates and counts (refer to Appendix B). We have added the information about sequencing depth to Appendix A.2.

W4. The authors show that DEL-Compose performs better for off-DNA data. It would be helpful to discuss the potential biases due to the DNA barcode.

We believe that DEL-Compose performs better for off-DNA data, because it contains the right inductive biases to regularize the model predictions. We know that individual data points in DEL experiments can be noisy, so it is important to not overfit onto individual molecules. DEL-Compose predicts a more conservative estimate of a molecule’s binding affinity, due to incorporating uncertainty directly in the predicted output distribution, which is why we believe DEL-Compose performs better on off-DNA data. We discuss briefly the differences between on- and off-DNA data in Section 4.

W5. Why does the RF method become worse for the disynthon split?

All models perform worse in the case of disynthon spits, and different models will observe different changes in these settings. Since specific structures are not observed as frequently in the disynthon splits, perhaps the random forest model is overfitting onto specific features, leading to a worse generalization.

---

We hope that we have properly addressed your concerns. Please let us know if you have any further insights or questions answering which would make you feel more positive about our paper. Thank you again for your time and valuable feedback.

We sincerely appreciate the Reviewer's recognition of the value our benchmark brings to the research community. We are committed to presenting a comprehensive comparison of the machine learning models that are frequently employed for analyzing DEL libraries. In our study, we have incorporated a diverse array of methods, including fingerprint-based methods (e.g. RF, XGBoost, DNN), graph neural networks (GIN) and synthon-based models (DEL-Compose). We are in the process of adding Chemprop (a SOTA GNN) to the benchmark. We would greatly appreciate any suggestions you might have on additional models that would enhance our comparison!

Q1. It might be possible that these existing datasets could be adapted to resemble KinDEL.

The data from Iqbal et al. [4] is a useful resource for developing new models, but it is still missing a lot of information that we provide in KinDEL. Their data encompasses 3 different DEL datasets, DOS-DEL, HitGen, and MSigma. Two of the datasets, DOS-DEL and MSigma, are not uploaded yet (currently missing from their Github), and their HitGen data only has 700k examples out of the 1B member library, so unfortunately it is difficult to assess the properties of their data. Furthermore, only full SMILES strings are provided, and no decomposed synthon structures, which can be important for modeling (as we have demonstrated in our experiments). This dataset also does not include pre-selection information, which can also be useful when modeling. Their data does, however, include the inhibitor condition which is useful to distinguish between allosteric/orthosteric/cryptic binders, which can be valuable.

Given the limited amount of available data, we do not see a way that this dataset could be used to supplement or replace KinDEL. One important distinguishing feature of our data is our inclusion of replicate data. This is very important due to the potential noisiness of the experimental data. We feel that including this in our dataset makes it a valuable testbed for future machine learning experiments.

Q2. How does KinDEL compare to the existing DEL datasets in terms of diversity, and how well does it reflect the performance of existing methods in predicting Poisson enrichment? Does KinDEL offer a more comprehensive or representative testbed for evaluating these methods?

Chemical diversity is difficult to characterize especially on libraries of this size, and is highly task-dependent. For instance, chemical diversity measured through Morgan Fingerprints can fail to distinguish property cliffs, which is a very challenging problem for small molecule tasks [6]. To make the computation of diversity more feasible for such large libraries, one can evaluate the diversity of each synthon group separately. However, as mentioned in the discussion, most DEL datasets do not release their synthon structures. Nevertheless, we would be happy to measure the similarity between their library synthons and ours if the data were made available. Moreover, some public datasets (e.g. [5]) provide binarized evaluation labels. While these datasets are great resources for the community, we feel that it is even more valuable to measure how well models rank compounds (Spearman on enrichment) than it is to report accuracy on a binary evaluation set. While Hou et al. and Gerry et al. both provide enrichment based methods, their libraries are smaller and thus lack the coverage and diversity of ours.

---

We hope that our answers clarify the importance of the proposed benchmark. Do you have any further questions in the meantime while we are testing more models? Thank you again for your time and valuable feedback.

[5] Iqbal, Sumaiya, et al. "DEL+ ML paradigm for actionable hit discovery–a cross DEL and cross ML model assessment." (2024).

[6] Van Tilborg, Derek, Alisa Alenicheva, and Francesca Grisoni. "Exposing the limitations of molecular machine learning with activity cliffs." Journal of chemical information and modeling 62, no. 23 (2022): 5938-5951.

I appreciate your efforts to address the points I raised.

The utilization of state-of-the-art (SOTA) methods presents an opportunity to examine whether their established strengths and limitations are accurately represented. Actually, the study by Iqbal et al. (2024) also suffers from this issue by incorporating fundamental machine learning (ML) approaches, such as Random Forest (RF), Support Vector Machines (SVM), and Multilayer Perceptron (MLP), despite the significant advancements in the ML field. It is essential to acknowledge that these methods may still outperform others in the target problem, although this assumption warrants further investigation.

Regarding my second question, diversity can be explored through two primary avenues: (1) explicit, probably chemical, features representing the data points, and (2) the performance of the tested ML algorithms. A crucial concern with existing datasets across various domains is the lack of assessment regarding dataset diversity. Specifically, when a dataset exhibits high similarity among its data points, an algorithm's performance may be attributed to its ability to cater to these similarities rather than demonstrating generalization. This oversight can lead to inaccurate claims of SOTA algorithms, highlighting the need for a more comprehensive evaluation of dataset diversity.

We appreciate your detailed feedback and have addressed each point to clarify and improve our manuscript.

W1. The evaluation is limited to only two kinase targets.

We have added an additional target for the purposes of testing generalization. Please, refer to the general response above.

W2. The data-splitting method may ensure that compounds with the same disynthon do not end up in the same split, but it doesn’t fully prevent similar compounds from being grouped together.

We agree that segregating similar compounds into a single split is extremely important. In our experience, traditional scaffold methods (Bemis-Murcko) have substantial weaknesses in DEL data due to the combinatorial nature of the chemistry. To illustrate this we have calculated the number of unique scaffolds and their frequency in the top 1M for mapk14. There are ~300k unique scaffolds, and the most frequent one occurs 10,061 times and is a benzene. Please see the new appendix section “Dataset splitting” where we attach a plot of the frequency of the top 100 most common scaffolds (showing a rapid decline in frequency). We also show the six most common scaffolds, all of which are generic ring structures.

Internally we have created a similarity based splitting method. It first uses UMAP to reduce 1024 ECFPs to 10 dimensions and then uses HDBSCAN to cluster them before constructing splits out of the scaffolds. This method is elaborated on in the new Appendix E. We will include an additional split from this method in our updated dataset. We are currently training models on this new data split.

W3. Presentation Improvements.

We apologize for any confusion caused by the table headers and appreciate your pointing out the typographical error in Figure 3. We have revised the table headers to improve clarity by adding the metric name “Spearman’s $\rho$”. We have also corrected "SP³" to "sp³."

---

Q1. Why use AI to reduce data noise?

DEL data efficiently generates hundreds of millions of data points, which is traditionally not possible through other screening methods because it would be prohibitively expensive. Despite its bias, DEL data routinely uncovers high affinity binders [3], which is why the data is a very valuable resource. Thus, even models that simply replicate DEL data (with inherent noise) are useful. Given the combinatorial nature of the library, we hope that future methods will demonstrate further denoising. While we know individual data points might have noise, we are confident that aggregates of molecular clusters should reveal the correct signals. By leveraging information in these aggregations, we believe AI models can recover information about the underlying affinities of the molecules tested.

Q2. More explanation is needed for why certain chemical properties in Figure 3 are considered "drug-like".

In Figure 3 we show the distributions of properties for KinDEL with ranges from Shultz (cited in the figure caption) as a reference. His paper updates the classic rule of 5 based on all drugs approved in the years since Lipinski et al. published their paper (1998-2017). In this paper, we see that the properties of “drug-like” molecules change over time. From 1998-2007, the 10th to 90th percentile range for molecular weight is 201 to 525, while from 2008-2017, this range shifted to 235 to 607. Generally, the trend in recent years has been towards larger drugs. Therefore, there is not necessarily a right or wrong range of molecules, and that’s why we think our data is still highly applicable for drug discovery.

There is no discussion of QED within Schultz so we do not include ranges for it. While QED is a popular metric, it was derived from pre-2012 approved drugs and thus also is no longer entirely representative of current paradigms. Many of its constituent metrics are impacted by the current shifts in MW (HBA,HBD,PSA,ROTB,AROM). In this context, “low” scores mostly mean that DEL molecules do not tightly match the characteristics of 771 pre-2012 drugs.

Generally, we intend Figure 3 as a reference to show that the distribution of properties of the molecules in KinDEL overlaps with those traditionally considered “druglike”. Of note is that Figure 2 of [4] also shows non-complete overlap with the “rules of 5” which suggests that this divergence is common in DELs.

[3] Peterson, Alexander A., and David R. Liu. "Small-molecule discovery through DNA-encoded libraries." Nature Reviews Drug Discovery 22, no. 9 (2023): 699-722.

[4] Gerry, Christopher J., et al. "DNA barcoding a complete matrix of stereoisomeric small molecules." Journal of the American Chemical Society 141.26 (2019): 10225-10235.

Thank you for addressing my concerns. While your responses have clarified several points, there are still issues I would like to discuss further.

Regarding Q1: Noise Reduction in DEL Data

I appreciate your explanation regarding the potential of future methods to address noise. However, I remain uncertain about this. What did the authors mean by "Given the combinatorial nature of the library, we hope that future methods will demonstrate further denoising."? What is the connection between "the combinatorial nature of the library" and the possibility of denoising? Could you elaborate on how these aspects are causally related?

Moreover, could you outline the current widely accepted approaches—whether machine learning-based or manual/rule-based—that are used to alleviate noise in DEL data? From your response, it seems such methods may not yet exist or are not well-established. Is this an accurate interpretation? If so, how does this impact the current utility of DEL data for model training and downstream applications?

Regarding Q2: The Term "Drug-like"

I agree that limiting "drug-like" criteria to the Lipinski rule of five is outdated, and I commend your efforts to provide an updated context. However, I note that you have not identified specific or quantitative ranges to support the claim that KinDEL’s properties fall within established "drug-like" boundaries. Without such evidence, I find the use of the term "drug-like" to describe this dataset potentially misleading.

Regarding Q3: On-DNA Data and Relevance to ML Research

Thank you for your detailed response, which I found interesting. Your explanation highlights how KinDEL brings attention to challenges within the DEL and chemical biology communities, such as noise reduction, the combinatorial nature of DEL data, and the unique issues with on-DNA data. These points are particularly relevant for the machine learning community. If your intent is to engage ML researchers, I suggest discussing these challenges explicitly in the manuscript, especially in terms that are accessible to researchers from pure ML backgrounds. This could enhance the impact of your work by framing the dataset as not just a resource, but as an invitation to tackle unresolved problems in the DEL domain. Could you share your thoughts on this?

Regarding Q4: End-to-End Models

By "end-to-end models", I refer specifically to methods that use raw molecular representations (e.g., SMILES strings, 2D graphs, or 3D atomic coordinates) as inputs, rather than relying on precomputed molecular features like Morgan fingerprints. I understand now that you have included models with molecular graphs as inputs. Additionally, I am curious about the performance of widely used denoising methods—both ML-based and non-ML-based—on KinDEL. If applicable, could you provide comparative results or insights into how these methods perform relative to the approaches you tested?

Thank you again for your detailed responses. I look forward to your insights on these remaining points.

Q3. These points are particularly relevant for the machine learning community. If your intent is to engage ML researchers, I suggest discussing these challenges explicitly in the manuscript, especially in terms that are accessible to researchers from pure ML backgrounds. This could enhance the impact of your work by framing the dataset as not just a resource, but as an invitation to tackle unresolved problems in the DEL domain. Could you share your thoughts on this?

We appreciate your suggestion, and we have explicitly added a paragraph in the discussion section labeled “Challenges and Future Directions” to address this:

“DEL data is powerful in that it specifically densely samples particular chemical spaces, which can be leveraged to learn more powerful representations. However, DEL data suffers from experimental noise to compensate for the scale of data this technology can generate. In particular, there are unobserved factors such as synthesis noise that makes it difficult to separate out signal from noise in the data (Zhu et al., 2021). Additionally, since our observations are sequencing read counts rather than actual binding affinity, the measurements also suffer from PCR bias (Aird et al., 2011). While we have presented several benchmark methods that try to learn a denoised enrichment from structure-based models, this is still an open question in the field, and we hope that our dataset release will enable the development of more denoising methods.”

Additionally, we have included more information about the differences between on- and off-DNA data in the description of the held-out sets in Section 3.1. The modified paragraph says:

“The observed count data in DEL experiments are an approximation of the true on-DNA binding affinity ($K_D$). The count data are influenced by multiple sources of noise (see Section 4). We ultimately wish to rank molecules by binding affinity, so we use compounds with measured $K_D$ (from biophysical assays) as a test set. Performance on these compounds assesses if the models correctly rank compounds by $K_D$. This can be viewed as measuring how well models can remove the noise inherent to DELs.

For both our targets, MAPK14 and DDR1, the selected compounds contained in the DEL library were resynthesized on- and off-DNA to create an in-library held-out test set. For hit finding, we would like to be able to predict off-DNA $K_D$. This is challenging because the DEL data comes from DNA bound molecules, and is biased by the DNA. The on-DNA $K_D$ more closely aligns with DEL data since the molecules in the training data are bound to the same DNA in the same way. A few additional compounds were added from outside the library (and tagged with DNA) to create an additional held-out test set that we refer to as "Extended". The $K_D$ data from these biophysical assays are also released with our dataset. A UMAP visualization of the DEL including the in-library and external test set compounds is depicted in Figure 5b.”

Q4. I understand now that you have included models with molecular graphs as inputs. Additionally, I am curious about the performance of widely used denoising methods—both ML-based and non-ML-based—on KinDEL. If applicable, could you provide comparative results or insights into how these methods perform relative to the approaches you tested?

Thank you for the clarification. One of the typical ways to analyze DEL data is with the Poisson enrichment developed by [1], which computes a ratio of Poisson distributions fit over the target and control data. We have included this in our benchmarks, under the row labeled “Poisson”. This is a computed metric of the data, and does not have any inference capabilities, so we use this number to understand the performance of trained structure-based models. What is exciting about these results is that some of our models, such as DEL-Compose, can make predictions with better performance than the Poisson enrichment, which is an indication that some models do have good denoising properties.

We have also enhanced our benchmark by incorporating Chemprop, a model that utilizes molecular graphs as inputs, rather than traditional fingerprints (refer to Tables 1 and 2 in the updated revised paper). We hope that the inclusion of Chemprop addresses your concerns and improves the comprehensiveness of our benchmark.