ComputAgeBench: Epigenetic Aging Clocks Benchmark

Q1: ...developing new methods on epigenetic aging clocks...

The dataset, gathered for benchmarking, is also suitable for differential methylation analysis, clustering, and developing epigenetic clocks. It enables exploration of novel clocks targeting CpG sites distinguishing aging-accelerating conditions (AACs) from healthy controls (HCs). Spanning 19 AACs, both genders, and a 90-year age range, it offers valuable opportunities for clock research. While the primary aim was to validate clocks for estimating accelerated aging, not training them, the HC cohort can still support clock training, complementing ongoing open-access efforts [R1].

Reference:

R1. Ying, K., ... & Gladyshev, V. N. (2023). A Unified Framework for Systematic Curation and Evaluation of Aging Biomarkers. bioRxiv, 2023-12.

Q2: How does the dataset account for demographic and biological diversity?...

Thank you for your comment. While collecting the dataset, we strived to find balance between data quantity and diversity. Because the published sets of DNA methylation (DNAm) data come from different studies with varying goals, they often lack a thorough annotation of patient health and other conditions. By aggregating as much data sources as possible, without excluding any genders or ethnicities, we presented the most representative collection of human DNAm profiles in health and age-related disease. The details on age and gender distribution within AACs and HCs per dataset are provided in Appendix figures A1, A2 (Ref. section A.8). Clearly, there is room for improvement in terms of data sources, which we outlined explicitly in section A.1. Additionally, we have calculated the following Table R1 with sample counts, showing that the majority of data is unlabeled by ethnicity (and sometimes gender), even though we did our best to find quality data.

Table R1. Sample counts across by ethnicities and genders (M=male, F=Female, U=Unknown) in the aggregated dataset.

Q3: ...to balance the dataset across ... AACs and healthy controls, or to mitigate known biases in the sample selection process?

To ensure that the datasets were not biased towards specific AACs, we first defined a set of criteria for AACs and datasets, targeting major human organ systems and then acquired 66 datasets, covering 19 of the 32 identified conditions (Ref. Table A2 for details). We aimed to assemble a comprehensive dataset to validate aging clocks, while mitigating any bias in sample selection through the diversity and large number of studies included. To prevent scenarios where a given aging clock might favor better predictions for a particular class of conditions (e.g., ISD), we present a decomposition of the clock’s AA2 and AA1 scores, as shown in Figure 3E,F. Additionally, we address the balance across categories by presenting sample distributions across conditions, ages, and demographic groups in Appendix Figures A1 and A2. These decompositions effectively prevent misinterpretations and reduce potential clock bias caused by dataset imbalances across the AACs.

Q4: ...datasets already exist publicly, reduces the novelty of the benchmark. However, ... putting together 66 datasets ... is a contribution...

Indeed, gathering and curating a new dataset of this size is a complex and a time-intensive endeavor (took us three years), because there are no options for an immediate selection of data relevant to the clock research. Collecting a large number of new DNAm samples from humans is increasingly difficult due to the high costs, significant time investment, and ethical challenges. This is also the reason why many clock-related papers use chaotically varying sets of data: everyone uses what they could find, with different teams managing to pre-process and split the data differently. Unfortunately, the rationale behind including or excluding data on a particular health/disease condition is rarely articulated as well.

Hence, by introducing a clearly stated methodology for conditions selection and by consolidating open-access datasets, we aspire to remove these barriers and to provide the research community with a valuable, easily accessible resource to accelerate the validation of aging clocks in a standardized fashion.

"Q1: ..better suited for a forum more specific to biological age.."

Although we hear the concern, we respectfully argue that the biological age, as presented in our study, aligns closely with the goals of representation learning, because the community looks for an interpretable univariate representation of complex biomarker data. Biological age encapsulates a high-dimensional array of biomarkers into a single, accessible latent variable. This process is achieved by training models under specific assumptions that have evolved across the generations of aging clocks, similar to how representation learning frameworks have refined their assumptions to improve the quality and relevance of latent representations in other biologically-relevant tasks presented at ICLR previously [R1, R2, R3].

Furthermore, the utility of biological age extends into numerous downstream applications, such as predicting all-cause mortality, estimating multi-morbidity risk, and evaluating general health status. These applications literally highlight its role as an interpretable and practically valuable representation with real-world impact [R4, R5]. In this way, biological age enriches traditional representation learning by adding interpretability and ease of use, bridging machine learning and healthcare. Regarding the tasks proposed in our benchmark, we emphasize that the task selection is inspired by integrating the established practices of aging clock evaluation found in numerous studies. We employ a clear, interpretable approach for testing aging clocks, which is relevant to a broader research community in data and life sciences.

Q2: ..normalization..

During the data pre-processing step of our study, we were primarily oriented at the most common use case of applying aging clocks by anyone other than the clockmaker. That is, when a researcher collects and processes their data in a way they find the most appropriate (for data, not for the aging clocks they plan to use), and then apply an aging clock model trained on other data, already pre-processed by the clock authors. As there is no gold standard for DNA methylation pre-processing, each research group carries out their own pre-processing that does not necessarily match the pre-processing pipeline used for training the clock model. For example, this is almost always the case when older clock models, such as Horvath2013 or Hannum2013, are applied to recently acquired data.

Therefore, so as to retain this typical workflow and not to put any clock model into advantage by choosing the same pre-processing that matches its own pipeline for every dataset, we decided to include already pre-processed datasets. In doing so, we also relied on two existing papers. First, compiling already pre-processed datasets without performing the same pre-processing for all of them was done in [R6], another notable effort in the aging clock community. Second, we were also encouraged by a recent paper by Varshavsky et al. [R7] who managed to create an accurate clock model by combining several blood datasets—without any additional normalization or correction procedure, using already pre-processed data from previous studies (some of which are included in our dataset as well), and thus demonstrating that the between-dataset normalization is not critical for this type of data. We discussed our reasoning in the Section 3.3 of our Benchmarking Methodology chapter and in the Appendix section A.9, and we thank you for pointing out the potential misunderstanding that we will clarify in the revised text.

Q3: A minor issue...

Admittedly, we had considered removing this list in the Appendix, but eventually kept it in the Methodology section so that all the respective studies of DNA methylation profiling would be cited in the main references list. However, this issue is indeed minor, and we are open to replacing the references in the revised version.

References

R1 Marin, F. I., et al. Bend: Benchmarking dna language models on biologically meaningful tasks. // ICLR 2024

R2 Pandeva T., Forré P. Multi-View Independent Component Analysis for Omics Data Integration //2023 ICLR

R3 Zhou Z. et al. Dnabert-2: Efficient foundation model and benchmark for multi-species genome //arXiv preprint arXiv:2306.15006. – 2023.

R4 Pyrkov T. V. et al. Longitudinal analysis of blood markers reveals progressive loss of resilience and predicts human lifespan limit //Nature communications. – 2021.

R5 Pierson E. et al. Inferring multidimensional rates of aging from cross-sectional data //The 22nd International Conference on Artificial Intelligence and Statistics. – PMLR, 2019.

R6 Ying, K., et al. (2023). A Unified Framework for Systematic Curation and Evaluation of Aging Biomarkers. bioRxiv, 2023-12.

R7 Varshavsky, M., et al. (2023). Accurate age prediction from blood using a small set of DNA methylation sites and a cohort-based machine learning algorithm. Cell Reports Methods.

Q1: Will you make your benchmarking dataset publicly available?

Yes, we will definitely make it fully available, and the links to both the dataset and the code required to reproduce all figures are already prepared, but are omitted in the anonymized version of this submission.

Q2: Can you please confirm that your evaluation tasks/metrics are original, and add citations if not? & W2: ..cumulative score requires more justification

We write in the article: “Two approaches we propose as essential tasks in our benchmark entail related prior art. For example, Porter et al. [41] and Mei et al. [34] used one-sample or two-sample aging acceleration tests for clock validation”, demonstrating that the AA2 (see [R1, R2]) and AA1 (see [R3]) tasks were well-established and actively utilized previously. Hence, constructing a score based on these approaches for clock validation was a natural choice. Furthermore, calculating the median absolute error (as in [R4, R5]) to measure the accuracy of chronological age prediction is also a well-known approach.

The reason a weighted sum of the four metrics is not applicable in our case is that only the AA2 and AA1 tasks are essential for testing clock validity, as outlined in the requirements provided in the Background section. The other two tasks serve as auxiliary measures, providing additional insights into the degree of clock calibration to chronological age.

Please kindly refer to our response to Reviewer PX5Y Q2 for additional details regarding the rationale behind the design of the BenchScore metric.

Q3: Can you make a case for why the paper is a strong fit for ICLR, despite not truly being in the representation learning space?

Please kindly refer to our response to Reviewer 3vvy Q1, where we explain why the biological age is indeed a learnable representation. Think of the task of BA prediction that has no ground truth. It bears similarities with unsupervised learning task when the latent representation (BA) is obtained within a particular training procedure. For example, in the 1st generation clocks (or even unsupervised clocks), the BA cannot be verified with a ground truth – and yet – these clocks are still used in downstream clinical tasks. Likewise, a significant portion of ICLR submissions is concerned with tasks that involve unknown targets, metrics, and data that lack annotation. That is why we believe that the ML community is the best place where aging researchers could seek help from.

W1: ...weren't all re-trained on a standardized training dataset...

Indeed, the standardized training data are important, but often missing from clock studies, with the Biolearn effort [R1] being one of the few sources to provide it. Taking pre-trained models was exactly the point of doing our comparison, because most published aging clock models employ the same architecture (most often linear regression-based). The outcomes are unique only thanks to their training data. Therefore, re-training clock models on a single training dataset would create a completely novel clock that would have little in common with the published ones. Moreover, the second-generation clocks rely on data combining mortality and DNAm values, which, as noted in our manuscript, are either unavailable or restricted due to ethical concerns.

W3: Requires clarification: ..."Clearly, the first task [AA1] provides a more rigorous way to test aging clocks [compared to AA2]"...

Thank you for highlighting a potential misunderstanding. We will clarify by mentioning the tasks explicitly: “Clearly, the first task (AA2) provides a more rigorous way to test aging clocks (compared to AA1) …”. AA2 is more rigorous because it compares predictions between diseased and healthy patients, accounting for possible age acceleration due to batch effects, rather than comparing diseased patients to zero acceleration. This ensures that clocks systematically predicting accelerated ages for healthy subjects gain no advantage in AA2, but might appear successful in AA1. To address this, we penalize the AA1 score for prediction bias, which justifies the need for a cumulative benchmarking score.

References:

R1 Mei X et al. Fail-tests of DNA methylation clocks, and development of a noise barometer for measuring epigenetic pressure of aging and disease. Aging (Albany NY), 2023.

R2 Ying K et al. Causality-enriched epigenetic age uncouples damage and adaptation. Nature Aging, 2024.

R3 Porter HL et al. (2021). Many chronological aging clocks can be found throughout the epigenome: Implications for quantifying biological aging. Aging cell, 20(11), e13492.

R4 Horvath S (2013). DNA methylation age of human tissues and cell types. Genome biology, 14, 1-20.

R5 Thompson MJ et al. A multi-tissue full lifespan epigenetic clock for mice. Aging (Albany NY), 10(10), 2832.

R6 Ying K et al. (2023). A Unified Framework for Systematic Curation and Evaluation of Aging Biomarkers. bioRxiv, 2023-12.

We sincerely thank you for your additional reflection on our work and we hear your concern!

First, the major contribution that we strived to make was not the dataset itself. It was an explicit methodology of selecting the specific conditions and datasets, based on clear, clinically relevant assumptions about how clocks should behave if we want them to be truly indicative of a person's biological age. The main problem with epigenetic aging clocks now, in our opinion, is that there is no consensus way that they can be validated with using open-source data, as biological age is a latent variable. By presenting our benchmark, we are, for the first time, providing researchers with a means of reliably comparing clock models, not just by how well they predict chronological age, but by how well they can perform in conditions that are proven to substantially decrease life expectancy. Our approach may not become consensus in the end, but, currently, it is the only one that exists for omics data, and we hope it will generate fruitful discussion in the field.

Second, while most aging clocks indeed rely on linear models, their training methodologies differ significantly, with first-generation clocks predicting chronological age and second-generation clocks predicting mortality using Cox proportional hazard models, requiring differing assumptions about the biological age as a learnt representation.

After three years of curating in-human data for our unifying benchmark, we see a critical need for ML expertise to develop robust biological age predictors, and we are well aware of the expectations at traditional ML conferences. It is to bridge this gap that we created our fully open-access benchmarking approach and repository. We hope that the optimal longevity markers will emerge through collaboration between the biological and ML communities.

Lastly, please kindly note a trend in ICLR and other top conferences to accept similar works:

-- Marin, et al. “Bend: Benchmarking dna language models on biologically meaningful tasks”, ICLR 2024.

-- Sihag, et al. “Explainable brain age prediction using covariance neural networks”, NeurIPS 2023.

-- Pandeva T., Forré P. “Multi-View Independent Component Analysis for Omics Data Integration”, ICLR 2023.

-- Zhou Z. et al. “DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes”, ICLR 2024.

-- Weinberger, E., & Lee, S. I. “A deep generative model of single-cell methylomic data”, NeurIPS 2023 (GenBio Workshop).

Q1: The selection of metrics ... "presence of covariate shift," but this shift is not clearly explained.

First, we want to thank you for pointing out a typo: in the description of the AA1 task (Section 3.5), we indeed meant “...mean aging acceleration…” instead of “median,” as the applied Student’s t-test is used to determine whether the difference in means between two groups is significant. We will correct this typo in the revised version of the text.

Second, for tasks 3 and 4, the choice of the median is more appropriate due to its robustness to outliers compared to the mean. Thus, simplicity and robustness are the two primary reasons for favoring the median.

Third, covariate shift, also referred to as a batch effect in bioinformatics, denotes the shift between the distributions of covariates in two datasets. For instance, the distribution of methylation values for a given CpG site could be centered around 0.45 in one dataset and around 0.55 in another—a common scenario in DNA methylation and other types of omics data. To evaluate the robustness of a given aging clock model to covariate shift (batch effect) between the original clock training dataset and datasets from the proposed benchmark, we introduced a prediction bias task. In this task, we calculate the median age acceleration, which reflects the systematic shift in clock predictions caused by differences between datasets. We will add explicit clarifications for the covariate shift in the revised version of the manuscript.

Q2: The rationale behind the summary benchmark score..

While designing our metric, we aimed for simplicity and interpretability. At the same time, we sought to include more data in the benchmark to address the data scarcity caused by the underrepresentation of certain AACs. In the simplest case, we could sum up the AA2 and AA1 scores; however, this approach would be unfair. Clocks exhibiting a large systematic bias in their predictions might automatically perform better in the benchmark due to their advantage in the AA1 task. Since we evaluate only aging-accelerating conditions, a positive systematic bias (where "positive" means that the predicted age acceleration tends to be statistically higher for healthy controls, whereas we expect it to be zero) should not be too large. Such bias gives an unfair advantage to the model. To account for this, we introduced a bracketed term in the BenchScore, which penalizes clocks with excessive systematic bias in their AA1 scores.

Additionally, we provided a full decomposition of our metric in the form of Figures 3E,F and in Table 1. This allows for a detailed examination of each clock's performance. As the authors of the first aggregating metrics, we hope this work sparks active discussion and contributes to the development of more advanced metrics in the future.

Q3: It appears that plots C and D in Figure 3 may be incorrectly presented. Plot D should likely represent Med(|∆|) rather than Med(∆), as all points are above the diagonal. Please clarify if this is a mislabeling or if I have misunderstood the data shown.

No mislabeling here, but thank you for pointing to a potential issue with the readability of the figure. We will rewrite the caption in Fig 3C and D to be less confusing. The meaning of Chronological age prediction accuracy (Fig. 3C) is to measure the absolute error for each data sample (i.e. Med(|∆|)). In Chronological age prediction bias (Fig. 3D), we measure the shift of the overall prediction (i.e., Med(∆), as written in the paper) and it can be of negative or positive value. To better demonstrate the concept of “prediction bias” we sketched a limiting case in the Figure 3D when all samples were predicted with a positive age acceleration. This casts a strictly positive value of Med(∆), which is graphically represented as a red arrow on the figure.