Hi-TPH: A Large-Scale Hierarchical Dataset for TCR-pHLA Binding Prediction

The rationale for including specific components at different levels, such as why certain levels exclude the HLA component, could be elaborated to clarify the dataset structure.

Many of the papers listed in Table 1 claim to pull from exactly the same, or a very similar, set of original datasets as Hi-TPH (VDJdb, IEDB, McPAS, MIRA). The reader might therefore expect a similar number of samples in the datasets used in these papers as in Hi-TPH. I suspect it may have to do with the inclusion of murine data in Hi-TPH, but the paper could be improved by including a thorough discussion of this discrepancy. The introduction of this dataset would be more impactful if the authors could show that including this additional data improves performance at the binding prediction task.

The dataset is "randomly" split into training, validation, and test sets, but it is not clear the steps taken to prevent data leakage: can the same CDR3$\beta$/peptide/etc. occur in the same splits? This is a non-trivial task for the datasets lower in the hierarchy, and could do with more discussion. Leakage of sequences lower down in the hierarchy that cannot exist higher up the hierarchy may impact the benchmarking results. Moreover, there is a discussion of unseen peptides in the test set, but it is unclear how these unseen peptides are chosen: are they just from peptides which have by chance not appeared in the training dataset? If so, they are more likely to belong to the long tail described in Section 3.2, which could introduce a bias, and the unseen benchmark might improve by selecting the peptides in a more appropriate way.

Although the "on-the-fly mispairing" is used in training, it is not clear how the negatives in the test set are constructed - is the test set deterministic?

This paper lists several models in Table 1, but these are not included in the benchmark.

The number of papers published on this topic has increased during the recent years. Authors try to be extensive in describing some of them but also ignore several recent contributions. Authors note for example (line 70), that majority of previous works have focused on using only the complementarity determining region 3 (CDR3). There are several earlier works, some of which are also cited in this manuscript, that try to utilize all parts of the TCR alpha and beta chains, such as ERGO-II (cited here), Titan, TCRGP, epi-TRACE, DEEPTCR, and perhaps some others.

Manuscript is a bit unclear of whether it is primarily the CRD3 that is the key determinant of the interaction, or do other TCR parts also contribute to the interactions. Glanville et al. (cited here) carried out a structural analysis of contacts between amino acids between different parts of the TCR, peptide MHC, using crystal structures, suggesting that also parts beyond CDR3 are important, and earlier studies have observed similar in terms of features important in their ML methods (e.g. ERGO-II and epi-TRACE, perhaps others that I do not remember now).

As mentioned above, the number of TCR, peptide-MHC pairs analysed in previous studies has been affected by pre-processing steps that the authors have decided to use, as well as the number of data points in the VDJdb and IEDB databases at the time of earlier studies. Claiming significant increase in the number of data points is somehow subjective, because essentially all previous papers have been using the exact same datasets. If one downloads the VDJdb and IEDB datasets, one immediately gets about 200k data points (this is an estimate, I didn’t do check the numbers now), or about 20k datapoints that would be in level 4 (using the terminology of this manuscript), which are comparable numbers reported here. Previous papers that have released their code have also made data processing scripts available, so earlier scripts makes such datasets also automatically available. I understand that this manuscript tries to make data even more easliy accessible to ML community, but I am not sure the contribution of this manuscript is significant enough to be a separate publication, as many earlier works have done exactly the same task.

One challenge in this ML prediction problem is that the experimental dataset contains only positive data points and negatives are typically artificially generated. Authors propose here to use so-called on-the-fly mispairing for that purpose. In practise, that means doing the mispairing randomly e.g. for each training epoch. Some of the previous methods may do that as well, e.g. if I remember correctly ImReg tool only takes the positive data points as an input. Whether it resamples the negative for each epoch, I would need to check from the code.

Some of the earlier methods, can also utilize partial information about TCRs (i.e., when the full length protein is not available for each data point), e.g. a recent method called TULIP.

Long-tail distribution of binding TCRs for different peptides is a known challenge in these datasets, but authors do not provide any solutions.

Authors conclude that HLA information (that encodes MHC genes) may not provide additional predictive capabilities. This may be too strong of statement, and perhaps rather reflect limitations of the current datasets, where peptide is typically measured only in a single HLA context, resulting in little variability for ML models to learn from.

Results. Authors provide baseline comparison results for the prediction task using different ML methods, including random forest, MLP, and language-based methods. It seems that apart from TAPE-BERT, none of the baseline methods used exactly correspond to any of the published methods. The comparisons would make more sense and would be more valuable it they were carried using published methods that are all tailored for this task.

The dataset lacks continuous confidence scores that differentiate between strong and weak bindings, which may limit the models' ability to capture the full spectrum of binding affinities. The benchmark may have not been well tuned. Why the larger ESM model performs worse than small models? While the dataset is large, the ability of models to generalize to new peptides remains a challenge, indicating that the current dataset may not fully capture the diversity of TCR-pHLA interactions.