Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcity

It is not clear how significant the results are in the private dataset section.

We agree with the reviewer that only online experiments can reveal the true impact of an LTR model due to the counterfactual nature of the LTR problem. That is from offline experiments, it is not clear how the user might have responded to the ranking provided by the model. However running online experiments on real-world user traffic needs to be carefully considered as it has a potential opportunity cost. Unfortunately, due to superseding business priorities we could not run online experiments. Similarly, unfortunately at this point we cannot reveal more details of the baseline, but we will try our best to provide more details in the camera ready version.

To provide more confidence in our private dataset results, we updated our Table 3 to provide error bars to our results.

Tabular (a.k.a.numerical) ML (e.g. LTR) systems are important for modern enterprises and therefore validating research hypotheses on large-scale practical datasets is highly valuable. Unfortunately, this is challenging due to the expensive and proprietary nature of such data and most LTR papers [2,3,4] do not prove their hypotheses on practical large-scale datasets. We demonstrate pretraining rankers achieve SOTA on public datasets but also emphatically on a large-scale commercial dataset. Therefore, we believe our results will be useful to the LTR community and the wider Tabular-DL community.

We thank the reviewer for their consideration, and are happy to help with any further concerns. If we have addressed the concerns we would appreciate it if the reviewer could consider raising their score.

[1] Transfer Learning with Deep Tabular Models. Roman Levin, Valeriia Cherepanova, Avi Schwarzschild, Arpit Bansal, C. Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, Micah Goldblum. ICLR 2023.
[2] Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees?, Qin et al., ICLR 2021.
[3] Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networks., Ai et al., SIGIR 2019. [4] SetRank: Learning a Permutation-Invariant Ranking Model for Information Retrieval., Pang et al., SIGIR 2020

Thank you for your review and feedback. We would like to first take an opportunity to clarify the impact and contributions of our work.

We strongly believe our work will significantly impact industries that work with LTR problems (search, recommendation), by causing them to revise, or at least revisit, current SOTA methods. Today many large companies use GBDTs for ranking tabular data, and this approach is SOTA even in the research community. The novel starting point of our work is the often ignored fact that most of the data collected in LTR systems is unlabeled. We have shown through extensive experiments on standard public datasets and a large-scale private dataset that pretrained deep rankers can leverage unlabeled data to consistently outperform GBDTs, with outsized gains in robustness towards outlier data.

All reviewers agree we have convincingly demonstrated this. Reviewer 1XYq: “Experimentally, authors show that pretrained models can outperform GBDTs, which is one of the strongest baselines, under label scarcity setting.” Reviewer oW1i: “the paper highlights the superior performance of pretrained deep rankers, especially on outlier queries, in scenarios with limited labeled data.” Reviewer i3NM: “The major contribution of the paper is to identify areas where [pretrained deep rankers] help, including query sparsity and label sparisity scenarios.”

We now address the concerns of the reviewer.

Will decreasing the number of negatives in SimCLR be sufficient to apply SimCLR to ranking problems?

Thank you for asking this insightful question. First, we would like to clarify that our primary goal is to demonstrate that DL rankers built using pre-training techniques can achieve SOTA performance in LTR settings, and consequently beat GBDT models (prior SOTA). Only as a secondary goal, we investigate a few simple but effective pre-training strategies such as SimSiam, SimCLR, and our variant SimCLR-Rank. While we identified SimCLR-Rank has a positive impact in a wide variety of LTR scenarios, we also show that SimSiam works better for our large-scale proprietary dataset. Our goal is not to show that any one particular method is the best for pretraining in LTR.

As the reviewer correctly identified, SimCLR-Rank is different from SimCLR in two aspects: (1) it uses items from the same query group as freely available hard-negatives, (2) it reduces the number of negatives from batchsize to at most query group size and thus allow better computational scaling with batchsize. To identify which of these contribute more towards SimCLR-Rank’s performance, we performed an additional experiment below (as suggested by the reviewer), comparing SimCLR-Rank with a variant of SimCLR (which we call as SimCLR-sample). In SimCLR-sample, instead of using all the other items as negatives, we uniformly sample a constant number of negatives from the batch. This reduces the computational complexity from quadratic in batchsize (of SimCLR) to linear in batchsize. In the below experiments we follow the setup in Subsection 4.3.2 and SimCLR-Rank and SimCLR-sample use the same number of negatives and batchsize.

We find that SimCLR-Rank performs better on MSLR/Istella, while SimCLR-sample performs better on Yahoo. We note that MSLR/Istella are much sparser than Yahoo (see Table 12 in the paper), and are more representative of typical search and recommendations applications. Therefore we still believe that SimCLR-Rank which selects hard-negatives from the same query group is a useful pre-training method for the LTR toolbox and can be highly effective in many situations.

Pretrained models in tabular datasets are not very extendable.

The reviewer’s observation that our pretrained models cannot take advantage of transfer learning is correct. Tabular datasets have specific features which can be very different. In fact, each tabular dataset is its own domain. Because in general there are no common features between tabular datasets, it is not clear how to have common knowledge. Levin et al. [1] show that if features are shared, it is possible to transfer knowledge between datasets, but in general it is not known how to do transfer learning in tabular data. The same shortcoming is true of GBDTs. Transfer learning for tabular data is a question of independent interest, and beyond the scope of our paper.

We thank the reviewer for their helpful feedback during this rebuttal process.

A follow-up W1: we produced a comprehensive comparison between SimCLR-Rank, SimSiam, and many SOTA tabular pretraining baselines in a reply to another reviewer: https://openreview.net/forum?id=Dk1ybhMrJv&noteId=1Z4qMYixVv. We hope this, combined with our earlier responses, gives the reviewer more confidence in our results.

We thank the reviewer for their time. Please kindly let us know if there are additional comments you have for us.

We would like to thank the reviewer again for their encouraging feedback. While we have an opportunity, we would like to make an additional point about W2 (extendability):

It is common for unsupervised pretraining (self-supervised learning) papers to not consider transferability (including in very high-impact and influential works). Examples include: tabular classification [1,2,3,7], image classification [4,5,6], and text classification [4,5]. Like these papers, we show that pretraining in the same dataset provides significant benefits, and transfer learning is not needed to achieve these results.

Extendability is even harder to prove in LTR than the general tabular setting. In tabular data (where transferability is hard, as noted in Ucar et al. [1]), overlapping columns seems to be what is needed to achieve transferability [8]. Getting overlapping columns seems to be nearly impossible in the three benchmark LTR public datasets, two of which (Yahoo, Istella) [9,10] do not specify what the columns even mean (due to the need to protect business trade secrets). The most transparent, MSLRWEB30K [11], also has many columns whose meanings may not make sense outside of the context of the dataset (for example, features that are the outputs of Microsoft internal models).

We hope our discussion has adequately addressed your feedback on (1) SimCLR-Rank's utility, (2) extendability of results, (3) the significance of the results. Please kindly let us know if there are additional comments you have for us.

[1] SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning. Talip Ucar, Ehsan Hajiramezanali, Lindsay Edwards. NeurIPS 2021.
[2] VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain. Jinsung Yoon, Yao Zhang, James Jordon, Mihaela van der Schaar. NeurIPS 2020.
[3] Towards Domain-Agnostic Contrastive Learning. Vikas Verma, Minh-Thang Luong, Kenji Kawaguchi, Hieu Pham, Quoc V. Le. ICML 2021.
[4] Contrastive learning with hard negative samples. Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, Stefanie Jegelka. ICLR 2021.
[5] Debiased Contrastive Learning. Ching-Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, Stefanie Jegelka. NeurIPS 2020.
[6] Robust Contrastive Learning Using Negative Samples with Diminished Semantics. Songwei Ge, Shlok Mishra, Chun-Liang Li, Haohan Wang, David Jacobs. NeurIPS 2021.
[7] SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption. Dara Bahri, Heinrich Jiang, Yi Tay, Donald Metzler. ICLR 2022.
[8] Transfer Learning with Deep Tabular Models. Roman Levin, Valeriia Cherepanova, Avi Schwarzschild, Arpit Bansal, C. Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, Micah Goldblum. ICLR 2023.
[9] Yahoo! Learning to Rank Challenge Overview. Olivier Chapelle, Yi Chang.
[10] Fast Ranking with Additive Ensembles of Oblivious and Non-Oblivious Regression Trees. Domenico Dato et al.
[11] Introducing LETOR 4.0 Datasets. Tao Qin, Tie-Yan Liu.

[1] SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning. Talip Ucar, Ehsan Hajiramezanali, Lindsay Edwards. NeurIPS 2019.

The potential improvement of SimCLR-Rank over SimCLR might be limited in practical situations.

We thank the reviewer for their insightful question. As the reviewer correctly identified, all the negative samples in the SimCLR-Rank loss are present in the SimCLR loss as well, so the qualitative and quantitative behaviors of the two algorithms are likely similar.

However, SimCLR is difficult or impossible to use in large-scale big-data settings because of its high runtime and space complexity (which is quadratic in the batch size). Reviewer i3NM also noted that SimCLR has this same shortcoming: “the major weakness of SimCLR for non-ranking problems was complexity.”

To overcome this issue, we leverage the fact that in LTR the hard negatives are in the same query group (as the reviewer correctly notices), so we do not have to use the entire batch as negatives. This (1) provides orders of magnitude improvement in speed (Table 6 in the paper, page 16), and (2) performs slightly better than SimCLR even with the runtime and memory savings (Table 5 in the paper, page 9).

To summarize, we do not claim that SimCLR-Rank’s loss will produce significantly different outcomes than SimCLR. However, SimCLR-Rank is significantly faster, which makes it applicable in real-world big data settings for ranking.

We thank the reviewer for their consideration, and are happy to help with any further concerns. If we have addressed the concerns we would appreciate it if the reviewer could consider raising their score.

Could we use features from the retrieval model to improve GBDT performance over SimCLR-Rank?

We thank the reviewer for this very interesting conjecture.

First, we would like to note that the features of the retrieval model are in fact used in our LTR experiments on the private dataset. Despite this, SimCLR-Rank still outperforms the baseline model. Second, in the public datasets we do not have access to the upstream retrieval model features to test this hypothesis.

The last observation we would like to make is that retrieval models are typically not large or powerful in applications like search or recommendations. They are typically designed to be fast models (e.g. KNNs or even handcrafted rules) with high recall (and potentially low precision) to quickly filter millions or billions of items into hundreds of possibly relevant items. Since they are trained for recall and not for precision, these models and their features may find it difficult to differentiate and rank the relevant items in the query group. Therefore, we conjecture that distilling from retrieval to rankers might not work well.

Thank you for your review and feedback. We would like to first take an opportunity to clarify the impact and contributions of our work.

We strongly believe our work will significantly impact industries that work with LTR problems (search, recommendation), by causing them to revise, or at least revisit, current SOTA methods. Today many large companies use GBDTs for ranking tabular data, and this approach is SOTA even in the research community. The novel starting point of our work is the often ignored fact that most of the data collected in LTR systems is unlabeled. We have shown through extensive experiments on standard public datasets and a large-scale private dataset that pretrained deep rankers can leverage unlabeled data to consistently outperform GBDTs, with outsized gains in robustness towards outlier data.

All reviewers agree we have convincingly demonstrated this. Reviewer 1XYq: “Experimentally, authors show that pretrained models can outperform GBDTs, which is one of the strongest baselines, under label scarcity setting.” Reviewer oW1i: “the paper highlights the superior performance of pretrained deep rankers, especially on outlier queries, in scenarios with limited labeled data.” Reviewer i3NM: “The major contribution of the paper is to identify areas where [pretrained deep rankers] help, including query sparsity and label sparisity scenarios.”

We now address the concerns of the reviewer.

The paper is missing important comparisons with other tabular SSL methods.

We thank the reviewer for bringing this to our attention. We added a comprehensive and expanded related works section to Appendix A.1 (page 14 in the appendix, colored in blue) discussing the current state of pretraining for tabular (a.k.a numerical) dataset. After the paper decision, we plan to reorder the paper to place it into the main text.

First, we would like to clarify that our primary goal is to demonstrate that DL rankers built using pre-training techniques can achieve SOTA performance in LTR settings, and consequently beat GBDT models (prior SOTA). Only as a secondary goal, we investigate a few simple but effective pre-training strategies such as SimSiam, SimCLR, and our variant SimCLR-Rank. While we identified SimCLR-Rank has a positive impact in a wide variety of LTR scenarios, we also show that SimSiam works better for our large-scale proprietary dataset. Our goal is not to show that any one particular method is the best for pretraining in LTR.

SimCLR-Rank is different from SimCLR in two aspects: (1) it uses items from the same query group as freely available hard-negatives, (2) it reduces the number of negatives from batchsize to at most query group size and thus allow better computational scaling with batchsize. To identify which of these contribute more towards SimCLR-Rank’s performance, we performed an additional experiment below (as suggested by Reviewer i3NM), comparing SimCLR-Rank with a variant of SimCLR (which name SimCLR-sample). In SimCLR-sample, instead of using all the other items as negatives, we uniformly sample a constant number of negatives from the batch. This reduces the computational complexity from quadratic in batchsize (of SimCLR) to linear in batchsize. In the below experiments we follow the setup in Subsection 4.3.2 and SimCLR-Rank and SimCLR-sample use the same number of negatives and batchsize.

As an additional comparison (suggested by reviewer 1XYq), we also evaluate SubTab [1] in the setting of Subsection 4.3.2. SubTab’s pretraining objective divides input features into multiple subsets (in the language of computer vision, “multiple views”) and trains an autoencoder to reconstruct the original input features. To pretrain using SubTab, we divide the input features into 4 subsets with 75% overlap, and with input corruptions of 15% masking probability and Gaussian noise of scale 0.1 (suggested in the SubTab paper as a good setting). The finetuning strategy and the choice for encoder model is the same as for SimCLR-Rank and SimCLR-sample. We find that SubTab does very poorly on the MSLR and Istella datasets while performing respectably on Yahoo (though worse than both SimCLR-Rank and SimCLR-Sample).

It becomes challenging to attribute a substantial level of innovation to this work.

We thank the reviewer for an opportunity for us to clarify the innovation in our work. SimCLR [14,15,16,17] is known to need large batch sizes to work well, which combined with its quadratic compute scaling in batch size makes it too inefficient. For example, pretraining SimCLR on CIFAR100/ImageNet requires many nodes of V100 GPUs [15].

The naive way to make SimCLR more efficient is to sample the negatives, which prevents quadratic scaling in batch size. This is SimCLR-sample, which we already showed does not perform very well in MSLR/Istella (the two harder LTR public datasets). A more sophisticated approach is to only include hard negatives in the contrastive loss, which Robinson et al. [18] find makes contrastive learning much faster and also perform better.

Our main technical insight (SimCLR-Rank) is that the hard negatives in LTR are already known–we do not have to mine for them at all because they are just samples in the same query group. Effectively, we utilize the structure of the LTR problem to obtain hard negative examples for free. Previous works showing how to mine hard negatives have been considered worthy contributions to the ML literature [18,19,20]. Note that the simplicity of SimCLR-Rank makes it more adaptable and thus more impactful: one can for example combine SCARF/DACL with SimCLR-Rank.

We hope this response explains our position better on the technical contribution of our paper.

Confining the scope of this paper solely to its application in the niche domain of learning-to-rank significantly diminishes its impact.

We thank the reviewer for the opportunity to elaborate on the state of LTR in research and industry. LTR is a central component of highly important applications like search and recommendation, with large impacts on the economy and society. Many works solely on LTR have been published at top ML conferences [5,6,7,8,9,10,11,12, 28, 29, 30, 31]. There was also recently a workshop on ranking at ICML 2023 [13] (https://icml.cc/virtual/2023/workshop/21495). Burges et al. [11] won the ICML 2015 Test of Time award for their publication on LTR. So we believe significant results in the space of LTR are likely to achieve significant impact (and have done so in the past). LTR is a significant area in both machine learning research and industry.

Recently, LTR has also garnered significant attention due to the use of new LLM (large language model) applications like Retrieval Augmented Generation (RAG) [25]. In RAG, a document retrieval mechanism is paired with an LLM to enhance the factual accuracy of an LLM’s response to a query. Recently, the document retrieval mechanism has included re-ranking [26] and tabular features [27]. As a result, we believe LTR is not only a significant area now but also will become even more important in the future.

We appreciate the reviewer’s comments, as they have been highly helpful in improving our paper. If the concerns have been addressed, we would appreciate it if the reviewer could consider raising their score.

[1] SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning. Talip Ucar, Ehsan Hajiramezanali, Lindsay Edwards. NeurIPS 2019.
[2] SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption. Dara Bahri, Heinrich Jiang, Yi Tay, Donald Metzler. ICLR 2022.
[3] VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain. Jinsung Yoon, Yao Zhang, James Jordon, Mihaela van der Schaar. NeurIPS 2020.
[4] Towards Domain-Agnostic Contrastive Learning. Vikas Verma, Minh-Thang Luong, Kenji Kawaguchi, Hieu Pham, Quoc V. Le. ICML 2021.
[5] Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees?, Qin et al., ICLR 2021. [6] Toward Understanding Privileged Features Distillation in Learning-to-Rank., Yang et al., NeurIPS 2022. [7] PiRank: Scalable Learning To Rank via Differentiable Sorting. Robin Swezey, Aditya Grover, Bruno Charron, Stefano Ermon. NeurIPS 2021.
[8] On the Value of Prior in Online Learning to Rank. Branislav Kveton, Ofer Meshi, Masrour Zoghi, Zhen Qin. AISTATS 2022.
[9] Learning to rank using gradient descent. Learning to rank using gradient descent. Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, Greg Hullender. ICML 2005.
[10] Learning to rank: from pairwise approach to listwise approach. Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, Hang Li. ICML 2007.
[11] Listwise approach to learning to rank: theory and algorithm. Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, Hang Li. ICML 2008.
[12] Ranking measures and loss functions in learning to rank. Wei Chen, Tie-Yan Liu, Yanyan Lan, Zhi-ming Ma, Hang Li. NeurIPS 2009.
[13] The Many Facets of Preference-Based Learning. Aadirupa Saha, Mohammad Ghavamzadeh, Robert Busa-Fekete, Branislav Kveton, Viktor Bengs. ICML 2023.
[14] A Simple Framework for Contrastive Learning of Visual Representations. Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton. ICML 2020.
[15] Exploring Simple Siamese Representation Learning, invited talk https://rosanneliu.com/dlctfs/dlct_210326.pdf. [16] Exploring Simple Siamese Representation Learning. Xinlei Chen, Kaiming He. CVPR 2021.
[17] Bootstrap Your Own Latent A New Approach to Self-Supervised Learning. Grill et al. NeurIPS 2020.
[18] Contrastive learning with hard negative samples. Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, Stefanie Jegelka. ICLR 2021.
[19] Debiased Contrastive Learning. Ching-Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, Stefanie Jegelka. NeurIPS 2020.
[20] Robust Contrastive Learning Using Negative Samples with Diminished Semantics. Songwei Ge, Shlok Mishra, Chun-Liang Li, Haohan Wang, David Jacobs. NeurIPS 2021.
[21] Self-training with Noisy Student improves ImageNet classification. Qizhe Xie, Minh-Thang Luong, Eduard Hovy, Quoc V. Le. CVPR 2020.
[22] STab: Self-supervised Learning for Tabular Data. Ehsan Hajiramezanali, Nathaniel Diamant, Gabriele Scalia, Max W. Shen. [23] Transfer Learning with Deep Tabular Models. Levin et al., ICLR 2023.
[24] MET: Masked Encoding for Tabular Data. Kushal Majmundar, Sachin Goyal, Praneeth Netrapalli, Prateek Jain.
[25] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
[26] Re2G: Retrieve, Rerank, Generate. Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Rajaram Naik, Pengshan Cai, Alfio Gliozzo.
[27] T-RAG: End-to-End Table Question Answering via Retrieval-Augmented Generation. Feifei Pan, Mustafa Canim, Michael Glass, Alfio Gliozzo, James Hendler.
[28] Beyond Greedy Ranking: Slate Optimization via List-CVAE. Ray Jiang, Sven Gowal, Timothy A. Mann, Danilo J. Rezende. ICLR 2019.
[29] Towards Amortized Ranking-Critical Training for Collaborative Filtering. Sam Lobel, Chunyuan Li, Jianfeng Gao, Lawrence Carin. ICLR 2020.
[30] Individually Fair Ranking. Amanda Bower, Hamid Eftekhari, Mikhail Yurochkin, Yuekai Sun. ICLR 2021.
[31] Adversarial Retriever-Ranker for dense text retrieval. Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, Weizhu Chen. ICLR 2022.

We thank the reviewer for their time and consideration.

LTR is a general problem and has been applied to tabular, text, images and various other domains. Among the papers we cited these focus exclusively on tabular LTR: [4,5,6,7,8,9,10,11,12] (all papers were published at ICLR, NeurIPS, ICML). One of these is Burges et al. [9], which won the ICML 2015 Test of Time Award. To give some more context, a significant portion of research and real-world applications of LTR have focused on tabular LTR due to its efficiency at web-scale and ubiquitous nature of tabular data.

We would also like to note that many impactful papers on pretraining published at top ML conferences also focused on only one modality and application: images [14,16,17,20,21], text [13,15,18,19], tabular [1,2,3]. Hence we strongly believe that studying the very practical (as illustrated in experiments) topic of pretraining for tabular LTR can also be similarly impactful and useful to the community.

We thank the reviewer for their continued engagement throughout the discussion period, and we hope that we have addressed the reviewer’s concerns.

[1] SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning. Talip Ucar, Ehsan Hajiramezanali, Lindsay Edwards. NeurIPS 2019.
[2] SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption. Dara Bahri, Heinrich Jiang, Yi Tay, Donald Metzler. ICLR 2022.
[3] VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain. Jinsung Yoon, Yao Zhang, James Jordon, Mihaela van der Schaar. NeurIPS 2020.
[4] Individually Fair Ranking. Amanda Bower, Hamid Eftekhari, Mikhail Yurochkin, Yuekai Sun. ICLR 2021.
[5] Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees?, Qin et al., ICLR 2021.
[6] Toward Understanding Privileged Features Distillation in Learning-to-Rank., Yang et al., NeurIPS 2022. [7] PiRank: Scalable Learning To Rank via Differentiable Sorting. Robin Swezey, Aditya Grover, Bruno Charron, Stefano Ermon. NeurIPS 2021.
[8] On the Value of Prior in Online Learning to Rank. Branislav Kveton, Ofer Meshi, Masrour Zoghi, Zhen Qin. AISTATS 2022.
[9] Learning to rank using gradient descent. Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, Greg Hullender. ICML 2005.
[10] Learning to rank: from pairwise approach to listwise approach. Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, Hang Li. ICML 2007.
[11] Listwise approach to learning to rank: theory and algorithm. Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, Hang Li. ICML 2008.
[12] Ranking measures and loss functions in learning to rank. Wei Chen, Tie-Yan Liu, Yanyan Lan, Zhi-ming Ma, Hang Li. NeurIPS 2009.
[13] Universal Language Model Fine-tuning for Text Classification. Jeremy Howard, Sebastian Ruder. ACL 2018.
[14] A Simple Framework for Contrastive Learning of Visual Representations. Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton. ICML 2020.
[15] Multilingual Constituency Parsing with Self-Attention and Pre-Training. Nikita Kitaev, Steven Cao, Dan Klein. ACL 2019.
[16] Exploring Simple Siamese Representation Learning. Xinlei Chen, Kaiming He. CVPR 2021.
[17] Bootstrap Your Own Latent A New Approach to Self-Supervised Learning. Grill et al. NeurIPS 2020.
[18] Adversarial Multi-task Learning for Text Classification. Pengfei Liu, Xipeng Qiu, Xuanjing Huang. ACL 2017.
[19] Unsupervised Sparse Vector Densification for Short Text Similarity. Yangqiu Song, Dan Roth. NAACL 2015.
[20] Robust Contrastive Learning Using Negative Samples with Diminished Semantics. Songwei Ge, Shlok Mishra, Chun-Liang Li, Haohan Wang, David Jacobs. NeurIPS 2021.
[21] Self-training with Noisy Student improves ImageNet classification. Qizhe Xie, Minh-Thang Luong, Eduard Hovy, Quoc V. Le. CVPR 2020.

Thank you for your feedback. We would like to clear up one last bit of confusion.

SimCLR-Rank is 7-14x faster wall-clock time and has an order-wise (big-O) speed/memory improvement over SimCLR (see Subsection 3.1). This is a significant improvement over SimCLR, which is known to require many nodes of V100 GPUs to train datasets like ImageNet (check the SimSiam talk slides https://rosanneliu.com/dlctfs/dlct_210326.pdf).

SimCLR is simply not practical in the big data regime (like large-scale recommendations/search/LTR), while SimCLR-Rank is, due to the speed difference. We hope the reviewer can incorporate this into their final evaluation.

Why does pretraining improve outlier NDCG?

It is a well-known phenomenon that pretraining helps improve robustness and out of distribution performance in domains like image and text [5, 6]. The same phenomenon, we find, is also true in the tabular LTR setting. Liu et al., [7] find that pretraining with data augmentation allows models to learn richer features they can generalize to outlier or out of distribution examples better–merely supervised training only incentivizes models to attend to highly predictive features in the training set. We conjecture that pretraining with augmentations in LTR (as we do in our paper) improves performance on outlier NDCG for similar reasons.

Can pretraining in one downstream dataset transfer to other datasets?

We thank the reviewer for bringing up this point. Tabular datasets have specific features which can be very different. In fact, each tabular dataset is its own domain. Because in general there are no common features between tabular datasets, it is not clear how to have common knowledge. Levin et al. [8] show that if features are shared, it is possible to transfer knowledge between datasets, but in general it is not known how to do transfer learning in tabular data.

Do the pretraining methods still outperform GBDT with semi-supervised learning?

We thank the reviewer for bringing up important baselines. Rubachev et al. [9] find that consistency regularization does not help GBDTs (https://openreview.net/forum?id=kjPLodRa0n&noteId=E4FT6VCltG). As suggested by the reviewer, we ran new experiments on pseudolabeling for GBDTs, and found that it (typically) performs worse than the GBDT baselines themselves.

Methodology: we allow 0.2% of the query groups to be labeled in each dataset, we train a GBDT on the labeled set and then use the GBDT to label the unlabeled train set.

A GBDT ranker outputs real-valued scores. Given that pseudolabeling datasets using GBDTs in LTR is not a well-studied problem, we propose the following approach. To convert real-valued scores into relevance scores, we take the lowest output score when labeling the unlabeled train set, and subtract it from all real scores. Then we round these positive floats into their nearest integers, and let these be the pseudolabeled relevance scores. In MSLR, this results in the pseudolabels being integers from 0-7, in Istella 0-12. For Yahoo we increase the spread of the scores by multiplying the positive floats by 100 before rounding to the nearest integer (resulting in the pseudolabels being numbers from 0-5), since without doing this all the pseudolabels would be 0.

We find that overall, pseudolabeling decreases the performance of GBDTs on MSLR and Istella (the sparser datasets which are typically more representative of search and recommendations applications, check Table 10 in the paper) while slightly improving the performance for Yahoo, which is somewhat an “easy” dataset. All the GBDT results below underperform pretrained deep models (we use the methodology in Section 4.1.1). Note that due to the size of the dataset, the GBDT results are deterministic (no stderr).

We thank the reviewer for their consideration, and are happy to help with any further concerns. If we have addressed the concerns we would appreciate it if the reviewer could raise their score.

Lacking comparison and discussion with existing tabular SSL methods.

We thank the reviewer for bringing up the comparison with existing tabular SSL methods. We add a comprehensive and expanded related works section to section A.1 (in the appendix of the paper, colored in blue) discussing the current state of tabular SSL. After the paper decision, we plan to reorder the paper to place it into the main text.

We would like to clarify that our primary goal is to demonstrate that DL rankers built using pre-training techniques can achieve SOTA performance in LTR settings, and consequently beat GBDT models (prior SOTA). Only as a secondary goal, we investigate a few simple but effective pre-training strategies such as SimSiam, SimCLR, and our variant SimCLR-Rank. While we identified SimCLR-Rank has a positive impact in a wide variety of LTR scenarios, we also show that SimSiam works better for our large-scale proprietary dataset. Our goal is not to show that any one particular method is the best for pretraining in LTR.

Regarding the suggested tabular SSL methods: STab [12] applies a SimSiam-like method to tabular self-supervised learning, RegCLR [13] aims to extract tables from images and is not applicable to our setting, and SubTab [14] is a well-known and influential tabular SSL technique. Given that STab has a similar strategy as SimSiam which we already evaluate in our paper (and because there was no published code for STab), we focus on evaluating SubTab in this response (respecting the time constraints of the rebuttal).

We evaluate SubTab [14] in the setting of Subsection 4.3.2. SubTab’s pretraining objective divides input features into multiple subsets (in the language of computer vision, “multiple views”) and trains an autoencoder to reconstruct the original input features. To pretrain using SubTab, we divide the input features into 4 subsets with 75% overlap, and with input corruptions of 15% masking probability and Gaussian noise of scale 0.1 (suggested in the SubTab paper as a good setting). The finetuning strategy and the choice for encoder model is the same as for SimCLR-Rank. We find that SubTab performs respectably on Yahoo, but fails to learn the MSLR and Istella datasets.

What’s the intuition behind SimCLR-Rank working better?

Thank you for giving us the opportunity to clarify. Recalling the response to the previous concern, our goal is to show that pretraining helps in LTR, not that any particular pretraining method (including SimCLR-Rank) is better–although SimCLR-Rank does perform well in many different settings in our experiments.

SimCLR-Rank is different from SimCLR in two aspects: (1) it uses items from the same query group as freely available hard-negatives, (2) it reduces the number of negatives from batchsize to at most query group size and thus allow better computational scaling with batchsize. To identify which of these contribute more towards SimCLR-Rank’s performance, we performed an additional experiment below (as suggested by reviewer i3NM), comparing SimCLR-Rank with a variant of SimCLR (which we call as SimCLR-sample). In SimCLR-sample, instead of using all the other items as negatives, we uniformly sample a constant number of negatives from the batch. This reduces the computational complexity from quadratic in batchsize (of SimCLR) to linear in batchsize. In the below experiments we follow the setup in Subsection 4.3.2 and SimCLR-Rank and SimCLR-sample use the same number of negatives and batchsize.

We find that SimCLR-Rank performs better on MSLR/Istella, while SimCLR-sample performs better on Yahoo. We note that MSLR/Istella are much sparser than Yahoo (see Table 12 in the paper), and are more representative of typical search and recommendations applications. Therefore we still believe that SimCLR-Rank which selects hard-negatives from the same query group is a useful pre-training method for the LTR toolbox and can be highly effective in many situations.

Thank you for your review and feedback. We would like to first take an opportunity to clarify the impact and contributions of our work.

We strongly believe our work will significantly impact industries that work with LTR problems (search, recommendation), by causing them to revise, or at least revisit, current SOTA methods. Today many large companies use GBDTs for ranking tabular data, and this approach is SOTA even in the research community. The novel starting point of our work is the often ignored fact that most of the data collected in LTR systems is unlabeled. We have shown through extensive experiments on standard public datasets and a large-scale private dataset that pretrained deep rankers can leverage unlabeled data to consistently outperform GBDTs, with outsized gains in robustness towards outlier data.

All reviewers agree we have convincingly demonstrated this. Reviewer 1XYq: “Experimentally, authors show that pretrained models can outperform GBDTs, which is one of the strongest baselines, under label scarcity setting.” Reviewer oW1i: “the paper highlights the superior performance of pretrained deep rankers, especially on outlier queries, in scenarios with limited labeled data.” Reviewer i3NM: “The major contribution of the paper is to identify areas where [pretrained deep rankers] help, including query sparsity and label sparisity scenarios.”

We now address the listed concerns one by one.

Technical novelty is limited.

We thank the reviewer for their feedback. We would like to bring to the reviewer’s attention these following novel contributions:

1. We make the observation that in Learning-To-Rank (LTR) applications like search and recommendations, there is an abundance of unlabeled data. Before our work, this LTR problem setting has not been studied, to the best of our knowledge. Reviewers agree on this point. Reviewer 1XYq: “Representation learning in tabular LTR has not been studied much.” Reviewer oW1i: “The paper studies pre-trained DNN for tabular LTR problems, which is an underexplored problem.” Reviewer i3NM: “To the reviewer’s knowledge, this is the first work that shows some promises for pre-trained DNNs for the LTR task.”

2. SimCLR-Rank is the first pretraining method that specifically leverages LTR problem structure. SimCLR, as noted by reviewer i3NM, is very slow: “the major weakness of SimCLR for non-ranking problems was complexity.” Our method SimCLR-Rank leverages the unique structure of the LTR problem to gain orders of magnitude speed improvements (Table 6 on page 16 of the paper) while performing slightly better than SimCLR (Table 5 on page 9 of the paper). We believe this is technically novel.

3. We would finally like to emphasize that our results on the large-scale industry-size dataset are a big contribution. Most LTR papers [1,2,3] do not give results on industry-scale datasets, and do not prove their hypotheses on true large scale datasets because of lack of availability of such data and prohibitively large expense of running large-scale experiments. The fact that we demonstrate pretraining rankers achieve SOTA on public datasets in LTR but also emphatically on a large-scale commercial dataset is highly impactful. Therefore we believe our results will help the LTR and the wider Tabular-DL communities tremendously.

Datasets in experiments are limited to three datasets and one private dataset.

We thank the reviewer for giving us an opportunity to clarify the experimental breadth. In LTR literature experiments these three datasets are generally considered the standard, with many influential and peer-reviewed work only evaluating at most the three public datasets we experiment on [1, 2, 3, 4]. Further, tabular (a.k.a. numerical) ML (e.g. LTR) systems are important for modern enterprises and therefore validating research hypotheses on large-scale practical datasets is valuable. Unfortunately, this is challenging due to the expensive and proprietary nature of such data. Given that we evaluate on all three of the standard public datasets—plus a large private dataset—we believe our empirical evaluation is stronger than comparable papers in the literature and could be valuable for the vast LTR and Tabular-DL communities.

[1] Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees?, Qin et al., ICLR 2021.
[2] Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networks., Ai et al., SIGIR 2019.
[3] SetRank: Learning a Permutation-Invariant Ranking Model for Information Retrieval., Pang et al., SIGIR 2020.
[4] Toward Understanding Privileged Features Distillation in Learning-to-Rank., Yang et al., NeurIPS 2022
[5] Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty., Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, Dawn Song. NeurIPS 2019.
[6] Pretrained Transformers Improve Out-of-Distribution Robustness., Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, Dawn Song. ACL 2020.
[7] Self-supervised Learning is More Robust to Dataset Imbalance., Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, Tengyu Ma. ICLR 2022.
[8] Transfer Learning with Deep Tabular Models. Roman Levin, Valeriia Cherepanova, Avi Schwarzschild, Arpit Bansal, C. Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, Micah Goldblum. ICLR 2023.
[9] Revisiting Pretraining Objectives for Tabular Deep Learning. Ivan Rubachev, Artem Alekberov, Yury Gorishniy, Artem Babenko.
[10] A Simple Framework for Contrastive Learning of Visual Representations. Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton. ICML 2020. [11] Exploring Simple Siamese Representation Learning. Xinlei Chen, Kaiming He. CVPR 2021.
[12] STab: Self-supervised Learning for Tabular Data. Hajiramezanali, E., Diamant, N. L., Scalia, G., & Shen, M. W. NeurIPS 2022 First Table Representation Workshop.
[13] RegCLR: A Self-Supervised Framework for Tabular Representation Learning in the Wild.Wang, W., KIM, B. H., & Ganapathi.
[14] Subtab: Subsetting features of tabular data for self-supervised representation learning. Ucar, T., Hajiramezanali, E., & Edwards, L.. NeurIPS 2021. [15] Improving Out-of-Distribution Robustness via Selective Augmentation. Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, Chelsea Finn. ICML 2022