Dear Reviewer Dd6v,

we appreciate your valuable feedback. We are happy that you recognized the importance of our benchmark in addressing the lack of consistent frameworks for evaluating active learning strategies. We refer to our general statement above and like to address your raised points in the following.

Weaknesses.

• W1: We are well aware that ICLR traditionally focuses on deep learning and, therefore, primarily on areas such as computer vision and natural language processing. Notably, there is a growing research direction focused on applying deep learning to tabular data, particularly for small datasets, leading to, e.g., the recently proposed transformer models like TabPFN, TabNet, or TabTransformer. Thus, we believe exploring active learning within the tabular data domain is an interesting and relevant contribution to this community. Furthermore, we also set the scope of our package explicitly to classification tasks on tabular data, as different data domains come with different requirements regarding types of learning algorithms and active learning settings. Please also refer to the "Tabular data" section in the general remark.

• W2: We agree that including additional learning algorithms, such as TabTransformer, would certainly be valuable for expanding the benchmark, and we plan to do it. For this submission, however, our selection of learners was based on [8], aiming to cover a diverse set of algorithms from different model types. In this context, we included TabPFN as a representative of transformer-based models designed for tabular data.

• W3: We completely agree that discussing the surprising findings by our benchmark would add significant value to the paper. Therefore, we will add the "Lessons Learned" section to the introduction, as also suggested by Reviewer 3njW.

Questions.

• Q1: ALPBench can be extended by other machine learning models and to different modalities. Adding custom models is already natively supported, provided they adhere to the scikit-learn API (i.e., implementing the .fit() and .predict_proba() methods). However, extending the benchmark to other data modalities involves additional implementation overhead, as different data modalities require different learning algorithms, preprocessing steps, training procedures, etc.

• Q2: Thank you for this suggestion. We summarized our key findings in the "Lessons Learned" section that we added to the introduction.

We hope that we have effectively communicated the changes we wish to make to our manuscript and addressed any remaining concerns and questions.

Dear Reviewer PLPc,

thanks for your valuable feedback. We are pleased that you found our systematic approach to evaluating different downstream models in different settings and metrics interesting and see value in our user-friendly, pip-installable Python package. We refer to our general statement above and like to address your raised points in the following.

Weaknesses.

• W1: The primary limitation of current active learning benchmarks for tabular data [1-4] is their restriction to a single downstream model. Notably, they consider rather weak models, such as logistic regression, SVMs, and simple MLPs, rather than gradient-boosted trees or more sophisticated deep learning approaches to tabular data. While this may seem like a minor point, we argue that including multiple SOTA downstream models is essential for drawing valid conclusions and have discussed this in more detail in the general remark in the "Novelty" section.

• W2: In Appendix A.3, we discuss the criteria behind our choice of settings; however, we will move this information to the main paper for clarity. You are correct that the number of samples required varies with dataset difficulty. This is one reason why we made the number of samples in the dynamic settings (used in our empirical evaluation) dependent on the number of classes, as this measure partially reflects the difficulty of a dataset.

• W3: In principle, it would be possible to use an existing benchmark and adapt it to run with our eight downstream models. However, this would require substantial implementation effort. Also, in contrast to existing benchmarks, our package offers standardized evaluation protocols, pre-defined settings, and logging of seeds, exact splits, and model hyperparameters. This is crucial because active learning outcomes can vary widely, as performance is influenced by numerous design choices, including the learner, query strategy, experimental setting, dataset, and evaluation metric.

• W4: It is indeed unsurprising that gradient-boosted trees perform strongly as downstream models. However, these models have, to the best of our knowledge, never been evaluated in tabular active learning benchmarks so far. We aimed to address the question of performances of QS conditioned on learners by presenting win-win matrices (Figure 3 and Figure 7 in the Appendix), where we observed that the suitability of query strategies varies notably across learners. To clarify and emphasize these findings, we will add a "Lessons Learned" section to the introduction, summarizing these key insights.

Further Remarks.

• I1: By subsampling our benchmark, we can provide more specific and also useful insights, such as recommending a certain QS for a given learning algorithm and setting, rather than offering broad, general recommendations as identifying the query strategy that performs best on average.

• I2: Our incentive was to summarize results through these tables. However, we acknowledge that a clearer decision criterion could enhance usability, and we have addressed this by adding the "Lessons Learned" section in the introduction.

We hope we have effectively communicated the changes we will make to our manuscript and addressed any remaining concerns and questions.

Dear Reviewer 3njW,

thanks for your valuable feedback and your suggestions for improvement. We also thank you for your appreciation of our work, specifically its potential impact on standardizing future active learning evaluations across different settings using different learners. In the following, we would like to address your proposed opportunities for improvements and answer your questions.

• We will add a "Lessons Learned" section at the end of the Introduction. This section summarizes our insights on the best-performing learner and query strategy combinations for various conditions, such as binary vs. multi-class datasets, small vs. large settings, and AUC vs. Accuracy metric.

• We appreciate the idea of including an illustrative, step-by-step example of using ALPBench in real-world scenarios, and we will incorporate this. Additionally, we will include a guided example in the Appendix, demonstrating how to add new datasets and query strategies.

• We will also include a table showing the state-of-the-art (SOTA) performance of learners on the full training dataset, as well as use these results to assess how quickly the best active learning approaches reach SOTA performance.

Questions.

• Q1: In Table 3 of Appendix A.2, we present an expanded version of this table, providing a detailed explanation of the query strategies (QS) included in our work and in previous approaches. For our selection of QS, we prioritized widely used, prominent strategies (such as Entropy Sampling, Margin Sampling, and Query-By-Committee) and more recent approaches, e.g. [5,6].

• Q2: In Appendix A.3 (Datasets), we provide details on the excluded datasets and the reasons for their exclusion.

• Q3: A more detailed explanation of the learning algorithm configurations is provided in Appendix A.3. However, we agree that this information would be better placed in the main paper, and we will make this adjustment. Regarding the base learners, it was primarily the TabNet classifier that reached the time limit only in the large setting.

• Q4: Thank you for pointing out that reference. We will add it to Section 3.3 and incorporate it into our benchmark package.

• Q5: For the dataset with ID 846, the Random Forest achieves better performance. For the dataset with ID 1053, KNN does outperform XGBoost; however, this dataset appears to be challenging for Active Learning in general, as performance tends to either decrease or remain stable across both learners (KNN and XGBoost) and all query strategies.

• Q6: No, unfortunately, ALPBench does not automatically detect or report situations where AL hurts. However, implementing such a feature is challenging, as we typically do not have a validation set available due to the limited amount of labeled data. This question is more closely related to the research area of "early stopping in AL," which is explored, for example, in "Hitting the target: stopping active learning at the cost-based optimum".

• Q7: Simulating a stream-based scenario is straightforward - one only needs to define a new setting and set the batch size to 1, which requires minimal effort. Including semi-supervised learning is indeed an interesting idea, and we have considered it; however, we chose to focus exclusively on AL for this submission. However, as we are still developing the package, we definitely plan to incorporate this feature. From an implementation perspective, it would be relatively simple to replace the Oracle with a “SemiSupervisedLabeler.”

• Q8: Epistemic uncertainty refers to the uncertainty arising from a lack of knowledge, and it can be reduced by gathering more data. The argument in the paper is that one should focus on sampling data points with high epistemic uncertainty, as those with high aleatoric uncertainty are unlikely to provide useful information for the learner.

We hope that we have effectively communicated the changes we want to make to our manuscript and addressed any remaining concerns and questions.