Is margin all you need? An extensive empirical study of deep active learning on tabular data

As we note in the main text, the SCARF paper compares deep NNs to gradient boosted decision trees and shows that on the contrary, deep NNs are competitive (especially when pre-trained). Could you please post experimental results or references that contradict this?

The scope of the paper is tabular data, as we explicitly state quite a few times. Others have studied images and text, as we mention in the Related Works: "There have also been empirical evaluations of active learning procedures in the non-tabular case. Hu et al. (2021) showed that margin attained the best average performance of the baselines tested on two image and three text classification tasks across a variety of neural network architectures and labeling budgets."

Thank you for the review.

To clarify, the model architecture is a feedforward ReLU network. SCARF is an architecture agnostic pre-training algorithm. We show results with and without SCARF pre-training. Since our domain is tabular data, within the realm of deep models, feedforward networks are the standard choice (though niche alternatives exist) -- which is to say, we don't think it'd make much sense to try CNNs or Transformers here. Our paper precisely ablates the effect of SCARF pre-training in every experiment. SCARF is the most appropriate pre-training scheme to consider for this paper as it is agnostic to modality and was shown to work well precisely for the type of real-world tabular data we use here.

RE: "To enhance the persuasiveness of the conclusions, it is crucial to conduct more experiments encompassing a broader range of active learning settings.". We believe the analysis is pretty thorough; could you suggest specifically which settings are missing?

RE: "...the paper should strive for innovation and provide additional theoretical analysis and in-depth discussions." We note in paragraph 3 of the Related Works that AL methods like Margin can be difficult to analyze theoretically and that there is little theoretical understanding of the method. We attempt to provide a deeper understanding in the "Deeper Analysis" section -- we say, "While it can be challenging to pinpoint why one method outperforms another, we attempt to provide some insight by comparing which examples each method chooses to acquire." We think the guidance this paper provides is clear -- practitioners doing active learning on tabular data should opt for Margin sampling.

Thank you for the review.

We benchmarked 16(!) diverse AL methods that are popular or purported as state-of-art. We believe these baselines provide great coverage.

RE: "As far as I know, there are other kinds of deep active learning methods". Could you please specify precisely which ones and why you think it is worthwhile to include them?

RE: BADGE. Indeed, BADGE performs well in our experiments. However, looking at both the win and relative improvement plots in Figure 1, the difference with Margin is not too significant in our opinion. We would like to echo what Andreas Kirsch (a public reviewer) noted about BADGE. While Margin is linear as it involves doing a single forward pass for unlabeled points and sorting them by a single scalar value (the margin), BADGE requires a forward pass to infer labels and a backward pass to compute loss gradients for each unlabeled sample. It then needs to cluster these gradients using k-means, which typically takes $O(nkdi)$, where $n$ is the number of $d$-dimensional vectors ($d$ is the dimensionality of the penultimate layer embedding), $k$ is the number of samples to acquire (i.e. number of clusters), and $i$ is the number of iterations needed to converge. $i$ can vary greatly in practice, and can make the effective runtime of k-means either linear or super-polynomial. Given the little performance difference between Margin and BADGE but the significant additional computational overhead of the latter, our recommendation to practitioners remains Margin. We will update the paper with discussion on this point.

Thank you for the detailed review.

The focus of this work is a large-scale analysis specifically on real-world tabular data. We think this is much needed as a lot of the other active learning papers have focused solely on images or text. We have good evidence that Margin works well in these other domains. For example, in paragraph 2 of the Related Works, we note that "Hu et al. (2021) showed that margin attained the best average performance of the baselines tested on two image and three text classification tasks across a variety of neural network architectures and labeling budgets."

The actionable recommendation the paper is making for practitioners is precisely that they should use Margin sampling: "Margin has no hyper-parameters and is consistently strong across all settings explored. We cannot recommend a single better method for practitioners faced with data labeling constraints, especially in the case of tabular data, or baseline to be benchmarked against by researchers in the field."

Andreas, we really appreciate your constructive feedback!

Sorting the legends in the line plots and the order of methods in the win matrices & box plots are great suggestions and we will try to do this for the final version.

You make a great point about Margin's runtime advantage over BADGE. We generally did not consider runtime, but given this paper is targeting practitioners in the real world, some discussion around this is apt. Reporting wall-clock times can often be misleading because of a strong dependence on implementation. In the final version, we will add a discussion on the runtime complexities.