Salutary Labeling with Zero Human Annotation

Larger query rounds R or larger query budget b. We provide our rationale for selecting a smaller query budget b in Appendix D, where we demonstrate that a larger budget causes the classification performance to reach a ceiling on the datasets. Regarding additional rounds, we invite Reviewer qHRp to refer to the second question in Section 5.3, where we conduct more learning rounds and annotate 50% of the pool set with salutary labels. For the convenience of the reviewer, we include the relevant results here. As shown in Table 2, our method achieves comparable or even higher performance than fully supervised training on the entire pool data, highlighting its effectiveness over more rounds of active learning.

Table: Accuracy (%) of the logistic regression model on the test set after querying 50% of the pool set.

Presentation suggestions. Thank the reviewer for the insightful suggestions for improving our presentation. We will split the first paragraph of Section 2 to make it easier to follow and understand. We will also make refinements in Figure 1 to highlight the polylines.

If there is anything in our response remains unclear to the reviewer, we are happy to provide more evidence and explanations.

Thank Reviewer qHRp's kind efforts in reviewing our paper. We are happy to see the reviewer's recognition of the beauty of simplicity inherent in our approach. We would like to provide point-to-point responses as follows.

Running time. Great suggestion! Here we listed the average running for 10 active learning rounds in one experimental run in the following table. As shown in Table A2, our method achieves shorter running times on datasets with 2 or 3 categories. However, for CIFAR10 and MNIST, it runs slower than the baselines due to the significant increase in the number of classes, which proportionally raises the number of parameters in the model (d). Despite this, we emphasize that our method eliminates the need for human annotations, which can potentially save substantial time in real-world applications.

Additionally, various research efforts [1, 2, 3] have focused on reducing the time required for influence calculations. These methods have achieved promising results, enabling efficient influence estimation for large datasets and complex models. As optimizing the computation of influence functions lies outside the scope of this work, we defer the acceleration of influence function calculations to future research.

Table A2: Running time (in seconds) of 10 rounds of active learning for different methods.

[1] Pruthi, Garima, et al. "Estimating training data influence by tracing gradient descent." NeurIPS 2020.

[2] Kwon, Yongchan, et al. "DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models." ICLR. 2023.

[3] Grosse, Roger, et al. "Studying large language model generalization with influence functions." arXiv preprint arXiv:2308.03296. 2023.

We will continue our discussion in the comment section (2/2), due to the space limit.

Fair comparison regarding validation set. Thanks to the reviewer for the suggestion to explore different sizes of validation sets.

i) We conducted the experiments for smaller validation sets. We randomly sampled validation sets as 10%, 5% and 1% of the data, and conducted the experiments for all baselines. We left out the Diabetic dataset as it is ready a small dataset its validation set only contains 100 samples. For traditional methods without the influence functions, i.e., the baselines other than ISLA and IBDS, the performance of different-sized validation sets is quite small (with 1% accuracy), so we reported the overall average accuracy here.

Our performance remains robust for validation sets comprising 10% and 5% of the data as shown in the table. While we observe a performance drop with only 1% validation data, our method still slightly outperforms other baselines, except on the Electric, Bank, and CIFAR10 datasets, as highlighted with underlines. This suggests that the size of the validation set does not significantly impact our method's effectiveness, as long as it provides a relatively accurate direction for influence estimation.

However, using very small validation sets might pose a limitation, as they may not provide sufficient information to maximize the guidance required for accurate influence estimation. Despite this, our method achieves comparable performance without any reliance on human annotations, demonstrating its robustness even under constrained validation settings.

Table A1: Accuracy (%) of the logistic regression model on the test set after 10 rounds of active learning with different validation sizes.

We will continue our discussion in the comment section (4/4), due to the space limit.

Fair comparison regarding validation set (Continued). ii) Various existing active learning methods [3, 4, 5] also utilize an unseen validation set to select the best hyperparameters. Traditional active learning techniques primarily rely on the information from the validation set to pick up the best hyperparameters, which then influence the final model parameters to achieve the best performance. Essentially, these use the validation set to help find the optimal model configuration. Our method uses the validation set to guide the selection of salutary labels, focusing on identifying the data most beneficial to the model. In this sense, we view the use of a validation set in our method as a novel approach that retrieves more information than traditional methods. Therefore, we humbly do not consider our comparison to be unfair. The experimental results for the 1% validation size also demonstrate that our method achieves performance comparable to the baselines without relying on human annotations, even under such a small validation size.

Related works in noisy labeled data methods. Great suggestion! We will incorporate analysis and comparison with methods for handling noisy labeled data, including our previous response discussing the novelty of our approach.

Analysis of the assigned salutary labels. We share Reviewer otRr’s interest in exploring the difference between ground truth and salutary labeling. Accordingly, we analyzed the impact of salutary labels that deviate from the ground truth. We kindly invite the reviewer to refer to the last three rows in Table 1 once more. For the convenience of the reviewer, we include the relevant results here. The performance improvements achieved using a small subset of such salutary labels demonstrate our method’s ability to identify key instances and assign labels that are more beneficial to the model than human-assigned ground truth.

Table: Accuracy (%) of the logistic regression model on the test set after 10 rounds of active learning of salutary labeling and ground truth. The last row represents the number of different annotations assigned by salutary labels and ground truth labels out of 100 new annotations.

Here, we emphasize that our method does not aim to distinguish "noisy" labels from other labels in the traditional sense. The differences between salutary labels and ground truth labels may arise from what could be considered "noise" or simply from the varying benefits provided by different annotations. From the model’s perspective, differentiating between the two is outside the scope of our work. Nevertheless, we appreciate the reviewer for raising this point, as it could indeed become an interesting direction for future work.

If there is anything in our response remains unclear to the reviewer, we are happy to provide more evidence and explanations.

[3] Kirsch, Andreas, Joost Van Amersfoort, and Yarin Gal. "Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning." NeurIPS. 2019.

[4] Liu, Zhuoming, et al. "Influence selection for active learning." ICCV. 2021.

[5] Mussmann, Stephen, et al. "Active learning with expected error reduction." arXiv preprint arXiv:2211.09283 (2022).

Novelty of our work compared with other label-efficient methods. Our salutary labeling can be regarded as a novel labeling method with broad applicability across various tasks, particularly in active learning and semi-supervised learning. Here we would like to clarify that we chose to showcase its effectiveness mainly within the active learning context in this work for two main reasons: i) salutary labeling demonstrates a marked improvement in model performance when applied to active learning, underscoring its value in this area; ii) By generating high-quality salutary labels, our method significantly reduces the need for human annotations. Here, we would like to point out that our novelty comes from our insight into the research problem and the fresh perspective we bring to addressing its challenges. We humbly believe this constitutes a key novelty of a research paper.

Specifically, our method shifts away from a human-centric perspective of existing label-efficient methods in active learning or noisy label tasks. Instead, our work focuses entirely on the model's viewpoint in annotating data, which aims to identify the data most beneficial for the model. Following this model-driven perspective, we extend the influence function to the context of active learning, enabling the model to directly estimate the impact of each unlabeled data point and its associated salutary label on model performance. This enables our method to address two fundamental challenges of active learning: estimating the impact of data points and reducing human annotation costs. We believe this new perspective fundamentally differs from other related methods in active learning or noisy label tasks. Technically, our method differs from [a, b, c] as it evaluates unseen data without requiring annotations and does not focus on the "correctness" of labels.

We will continue our discussion in the comment section (3/4), due to the space limit.

Thank you for your kind efforts in reviewing our paper. We are happy to see the reviewer's informative comments. We would like to provide point-to-point responses as follows.

Why not applying salutary labels to all available. We follow the active learning settings, where data is selected and labeled iteratively rather than all at once. This iterative process enhances the accuracy of the influence function calculation by dynamically updating the model with newly labeled data points. Instead of relying on a fixed model, we continuously update the model after each annotation round, recalculating the influence estimation based on the updated model. This allows for more targeted and precise labeling, ensuring that each new annotation is maximally beneficial to the model. This dynamic adjustment enhances the effectiveness of salutary labels and optimally improves the model performance.

No human annotation. We agree with the reviewer that human-assigned ground truth labels intuitively seem to provide performance boost in active learning. Yet we believe that learning models inherently exhibit biases that do not always align with the human perspective. This misalignment can result in scenarios where ground truth labels, though accurate from a human perspective, fail to provide the most benefit to the model's learning process, causing the ground truth labels to yield suboptimal model performance. Notably, previous works on noisy labels [1,2] have demonstrated that incorporating a small number of labels considered noisy by humans can improve model performance. Our own observations, detailed in Figure 1 and Table 1 of the manuscript (See the table below), further support this argument, showing that selectively leveraging such "noisy" but salutary labels can enhance learning outcomes for the model.

Our method takes this perspective of the model, enables the active learning framework to identify the most impactful samples, and assigns their corresponding salutary labels. By prioritizing data that maximizes performance gains, this approach optimizes the model's learning potential without relying on human annotation efforts.

Table: Accuracy (%) of the logistic regression model on the test set after 10 rounds of active learning of salutary labeling and ground truth. The last row represents the number of different annotations assigned by salutary labels and ground truth labels out of 100 new annotations.

[1] Natarajan, Nagarajan, et al. "Learning with noisy labels." NeurIPS. 2013.

[2] Song, Hwanjun, et al. "Learning from noisy labels with deep neural networks: A survey." IEEE TNNLS. 2022.

We will continue our discussion in the comment section (2/4), due to the space limit.

I sincerely appreciate the time and effort the authors have dedicated to addressing my concerns. While some of my concerns have been resolved, I have outlined additional points for discussion regarding the remaining questions below.

Motivation of active learning with salutary labels

We follow the active learning settings, where data is selected and labeled iteratively rather than all at once.

I'm curious about why data sampling is necessary in this context. In Algorithm 1, Line 4, the proposed work compute salutary labels for all data, so why do this method still select a subset of these labeled samples in Lines 6–8? Given that all data already have salutary labels, it seems feasible to train with all labeled data in every round. In traditional active learning, iterative data selection is crucial due to the high cost of human annotation, but when annotations are unlimited, sampling appears unnecessary. On another note, if iteratively updating the influence function offers advantages, I’m curious about what budget size is optimal for this. How was the small budget size used in this study chosen, and how should it be determined for new datasets?

Novelty of salutary labeling task compared to other label-efficient learning task

Our salutary labeling can be regarded as a novel labeling method with broad applicability across various tasks, particularly in active learning and semi-supervised learning. Technically, our method differs from [a, b, c] as it evaluates unseen data without requiring annotations and does not focus on the "correctness" of labels.

As noted in my earlier comments on the strengths of your work, I agree that salutary labeling is a new method. However, if proposing a new task is one of the main contributions of the paper, as indicated in Line 55, there should be a clear distinction between the new task and existing ones. From the perspective of supervision (unlabeled or few-labeled datasets) and objective (maximizing test performance under the same distribution), salutary labeling appears to align with existing tasks. Thus, I view salutary labeling not as a new task but as a method addressing the same problem as label-efficient learning.

Fairness of the experiment

I agree that most active learning (AL) methods require a validation set for hyperparameter tuning. However, it may be unrealistic for AL methods to operate with only 10 labeled samples per round, particularly in a setting where the validation set comprises 20% of the data. In Figure 5, when only 6% of the 20% validation set is allocated for training, existing AL methods outperform the performance of salutary labeling. Reducing the validation set from 20% to 14% might not significantly impact the performance of existing AL methods.

Analysis of salutary labels

It is interesting that assigning different labels to samples that seemingly belong to a specific class can improve performance. Qualitative results for samples where salutary labels differ from the ground truth (GT) would provide valuable insights.

Thank you for dedicating your time and energy to reviewing our paper. We would like to provide point-to-point responses as follows.

Difference with the unsupervised learning methods. Our salutary labeling can be regarded as a novel labeling method with broad applicability across various tasks, particularly in active learning or semi-supervised learning.

In this work, we focus on active learning, which operates in a supervised setting, where the model is trained with labeled data, and the size of the training set grows incrementally as new labeled data is added. Our approach introduces an automated labeling method without requiring manual annotation. This makes our method fundamentally different from unsupervised settings, where models are trained without labeled data and typically aim to identify patterns within the data itself. As with the works referred by Reviewer CWJE, they primarily address unsupervised domain adaptation, which focuses on aligning data distributions across different domains.

We will include additional discussion in the manuscript to clarify this point.

Error correction and impact of incorrect labels. Our method does not aim to assign the "correct" labels in the same sense as human annotators, as we discussed in the Introduction Section of the manuscript. Instead, our method assignssalutary labels, a kind of pseudo labels, to unlabeled samples from the model's perspective, with the goal of maximizing the benefit to model performance.

In this context, we analyze the impact of the salutary labels that are different from the ground truth. We kindly invite the reviewer to refer to the last three rows in Table 1. For the convenience of the reviewer, we include the relevant results here. The performance improvement achieved by a small set of such salutary labels demonstrates that our method can identify key instances and assign more beneficial labels compared with human-assigned ground truth. Again, we emphasize that our method is not concerned with the "correctness" of the labels in the traditional sense. Instead, the focus is on the benefit of the model.

Table: Accuracy (%) of the logistic regression model on the test set after 10 rounds of active learning of salutary labeling and ground truth. The last row represents the number of different annotations assigned by salutary labels and ground truth labels out of 100 new annotations.

If there is anything in our response remains unclear to the reviewer, we are happy to provide more evidence and explanations.

Salutary labeling for complex models. Great question! We would first like to note that using surrogate models for deep networks is a common practice [1,2,3] to overcome the convexity requirement of the influence function. Following this established routine, we demonstrate the effectiveness of salutary labeling in applications with complex model structures in Appendix E.

Regarding direct influence calculation for deep models like ResNet, we believe it is achievable with some improvements. While there are concerns regarding the use of the influence function on non-convex models [4], several studies have successfully applied the influence function to non-convex models, demonstrating its effectiveness in these settings [5,6].

[1] Chen, Zizhang, Peizhao Li, Hongfu Liu, and Pengyu Hong. "Characterizing the Influence of Graph Elements." ICLR. 2022.

[2] Brophy, Jonathan, Zayd Hammoudeh, and Daniel Lowd. "Adapting and evaluating influence-estimation methods for gradient-boosted decision trees." Journal of Machine Learning Research 2023.

[3] Anshuman Chhabra, Peizhao Li, Prasant Mohapatra, Hongfu Liu. "What Data Benefits My Classifier?" Enhancing Model Performance and Interpretability through Influence-Based Data Selection. ICLR 2024.

[4] Basu, S., P. Pope, and S. Feizi. "Influence Functions in Deep Learning Are Fragile." ICLR. 2021.

[5] Epifano, Jacob R., et al. "Revisiting the fragility of influence functions." Neural Networks. 2023.

[6] Kwon, Yongchan, et al. "DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models." ICLR. 2024.

If there is anything in our response remains unclear to the reviewer, we are happy to provide more evidence and explanations.

Performances on different sizes of validation set. Thanks to the reviewer for the suggestion for exploring different sizes of validation sets. We conducted the experiments for smaller validation sets. We randomly sampled validation sets as 10%, 5% and 1% of the data, and conducted the experiments for all baselines. We left out the Diabetic dataset as it is ready a small dataset its validation set only contains 100 samples. For traditional methods without the influence functions, i.e., the baselines other than ISLA and IBDS, the performance of different-sized validation sets is quite small (with 1% accuracy), so we reported the overall average accuracy here.

Our performance remains robust for validation sets comprising 10% and 5% of the data as shown in the table. While we observe a performance drop with only 1% validation data, our method still slightly outperforms other baselines, except on the Electric, Bank, and CIFAR10 datasets, as highlighted with underlines. This suggests that the size of the validation set does not significantly impact our method's effectiveness, as long as it provides a relatively accurate direction for influence estimation. Here kindly invite Reviewer zEN4 to revisit Eq. (1) in the manuscript, where the validation set plays a crucial role in determining the direction of the first term. The size of the validation set is not a deciding factor as long as it provides an accurate direction for the gradients.

However, using very small validation sets might pose a limitation, as they may not provide sufficient information to maximize the guidance required for accurate influence estimation. Despite this, our method achieves comparable performance without any reliance on human annotations, demonstrating its robustness even under constrained validation settings. Again, we thank the reviewer for bring up this point, and we would like to add additional discussion in our Limitation Section regarding this point.

Table A1: Accuracy (%) of the logistic regression model on the test set after 10 rounds of active learning with different validation sizes.

We will continue our discussion in the comment section (3/3), due to the space limit.

Thank you for dedicating your time to reviewing our paper with a high level of expertise. We thoroughly read the comments and are happy to see that Reviewer ZEN4 is quite interested in our work and has provided informative feedback. We want to provide point-to-point responses as follows.

Alignment to semi-supervise learning. Thank you for the helpful suggestion, and we agree with the reviewer that salutary labeling can indeed be regarded as a novel labeling method with broad applicability across various tasks, where we choose the context of active learning to better demonstrate the effectiveness of our method. We will incorporate additional discussion in the manuscript to better reflect this point.

Here we would like to clarify that we chose to showcase its effectiveness mainly within the active learning context in this work for two main reasons: i) salutary labeling demonstrates a marked improvement in model performance when applied to active learning, underscoring its value in this area; ii) By generating high-quality salutary labels, our method significantly reduces the need for human annotations.

Lack of technical contributions. Thank you for the feedback regarding the technical contributions. We agree with the reviewer that our method does not propose fancy techniques, yet we also would like to kindly point out that we do not claim existing techniques related to the influence function as part of our contributions.

Instead, we would like to point out that our novelty comes from our insight into the research problem and the fresh perspective we bring to addressing its challenges. We humbly believe this constitutes a key novelty of a research paper. Specifically, our method shifts away from a human-centric perspective and focuses entirely on the model's viewpoint in annotating data, which aims to identify the data most beneficial for the model. Following this model-driven perspective, we extend the influence function to the context of active learning, enabling the model to directly estimate the impact of each unlabeled data point and its associated salutary label on model performance. This enables our method to address two fundamental challenges of active learning: estimating the impact of data points and reducing human annotation costs.

In sum, our technical contributions can be summarized into two points. 1) Novelty, no one has done this before, 2) Simplicity, if a task can be tackled well with a simple tool, there is no incentive to design complicated tools. Empirically, we demonstrated the effectiveness of our simple solution.

Results for additional rounds or a larger query budget. We provide our rationale for selecting a smaller query budget $b$ in Appendix D, where we demonstrate that a larger budget causes the classification performance to reach a ceiling on the datasets. Regarding additional rounds, we invite Reviewer zEN4 to refer to the second question in Section 5.3, where we conduct more learning rounds and annotate 50% of the pool set with salutary labels. For the convenience of the reviewer, we include the relevant results here. As shown in Table 2 of the manuscript, our method achieves comparable or even higher performance than fully supervised training on the entire pool data, highlighting its effectiveness over more rounds of active learning.

Table: Accuracy (%) of the logistic regression model on the test set after querying 50% of the pool set.

As for the performance differences in Appendix D, we have provided an honest report of our experimental results in Figure 5 to illustrate the performance ceiling on these datasets. The average performance for all methods falls within the standard deviation, so we humbly considered the differences between methods to be comparable, especially given that the focus of these experiments is not on evaluating different methods.

We will continue our discussion in the comment section (2/3), due to the space limit.