Understanding Impact of Human Feedback via Influence Functions

[Q1] Evaluation set size

We conducted ablation experiments with various evaluation set sizes, detailed in Appendix H and illustrated in Figure 8 of the original draft. The results of these experiments indicate that influence functions perform well even with as few as 50 samples. These empirical findings suggest that efficient, curated validation sets can be achieved with approximately 50-100 samples. Increasing the size of the validation set beyond 100 results in marginal performance improvements, which tend to saturate around 100 samples.

---

[Q2] Considering different bias types than length and sycophancy

Thank you for this important question. We considered various biases identified in previous research on preference datasets. We chose to focus on length and sycophancy because they are prevalent across diverse datasets and models, including LLMs and reward models.

For instance, Singhal et al. [5] noted that LLMs fine-tuned with RLHF tend to produce lengthier responses, a tendency attributable to datasets that favor longer text. Similarly, sycophancy bias is widely recognized in state-of-the-art LLMs, such as GPT-4, where datasets biased towards sycophantic responses have been shown to influence model outputs after RLHF [6]. Given these findings, we believe addressing these biases is crucial for improving dataset integrity and quality of current LLMs.

Additionally, investigating other potential biases, such as self-preference bias among LLM evaluators [7], would be an interesting future direction. This bias involves evaluators favoring responses generated by themselves or similar LLM models over those produced by humans.

---

[Q3] Metric Choice

Thank you for your question. We have expanded our analysis to include additional metrics: the area under the precision-recall curve (AP) and the TNR value at TPR=0.8 (TNR80). The table below demonstrates that Influence functions consistently outperform all baselines across these metrics for both length and sycophancy bias detection. This comprehensive analysis confirms that our improvements are not at the expense of overfitting specific biases, thereby ensuring the overall robustness of our method. We have included these results, along with precision-recall curves, in Appendix E of the revised draft.

---

[Q4] Dependent Rewards

In Section 5.2, we treated the fine-grained objectives as independent to align with the framework used in the HelpSteer2 dataset [8], where different aspects of response quality are evaluated separately. We adopted HelpSteer2's methodology of modeling response quality as a linear combination of these objectives. Extending our setup to accommodate dependent objectives is an interesting direction for future research.

---

[1] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.

[2] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, 2022.

[3] Johnson Kuan and Jonas Mueller. Model-agnostic label quality scoring to detect real-world label errors. International Conference on Machine Learning DataPerf Workshop, 2022.

[4] Youden William John. Index for rating diagnostic tests. Cancer, 3: 32–35. 1950.

[5] Singhal, Prasann, et al. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023.

[6] Sharma, Mrinank, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548, 2023.

[7] Wataoka, Koki, Tsubasa Takahashi, and Ryokan Ri. Self-Preference Bias in LLM-as-a-Judge. arXiv preprint arXiv:2410.21819, 2024.

[8] Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673, 2024.

Dear Reviewer facp, we sincerely appreciate your valuable and helpful feedback. We address each comment in detail, one by one below.

---

[W1] Evaluation Set Requirement

Thank you for your insightful comment. We acknowledge that requiring targeted evaluation sets can be a barrier to using our method (as mentioned in Remark 4.1 of the original draft). However, it is important to highlight that influence functions can still be effective even with less curated sets. To demonstrate this, we conducted additional experiments on bias detection using the original validation set of the Anthropic HH dataset, denoted as ‘Full’. This dataset was used in its unmodified, uncurated form as provided by the Anthropic HH validation set. The table below shows that influence functions effectively detect biased samples using this uncurated set compared to the high-quality, carefully filtered validation set used in our study (see Section 5.1.1 for more details).

These results confirm that influence functions are robust even without well-curated validation sets, demonstrating their utility with more generalized samples. We also note that influence functions demonstrate reasonable performance even with relatively small validation sets (approximately 50-100 samples), as shown in Figure 8 and Appendix G. This suggests the potential for practical applications, using small-scale expert data.

We have revised Figure 4 (with Section 5.1.2) of our draft to reflect these findings, with the changes highlighted in red.

---

[W2] Computational Complexity

We agree that computational complexity arising from computing the inverse Hessian matrix poses a fundamental challenge in utilizing influence functions. This complexity can become a major bottleneck when applying influence functions to large models and extensive datasets. However, our work demonstrates that it is feasible to leverage efficient estimation methods for influence functions to a setup involving large human feedback datasets and models with billions of parameters. Furthermore, we anticipate continued algorithmic advancements in this area, which, under certain assumptions about datasets and models, may enable the development of more efficient approximation methods. Our approach is compatible with any efficient computation techniques related to influence functions.

---

[W3] Bias Detection Methodology

We acknowledge that relying on a specific threshold presents limitations in threshold-based detection systems. However, this approach has been commonly utilized in prior works [1-3] to measure the quality of score metrics. To complement these limitations, we also measure threshold-independent metrics, like the AUC, to assess overall performance. Finally, we remark that systematic methods such as maximizing the F1-score or Youden’s J-statistic [4] can be employed to set more objective thresholds.

---

[W4] Experimental concerns of labeler strategy oversight experiment

Thank you for your detailed comments. We address each point below:

(a) Expert labelers for every aspect: Our method does not require expert labelers for each fine-grained objective. The experiments in Section 5.2 serve as a proof-of-concept to demonstrate how influence functions can guide labeler strategies. In practical settings, labelers typically assess multiple aspects internally and integrate these assessments to determine their response preferences.

(b) Diversity of responses: Our approach involves reassigning preference labels within the existing dataset without altering the responses themselves. Therefore, we believe that our method does not reduce the diversity of the generated responses. It maintains the original variety of responses while ensuring the labels are accurately applied.

(c) Computational costs and SVM accuracy: Training the SVM for our experiment is efficient: classifying 4-dimensional vectors across 8,218 data points in mere seconds. The computational demands are minimal, and the accuracy is sufficient for our proof-of-concept.

(d) Updating labeler strategies: Our current setup requires an expert labeler to provide ground-truth preference labels on the validation set, as our objective is to align non-expert labels with those of the experts. We welcome further clarification on the experimental setup or alternative labeling strategy update methods you have in mind. We are eager to explore these suggestions in detail and conduct additional experiments if necessary.

Dear Reviewer facp,

Thank you again for your time and efforts in reviewing our paper.

As the discussion period draws close, we kindly remind you that two days remain for further comments or questions. We would appreciate the opportunity to address any additional concerns you may have before the discussion phase ends.

Thank you very much!

Many thanks, Authors

Dear Reviewer dPes, we sincerely appreciate your valuable and helpful feedback. We address each comment in detail, one by one below.

---

[W1] Novelty of our work

First, thank you for noting that this work provides a meaningful contribution to the field. Regarding the novelty of our method, we would like to further elaborate on our contributions.

Efficient estimation method: The influence function computation method suggested by Koh and Liang [1] is inefficient for application to modern LLMs or large preference datasets. We address this computational challenge by implementing the vector compression method OPORP (One Permutation + One Random Projection) [2] alongside DataInf [3], significantly enhancing computational efficiency.

Practical value of approach: Our work demonstrates practical applications using influence functions: enhancing the interpretability of human feedback. These applications address the growing need to provide reliable oversight to modern LLMs. The significance of our research topic was additionally acknowledged by reviewers CzsY, Wn7U, and roz9.

Bias detection: We have applied influence functions to detect prevalent types of labeler bias (i.e., length and sycophancy biases) in RLHF. Our approach demonstrates superior performance in identifying biased samples compared to other baselines, including GPT-4o.

Labeler strategy oversight: Through a proof-of-concept experiment, we provide preliminary evidence that influence functions can improve non-expert labeling strategies by leveraging small sets of expert-labeled samples. This suggests a scalable path for enhancing oversight using influence functions.

We hope this clarifies the novelty and practical value of our work. Thank you for the opportunity to address this point.

---

[W2] Dependency on validation set composition

Thank you for your insightful comment. We acknowledge that requiring targeted evaluation sets can be a barrier to using our method (as mentioned in Remark 4.1 of the original draft). However, it is important to highlight that influence functions can still be effective even with less curated sets. To demonstrate this, we conducted additional experiments on bias detection using the original validation set of the Anthropic HH dataset, denoted as ‘Full’. This dataset was used in its unmodified, uncurated form as provided by the Anthropic HH validation set. The table below shows that influence functions effectively detect biased samples using this uncurated set compared to the high-quality, carefully filtered validation set used in our study (see Section 5.1.1 for more details).

These results confirm that influence functions are robust even without well-curated validation sets, demonstrating their utility with more generalized samples. We also note that influence functions demonstrate reasonable performance even with relatively small validation sets (approximately 50-100 samples), as shown in Figure 8 and Appendix G. This suggests the potential for practical applications, using small-scale expert data.

We have revised Figure 4 (with Section 5.1.2) of our draft to reflect these findings, with the changes highlighted in red.

---

[W3] labeler guidance experiment remains as proof-of-concept

Thank you for your insightful feedback. Our labeling oversight experiment serves as a proof-of-concept demonstrating how influence functions might guide labelers. As noted in the 'Limitations' section, we acknowledge the limitation that our setup does not perfectly model real-world conditions. However, despite these constraints, we believe our results provide valuable preliminary evidence that influence functions can be beneficial. We hope our work encourages further research to test influence functions in more realistic settings. We have also revised the introduction section of our draft for clarification.

[Q1] Clarifying reference on L.140

Thank you for your detailed comment. We have revised our draft accordingly.

---

[Q2] Reason for Mahalabobis distance baseline choice

The Mahalanobis distance is a proven metric for detecting out-of-distribution (OOD) samples [4] and noisy labels [5]. Given that the labeler bias detection problem shares similarities with these issues, we utilized it as a baseline measure.

---

[1] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, 2017.

[2] Ping Li and Xiaoyun Li. Oporp: One permutation+ one random projection. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023.

[3] Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models. In International Conference on Learning Representa- tions, 2024.

[4] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.

[5] Lee, Kimin, et al. Robust inference via generative classifiers for handling noisy labels. International conference on machine learning. PMLR, 2019.

Dear Reviewer Wn7U, we sincerely appreciate your valuable and helpful feedback. We have carefully considered your comments, and provide responses below.

---

[S3] Promising empirical results

Thank you for acknowledging the strength of our empirical results. If you have any specific concurrent works in mind, we would be happy to conduct additional experiments for comparison. Additionally, in our revised draft, we have included comparisons with another baseline, KNN [1], where our influence functions significantly outperform it.

---

[W1] Novelty of our work

Thank you for also kindly noting that the technical novelty is not a serious issue. We would like to elaborate on our contributions further.

Efficient estimation method: The influence function computation method suggested by Koh and Liang [2] is inefficient for application to modern LLMs or large preference datasets. We address this computational challenge by implementing the vector compression method OPORP (One Permutation + One Random Projection) [3] alongside DataInf [4], significantly enhancing computational efficiency.

Practical value of approach: Our work demonstrates practical applications using influence functions: enhancing the interpretability of human feedback. These applications address the growing need to provide reliable oversight to modern LLMs. The significance of our research topic was additionally acknowledged by reviewers CzsY, roz9.

Bias detection: We have applied influence functions to detect prevalent types of labeler bias (i.e., length and sycophancy biases) in RLHF. Our approach demonstrates superior performance in identifying biased samples compared to other baselines, including GPT-4o.

Labeler strategy oversight: Through a proof-of-concept experiment, we provide preliminary evidence that influence functions can improve non-expert labeling strategies by leveraging small sets of expert-labeled samples. This suggests a scalable path for enhancing oversight using influence functions.

We hope this clarifies the novelty and practical value of our contributions. Thank you for the opportunity to address this point.

---

[W2] Mistakes in Fig 4 legend

Thank you for catching this mistake. We have corrected Fig. 4 accordingly. The AUC values are 0.800 for the Concise set and 0.202 for the Verbose set.

---

[W3 & Q2] Discussion of Data Quality Improvement on real-world datasets

Thank you very much for your constructive feedback. To assess whether our bias detection method enhances data quality and reward model performance in real-world datasets, we conducted additional experiments outlined in Section 5.1. We utilized a 15k helpful subset of the Anthropic HH dataset without flipping any labels to introduce synthetic bias. Using our curated Concise validation set, designed to capture length bias, we calculated the influence for training samples and identified the top 100 samples exhibiting positive influence. Upon manual inspection (by authors) to verify label accuracy, we found a significant discrepancy in labeling compared to a control group of 100 randomly selected samples:

This result indicates that the top-influenced samples, predominantly longer responses, were more frequently mislabeled. To test the impact on model performance, we trained reward models on a dataset corrected for the 47 mislabeled samples. The reward model trained on this refined dataset achieved a score of 56.1 on Rewardbench [5], showing a modest improvement over the 54.0 scored by models trained on the unaltered dataset. We believe that these results highlight the potential effectiveness of our method when applied to real-world datasets.

We will open-source the human-labeled results, including both the original and updated training datasets.

---

[Q1] Reason why gradient vector dimension is 42M

The size of a single gradient vector is 42M dimensions (160MB for storage) because we utilized Low-Rank Adaptation [6] fine-tuning with a rank of 16 as mentioned in the ‘Reward model training’ paragraph of Section 5.1.1. For clarity, we have included a footnote explaining this in Section 4.2 of our revised draft.

---

[Q3] Reason for using 4 dimensions of HelpSteer2

Thank you for this detailed question. As you have correctly noted, HelpSteer2 [7] evaluates responses based on five criteria: helpfulness, correctness, coherence, complexity, and verbosity. In our study, we utilize four of these criteria, excluding ‘helpfulness.’ This exclusion is because the helpfulness score is considered an overall evaluation of the response, unlike the other four criteria that measure specific sub-aspects (refer to Appendix G.3: Per-axis ratings in the HelpSteer2 paper). We believe the helpfulness score is unnecessary for our experimental motivation, which requires labelers to make decisions based on more fine-grained objectives. This explanation has been added to Appendix B.2 in red for further clarification.

---

[1] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, 2022.

[2] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, 2017.

[3] Ping Li and Xiaoyun Li. Oporp: One permutation+ one random projection. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023.

[4] Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models. In International Conference on Learning Representa- tions, 2024.

[5] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024.

[6] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.

[7] Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673, 2024.

Dear Reviewer Wn7U,

Thank you again for your time and efforts in reviewing our paper.

As the discussion period draws close, we kindly remind you that two days remain for further comments or questions. We would appreciate the opportunity to address any additional concerns you may have before the discussion phase ends.

Thank you very much!

Many thanks, Authors

[Q2] Scalability of method regarding dataset size

We thank the reviewer for this important question. As shown in Figure 5 of our original draft, time consumption scales linearly with dataset sizes up to 100k samples on a single NVIDIA A6000 GPU. This linear scaling is expected to extend to even larger datasets, as the primary computational cost lies in the backpropagation step, which is proportional to dataset size. Importantly, this assumes adequate storage for the compressed gradient vectors of the training and validation datasets. By employing OPORP, we have compressed gradient vectors to $2^{16}$ dimensions, facilitating the storage of millions of samples, e.g., 5M samples require approximately 1TB of storage. If even larger datasets are considered, smaller compression dimensions could be used to further reduce storage needs, though at the cost of increased estimation variance, which according to the OPORP paper, is inversely proportional to the compressing dimension $\text{K}$ ($\text{Var} \propto 1/\text{K}$) [2].

---

[1] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, 2017.

[2] Ping Li and Xiaoyun Li. Oporp: One permutation+ one random projection. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023.

[3] Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models. In International Conference on Learning Representa- tions, 2024.

[4] Team, Gemma, et al. Gemma 2: Improving open language models at a practical size, arXiv: 2408.00118, 2024.

[5] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2024.

Dear Reviewer 71Vm, we sincerely appreciate your valuable and helpful feedback. We address each comment in detail, one by one below.

---

[W1] Novelty of our work

We would like to clarify and emphasize our contributions as follows:

Efficient estimation method: The influence function computation method suggested by Koh and Liang [1] is inefficient for application to modern LLMs or large preference datasets. We address this computational challenge by implementing the vector compression method OPORP (One Permutation + One Random Projection) [2] alongside DataInf [3], significantly enhancing computational efficiency.

Practical value of approach: Our work demonstrates practical applications using influence functions: enhancing the interpretability of human feedback. These applications address the growing need to provide reliable oversight to modern LLMs. The significance of our research topic was additionally acknowledged by reviewers CzsY, Wn7U, and roz9.

Bias detection: We have applied influence functions to detect prevalent types of labeler bias (i.e., length and sycophancy biases) in RLHF. Our approach demonstrates superior performance in identifying biased samples compared to other baselines, including GPT-4o.

Labeler strategy oversight: Through a proof-of-concept experiment, we provide preliminary evidence that influence functions can improve non-expert labeling strategies by leveraging small sets of expert-labeled samples. This suggests a scalable path for enhancing oversight using influence functions.

We hope this clarifies the novelty and practical value of our contributions. Thank you for the opportunity to address this point.

---

[W2] Evaluating influence functions for different models

Thank you for your valuable feedback. To assess the robustness of the influence functions across different models, we added a new model, Gemma-2-9B [4], in the length bias detection experiment. As shown in the table below, the influence functions demonstrate competitive performance for both Llama and Gemma, outperforming the GPT-40 baseline. We report the TPR values at tied FPR rates with GPT-4o (FPR=0.283)

We also recognize the importance of evaluating more downstream alignment methods, such as DPO [5]. Although time constraints prevented us from testing our method with these algorithms, we plan to explore this in future studies to provide a more comprehensive validation of our approach.

---

[Q1] Handling misleading feedback

Thank you for the intriguing question. In our labeler bias detection experiments, we specifically aim to identify intentionally misleading feedback with certain biases, such as those favoring lengthy or flattering responses. Our method has demonstrated strong detection performance, effectively identifying and mitigating such biases. We believe that our method can be effectively applied to other types of biases and adversarial feedback.

Dear Reviewer 71Vm,

Thank you again for your time and efforts in reviewing our paper.

As the discussion period draws close, we kindly remind you that two days remain for further comments or questions. We would appreciate the opportunity to address any additional concerns you may have before the discussion phase ends.

Thank you very much!

Many thanks, Authors

Dear Reviewer CzsY, we sincerely appreciate your valuable and helpful feedback. We address each comment in detail, one by one below.

---

[W1] Dependence on Domain Knowledge

Thank you for your insightful comment. We acknowledge that requiring targeted evaluation sets can be a barrier to using our method (as mentioned in Remark 4.1 of the original draft). However, it is important to highlight that influence functions can still be effective even with less curated sets. To demonstrate this, we conducted additional experiments on bias detection using the original validation set of the Anthropic HH dataset, denoted as ‘Full’. This dataset was used in its unmodified, uncurated form as provided by the Anthropic HH validation set. The table below shows that influence functions effectively detect biased samples using this uncurated set compared to the high-quality, carefully filtered validation set used in our study (see Section 5.1.1 for more details).

These results confirm that influence functions are robust even without well-curated validation sets, demonstrating their utility with more generalized samples. We also note that influence functions demonstrate reasonable performance even with relatively small validation sets (approximately 50-100 samples), as shown in Figure 8 and Appendix G. This suggests the potential for practical applications, using small-scale expert data.

We have revised Figure 4 (with Section 5.1.2) of our draft to reflect these findings, with the changes highlighted in red.

---

[W2 & Q1] Additional metrics for experiments

Thank you for your suggestions. We have expanded our analysis to include additional metrics: the area under the precision-recall curve (AP) and the TNR value at TPR=0.8 (TNR80). The table below demonstrates that Influence functions consistently outperform all baselines across these metrics for both length and sycophancy bias detection. This comprehensive analysis confirms that our improvements are not at the expense of overfitting specific biases, thereby ensuring the overall robustness of our method. We have included these results, along with precision-recall curves, in Appendix E of the revised draft.

---

[W3] Limitations of labeling strategy guidance experiment

Thank you for your insightful feedback. Our labeling oversight experiment serves as a proof-of-concept demonstrating how influence functions might guide labelers. As noted in the 'Limitations' section, we acknowledge the limitation that our setup does not perfectly model real-world conditions. However, despite these constraints, we believe our results provide valuable preliminary evidence that influence functions can be beneficial. We hope our work encourages further research to test influence functions in more realistic settings. We have also revised the introduction section of our draft for clarification.

---

[Q2] Performance degradation of hessian approximation

Thank you for this important question. The hessian approximation using DataInf [1] does come with approximation error. We could not perform an error analysis using our method as computing Hessians is computationally infeasible for large language models with sizes that we consider. Instead, we refer the reviewer to the DataInf paper (Section 4.1, Figure 1), which evaluates approximation error on a smaller scale using the RoBERTA-large model [2] (355M parameters). According to their experiments, DataInf achieves a correlation of approximately 0.35-0.65 with the exact influence estimate on the GLUE dataset [3]. Although this correlation does not reach optimality, it surpasses that of other estimation methods available to us. We therefore utilize DataInf to estimate influence functions, accepting this compromise as the most effective available solution within our computational constraints.

[Q3] Underperformance of Entropy

We expect that Entropy shows poor performance in detecting sycophancy bias primarily because this metric does not use a validation set, which is crucial for identifying sycophancy bias. Additionally, unlike other baselines that measure scores based on activations from intermediate layers of the reward model, Entropy relies solely on logit values. This approach may fail to capture the more complex patterns in the data that are essential for recognizing sycophancy bias.

---

[Q4] Procedure of weight selection

The five initial weights for model B were selected through a random procedure, with specific considerations in mind. Firstly, we aimed to ensure that the initial labeling accuracy of the model fell within the 70-80% range. Additionally, we avoided weights that were too similar, to ensure that the selected weights captured distinct aspects of our fine-grained objectives. We have updated Appendix B.2 in our draft to include these details.

---

[1] Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models. In International Conference on Learning Representa- tions, 2024.

[2] Liu, Yinhan. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 364, 2019.

[3] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.

Dear Reviewer roz9, we sincerely appreciate your valuable and helpful feedback. We address each comment in detail, one by one below.

---

[W1] Limitations in sharing sub-objective scores

Thank you for your insightful feedback. Our labeling oversight experiment serves as a proof-of-concept demonstrating how influence functions might guide labelers. As noted in the 'Limitations' section, we acknowledge the limitation that our setup does not perfectly model real-world conditions. However, despite these constraints, we believe our results provide valuable preliminary evidence that influence functions can be beneficial. We hope our work encourages further research to test influence functions in more realistic settings. We have revised the introduction section of our draft to make it clear that our labeler strategy guidance experiment is a proof-of-concept.

---

[W2] Sycophancy bias is challenging

We agree with the reviewer that detecting sycophancy bias presents significant challenges due to its reliance on understanding implicit and context-dependent human preferences. Despite these challenges, it is important to note that influence functions continue to perform relatively robustly compared to other baselines. For example, when matched at equal false positive rates, influence functions outperform GPT-4o, the strongest baseline, by a significant margin of 14.8% in sycophancy bias experiments. These results underscore the effectiveness of influence functions in managing more complex biases

---

[Q1] On more generalized validation samples

Thank you for your insightful comment. We acknowledge that requiring targeted evaluation sets can be a barrier to using our method (as mentioned in Remark 4.1 of the original draft). However, it is important to highlight that influence functions can still be effective even with less curated sets. To demonstrate this, we conducted additional experiments on bias detection using the original validation set of the Anthropic HH dataset, denoted as ‘Full’. This dataset was used in its unmodified, uncurated form as provided by the Anthropic HH validation set. The table below shows that influence functions effectively detect biased samples using this uncurated set compared to the high-quality, carefully filtered validation set used in our study (see Section 5.1.1 for more details).

These results confirm that influence functions are robust even without well-curated validation sets, demonstrating their utility with more generalized samples. We also note that influence functions demonstrate reasonable performance even with relatively small validation sets (approximately 50-100 samples), as shown in Figure 8 and Appendix G. This suggests the potential for practical applications, using small-scale expert data.

We have revised Figure 4 (with Section 5.1.2) of our draft to reflect these findings, with the changes highlighted in red.

---

[Q2] Scalability of current method

Scalability with increasing dataset sizes As shown in Figure 5 of our original draft, time consumption scales linearly with dataset sizes up to 100k samples on a single NVIDIA A6000 GPU. This linear scaling is expected to extend to even larger datasets, as the primary computational cost lies in the backpropagation step, which is proportional to dataset size. Importantly, this assumes adequate storage for the compressed gradient vectors of the training and validation datasets. By employing OPORP, we have compressed gradient vectors to $2^{16}$ dimensions, facilitating the storage of millions of samples, e.g., 5M samples require approximately 1TB of storage. If even larger datasets are considered, smaller compression dimensions could be used to further reduce storage needs, though at the cost of increased estimation variance, which, according to the OPORP paper, is inversely proportional to the compressing dimension $\text{K}$ ($\text{Var} \propto 1/\text{K}$) [1].

Scalability with increasing model size The computational expense also escalates with the model size due to the scaling of backpropagation costs with the number of model parameters. A trade-off here is the increased approximation error in DataInf, which arises from the need to approximate the Hessian matrix. This error potentially impacts performance of influence functions, particularly in scenarios demanding high precision.

[1] Ping Li and Xiaoyun Li. Oporp: One permutation+ one random projection. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023.

Dear Reviewer roz9,

Thank you again for your time and efforts in reviewing our paper.

As the discussion period draws close, we kindly remind you that two days remain for further comments or questions. We would appreciate the opportunity to address any additional concerns you may have before the discussion phase ends.

Thank you very much!

Many thanks, Authors

Dear Reviewer roz9,

Thank you for your response. We are pleased to hear that your concerns have been resolved and sincerely appreciate your decision to raise the score.

If there are any remaining concerns or suggestions for improvement, we would be more than happy to address them.

Best regards, Authors