More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness

We genuinely appreciate your feedback on our paper, and here we would like to address your questions and concerns raised in the review. We also include our responses for common questions shared by multiple reviewers in a separate section above.

$**Q4.1 The cause of the misalignment issue and potential mitigation strategies**$

$**A4.1**$

Thank you for your feedback. We addressed a similar question in A2.2, and we encourage you to refer to it for additional context. The purpose of section 5 in our paper is to explain the misalignment issue from a data-centric perspective, where we propose adapting data attribution methods to the RLHF setting. This approach enables users to identify influential data points that may negatively impact trustworthiness and provides a pathway to address these issues through dataset pruning.

However, we acknowledge that fully validating the effectiveness of this approach requires significant computational resources, such as re-running RLHF and conducting evaluations multiple times on pruned datasets. Due to these constraints, we have decided to leave the comprehensive empirical validation of this mitigation strategy to future work. We believe this is a promising direction that could systematically address misalignment issues and complement our current findings.

$**Q4.2 Dataset limitations**$

$**A4.2**$ This is answered in global response A0.2. Please let us know if you have further questions.

$**Q4.3 Model scale constraints**$

$**A4.3**$ This is answered in global response A0.1. Please let us know if you have further questions.

$**Q4.4 Additional fine-tuning combined with RLHF to mitigate observed trustworthiness degradation**$

$**A4.4**$ Yes, this is certainly possible. As proposed by Haider et al. (2024), such post-training requires a “break-fix” cycle, involving multiple rounds of dataset curation, safety post-training, benchmarking, red teaming, and vulnerability identification to address various harm areas systematically. While algorithm-based mitigation strategies are viable, our intuition is that they need to be combined with (or relied on) high-quality and trustworthiness-aware preference datasets. These datasets provide supervision specifically aligned with trustworthiness goals, serving as a critical foundation for addressing observed degradations after RLHF.

$**Q4.5 Possibility of using smaller, targeted RLHF datasets or combinations of general-purpose and aspect-specific datasets**$

$**A4.5**$ This is also answered in global response A0.2.

$**Q4.6 Influence of dataset size (used for RLHF) on model trustworthiness metrics**$

$**A4.6**$

While this is not the primary focus of our work, we are happy to share some preliminary thoughts. From a dataset perspective, two key factors influence a model's performance on trustworthiness metrics: (1) the diversity of the fine-tuning dataset, which ensures the model has sufficient generalizability, and (2) the alignment between the dataset and the trustworthiness aspects being evaluated. In terms of dataset size, it typically needs to be sufficiently large to achieve adequate diversity. For widely used general-purpose preference datasets, such as Anthropic-HH, diversity is less of a concern due to their broad coverage. However, our findings highlight a misalignment between Anthropic-HH and several trustworthiness metrics, leading to decreased performance on those tasks.

For your question, the impact of dataset size on trustworthiness depends heavily on the level of alignment between the dataset and the trustworthiness aspects. If alignment is strong and computational resources are not a constraint, larger datasets generally lead to better performance by providing more comprehensive training signals. Conversely, if the dataset is misaligned with the trustworthiness dimensions, increasing its size may not improve performance and could even exacerbate issues.

We did not conduct experiments varying dataset sizes, as this would be computationally intensive and not directly related to our main research question. However, future work could explore this relationship more systematically to better understand the trade-offs between dataset size, alignment, and trustworthiness outcomes.

Thanks for the clarification. I will keep my positive score.

We genuinely appreciate your feedback on our paper, and here we would like to address your questions and concerns raised in the review. We also include our responses for common questions shared by multiple reviewers in a separate section above.

$**Q3.1 Generalizability of the findings to larger models**$

$**A3.1**$ This is answered in global response A0.1. Please let us know if you have further questions.

$**Q3.2 More specialized preference datasets**$

$**A3.2**$ This is answered in global response A0.2. Please let us know if you have further questions.

$**Q3.3 Guidance for balancing helpfulness and trustworthiness**$

$**A3.3**$ Relevant discussions are briefly included in Sections 5 and 6, but we are happy to expand on them in the updated version. Broadly speaking, addressing the limitations of RLHF requires a more nuanced approach than simply applying it to crowdsourced, general-purpose datasets. For example, using datasets tailored to individual trustworthiness aspects (as some reviewers have suggested) might be able to improve alignment with specific goals. Additionally, performing dataset pruning (e.g. using our proposed data attribution method) might help identify and remove samples likely to negatively impact trustworthiness dimensions. These strategies offer promising avenues to enhance alignment while mitigating unintended consequences.

Thanks for the explanation, I will keep my positive score.

$**Q2.8 Sensitivity to different prompts**$

$**A2.8**$ For specific examples of prompts being used in our evaluations, please refer to appendix A. We did evaluate the model sensitivity to different system prompts for toxicity, stereotypical bias, and machine ethics tasks, and the results are reported in appendix D and E. For toxicity, we also evaluated on toxic and non-toxic user prompts, and the details can be found in section 4.1. The general observation is that the trends are consistent across all these different settings. For truthfulness evaluation, we experimented with different prompting strategies and found most success with directly asking the model to repeat its answer among all multiple-choice options. We used the same prompt for all different model configurations.

$**Q2.9 Motivation and practicality of the privacy evaluation**$

$**A2.9**$ We included this privacy evaluation in our work because it exemplifies cases where helpfulness and trustworthiness/safety are in direct conflict, based on human intuition. While RLHF fine-tuning seeks to make models more helpful and aligned with user instructions, it can inadvertently increase compliance with harmful requests, such as disclosing sensitive information. Evaluating privacy leakage underscores this trade-off, showing how prioritizing helpfulness without explicitly addressing safety risks can undermine trustworthiness. These scenarios are highly relevant in real-world applications, where users often share private data with the expectation that the system will safeguard it.

$**Q2.10 Verification for the claim that “the HH dataset used for RLHF is out-of-domain for the evaluation tasks”**$

$**A2.10**$

Thank you for your suggestion. In the table below, we report the changes in average perplexity scores across all evaluated prompts for five trustworthiness tasks after the initial SFT with the Anthropic-HH preference data, using Pythia-6.9B. The results show that perplexity scores for all five evaluation tasks slightly increase after SFT. This indicates that there are indeed out-of-domain (OOD) issues between the fine-tuning data and the evaluation datasets, even for the ethics benchmark, where we observe improvements in model trustworthiness.

That said, we would like to clarify that while perplexity is a useful measure for identifying surface-level OOD issues (e.g., unfamiliar vocabulary or syntax), it may not adequately capture deeper alignment or misalignment between datasets from a trustworthiness perspective. By describing the HH dataset as "OOD for evaluation tasks," we mean that it may include samples that are not beneficial—or may even be detrimental—to specific trustworthiness aspects. This higher-level misalignment is likely beyond what perplexity scores alone can reveal.

To better illustrate this OOD issue, particularly from a trustworthiness perspective, we have provided concrete examples of fine-tuning samples that negatively impact evaluation tasks. These examples are included in Figure 6.

$**Q2.11 Statistical significance of differences in log likelihood values reported in Table 5**$

$**A2.11**$ Thank you for pointing this out. These log-likelihood values are essentially negative perplexity scores, but they are computed on the model's self generations rather than a given corpus of texts. While the changes in these values might appear small, they are statistically significant, especially given that they are averaged over thousands of prompts and five generations per prompt. We will include the confidence intervals based on these five generations in the updated paper to better show the statistical robustness of this particular analysis.

$**Q2.12 Statistical significance supporting the claim “toxicity first increases notably after SFT"**$

$**A2.12**$ Thanks for pointing this out and you are right that we should not use the phrase “increases notably” here based on the overlapping error bars. We will make this revision in the updated paper.

$**Q2.5 Different generation configs compared with prior work in toxicity evaluation**$

$**A2.5**$

The specific configuration of 25 generations and temperature = 1, as used by Wang et al. (2024), is tailored for evaluating very large models like GPT-3.5 and GPT-4. Since our models are significantly smaller, we do not believe their output distributions are directly comparable. Moreover, it is unclear why maintaining the same distribution as a prior work with different models would be necessary (or could you clarify why this is important?). As long as we consistently apply the same generation configuration across all toxicity evaluations, our experiments remain in a highly controlled setting.

The prior work (Wang et al., 2024) also relied on a single model configuration for toxicity evaluation. We believe the robustness of our findings is further supported by the fact that we took the average across 1,200 prompts. Our choice of num_generations=5 is primarily due to computational constraints and the rate limits of Perspective API calls. To comply with these limits, we also opted for a lower temperature of 0.5 to produce a more centralized distribution.

Since our focus is on expected maximum toxicity (i.e. the maximum among the 5 generations) rather than the mean, increasing the number of generations from 5 to 25 would not enhance the robustness of the results. Under this metric, toxicity may slightly increase (with diminishing returns) as the number of generations or the temperature value increases. However, this expected increase does not compromise the robustness of our claims. A highly systematic sensitivity analysis would be time-consuming and not feasible within the discussion period, and in our opinion, it is not strictly necessary.

$**Q2.6 Sycophantic tendency of language models**$

$**A2.6**$

This is indeed an interesting question to discuss. We agree that the observed model behaviors (especially in bias and ethics) could be potentially decoupled into true model understanding and it’s inherent tendency to agree with user beliefs, but we’d like to argue that sycophancy in language models is not merely an artifact of evaluation, but a reflection of the model's alignment with user expectations. In human interactions, individuals often conform to social norms and expectations to build rapport and trust. Similarly, LLMs are designed to align with user inputs, which can manifest as agreement with presented statements. Consider a scenario where a person is asked, "Do you think people from [specific group] are less competent?" Even if the individual does not hold this belief, social pressures or a desire to conform might lead them to agree, especially if they perceive that agreement is expected or will be favorably received. This human tendency to conform or agree, even against personal beliefs, parallels the sycophantic behavior observed in LLMs. In other words, this component is part of the model's trustworthiness behavior, and we should not exclude it in our evaluation, as is consistent with prior works like Wang et al. (2024).

As a side note, the decoupling to evaluate true model beliefs could be done by using a balanced set of user prompts (e.g. constructing the statements with opposite meanings, or simply taking the average between “do you agree…” and “do you disagree”), and this would also be an interesting follow-up work for future research that requires rigorous experimental design.

$**Q2.7 Definition of FPR in machine ethics evaluation**$

$**A2.7**$

We appreciate the reviewer’s observation and agree that our current framing of the false positive rate (FPR) as the primary metric could be confusing given the context. The evaluation setup we referenced from Wang et al. considers morally neutral or positive actions as the “positive” class and immoral actions as the “negative” class. This led us to adopt FPR for consistency. However, we acknowledge that our goal is to evaluate the model’s ability to correctly identify morally wrong actions, so it’s more appropriate to switch the definitions of positive and negative cases and then call our metric “false negative rate” (FNR). FNR reflects the rate at which morally wrong actions are mistakenly overlooked, which is critical because failing to recognize such actions undermines the model’s trustworthiness in ethical decision-making tasks.

$**Q2.3 Contributions compared with prior works**$

$**A2.3**$

We consider our evaluation to be “the first systematic evaluation of RLHF’s impact on key trustworthiness aspects.” By this, we mean:

1. It uses the most widely adopted general-purpose RLHF dataset (Anthropic-HH), popular open-source models, and standard RLHF procedures.

2. It provides clear, stage-wise comparisons of model trustworthiness across different RLHF steps (SFT, PPO, and DPO).

3. It evaluates five trustworthiness aspects using standard, widely accepted benchmarks, enabling direct comparisons.

Here we’d also like to highlight key differences between our evaluation and each previous work you referenced above:

1. Askell et al. 2021: This work focuses on SFT and separate ranked preference modeling tasks, rather than standard RLHF workflows used nowadays. Moreover, it does not include comprehensive trustworthiness evaluations—its overlap with our work is limited to some ethics benchmarks.

2. Glaese et al. 2022: The goal of this work is to build a highly aligned dialogue agent by collecting preference data with fine-grained and targeted rules (23 rules in total), which is quite different from our general-purpose preference dataset. While some rules partially overlap with our trustworthiness aspects, their approach fundamentally differs by augmenting standard RLHF with multiple reward models designed for these specific rules. Our goal is completely different: to identify misalignments between a general-purpose dataset and trustworthiness aspects without such targeted augmentations for RLHF.

3. Wang et al. 2024: This study evaluates trustworthiness on closed-source GPT-3.5 and GPT-4 models without explicit RLHF experiments, making direct before-and-after RLHF comparisons impossible.

4. GPT-4 Tech Report (2024): The RLHF dataset is proprietary, and while the report compares base and fine-tuned models, the focus is not on trustworthiness aspects. Discussions around safety primarily involve comparisons against earlier model versions, not pre-RLHF baselines.

5. LLama 2 (Touvron et al 2023): Their internal RLHF dataset is proprietary and incorporates fine-grained supervision (e.g., safety labels), which differs from the general-purpose Anthropic-HH dataset used in our study. Additionally, trustworthiness evaluations in their work didn’t include critical aspects like machine ethics and privacy leakage.

In addition to much more controlled trustworthiness comparisons before and after RLHF when compared with prior works, our work has another novel contribution: adapting data attribution methods to the RLHF setting. This approach provides unique, data-level insights into the relationship between training data and trustworthiness outcomes. We will further emphasize these distinctions and contributions in the introduction and related work sections to clarify how our methodology and findings extend previous research. Thank you for your valuable feedback.

$**Q2.4. Are the base models capable enough to follow task-specific instructions during evaluation?**$

$**A2.4**$

Thank you for pointing this out. Our decision to use 1.4B as the smallest model size stems from our observation that this size is the minimum required for consistent adherence to output formats across the evaluation tasks. Specifically:

Toxicity and privacy tasks: These do not require explicit generation formats, so adherence is not an issue.

Bias and ethics tasks: Models are prompted to respond with “Yes” or “No,” followed by reasoning. All four models reliably follow this format for all evaluated prompts (i.e. 100% success).

Truthfulness evaluation: This is the only task where we observed occasional format inconsistencies. In this task, the model is presented with a multiple-choice question and instructed to repeat the correct answer explicitly. Failures occur when the model does not repeat any of the provided options. We report the percentage of base model generations that correctly adhere to this instruction in the table below. These observations indicate that the base models are generally capable of following task-specific instructions even in zero-shot settings, and we will add these details to the appendix.

We appreciate your detailed review and valuable feedback for our paper, and here we would like to address the main questions and concerns raised in the review. We also include our responses for common questions shared by multiple reviewers in a separate section above. Additionally, we will incorporate presentation-related suggestions into the updated paper as much as possible.

$**Q2.1 Generalizability to newer models like OLMo and Llama 3 models**$

$**A2.1**$ This is answered in global comment A0.1. Please let us know if you have further questions or comments.

$**Q2.2 Insights into the data attribution results and mitigation strategies for the misalignment issue**$

$**A2.2**$

As we mentioned in our response A1.2, it’s hard to make generalized claims about the observed patterns of contribution scores shown in Figure 5 and Figure 11 without considering the specific model and the current RLHF step, as the trends vary across different configurations.

As partially hinted in our intro and conclusion sections, a key application of our data attribution approach is its potential to identify influential data points that negatively impact specific trustworthiness aspects. These data points can then be removed from the RLHF dataset (i.e. dataset pruning) to address the misalignment issue. Similar discussions are also included in A4.1, A4.4, and A4.6. However, rigorously testing the effectiveness of this approach would require re-running RLHF and trustworthiness evaluations multiple times, which is computationally prohibitive and also not directly related to the focus of this work. For this reason, we have decided to leave such empirical validations for future research.

The main contribution of this work is to shed light on the misalignment problem in RLHF and provide data-level insights through our attribution analysis. We recognize that further exploration is needed to fully leverage this approach, and we will add more discussion on its potential applications and long-term utility in addressing misalignment in the updated paper. We greatly appreciate your valuable feedback.

As for the general comment on why you don't include newer models, I don't fully agree with the claim "We chose Pythia and older Llama models because they are most widely used within the research community, supporting reproducibility and cross-comparisons." In particular, OLMo won the best theme paper award in ACL which emphasizes the importance of OLMo for the research community. Perhaps the authors can convince me by providing examples of uses of Llama or Pythia models in the community that are still widely used and relevant in the research community.

However I do agree that the core principles of RLHF and the findings of this paper remain relevant across architectures and that performing such studies across multiple models would be infeasible.

Based on the discussed changes to the camera ready version, I'm raising my score as I believe this work is important for the community and the authors addressed the weaknesses and questions.

We appreciate your detailed review and recognition of our paper's contributions. Our responses to your questions and concerns are provided below. We also include our responses for common questions shared by multiple reviewers in a separate section above.

$**Q1.1 Noises in the black-box API for toxicity evaluation**$

$**A1.1**$ We acknowledge that inherent noise exists in toxicity evaluations using black-box APIs like Perspective API, and we appreciate you pointing this out. The two papers cited in the review highlight two specific sources of noise: (1) discrepancies in toxicity scores for the same model generations over extended periods (e.g., 2020 vs. 2023), likely due to API updates, and (2) differences across languages (e.g., German translations receiving higher toxicity scores). However, these issues are unlikely to affect our experiment results because (1) all our toxicity evaluations using Perspective API were conducted within approximately two weeks, minimizing the impact of potential API updates, and (2) the RealToxicityPrompts dataset consists entirely of English prompts, eliminating cross-language inconsistencies.

To the best of our knowledge, Perspective API remains the most widely used toxicity evaluation tool in the research community. While human evaluators might seem like an alternative, they are more prone to errors for such rating tasks, and their judgments are also subject to variability over time and across languages. Given these factors, we believe Perspective API is probably the most reliable and practical option for our study.

$**Q1.2 Potential trade-offs between the five trustworthiness aspects**$

$**A1.2**$ We believe it’s challenging to make high-level claims about trade-offs between the five dimensions of trustworthiness without considering specific scenarios. In this work, the impact on any trustworthiness aspect depends on the fine-tuning data, the target model, and the specific RLHF step (SFT, PPO, or DPO). Figures 5 and 11 show the aggregated effects across the entire fine-tuning dataset, highlighting these dependencies. For any individual trustworthiness aspect, typically only a small fraction of the data points are relevant, so it’s more meaningful to focus on specific data samples or subsets based on estimated contribution scores. For example, we can make microscopic observations like “this subset of fine-tuning data positively impacts bias but negatively impacts truthfulness.” However, this does not imply that bias and truthfulness are inherently in conflict. Ultimately, such trade-offs are highly contextual, and generalizing without grounding the discussion in specific data or scenarios risks oversimplification. We believe a data-driven, granular approach is the most effective way to analyze these dynamics.