Response to Weakness 1

Thank you for your feedback. As you kindly pointed out, the supervised fine-tuning methods are indeed only part of this paper. The primary contribution of our work is the development of a comprehensive and feasible framework for "alignment for honesty": this includes establishing a precise problem definition, proposing effective evaluation metrics, and also demonstrating the effectiveness of several straightforward training methods. We hope our first step on alignment for honesty can inspire more excellent future work.

Response to Weakness 2

Thank you for your valuable comment. We would like to take this opportunity to provide some clarifications regarding our proposed methods.

(1) Regarding

..., it cannot be ruled out that multiple samples perform better because they trained the correct answer more times.

We would like to kindly clarify that the Multisample method does not perform better in terms of "Accuracy" although it "trained the correct answer more times", as shown in Tab. 3. However, we would like to note that there is an improvement in the honesty score, and we are more than willing to explain why it works for "Honesty": By implicitly learning from the proportions of correct answers and idk responses among the m sampled responses in the expanded training data for Multisample, the model can better recognize its knowledge boundaries in a fine-grained manner, leading to the improved honesty score.

(2) Regarding

... Intuitively from the prompt, such training does not directly make the model learn to say "I don't know."

First, we would like to respectfully remind you that the training data for both the Absolute and Confidence methods includes an equal number of "unknown questions" (as shown in Eq. 9) where the output is an idk response. This means that in both methods, the model has the same opportunity to directly learn to say "I don't know".

Additionally, compared to Absolute, Confidence provides prefixed confidence expressions for "known questions". These prefixes serve as more fine-grained supervision signals, enabling the model to implicitly learn to more precisely capture its knowledge boundaries. The experimental results presented in Tab. 3 and 4 demonstrate that these additional hints are indeed beneficial for improving the honesty score, as we discuss in detail in Lines 256-262 of our paper.

Response to Weakness 3

We sincerely appreciate you bringing this valuable reference to our attention. However, we would like to kindly clarify that [1] has a different purpose from our work. Section 4.3.3 of [1] introduces adding honesty reading vectors to elicit truthful answers to known questions, which corresponds to the concept of "eliciting latent knowledge" as we discuss in Lines 149-154. In contrast, our study aims to adjust the model's behavior for both known and unknown questions. We will address this distinction in our related work section in the revised version. Once again, thank you for sharing this reference, and we look forward to incorporating exploration at the representation level in our future work.

[1] Representation Engineering: A Top-Down Approach to AI Transparency

Response to Question 1

Thank you for your thoughtful question. We would like to explain that the process of aligning the model with honesty does not explicitly introduce any instructions that would compromise safety, so that we did not conduct experiments on how increased honesty impacts harmlessness. However, we are more than willing to exploring this further through empirical research.

Specifically, we utilize the 700 test prompts from BeaverTails-Evaluation that can potentially elicit harmful responses, and employ GPT-4o to assess whether the model responses are safe, unsafe, or controversial (in terms of safety). As shown in Table 1 in the global response, honesty-oriented supervised fine-tuning has almost no impact on the model's inherent harmlessness.

Response to Question 2

Thank you for your insightful question. We are very pleased to analyze the relationship between confidence expressions and accuracy of the model trained using Confidence-Verb.

Specifically, we bin M_t+1's responses (the model after alignment using Confidence-Verb) by its expressed confidence and measure the average accuracy of responses in each confidence bin. From Figure 1 in the global response, we observe that:

1. In areas of high confidence, accuracy is slightly lower than its corresponding confidence, reflecting the model's over-confidence, which is consistent with related work [2].
2. In areas of low confidence, particularly at confidence=0.2, accuracy is actually higher than its corresponding confidence. This can be explained by the fact that the training data for Confidence-Verb pairs "confidence expression" with "correct response", encouraging the model to prioritize correct answers over calibrated confidence. This suggests a potential issue for future work on calibration: how to balance calibration and performance.

We acknowledge that our Confidence method, while leading the model to respond more prudently, has potentially limited effects on calibration due to the nature of our constructed training data. We encourage future work to handle known questions in a more fine-grained manner (as stated in Lines 42-44) and strike a balance between calibration and performance.

If we have misinterpreted any aspect of your questions, or if you have suggestions for more targeted experiments, please feel free to tell us.

[2] Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

Response to Limitations

Thank you for your reminder. We would like to clarify that we have included a Limitations section in Section 5. Additionally, as outlined in 979-985, we have explained why this paper does not address societal impact. However, we can reorganize a Limitations & Social Impacts section.

Response to Weakness 1 and Question 1

... The paper only explores the impact of one hyperparameter in Section D.5.1. It is believed that further exploration of related hyperparameters could be quite interesting.

Is there further investigation and case study of the relation between the hyperparameter and model behavior?

Thank you for your thoughtful comment. We have provided a practical and enlightening analysis of how hyperparameters impact model behavior in Appendix D.5.1. Nevertheless, we are quite open to conducting additional experiments to discover further meaningful insights.

Specifically, we explore how the number of unknown questions affects Multisample's performance. We define "absolute unknown questions" (AUQs) as those where all m=10 sampled responses are incorrect, and then gradually reduce the number of AUQs among the 8,000 training questions. Notably, even with AUQs reduced to 0, the expanded training dataset for Multisample (please refer to Lines 216-218) still contains samples whose output is an idk response, allowing the model to directly learn this refusal expression. The results are shown in the table below, where "AUQ reduction prop." indicates the proportion of reduced AUQs, and "Training idk prop." indicates the proportion of samples whose output is an idk response relative to the remaining training data.

From the results, we can see that overall, as the number of unknown questions decreases, the model becomes less conservative but also less reliable when answering unknown questions, which underscores the need for empirical optimization of the ratio between unknown and known questions in the training dataset. We can include these results in the Appendix.

If we have misinterpreted any aspect of your questions, or if you have suggestions for more targeted experiments, please feel free to tell us.

Response to Weakness 2 and Question 2

The evaluation metrics proposed in the paper rely on a baseline model (M_0). However, obtaining this M_0 in a public benchmarking setting may be difficult, thereby limiting the widespread adoption of the evaluation methods proposed. Moreover, determining which M_{t_1} is base and evaluating it appropriately can also be challenging. For instance, if a model at one stage of training "learns" to answer a question and then loses this ability later, I can be confused about whether it possesses knowledge and honesty.

Thank you for your insightful feedback. We would like to clarify that the flexibility of our proposed evaluation metrics allows us to freely designate a starting point M_t and an endpoint M_t+1 to assess whether the alignment process is beneficial in terms of honesty, without the need to trace back to M_0. For instance, if M_t can correctly answer a question but M_t+1 refuses to answer, then this over-conservativeness is undesirable; if M_t answers incorrectly and M_t+1 refuses to answer, then this prudence is worth encouraging. In Section 4.4.2, we have demonstrated the adaptability of this framework across multiple open-source LLMs whose specific training stages (M_x) are unknown.

Regarding the provided case, suppose M_t-1 can correctly answer a question, but M_t cannot:

1. If we have access to M_t-1, we can use it as a starting point and expect a better aligned M_t' to retain the ability to answer the question correctly.
2. If we only have access to M_t, then with M_t as the starting point, we would prefer M_t+1 to refuse to answer the question rather than answer it incorrectly, thus ensuring its reliability.

However, as we mention in Lines 149-154, we acknowledge that there are more complex scenarios that need to be explored in future work, such as whether we can elicit latent knowledge for M_t that M_t-1 possesses but M_t seems to have lost.

Is there further analysis of the model behavior and honesty dynamics during the alignment process?

We have demonstrated the effectiveness of our honesty-oriented supervised fine-tuning methods across in-domain (Section 4.4), out-of-domain (4.5), and helpfulness-related tasks (4.6), with two real cases provided in Appendix D.8. Our findings show that, although there is a minor reduction in informativeness, the model becomes more reliable and trustworthy, which is a positive outcome from an alignment perspective. Additionally, we find that non-honesty-oriented fine-tuning leads LLMs to hallucinate, whereas honesty-oriented fine-tuning performs better. This highlights the importance of integrating honesty considerations in the model training and alignment process.

Overall, these extensive experiments not only depict the current state of alignment for honesty but also guide the direction for future developments in this field.

We would be eager to engage in a discussion if there are specific experiments or particular aspects that interest you.

---

Thank you for reviewing our paper. We greatly appreciate your valuable feedback and are ready to address any further questions or concerns you may have.

Response to Question 1

While the article emphasizes the importance of honesty, it does not seem to discuss in detail how to maintain the model's helpfulness while improving honesty. Could this lead to the model being overly cautious in practical applications and unable to provide useful information?

Thank you for your thoughtful question. We share your concern regarding whether aligning the model with honesty compromises its helpfulness. Therefore, we have assessed the model's helpfulness before and after alignment using helpfulness-related requests in Section 4.6. The results indicate that "the cost of aligning LLMs for honesty does not significantly impact their overall helpfulness."

Meanwhile, we would like to share our understanding of these results:

1. Helpfulness-related practical tasks and knowledge-based QA tasks belong to different domains (e.g., "Summarize the following post." vs. "Who was the first president of the USA?"). Consequently, honesty-oriented supervised fine-tuning methods do not lead the model to reject requests associated with helpfulness.
2. On the other hand, honesty-oriented supervised fine-tuning does not cause severe model collapse or a loss of the model's original capabilities, as demonstrated by the robust accuracy on knowledge-based QA tasks presented in Tab. 3 and 4.

Response to Question 2

The paper has simplified some assumptions when defining honesty, such as the distinction between honesty and truthfulness. The focus is mainly on whether the model can express its internal knowledge, rather than whether its knowledge corresponds to objective facts, right?

Thanks for your question. You are correct that the focus of honesty is on whether the model can express its internal knowledge. By emphasizing honesty rather than truthfulness, we aim to explore the model's knowledge boundaries, instead of compelling it to provide accurate information without considering what it has learned.

Please feel free to reach out if you have any further questions or require additional clarifications.

Response to Question 3

How about a 'white lie'? Is it truly beneficial for a model to be brutally honest when informing someone they have cancer?

This question is critically important for the development of superintelligence and necessitates the involvement of the entire AI community. Nonetheless, we would like to share our perspective: it is essential that the model remains entirely honest (without white lie) with at least some individuals, such as developers. Without such transparency, combining deceit with superhuman capabilities could lead to immeasurable and unforeseen dangers.

Response to Limitations

Although the paper proposes methods to approximate the model's internal knowledge through external behaviors, it does not delve deeply into the working principles and knowledge representation within the model, which may limit a more profound understanding of the model's honesty.

Thank you for your insightful comment! We would like to respectfully note that fully understanding the model's internal workings is a non-trivial thing and depends on progress in other research areas, including model interpretability. We are pleased to have taken the first step toward aligning AI models with honesty by establishing a comprehensive and practical framework. As we mention in the Conclusion section: "We hope this work can inspire more thoughts on the development of honest AI models in the NLP community."

---

Thank you for taking the time and effort to review our paper. We are more than willing to provide any clarifications if you have further questions or concerns.