Thanks for your review and valuable feedback. We provide our responses below:

• We are considering adding knowledge-intensive QA datasets as a further validation scenario in our future versions. Moreover, we will release our code and data soon to facilitate future research.

• The questions in GSM8k are relatively complex, making it difficult for the model to learn which questions it can answer and which it cannot. Additionally, our use of self-consistency to generate preference data has also introduced some degree of noise. In the future, we will explore using different alignment objectives and data generation methods to enable the model to have better self-knowledge on complex tasks and avoid overconfidence.

• Reponse to the question:

  • We apologize for any confusion caused by the typos. "Acnt" in Table 2 indeed represents truthfulness.

Thanks for your review and valuable feedback. We provide our response below:

• On the one hand, our method shows a slight loss in accuracy for arithmetic tasks, but it significantly improves truthfulness, greatly enhancing the reliability of the model. On the other hand, the questions in GSM8k are relatively complex, making it difficult for the model to learn which questions it can answer and which it cannot. Additionally, our use of self-consistency to generate preference data has introduced some degree of noise. Despite this, our model has found a good balance between being overly cautious and overly confident in out-of-domain scenarios, significantly improving the overall reliability of the model. In the future, we will explore using different alignment objectives and data generation methods to enable the model to have better self-knowledge on complex tasks and avoid overconfidence.

Hi J8K1, can you check the authors' response and update your review if it addressed your concern (or participate in discussion with the authors if it did not)?

Thanks for your review and valuable feedback. We provide our responses below:

• Thanks for your suggestions. We will discuss more about how to apply our method to other tasks in the conclusion.

• Due to the relatively simple format of our arithmetic questions (template-synthesized questions), directly converting the question into an equation and sending it to a calculator achieves very high accuracy. However, we believe that the best way to integrate tools like calculators or search engines is for the model to determine whether it can answer the question itself. Similar to the Dual process theory in the cognitive process of humans [1-4], we consider the model's direct responses analogous to an implicit, unconscious, and intuitive System 1, which is faster but more prone to errors, while invoking external tools is akin to an explicit, conscious and controllable System 2, which has some overhead but is more accurate. We believe the model needs to learn its own capability boundaries to better determine when to call external tools, rather than making redundant tool calls that increase inference time. In this case, we treat the model's refusal to answer as a signal to call the calculator. When the model refuses, we have it convert the problem into an equation and use the calculator. As shown in the table below, our model effectively recognizes its capability boundaries and calls the calculator to solve the problem, achieving the highest accuracy.

[1] Evans, Jonathan St BT. "In two minds: dual-process accounts of reasoning." Trends in cognitive sciences 7.10 (2003): 454-459.

[2] Evans, Jonathan St BT. "Dual-processing accounts of reasoning, judgment, and social cognition." Annu. Rev. Psychol. 59 (2008): 255-278.

[3] Sloman, Steven A. "The empirical case for two systems of reasoning." Psychological bulletin 119.1 (1996): 3.

[4] Kahneman, Daniel. Thinking, fast and slow. macmillan, 2011.

directly converting the question into an equation and sending it to a calculator achieves very high accuracy

how much accuracy does this achieve? higher than 85.2?

ty

Thanks for your review and valuable feedback. Due to space constraints, we did not include specific experimental tables in the rebuttal. We will comment on the experimental results during the discussion phase.

About Calibration-Based Methods

We have already added some relevant references to the paper[1], and we will include the missing references you provided. Besides, the comparison with calibration methods will also be added to the experimental section of the paper.

We compared our model with several methods, including raw logits, P(True), verb. 1S/2S, and consistency-based agreement. Our method performs slightly worse than the consistency-based method but significantly better than other baselines. Additionally, our method achieves reliability comparable to the consistency-based method while significantly reducing inference costs. Further comparisons and analyses are as follows:

• Explicit Rejection: Calibration methods require manually set classification thresholds and rejection templates, which vary significantly across datasets, leading to performance degradation. In contrast, our method can reject out-of-knowledge questions automatically with personalized responses for all datasets.

• Inference Cost: Our method (rlkf) achieves a significant reduction in inference cost compared to consistency-based methods which require multiple samplings and potentially additional models for answer extraction. Some Verbalized-based methods (verb. 2S) also require the model to generate confidence through an additional round of response.

• Calibration Variance: Calibration methods show instability in accuracy, especially with logit-based and verbalized methods. For instance, logit-based methods are not quite reasonable when the model generates longer responses, and verbalized-based methods can fluctuate significantly (gsm8k results of verb. methods) due to prompt variability.

About broader instruction-tuning

We tested gpt-3.5-turbo-0125 on arithmetic questions and found that while ChatGPT exhibits high accuracy on simpler tasks, it struggles with rejecting out-of-knowledge questions and shows significantly lower reliability than our methods on harder tasks. Thus we believe that broader generic instruction-tuning alone does not resolve reliability issues.

Thank you for providing many other valuable comments on the writing and presentation of our paper. We will address these in future versions of the paper.

[1] Language Models (Mostly) Know What They Know

We further compare our method with calibration-based methods, and provide the results and our analysis below:

• Unable to Reject Explicitly: Calibration-based methods need to search and determine the best threshold for rejection and provide human-crafted rejection templates as responses. As we shown in the table, we search the threshold for arithmetic and gsm8k separately (on 100 validation cases from each dataset). However, the thresholds are quite different for different datasets which results in significant performance degradation with different thresholds. In contrast, our method can enable the model to reject out-of-knowledge questions with personalized responses for different prompts automatically.

• High Inference Cost: Consistency-based methods, on the one hand, require multiple samplings to obtain results, and on the other hand, may necessitate the use of additional models to extract answers for voting (we use ChatGPT to extract answers because rule-based methods may result in inaccurate extraction). This results in 5(sampling num) * 2(1 for answer generation, 1 for answer extraction) = 10 times (or at least 5 times) the inference cost than other methods. Some Verbalized-based methods (verb. 2S) also require the model to generate confidence through an additional round of response after generating the answer.

• High Calibration Variance: Utilizing calibration methods to determine the accuracy of answers is not stable. For instance, logit-based methods are not quite reasonable when the model generates longer responses, and Verbalized-based methods result in significant fluctuations in confidence scores and even prediction results (as shown in the gsm8k results of verb. methods) due to the variability in prompts.

In summary, calibration methods are more suitable for analyzing the uncertainty of model responses or constructing training data (such as the self-consistency[5] introduced in our paper). However, our alignment research on reliability aims to enable the model to acquire self-knowledge and explicitly refuse out-of-knowledge questions. Experimental results show our method can enable the model to reject automatically without additional inference costs and improve the accuracy of rejections compared to most calibration methods.

[1] Calibrating Large Language Models with Sample Consistency

[2] Calibration-Tuning: Teaching Large Language Models to Know What They Don’t Know

[3] Language Models (Mostly) Know What They Know

[4] Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

[5] Self-consistency improves chain of thought reasoning in language models.

Hi tasN, can you check the authors' response and update your review if it addressed your concern (or participate in discussion with the authors if it did not)?

Thanks for your comments! We give our responses below:

1. Re: Comment 1

• We construct the preference pairs following the same construction setting of in-domain preference data in our paper. When the model's multiple sampling results are all correct, we use pairs where the correct answer is preferred over a refusal. When all results are incorrect, we use pairs where a refusal is preferred over the incorrect answer. When the sampling results are mixed, we use pairs where the correct answer is preferred over the incorrect answer. We use the exact match metric instead of accuracy to determine if each sampling result is correct.
• Due to time constraints, we did not run the RLHF baseline. We will include the relevant experiments in future version. Additionally, since the reward model used in RLHF does not incorporate the preference data regarding refusals as in RLKF, we believe it cannot effectively improve the model's truthfulness, as demonstrated by our results on the arithmetic dataset in the paper.

2. Re: Comment 4

Thank you for your clarification! We indeed had a misunderstanding regarding the initialization of the reward model in Llama-2. We will correct the inaccurate claims in our paper. The core difference between our method and the RLHF method is that we explicitly introduced the rejection preference pairs in the training of the reward model. This allows the reward model to better understand when to refuse. As shown in Table 4 of our paper, the reward model trained with helpful preference data only achieved an accuracy of 49.9 on the "beyond" subset of the in-domain arithmetic reliable preference data (this task is binary classification, where the reward model chooses between refusing to respond and responding incorrectly, with the correct label being refusal). However, after adding the corresponding in-domain reliable preference data (i.e., various preference pairs regarding refusals), this classification accuracy significantly improved to 83.0. This partly explains why RLKF can better improve the model's reliability compared to RLHF, as the reward model after RLKF can better determine when it should refuse.