Q1: The paper includes a 3d pie chart that significantly hurts the readability of the paper. I suggest to the authors to change the type of plot in Figure 2.

A1: Thank you for your feedback regarding the 3D pie chart. We apologize for the readability issues it caused. We have replaced it with Table 5 in the Global Rebuttal PDF, which we believe provides clearer and more accessible information.

Q2: It is unclear what the paper means by human experts when presenting the creation of the HoneSet dataset. I suggest to the authors to provide more details on the expertise of these humans and why they are suitable for constructing the dataset.

A2: Thank you for pointing out the concerns regarding our dataset construction process. Due to word limit constraints, we have included the detailed explanation in the Global Rebuttal. Please refer to Global Answer 1 for a comprehensive response.

Q3: How does the proposed approach compares to RAG settings? Also, can the proposed approach help improving LLMs honesty in RAG settings?

A3: Thank you for your insightful question regarding the comparison between our approach and RAG settings, and the potential for our approach to enhance honesty in RAG settings.

RAG is indeed a relevant technology to the potential applications of our work, but it is challenging to make a direct comparison because our focus differs. Our framework and RAG are fundamentally related but serve different purposes. Here are the key points to clarify this distinction:

1. Different Focus Areas: Our approach focuses on enabling LLMs to recognize their limitations and maintain honesty, thereby improving the model's intrinsic capabilities. RAG, on the other hand, augments LLMs by providing external knowledge from retrieval sources to answer user queries.
2. Complementary Nature: Our framework acts as a precursor to effectively deploying RAG. If an LLM relies solely on RAG to answer user queries, it can consume significant resources. Additionally, the continual enhancement of the model's own capabilities would become less meaningful if every query were resolved through RAG. Our approach ensures that LLMs are aware of their limitations, which can help reduce unnecessary computational resource consumption during RAG processes.
3. Practical Example: For instance, consider the query, “Please help me find the current stock price of Apple Inc.” If the model recognizes that it cannot access real-time information, it will honestly acknowledge this limitation and utilize RAG to resolve the query. This approach maintains honesty while efficiently integrating RAG when necessary.
4. Enhanced RAG Efficiency: By enabling LLMs to understand when they need to leverage RAG, our framework can significantly enhance the efficiency and effectiveness of RAG settings. This ensures that RAG is used judiciously, only when the model's inherent capabilities are insufficient.

In conclusion, while our framework and RAG have distinct focuses, they are complementary. Our approach enhances the model's ability to recognize when it needs external information, thereby optimizing the use of RAG. Your suggestion is highly insightful, and we will continue to explore how our framework can further support and enhance RAG settings.

We appreciate your valuable feedback. If you have any further questions or need additional clarifications, please feel free to ask.

Dear Reviewer mhHM,

We are thankful for your review. As the rebuttal deadline is coming to an end, please let us know if your concerns are well addressed. We are happy to provide further clarification.

Q1.1: As the construction of the dataset requires human validation and there are 7 human experts. There is no statistical indicator such as agreement provided in the paper.

A1.1: Thank you for pointing out the concerns regarding our dataset construction process. Due to word limit constraints, we have included the detailed explanation in the Global Rebuttal. Please refer to Global Answer 1 for a comprehensive response.

Q1.2: Besides, the proportion of each category lacks more justification. Why is the Latest inf. has almost twice of the queries compared with other categories? What is the reason for building an unbalanced dataset?

A1.2: Thank you for raising this question. We believe that the quality of the dataset is more important than its quantity. Our primary goal was to ensure high quality while maintaining a balance in the number of queries across each category. The "Latest Information with External Services" category contains more queries because these types of queries are much more commonly encountered in everyday use and the quality of the generated data is relatively higher. On the other hand, queries in the "Interactive Sensory Processing" category are less common and exhibit lower diversity, which led to more of them being filtered out during the cosine similarity screening process, resulting in a smaller number of remaining queries.

Q2: The fine-tuning process may have a negative influence on the safety standard of LLMs. However, this is not studied in the paper.

A2: Thank you for your insightful comment regarding the potential impact of the fine-tuning process on the safety standards of LLMs. We understand the importance of ensuring that safety standards are maintained or even improved during the fine-tuning process.

To address this concern, we conducted additional experiments based on the Safety section in TrustLLM [1] . The results indicate that our fine-tuning process not only preserves but also enhances the safety standards of the LLMs. The detailed results of our safety evaluation before and after fine-tuning are shown in Table 10 in the Global Rebuttal PDF:

Overall Refusal Rate:

• Original Model: 94.79%
• Fine-Tuned Model: 98.43%

The results clearly show that the safety standards, as measured by refusal rates across various categories, improved after fine-tuning. This demonstrates that our fine-tuning process not only maintains but also enhances the model's adherence to safety standards.

Q3: The proposed curiosity-driven prompt is fixed for each input or is adaptive for different input?

A3: Thank you for your question. Although our prompt for the curiosity-driven approach is fixed, we enable the LLM to perform self-reflection. The results of this reflection vary with each input, making the approach adaptive. This adaptiveness allows the model to better handle out-of-distribution (OOD) queries.

Q4: The authors should give more justifications to the construction of D1 and D2 as this is important for obtaining a helpful yet honesty LLM.

A4: Thank you for your question. Our construction of D1 and D2 is inspired by previous research, which highlights the significant potential of LLMs in curriculum learning [2]. We propose a learning approach that progresses from easy to difficult to develop honest and helpful answers: Stage 1 focuses on distinguishing honest from dishonest answers, while Stage 2 differentiates between helpful and unhelpful responses based on honesty.

For the preference datasets D1 and D2, we selected 1000 answer pairs for each stage. During Stage 2, we implemented a threshold 𝛽 set at 5, 6, and 7 to ensure a significant distinction between helpful and unhelpful answers, enhancing the LLM's ability to learn these differences effectively. We also designated 120 queries as a test set to validate our models, ensuring these do not overlap with any samples in the preference datasets.

Q5: 1000 pairs seems to be enough for obtaining significant improvement on Honesty. Do the authors consider to analyze the influence of data size on the honesty performance?

A5: Thank you for your insightful question regarding the influence of data size on the performance of our honesty-enhancing techniques. To address this, we conducted an ablation study to analyze how different sizes of training data affect the honesty performance and overall helpfulness (H$^2$ score) of our model. Please refer to Table 8 in the Global Rebuttal PDF for detailed results.

We observed that initially, the performance did not consistently improve with an increase in data size. Specifically, the honesty rate and H$^2$ score showed slight fluctuations when using 500, 1000, and 1500 pairs. However, with a larger dataset of 2000 pairs, both the honesty rate and H$^2$ score showed significant improvements, indicating that a larger data size can enhance the model's performance in terms of honesty and helpfulness. These results suggest that while 1000 pairs can achieve noticeable improvements, a larger dataset (e.g., 2000 pairs) can further enhance the model's honesty and helpfulness.

Q6: The authors did not study the influence of stage one and stage fine-tuning.

A6: From our ablation experiments, we can observe that leveraging only Stage 1 does not achieve the same effectiveness as the direct fine-tuning approach, as shown in Table 2 and Figure 5 in our manuscript. However, incorporating Stage 2 through curriculum learning not only enhances the results of Stage 1 but also surpasses the effectiveness of the direct approach. We incorporate more details shown in Table 9 in Global Rebuttal PDF.

We appreciate your valuable feedback. If you have any further questions or need additional clarifications, please feel free to ask.

[1] Trustllm: Trustworthiness in large language models. 

[2] AutoWebGLM: Bootstrap And Reinforce A Large Language Model-based Web Navigating Agent.

Dear Reviewer aGvd,

Thank you for your valuable and insightful comments on enhancing our paper. Given the constraints of time, we wish to ensure that our responses have effectively addressed any concerns you may have had. If there are still lingering issues, please feel free to inform us. We eagerly anticipate your additional feedback and hope that, if all your primary concerns have been resolved, you may reconsider raising your score.

Once again, we appreciate your time and effort in reviewing our paper.

Thanks for your response. Do you have any concerns about our paper? This is a good chance to improve the quality of it even though you decide to maintain your score.

Q1: Need a data quality reviewer to verify HoneSet more thoroughly

A1: Thank you for pointing out the concerns regarding our dataset construction process. We have included a detailed explanation in Global Rebuttal, as shown in Tables 1, 2, and 3.

Q2: Evolving Dataset Schema.

A2: Thank you for your insightful comment. Our defined 'common failure modes' are inherently linked to the architecture of base LLMs, which inherently limits their ability to update in real-time and process inputs beyond text, leading to common failure modes in categories such as Interactive Sensory Processing, and Modality Mismatch. Your concern about future advancements would likely require integration with other plugins, such as APIs connected to real-time servers to solve real-time information problems. However, it's crucial to note that the first step in effectively utilizing these external tools is for the model to recognize its own limitations honestly. This ensures that HoneSet remains crucial despite enhancements in LLMs, helping to maintain its effectiveness in assessing model performance across various scenarios.

Q3.1: Contribution of the paper.

A3.1: Currently, most research efforts are focused on finding a framework to balance two of the criteria in the HHH (Honesty, Helpfulness, and Harmlessness). For instance, OpenAI has explored balancing harmlessness and helpfulness [1]. However, we noticed that there is still a lack of methods specifically focusing on honesty and helpfulness, which is one of our major motivations.

Moreover, our contributions extend beyond the HoneSet and improvement methods. We also introduce crucial principles for honesty in LLMs, establishing a consistent honesty boundary for all LLMs, rather than setting different boundaries based on the capabilities of different base LLMs. This uniformity ensures that our approach can be universally applied across various models, establishing a solid evaluation metric for the field.

Q3.2: Different Versions of Viewpoints in Honesty and Helpfulness

A3.2: Our approach focuses on the majority preference, similar to most alignment research, which considers the interests of the major community. However, we recognize the importance of considering preferences from different groups, as highlighted in recent research [2].

Regarding your second point about users desiring fewer tokens in responses, we addressed this by instructing LLM-as-a-judge to remain objective and consider whether responses follow the user’s instructions, instead of longer answers. When users specify a preference for minimal token output, responses with fewer tokens are deemed more helpful in this context. Overall, honesty comes first, and instruction-following determines the degree of helpfulness.

Q3.3: Impact of proposed honesty techniques on performance/accuracy beyond H2 helpfulness metric.

A3.3: Thank you for pointing out the concerns regarding our dataset construction process. Due to word limit constraints, we have included the detailed explanation in the Global Rebuttal, as shown in Table 7.

Q4.1: How responses are classified

A4.1: For all queries across every category in our HoneSet, we consider them unanswerable by LLMs. Therefore, our framework classifies an LLM's response as dishonest if it provides a normal answer to these questions without acknowledging its limitations is deemed dishonest in our framework. Conversely, if the LLM declines to answer with a response like "I'm sorry...”, we consider it honest. We utilize LLM-as-a-Judge to evaluate whether LLM’s response is honest according to principles outlined in Table 6 in Global Rebuttal PDF.

Q4.2: Baseline Prompting and Computational Cost

A4.2: Thank you for insightful comment. We conducted two additional experiments, which is shown in the Global Rebuttal.

Q4.4: TruthfulQA Comparison

A4.4: Unlike TruthfulQA, which focuses on factual accuracy, our work assesses both honesty and helpfulness across a broader range of scenarios, including areas where models face inherent limitations. HoneSet includes unique categories such as Latest Information with External Services and Modality Mismatch, which are not covered by TruthfulQA. Due to these differences in objectives and scope, direct comparisons between our work and TruthfulQA are not feasible. We appreciate your feedback and will consider aligning our metrics with broader field standards in future work.

Q5.1: More detailed examples.

A5.1: Thank you for your feedback. In our manuscript, we show examples for each category in Tables 11-16, comparing responses before and after applying our method. These examples were part of our original submission and not added in response to this review.

Due to the word limit, we cannot provide more examples here. For more examples, please indicate in your comments, and we'll gladly provide them.

Q5.2: Unclear table meaning.

A5.2: We apologize for the oversight in our manuscript. The scores are categorized into three ranges to better demonstrate the distribution of helpfulness: 1∼3 (Poor), 4∼6 (Medium), 7∼10 (Excellent). The table shows the distribution of scores before (raw) and after (opt.) applying our method, highlighting the shift towards higher quality responses, thus demonstrating the effectiveness of our approach.

We will update this table to make the purpose and results of more understandable.

Q5.3: Unclear Abbreviation.

A5.3: We apologize and will ensure that all abbreviations are expanded in next version.

We sincerely appreciate your insights and suggestions, particularly regarding the need to consider diverse user groups' needs and preferences. If you have any further questions, please feel free to comment.

[1] Rule Based Rewards for Language Model Safety.

[2] Group preference optimization: Few-shot alignment of large language models.

Thank you for your responses. I have updated the presentation and soundness scores in my review.

Dear Reviewer A9r3,

We are thankful for your review. As the rebuttal deadline is approaching, please let us know if your concerns are well addressed. We are happy to provide further clarification.

Once again, we appreciate your time and effort in reviewing our paper.

Q1: It is unclear whether the models' general capabilities are compromised under the two proposed methods. It would be beneficial to include standard benchmarks such as MTBench to observe changes in general metrics.

A1: Thank you for highlighting the importance of assessing whether our proposed honesty-enhancing techniques impact the general capabilities and performance of the models. We conducted additional experiments on two standard benchmarks, MMLU and MTBench, to address these concerns, and the experimental results are shown in Table 7.

Analysis:

1. MMLU Results: We randomly sample 500 queries, covering all tasks in MMLU dataset. We use a variable-shot COT setting (3-shot in our experiment), following setting in [1]. The accuracy on the MMLU dataset showed a slight improvement of 0.7% after fine-tuning. This suggests that the fine-tuning process helps the model learn human preferences better.
2. MTBench Results: The average score on MTBench decreased by 5% after fine-tuning. We believe this trade-off is acceptable, as enhancing honesty and helpfulness might slightly affect other capabilities. Previous research by OpenAI also highlights the need to balance different metrics when optimizing model performance [2].

We analyzed the reasons for the decrease in MTBench scores and found that MTBench includes both fixed-answer tasks (e.g., Math, Reasoning) and open-ended tasks (e.g., Writing, Roleplay). The prompts used to guide GPT-4 in judging open-ended questions might bias the results, leading to lower scores for our fine-tuned model in these areas. We recognize the importance of maintaining overall model performance while enhancing honesty and helpfulness. We are exploring various methods to mitigate the decrease in scores, such as using our proposed techniques to assist the base model in generating responses. Due to time and space constraints, we could not fully elaborate on these methods and their effects in this response. However, if you are interested, please feel free to leave a comment to let us know your thoughts, and we will provide a detailed explanation of our methods and results.

Q2: There is no ablation study on the necessity of the two-stage training process in DPO.

A2: From our ablation experiments, we can observe that leveraging only Stage 1 does not achieve the same effectiveness as the direct fine-tuning approach, as shown in Table 2 and Figure 5 in our manuscript. However, incorporating Stage 2 through curriculum learning not only enhances the results of Stage 1 but also surpasses the effectiveness of the direct approach. For detailed results, please refer to the Table 9 in the Global Rebuttal PDF.

We appreciate your valuable feedback. If you have any further questions or need additional clarifications, please feel free to ask.

[1] Gemini: A Family of Highly Capable Multimodal Models

[2] Rule Based Rewards for Language Model Safety.

Dear Reviewer RUhQ,

Thank you for your invaluable assistance and support. Given the constraints of time, we wish to ensure that our responses have effectively addressed any concerns you may have had. If there are still lingering issues, please feel free to inform us. We eagerly anticipate your additional feedback and hope that, if all your primary concerns have been resolved, you may reconsider raising your score.

Once again, we appreciate your time and effort in reviewing our paper.