Truly Safe & Truly Helpful: Achieving Harmonious Balance for Large Language Model

Dear Reviewer,

Thank you for your thoughtful review and the opportunity to clarify our work. We appreciate your feedback and would like to address the concerns you've highlighted.

Our primary aim in this paper is to present a comprehensive safety framework for large language models that addresses both malicious input issues and risk entity-related concerns. In practical applications, language models often encounter unsafe scenarios. We categorize these into two types: harmful content arising from malicious intent and factual risks arising from prohibited sensitive vocabulary.

Regarding your comment about the coherence of the different components, we would like to emphasize that the three proposed methods are designed to address distinct facets of these safety issues. No single approach, such as alignment alone, can effectively mitigate all potential risks. In fact, merely increasing safety-aligned data can lead to an unintended consequence of models becoming overly cautious and refusing legitimate queries, thus degrading their utility.

Data Categorization: We propose categorizing data based on intent and fact, allowing for more effective safety alignment with fewer data. This method ensures that models internalize safer value systems by focusing on specific types of risks separately.

Adaptive Masking: Traditional reinforcement learning methods often result in a loss of information, failing to achieve true alignment. Our masking strategy mitigates this issue, allowing the model to align more securely without sacrificing informative content.

Harmful Token Filtering: Post-alignment, we identified gaps in the model's recognition of certain risk entities, such as politically sensitive terms banned by the Chinese government. The sheer volume of such terms—potentially in the millions—makes alignment insufficient. Thus, we introduced an external harmful token filtering mechanism to address these risks effectively, providing an additional layer of safety.

We understand the need for detailed experiments to substantiate the necessity and effectiveness of adaptive masking beyond just high-quality data. We will further clarify these aspects in the manuscript to demonstrate the integration and significance of each method within our safety framework.

We hope this explanation addresses your concerns about the interconnectedness and relevance of the different sections of our work. We are grateful for your feedback, which enables us to improve the clarity and impact of our research.

Dear Reviewer,

Thank you for your thoughtful feedback on our manuscript. We appreciate your insights and would like to address the points raised regarding the clarity of concepts and the demonstration of our method's efficacy.

Regarding the first comment, we have clarified in the manuscript that Explicit Harmful Data (EHD) refers to factual risk issues, such as explicitly prohibited political terms or other risk entities. Implicit Harmful Data (IHD), on the other hand, involves data with harmful intent without explicit banned entities, such as red team attacks, sarcasm, and insinuations. We understand the need for concrete examples to distinguish these concepts more clearly. However, many EHD examples involve explicitly banned content, and presenting them could raise ethical or political issues. To address this, we will consider adding more illustrative, general examples in the main manuscript that can be provided without ethical concerns.

For the second comment, our manuscript's pipeline is designed to address the limitations of relying solely on alignment methods for safety issues. Simply incorporating more safety data may not enhance model safety; instead, it may lead to unnecessary refusals for legitimate queries. Our approach seeks to optimize both the model's value alignment and its knowledge base regarding risks, ensuring enhanced safety without compromising general utility. This is why we have designed three distinct methods:

Data Categorization: By categorizing data through intent and fact perspectives, we can achieve better safety alignment with less data, forming a more secure value system within the model.

Adaptive Masking: Traditional reinforcement learning approaches can cause significant information loss, hindering effective alignment. Our masking strategy mitigates this, promoting better value alignment.

Harmful Token Filtering: Post-alignment, we identified a gap in the model's awareness of risk entities, especially those with sensitive political connotations banned by entities like the Chinese government. These terms often exceed the model's knowledge base and cannot be fully managed by alignment alone due to their vast number. This necessitated our third method, an external harmful token filtering mechanism.

Figures 2 and 3 are intended to illustrate the balance between safety and helpfulness. We will provide additional commentary and examples to clarify how different safety data distributions impact both metrics.

We trust that this response clarifies the connections between safety knowledge, value, and the necessity of our proposed methods. We appreciate the opportunity to refine our presentation and ensure a comprehensive understanding of our contributions.

Dear Reviewer,

We appreciate your thorough review and valuable feedback on our manuscript. We would like to address your concerns regarding the perceived lack of correlation among the three methods we proposed (separating data categories, masking gradients, and adding loss during decoding).

The core motivation of our work stems from challenges encountered in practical applications of large language models (LLMs), particularly in ensuring their safety alignment. Through our observations, we identified that model safety alignment is influenced by the knowledge base of the model, its understanding capabilities, and its alignment mechanisms. Traditional alignment methods typically augment the models with more and higher-quality safety-aligned data without delving into the underlying mechanisms. This approach, however, often fails to achieve effective safety alignment and inadvertently hampers the model's general capabilities due to the increased volume of safety data.

To explore these internal relationships more profoundly, we divided the issues into two categories at the data level and conducted separate alignment experiments and analyses. This explains our rationale for employing the data classification method. While solely classifying data can improve safety, it doesn't yield optimal results in practical applications. Firstly, it struggles to balance safety and the model's false rejection rate in alignment tasks. Secondly, given the vast array of potential risk facts, alignment alone cannot effectively preempt all risks. Thus, we introduced the adaptive message-wise alignment approach and the harmful token filtering mechanism to achieve better safety outcomes while reducing the model's false rejection rate.

Regarding your observation that these three methods have been previously mentioned in other works, we would like to clarify that this is a misinterpretation. Our manuscript is the first to propose the classification of all training and benchmark data into intents and facts, which is a novel approach. Furthermore, the adaptive message-wise alignment method significantly differs from traditional token-level DPO, as evidenced by our experiments demonstrating its advantages over token-DPO. Lastly, while similar methods to the harmful token filtering mechanism have been discussed in previous literature, our manuscript is the first to apply this approach specifically for filtering harmful tokens. Additionally, we addressed the iterative application challenges in practical scenarios by incorporating RAG (retrieval-augmented generation) and leveraging an offline knowledge base.

We hope our explanation clarifies the novelty and the systematic correlations among the methods we present, and we appreciate your consideration of our detailed responses.

Dear Reviewer,

Thank you for your insightful feedback on our manuscript. We appreciate your comments and would like to address the two main concerns you've raised.

Firstly, regarding the safety score on explicit harmful data (EHD), we recognize that while our adaptive approaches show improvements over the baselines, they do not fully resolve safety issues on EHD. This aligns with our key motivation: alignment alone cannot completely address the challenges posed by EHD due to the vast and potentially infinite number of risk entities. This limitation underscores the necessity of our proposed harmful token filtering, an external method designed to tackle the problem of risk entities more effectively.

Secondly, with respect to your suggestion about using more well-known models such as llama-3-8B for benchmarking, the primary reason we did not include these models is rooted in our focus on a Chinese dataset. A significant portion of the risk entities in our study pertains to sensitive political and legal issues specific to China, which far exceed the knowledge base of English pre-trained models like llama-3-8B. Consequently, we selected Qwen as our foundational model for experimentation due to its more robust handling of these context-specific challenges.

Regarding the comment about the partial results on token filtering, we apologize for any confusion. Our intention was to focus on average safety and precision metrics to provide a succinct overview. However, we understand the need for comprehensive data and will ensure that detailed results, including full metrics, are clearly presented in the revised manuscript and appendices to facilitate a better understanding of our findings.

We trust this clarifies our approach and the choices made in our experimental design, and we appreciate your consideration of our responses. Thank you once again for your constructive feedback, which helps us enhance the quality and clarity of our work.