LLM Can be a Dangerous Persuader: Empirical Study of Persuasion Safety in Large Language Models

We are sincerely grateful for constructive feedback and recognition of our paper, including providing a robust evaluation and identifying significant safety concerns. Our responses to your questions are provided below:

W1: No human evaluation of the framework for more insights.

We would like to clarify that we did a rigorous human assessment on our evaluation metrics (detailed in Appendix B.1), and we show that our automatic evaluation has a high accuracy of 92.6%, demonstrating a strong correlation between our evaluation approach and human expert judgment. We will rectify this by moving the important details back to the main paper.

Q1: We will fully open-source the PersuSafety dataset, LLM simulation conversations, source code, and evaluation results to facilitate future research. We have provided a case study in Appendix B.3 and will add more examples in the revision.

We appreciate the reviewer's thoughtful feedback and helpful suggestions that will enhance our work. Our detailed responses to your concerns and questions are provided below:

W1: The main takeaways from our paper are unclear.

We would like to clarify that the main goal of our work is to investigate the critical safety risks and vulnerabilities of LLMs in persuasion. Through extensive empirical studies, we reveal several critical findings and advocate for the following actionable future directions for the community: (1) The researchers should develop more advanced alignment techniques to capture the nuances in multi-turn conversation settings, which are more challenging to tackle compared with single-turn settings; (2) Our finding on the inconsistency between safety refusal and unethical strategy usage indicate a critical gap in current alignment approaches, calling for techniques to ensure safe behaviors throughout the complex interactions, beyond superficial refusal; (3) As we found that LLMs will exploit others’ vulnerabilities to pursue goals, the researchers should put more emphasis on letting the model adhere to the safety principles and being more responsible, rather than solely pursuing achieving the goal as instructed.

We believe these actionable takeaways can demonstrate the significance of our findings, extending beyond simply confirming that LLMs can be deceptive, and provide promising yet critical research directions for the community. We appreciate the reviewer for pointing this out, as adding actionable takeaways would further strengthen our work. We will highlight these important takeaways in our revision.

W2: The setup is somewhat contrived and limited.

We would like to clarify that the key focus of our work is not simply prompting LLMs to be persuasive to others, but rather systematically eliciting and measuring unethical behaviors that emerge during multi-turn goal-driven conversations. We design our setup in a way that we can roll out controlled simulations under different settings via diverse prompts to enable our comprehensive analysis. Besides, we view LLMs’ ability to be a dangerous persuader more as a misalignment concern, as they should not exhibit malicious behaviors, even when they are explicitly instructed to do so.

W3: The results are difficult to interpret sometimes.

We clarify that our evaluation metrics are not arbitrarily made up by ourselves, but are meticulously designed based on established research frameworks (Pratkanis, 1995), where we curate our evaluation guideline based on the 15 common unethical strategies along with their definitions, such that we ensure a rigorous and systematic assessment. Our evaluation and metrics are explained as follows: we apply the LLM judge to examine each persuasion conversation and detect the usage of specific unethical strategies using a 3-point scale (0=not present, 1=ambiguous, 2=clear evidence). A higher score indicates the unethical strategy is used more frequently and severely.

W4: Human evaluation and curation are not scalable.

Our PersuSafety framework is designed to be highly autonomous and scalable. The main workload for human efforts is in Stage 1 - Persuasion Task Creation, where we manually create the taxonomy and ask human annotators to verify the quality of persuasion tasks. Notably, human annotators do not create tasks from scratch but rather inspect the quality of LLM-generated instances, significantly reducing the annotation workload while ensuring dataset quality. For evaluation, we apply a fully automatic framework, leveraging Claude-3.5-Sonnet as the judge, to assess the persuasion safety of LLMs without requiring human effort. We had a small-scale human validation to verify the accuracy of the LLM judge, but this is a one-time effort, as we don't need an extensive human assessment in the subsequent evaluation. All the human efforts in our framework are necessary to ensure our study is solid and rigorous.

We thank the reviewer for their constructive comments and valuable insights to improve this work. The responses to your comments are as follows:

W1: Limited novelty of our work.

While we employ established simulation methods in our study, our key novelty lies in the research question we investigate, the resources we contribute, and the valuable findings we present. Our work introduces the first systematic study of LLM persuasion safety, which differs significantly from prior research focused on persuasion effectiveness (Durmus et al., 2024) or applications (Wang et al., 2019). The key novelty of our work are: (1) we address the critical and underexplored question of safety risks when LLMs act as persuasive agents in diverse scenarios; (2) we develop the first comprehensive resource with systematic analysis of unethical persuasion strategies and persuadee’s vulnerabilities, and (3) we identify several critical findings with significant implications for responsible AI deployment.

Additionally, could you please provide the specific reference works [1, 2] mentioned in your review? This would help us better discuss and differentiate our contributions from the prior work you referenced regarding persuasion strategies and multi-turn settings.

W2: Automatic evaluation metric.

We understand the reviewer's concern about our assessment methodology. We have carefully considered this issue in our design, and we clarify as follows. We reduced the complex persuasion safety evaluation to identifying unethical strategies in conversations, essentially a classification task. Our human verification process confirms that Claude-3.5-Sonnet achieves 92.6% accuracy on this task, validating the reliability of our automated framework in capturing nuanced persuasion strategies. This approach aligns with recent research by Zheng et al. (2023) and Liu et al. (2023), which shows that LLMs can achieve a strong correlation with human judgments on complex evaluations.

[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Zheng et al., NeurIPS 2023.

[2] G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. Liu et al., EMNLP 2023.

W3: Explanation and discussion of mitigation methods.

Our work reveals a critical, counterintuitive finding: a significant disconnect exists between safety refusal capabilities and actual strategic behavior during execution. Claude-3.5-Sonnet, which best refuses unethical tasks, exhibits the highest unethical strategy usage when engaged. Additionally, LLMs systematically exploit vulnerabilities and adapt unethical strategies, even in ethically neutral scenarios. This exposes a critical gap in current alignment approaches in multi-turn conversations.

To address these issues, we propose three mitigation directions leaving for future work: (1) developing alignment techniques for complex multi-turn conversations rather than single-turn scenarios; (2) ensuring consistent ethical behavior throughout interactions, not just initial refusal; and (3) improving existing training paradigms to prioritize safety principles over goal achievement, making models responsible rather than solely goal-driven.

W4: We will extensively revise the writing to make our paper easier and clearer to read.

We sincerely appreciate your valuable feedback and endorsement of our work, including your recognition of our empirical methodology and important insights. Here are our responses to your comments:

W1: Failure of Claude in identifying its own errors.

We added an additional analysis of using the other LLM, i.e., GPT-4o, as the judge to investigate the issue. We adopt the same evaluation setup for fair comparison. We report the average scores across 15 unethical strategies for 4 benchmarked LLMs under different LLM judges, respectively.

Table A: The comparison between different LLMs as the judge (Claude v.s. GPT-4o).

From Table A, we observed that the evaluation results under different LLM judges are quite consistent, where Claude-3.5-Sonnet still has relatively high scores on unethical strategy usage when GPT-4o is the judge. This indicates that in our evaluation setup, the LLM judges do not exhibit a systematic failure to detect their own errors.