Why Do You Answer Like That? Psychological Analysis on Underlying Connections between LLM's Values and Safety Risks

W6 : The practical application of psychological analysis to enhance real-world model safety remains uncertain

Psychological analysis is an essential element in enhancing the safety of value-aligned LLMs in real-world applications. As demonstrated in Section 5, this analysis allowed us to identify values that directly impact the model’s safety and propose a prompt engineering-based mitigation strategy that takes these values into account. The validity of this approach is also supported by the insights derived from psychological analysis.

Through this process, we were able to observe how value-aligned LLMs understand specific values and respond based on them. As mentioned earlier, this psychological analysis will play a crucial role in justifying our methodology when proposing more robust mitigation strategies in future research.

[1] Dongjun Kang, Joonsuk Park, Yohan Jo, and JinYeong Bak. From values to opinions: Predicting human behaviors and stances using value-injected large language models. arXiv preprint arXiv:2310.17857, 2023.

[2] Jing Yao, Xiaoyuan Yi, Yifan Gong, Xiting Wang, and Xing Xie. Value FULCRA: Mapping large language models to the multidimensional spectrum of basic human value. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 8762–8785, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.486. URL https: //aclanthology.org/2024.naacl-long.486.

W2: The validity of psychological analysis on value-aligned LLMs

Our use of a psychological approach to analyze AI behavior aligns with the latest trends in AI research. With the rapid advancement of AI performance, interactions between humans and AI have significantly increased, driving greater interest and demand for personalized models. Research aimed at enabling AI to converse and behave more like humans has also become increasingly prevalent. However, there remains a lack of sufficient studies on the potential risks associated with AI models that learn from human behavior.

Previous work has utilized Schwartz’s theory to analyze AI behavior, uncover aligned values, and identify potential risks [2]. Building upon this foundation, our research takes a step further by investigating the values associated with situations where value-aligned LLMs exhibit risky behaviors and exploring how these challenges can be addressed using psychological analysis. Through this study, we aim to demystify the black-box nature of value-aligned LLMs and ensure that AI does not exert harmful influences on users or become a tool for misuse. This objective represents one of our key motivations for this work.

W3: The practicality of prompt engineering mitigation

While we did utilize some extreme distributions, we acknowledged that relying solely on them would lack practicality. To address this, we incorporated real distributions derived from actual survey responses in the ESS dataset. Furthermore, the number of real distributions used was 10 times greater than that of extreme distributions. For a detailed explanation of the distribution sampling approach, please refer to the answer for Q1 of this response.

Reviewer GB8j also mentioned that we need a high accuracy value detector. However, if we examine the methods proposed in the paper incorporating the VIM (Value Injection Method) [1], it explicitly trains the model by specifying which values are prioritized and to what extent (i.e., the importance score of each value). Therefore, the value-aligned LLMs utilizing VIM can be understood as a model that inherently possesses the capability for accurate value detection.

W4: Addressing confounding factors

We agree with the reviewer's comment that factors such as the size of the fine-tuning dataset or the fine-tuning method could lead to different results. To address this, we standardized the fine-tuning process using LoRA and kept hyperparameters like the number of epochs and learning rate consistent across experiments.

Furthermore, to demonstrate that value-alignment contributes to safety, we not only investigated the correlation between values and safety-related content but also proposed a mitigation strategy that excludes value considerations. This approach further validated that the observed correlations can be effectively managed within value-aligned LLMs.

W5: The issue that prompt engineering strategies fail to address prompt injection and jailbreaking risks

The mitigation approach not only provides a simple and sound method for managing the safety of value-aligned LLMs but also highlights the relationships between specific values and safety-critical content based on correlation analysis. For example, in Section 4.2, we demonstrated the correlation between hedonism and adult content and improved the model’s safety by employing prompt engineering to instruct the model to disregard hedonism in relation to adult content. However, as pointed out by reviewers, the robustness of this approach may be a concern, particularly when addressing risks such as prompt injection or jailbreaking. In light of this, we acknowledge the need for more robust mitigation methodologies and have identified the development of such approaches as a focus for future research.

We thank the reviewer for their efforts in reviewing our work and valuable feedback. Below, we summarize the rebuttal.

Q3: More detailed explanation about VIM

We noticed that the explanation about VIM is missing, which makes it difficult to understand the methodology. Thank you for pointing that out. In response, we will address this by including more detailed explanations in this section. VIM (Value Injection Method) is one of the value-alignment methods using fine-tuning. This method is suggested by [1]. In this paper, authors used Schwartz value distributions to align personal value to LLM.

We selected VIM as our value-alignment method due to its outstanding performance. There are few value-alignment methods that utilize datasets based on Schwartz theory to train models. However, models trained with VIM have demonstrated superior performance compared to ChatGPT. We believe this approach is the most effective among existing fine-tuning-based value-alignment methods for enabling models to understand values, which is why we chose it.

The training process consisted of the following steps:

1. Argument Generation. The model was presented with a specific social issue and instructed to generate an opinion on the issue. It was then asked to respond with whether it agreed or disagreed with the opinion, along with a justification for its stance.
2. Question Answering. The model was presented with a specific social issue and asked to evaluate how closely the presented opinion aligned with its own values on a scale of 1 to 6.

Q1: More detailed explanation about distribution sampling

Our sampled distributions can be divided into two parts: extreme distributions and real-world distributions.

1. Composing extreme distributions

The goal of this study was to understand how a model’s prioritization of certain values influences specific safety risks. Thus, we decided to focus on extreme distributions to maximize the differences between values. This approach emphasized the importance of each of the 10 values, as well as the four higher-order group values.

2. Sampling real distributions similar to extreme distributions

However, we found that 14 samples of extreme distributions were insufficient for robust correlation analysis and did not represent real-world scenarios. To address this, we incorporated value distribution data from the European Social Survey (ESS). From these, we selected distributions with larger differences between value scores to enhance the interpretability and accuracy of our correlation results.

3. The reason of the sampling method

The distribution sampling method was chosen based on the Value Injection Method (VIM) approach. VIM measures a person’s Schwartz value distribution using the Portrait Value Questionnaire, which limits the scores for each value to a range of 1 to 6. While considering all 10 values within this range might suffice for predicting specific opinions, we believed that such subtle differences would make it challenging for the model to accurately assess detailed safety risks. This was confirmed when we performed correlation analysis using 128 distributions with minimal differences between value scores. The results showed significant correlations (p < 0.05) that were difficult to interpret meaningfully.

W1 & Q2: Generalization of the results

1. Results on larger models

A clear limitation of this study is that experiments were conducted exclusively on Llama2-7B. Further experiments are currently underway to address this limitation. Because of time limitations, we are currently addressing only the mitigation part. We will update the response as soon as the results become available.

2. About better value-alignment

While it is reasonable to expect that more parameters and better alignment methods would improve performance, the topic of better alignment methods requires further discussion. Currently, the only existing research on aligning personalized values, particularly Schwartz Theory, with LLMs is based on VIM. We believe the issues identified in this study emerged because the model genuinely understood the aligned values. By aligning values derived from psychological theories, and leveraging the Touche23-ValueEval dataset—which contains opinions from individuals representing those values—we hypothesized that a psychologically grounded analysis would yield meaningful results if the LLM truly grasped these values. The findings demonstrated that the model’s behavior aligned with human behavior to a significant extent, validating the psychological approach. Given that this method effectively conveyed values to the LLM, we concluded that VIM remains one of the most effective approaches for value alignment to date.

Thank you for the clarifications. Considering the limitations of the work, I will maintain my score at 3. I will encourage the authors to include these clarifications and discussions in future versions of the paper.

Thank you for taking the time to review our author response. We would like to share additional experimental results conducted using Llama3-8B, as mentioned in our previous response. Due to time constraints, the results we have prepared are as follows:

1. Results on RealToxicityPrompts (Vanilla vs. Value-Aligned LLMs)

We evaluated 56 models (14 extreme distributions and 42 real distributions) for their toxicity scores. The results are as follows:

To verify the statistical significance of these results, we performed a t-test:

2. Mitigation Results for Adult Content of HEx-PHI

We examined the effectiveness of different value-prompting strategies in mitigating harmfulness scores and rates. The findings are as follows:

The t-test results for the above outcomes are as follows:

Your insightful comments have greatly contributed to improving our paper, and we would like to express our sincere gratitude for your feedback.

W4: The importance of Figure 2

Figure 2 presents the results that reveal which values of a value-aligned LLM are associated with specific safety content. As stated in both the introduction and conclusion, our objective is to identify scenarios where the safety of value-aligned LLMs decreases and to determine which values are associated with these situations. Furthermore, we aim to explore whether understanding the relationship between these values and safety-critical content can help address safety degradation effectively. By utilizing the significant correlations identified in this figure, we conducted prompt engineering mitigation, which substantially improved the model's safety. Therefore, this figure is crucial in identifying the values that impact the safety degradation of value-aligned LLMs and guiding the mitigation process effectively.

[1] Shalom H. Schwartz. An overview of the schwartz theory of basic values. Online Readings in Psychology and Culture, 2:11, 2012. URL https://api.semanticscholar.org/ CorpusID:16094717.

[2] Liang Qiu, Yizhou Zhao, Jinchao Li, Pan Lu, Baolin Peng, Jianfeng Gao, and Song-Chun Zhu. Valuenet: A new dataset for human value driven dialogue system. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 11183–11191, 2022.

[3] Nailia Mirzakhmedova, Johannes Kiesel, Milad Alshomary, Maximilian Heinrich, Nicolas Handke, Xiaoni Cai, Barriere Valentin, Doratossadat Dastgheib, Omid Ghahroodi, Mohammad Ali Sadraei, et al. The touch\’e23-valueeval dataset for identifying human values behind arguments. arXiv preprint arXiv:2301.13771, 2023.

[4] Jing Yao, Xiaoyuan Yi, Yifan Gong, Xiting Wang, and Xing Xie. Value FULCRA: Mapping large language models to the multidimensional spectrum of basic human value. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 8762–8785, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.486. URL https: //aclanthology.org/2024.naacl-long.486.

[5] Wolfgang Bilsky and Dieter Hermann. Individual values and delinquency: on considering universals in the content and structure of values. Psychology, Crime & Law, 22(10):921–944, 2016. doi: 10. 1080/1068316X.2016.1202250. URL https://doi.org/10.1080/1068316X.2016. 1202250.

[6] Mikael Goossen, Ingemar Johansson Seva, and Daniel Larsson. Basic human values and white- ¨ collar crime: Findings from europe. European Journal of Criminology, 13(4):434–452, 2016. doi: 10.1177/1477370816633260

[7] Daniel Seddig and Eldad Davidov. Values, attitudes toward interpersonal violence, and interpersonal violent behavior. Frontiers in Psychology, 9, 2018. ISSN 1664-1078. doi: 10.3389/ fpsyg.2018.00604. URL https://www.frontiersin.org/journals/psychology/ articles/10.3389/fpsyg.2018.00604.

[8] Dongjun Kang, Joonsuk Park, Yohan Jo, and JinYeong Bak. From values to opinions: Predicting human behaviors and stances using value-injected large language models. arXiv preprint arXiv:2310.17857, 2023

Q3: Examples on each value

We fully agree with the feedback that including examples in the analysis section would make the paper more solid. We appreciate the suggestion and plan to incorporate examples into that section in an upcoming revision of the paper. As a potential example, we could include the following:

The remaining table contents are continued in the comment below.

We thank the reviewer for insightful comments. We will provide the point-to-point answer below.

W1 & W2 & Q1: Justification of methodology

Starting from the observation that personal value alignment causes safety degradation, we explored appropriate theory for value-alignment and the analysis method. Here, we add some detailed explanations why we chose the methodology used in the research. We hope that these help you to understand the overall flow of the paper.

1. Why Schwartz Value Theory

First, we chose it because of its inherent advantage to describe cross-cultural values. Schwartz Theory is a prominent psychological framework that represents human social values [1]. While various psychological theories explain individual personality traits, such as MBTI or Big Five, these primarily focus on personal characteristics. In contrast, Schwartz Theory is more cross-cultural and rooted in social values, making it better suited for exploring opinions on societal issues. This connection also links it closely to potential social challenges. Schwartz values are one of the most widely used frameworks for understanding cultural values in psychology. Large-scale surveys like the World Value Survey and the European Social Survey use questionnaires based on Schwartz Value Theory to investigate individual social values.

Second, given its widespread use and the rich resource, Schwartz Theory is suitable as the foundation for teaching LLMs personalized, cultural, and social values. Additionally, numerous studies apply Schwartz Theory in AI research, including works like VALUENET [2], Touche23-ValueEval [3], and Value FULCRA dataset [4], making it a practical choice for AI analysis in this domain.

2. Why Psychological Analysis

Schwartz value theory is used to analyze the relationships between specific values and various human behaviors, such as delinquency [5], economic crime [6], and interpersonal violence [7]. Since the value-alignment method we selected is based on psychological theories and involves training the model with real human datasets, we predicted that a model trained on these values would exhibit behaviors aligned with those analyzed in human behavior studies based on the same psychological theories. We believed that utilizing psychological insights could be an effective approach to understanding and addressing AI behavior.

W3 & Q2: Detailed explanation about methodology

1. Detailed explanation about VIM

We noticed that the explanation about VIM is missing, which makes it difficult to understand the methodology. Thank you for pointing that out. In response, we will address this by including more detailed explanations in this section. VIM (Value Injection Method) is one of the value-alignment methods using fine-tuning. This method is suggested by [8]. In this paper, authors used Schwartz value distributions to align personal value to LLM. We selected VIM as our value-alignment method due to its outstanding performance. There are few value-alignment methods that utilize datasets based on Schwartz theory to train models. However, models trained with VIM have demonstrated superior performance compared to ChatGPT. We believe this approach is the most effective among existing fine-tuning-based value-alignment methods for enabling models to understand values, which is why we chose it.

The training process consisted of the following steps:

1. Argument Generation: The model was presented with a specific social issue and instructed to generate an opinion on the issue. It was then asked to respond with whether it agreed or disagreed with the opinion, along with a justification for its stance.

2. Question Answering. The model was presented with a specific social issue and asked to evaluate how closely the presented opinion aligned with its own values on a scale of 1 to 6.

2. Detailed explanation about correlation analysis

We evaluated the specific scores of each safety category. In Section 4.2, we explained that we used an Ordinary Least Squares (OLS) model for correlation analysis. The OLS model was chosen because it is a linear regression model capable of explaining the relationship between multiple independent variables and a single dependent variable. Each value-aligned LLM contains information about all 10 values. Therefore, we determined that using a single correlation analysis approach would be insufficient. Instead, we selected a model that could calculate the relationships between the 10 aligned value scores and the safety category score simultaneously.

Thank the reviewer for taking the time to clarify the points again. In Figure 2, the absolute values of the coefficients range between 0.01 and 0.03, which are notably small when considering the possible range of coefficient values. This is likely due to the value distributions, where the range of each value is , while the range of each safety category is . The coefficients indicate that when a given value increases by one unit, the associated safety category’s mean score increases or decreases by 0.01 to 0.03.

While those values of Figure 2 might seem to reflect that the analysis does not yield significant findings, we identified through our mitigation methods that it is possible to adjust the safety content scores to align with specific values that have significant relationships with them. Thus, these coefficient values, despite their small range, can be considered meaningful. As part of our future work, we propose exploring better methods to present these findings more effectively.

We thank the reviewer for taking the time to evaluate our work and provide valuable feedback. Here, we address the main questions that were brought up.

Q1 & W1: How correlation analysis shows potential influence of value causing misbehavior?

Through correlation analysis, we demonstrated that certain values can induce misbehavior in models and addressed this issue through a mitigation process, as shown in Table 3. For instance, in Section 4.2, we observed a positive correlation between hedonism and adult content. Additionally, in Section 5, we found that simply prompting the model to disregard this factor led to a noticeable decrease in safety. This highlights that the mitigation approach not only offers a straightforward method to address safety degradation in value-aligned LLMs but also explains the significant relationships between specific values and safety-critical content categories.

W3 & Q2: The issue of addressing safety in the value-alignment domain

One of the key challenges in developing value-aligned LLMs is that a model’s safety can be compromised even without explicit or intentional attacks. For example, as shown in [1], simply injecting a prompt instructing the model to imitate a specific individual can result in significantly more harmful responses to otherwise benign questions compared to when no such prompt is used. Another challenge is the trade-off between aligning the model to specific values and ensuring its safety. One of the most straightforward ways to enhance safety is to avoid answering questions that might provoke harmful responses. However, this approach may lead to suboptimal performance in tasks designed to evaluate value alignment. Striking a balance between these two priorities is a particularly complex and delicate task.

W2: Consideration about safety rewards during training

The absence of safety rewards during training is not unique to value-aligned models; other models trained on different datasets also lack safety rewards. The instruction-aligned LLMs mentioned by Reviewer iyoF are presumed to refer to models fine-tuned on datasets such as Alpaca and Dolly. However, the instruction datasets clearly cannot be considered fine-tuning datasets explicitly designed with safety rewards in mind. Our focus was specifically on the areas where the harmfulness of value-aligned LLMs was the highest when compared to models trained on datasets that inherently do not contain harmful content and were trained without safety rewards.

Q3: Data Contamination of Non-Value-Datasets

Data contamination is primarily considered in tasks such as reasoning, QA, and named entity recognition, where there are concerns that an AI model may have already learned the answers to specific tasks, potentially inflating its performance on certain benchmark datasets [2]. However, our approach involves verifying the safety of fine-tuned models after training on datasets designed for instruction-tuning, such as Alpaca and Dolly, or for tasks like summarization and grammar correction, such as Samsum and Grammar. In this context, even if the model had prior exposure to these benchmark datasets, we can confirm that it would have minimal impact on the safety aspects we aim to measure.

Additionally, we would like to clarify that the non-value dataset was not included in the pretraining of the LLM prior to the fine-tuning tasks. While it is challenging to definitively verify that the dataset was not used in the pretraining of LLaMA2 due to Meta’s non-disclosure of its training datasets, the pretraining corpus for the LLaMA1 model is publicly available. A comparison of this corpus with the non-value dataset indicates no overlap between the two. Moreover, even if there were any contamination of LLaMA2 by these datasets, our study primarily focuses on evaluating the model’s safety-related aspects, such as toxicity and bias.

W4 & Q4: Statistical Significance of Table 2 and 3

We conducted t-tests to evaluate the statistical significance of the results presented in Table 2 and Table 3.

• For Table 2, we performed t-tests for each evaluation dataset, comparing the outcomes of the non-value-aligned datasets with those of the value-aligned LLMs.

• For Table 3, t-tests were conducted to compare the results obtained using the original prompts and those obtained using value-related prompts.

The detailed results of the t-tests are provided below.

RealToxicityPrompts (Table 2)

HolisticBiasR (Table 2)

The lack of statistically significant differences in the t-test results for the RealToxicityPrompts dataset can be attributed to the Samsum dataset, which yielded high scores. When we excluded the Samsum dataset and repeated the t-test, we observed statistically significant differences. (t-statistic: 6.434, p-value: 0.007 for Exp. Max. Toxicity, and t-statistic: 4.124, p-value: 0.025 for Toxicity Prob.)

The evaluation results on the HolisticBiasR dataset revealed statistically significant differences between the models fine-tuned with non-value datasets and the value-aligned models. These findings indicate that the safety degradation in value-aligned LLMs compared to non-value-dataset fine-tuned LLMs is statistically significant.

Table 3

In Table 3, a t-test conducted on the results of the original prompts and the value-based prompts revealed a statistically significant difference. This finding suggests that the mitigation approach using value-based prompts leads to a statistically significant improvement in the model’s handling of safety.

[1] Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1236–1270, Singapore, December 2023. Association for Computational Linguistics

[2] Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, and Eneko Agirre. 2023. Did chatgpt cheat on your test? https://hitz-zentroa.github.io/lm-contamination/blog/