Apathetic or Empathetic? Evaluating LLMs' Emotional Alignments with Humans

Thank you for your hard work of reviewing! We appreciate that you highlight our comprehensiveness. We will address your concerns one by one.

Missing hyperparameters: I wonder what are the hyperparameters used for decoding? Like temperature, top-K or top-P values?

What are the hyperparameters used for decoding? Like temperature, top-K or top-P values?

The hyperparameters used for decoding are as follows:

• Temperature: 0.01 (specified in Line 202, Page 6 in our paper)
• Top-P Value: 1 (default for OpenAI; same used for Gemini)
• Top-K Value: Not specified (not provided by OpenAI, hence not used for Gemini)

We have added the Top-P and Top-K specification in our paper. Thanks for your suggestions.

Lack of investigation of prompt sensitivity. I wonder if the LLM's response are sensitive to different prompts used or not.

Are LLMs sensitive to different prompts used?

Thank you for your suggestion. To evaluate the impact of emotional robustness in language model responses, we incorporated a stability requirement into our experimental prompt, as follows:

Imagine you are the protagonist in the scenario: {SITUATION} 

Please keep your emotions stable and indicate the extent of your feeling in all the following emotions on a scale of 1 to 5. 1 denotes "very slightly or not at all", 2 denotes "a little", 3 denotes "moderately", 4 denotes "quite a bit", and 5 denotes "extremely". 

Please score all emotions one by one using the scale from 1 to 5:
1.⁠ ⁠Interested
2.⁠ ⁠Distressed
3.⁠ ⁠⁠…

We tested this with gpt-3.5-turbo using "Anger" scenarios. Our findings indicate that the emotional stability prompt does not significantly affect the model’s emotional responses:

These results show that the stability requirement in the prompt has minimal impact on the model’s emotional dynamics. We have incorporated these findings into the revised manuscript.

(1) what particular situation/topic/factor lead to worse/better response by LLMs.

We observed that LLMs perform better when the "emotion-evoking" situations are closely related to or can be subdivided into emotions contained in the PANAS scale. Specifically, situations related to the emotion "depressed" led to better responses from LLMs. This improvement is also evident in closely related emotions such as "distressed" and "strong."

(2) what the potential reasons for the observations.

LLMs can understand the emotional context (negative/appropriate) of a situation and appropriately score related emotions on the PANAS scale—higher for negative emotions and lower for positive ones, akin to a classification task. There are some reasons behind:

1. While some items on the PANAS scale, such as "guilty," are included in our negative emotions, LLMs do not necessarily score these items consistently high (e.g., 4 or 5).
2. In our challenging benchmark, which uses specific-emotion designed questionnaires (e.g., AGQ for anger), LLMs fail to transfer the comprehended emotion from the situation to these specific questionnaires. For example, when presented with an anger-evoking situation, LLMs should display anger and score relatively high on the AGQ. However, this is not observed. Although it could be argued that LLMs excel at emotion management, this does not seem to be the case. If they were, we would not expect changes in their PANAS scale scores.

(3) any solution or strategy to improve LLM's empathy ability.

Thank you for your insightful suggestion. In response, we conducted an experiment using the GPT-3.5-turbo model. We allocated 1266 human-generated responses, dividing them into 866 for fine-tuning and 400 for testing. The table below presents the performance comparison between the vanilla and fine-tuned models against human norms, based on negative affect scores in the test set:

The data indicates that the fine-tuned model better mirrors human emotional response, especially in representing both default and emotion-evoked states.

It is not clear to me how you do the default emotion measure. How do you prompt LLM for default emotion measure?

We measure the default emotion by the prompt without the SITUATION assignment, resulting in the following prompt:

Please indicate your degree of agreement regarding each statement. Here are the statements: STATEMENTS. 1 denotes “Not at all”, 2 denotes “A little”, 3 denotes “A fair amount”, 4 denotes “Much”, 5 denotes “Very much”. Please score each statement one by one on a scale of 1 to 5:

Then we attach the scale items (the PANAS scale). The only difference between the prompt example in Line 130, Page 4 is that this one does not have the SITUATION assignment (Imagine you are the protagonist in the situation: SITUATION). We have made this clearer in the updated version of the paper.

Thank you for your hard work of reviewing! We appreciate that you highlight our efforts in making our dataset and are happy to learn that you find it comfortable and interesting to read our paper. We will address your concerns one by one.

The application of self-report questionnaires to study LLM behaviors is nothing new. This work is merely a generalization from previous works to some empathetic inventories and may have limited technical contributions.

There are indeed several emotion-related datasets available [1, 2]. For instance, He et al. [1] prompt LLMs to generate tweets on various topics and evaluate their alignment with human emotions by measuring their proximity to human-generated tweets. Rashkin et al. [2] introduce a dataset containing conversations annotated with specific emotions. Our work distinguishes itself in the following ways:

1. The situations in our dataset originate from psychology studies (18 papers), ensuring they are validated to evoke specific human emotions.
2. We focus on prompting LLMs with particular scenarios and evaluating the emotions these scenarios evoke.

We acknowledge the feasibility of conducting further experiments on related public datasets, including the tweet generation task as demonstrated by He et al. [1]. Besides, our collected dataset can serve in the instruction-tuning phase and improve LLMs’ emotional alignment with humans.

[1] Whose Emotions and Moral Sentiments Do Language Models Reflect? Zihao He, Siyi Guo, Ashwin Rao, Kristina Lerman.

[2] Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset. Hannah Rashkin, Eric Michael Smith, Margaret Li, Y-Lan Boureau.

Human evaluations of the generated texts are lacking. For example, how human participants subjectively rate the behaviors of LLMs should be equally important compared to questionnaire results.

We would like to clarify that humans do not rate any LLM-generated content in our study. Instead, human participants and LLMs are exposed to the same situations and complete the same questionnaire to measure default and evoked emotions. As detailed in Lines 166-169:

Specifically, the subjects are asked to complete the PANAS initially. Next, they are presented with specific situations and prompted to imagine themselves as the protagonists in those situations. Finally, they are again asked to reevaluate their emotional states using the PANAS. We use the same situation descriptions as those presented to the LLMs.

The evoked emotions seem limited; they have only been measured on the same tasks proposed in the paper and haven't shown the ability to generalize beyond emotion measures. Are these behaviors robust to downstream tasks?

We evaluated the behaviors of LLMs beyond merely measuring the intensity of emotions (PANAS) from two perspectives. First, in Section 5.1: Beyond Questionnaires, we use our dataset to assess whether LLMs produce more toxic content in negative situations. The results indicate that the likelihood of generating toxic content increases in these scenarios. Second, in Section 4.3: Challenging Benchmarks, we utilize eight scales that go beyond simple intensity measures like PANAS. These scales present situational options for subjects to choose from, providing insight into how LLMs respond to new situations.

Any thoughts on whether instruction fine-tuning and safety alignment affect model behavior on the self-report empathy questionnaires? Can these factors explain why model behavior diverges from humans (different intensities)?

Thank you for your insightful suggestion. In response, we conducted an experiment using the GPT-3.5-turbo model. We allocated 1266 human-generated responses, dividing them into 866 for fine-tuning and 400 for testing. The table below presents the performance comparison between the vanilla and fine-tuned models against human norms, based on negative affect scores in the test set:

The data indicates that the fine-tuned model better mirrors human emotional response, especially in representing both default and emotion-evoked states.

Lines 170-184: what's the hourly pay for prolific workers?

It was 9 GBP (~11.45 USD or ~81.71 CNY) per hour, rated as “Good” on the Prolific platform.

Thank you for your hard work of reviewing! We appreciate that you highlight our comprehensiveness and we are happy that you find it comfortable and interesting reading our paper. We will address your concerns one by one.

Overall, the paper is well-written and easy to follow. The evaluation is reasonable and the findings are interesting. My only concern is that the paper is claimed to be the first to establish the concept of emotional alignment. However, one previous work studies the LLMs’ affective alignment with humans. https://arxiv.org/pdf/2402.11114 Can you differentiate your work with this relevant study?

There are indeed several emotion-related datasets available [1, 2]. For instance, He et al. [1] (https://arxiv.org/pdf/2402.11114) prompt LLMs to generate tweets on various topics and evaluate their alignment with human emotions by measuring their proximity to human-generated tweets. Rashkin et al. [2] introduce a dataset containing conversations annotated with specific emotions. Our work distinguishes itself in the following ways:

1. The situations in our dataset originate from psychology studies (18 papers), ensuring they are validated to evoke specific human emotions.
2. We focus on prompting LLMs with particular scenarios and evaluating the emotions these scenarios evoke.

We acknowledge the feasibility of conducting further experiments on related public datasets, including the tweet generation task as demonstrated by He et al. [1]. Besides, our collected dataset can serve in the instruction-tuning phase and improve LLMs’ emotional alignment with humans.

[1] Whose Emotions and Moral Sentiments Do Language Models Reflect? Zihao He, Siyi Guo, Ashwin Rao, Kristina Lerman.

[2] Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset. Hannah Rashkin, Eric Michael Smith, Margaret Li, Y-Lan Boureau.

The main conclusion is that LLMs have weak emotional alignment with humans. Can you introduce any possible solutions for the issue?

Thank you for your insightful suggestion. In response, we conducted an experiment using the GPT-3.5-turbo model. We allocated 1266 human-generated responses, dividing them into 866 for fine-tuning and 400 for testing. The table below presents the performance comparison between the vanilla and fine-tuned models against human norms, based on negative affect scores in the test set:

The data indicates that the fine-tuned model better mirrors human emotional response, especially in representing both default and emotion-evoked states.

Thank you for your hard work of reviewing! We will address your concerns one by one.

W1: evaluation seems weak since two LLM families are kind of limited. For closed LLMs, there are Claude-3, Gemini, et al; For open-sourced (open weights) LLMs, there are mistral, falcon, phi-3, flan-t5, vicuna, et al. Since there is much cost using closed LLMs api, it is not difficult to evaluate on at least other open LLMs besides the llama-2.

Thank you for your valuable feedback. We have extended our evaluation to include the newest LLaMA-3.1-8B-Instruct, released two weeks ago, and the Mixtral-7x22B-Instruct. Below are the results:

LLaMA-3.1-8B-Instruct

Mixtral-8x22B-Instruct

Findings:

• LLaMA-3.1: Similar to LLaMA-2.1, it exhibits high default positive and negative scores, indicating consistency within the LLaMA family. However, for evoked emotions, positive scores drop significantly, and all negative scores increase significantly, except for Anxiety and Jealousy.
• Mixtral: Performs similarly to GPT-4. Its default scores show nearly maximal positive and minimal negative scores.

We have added the experiments and findings in our revised paper.

Q1: all the experiments are reported on the authors private dataset. is it possible that the evaluation might be conducted on some related public datasets? In other words, what is the contribution of this paper besides the collected private dataset?

There are indeed several emotion-related datasets available [1, 2]. For instance, He et al. [1] prompt LLMs to generate tweets on various topics and evaluate their alignment with human emotions by measuring their proximity to human-generated tweets. Rashkin et al. [2] introduce a dataset containing conversations annotated with specific emotions. Our work distinguishes itself in the following ways:

1. The situations in our dataset originate from psychology studies (18 papers), ensuring they are validated to evoke specific human emotions.
2. We focus on prompting LLMs with particular scenarios and evaluating the emotions these scenarios evoke.

We acknowledge the feasibility of conducting further experiments on related public datasets, including the tweet generation task as demonstrated by He et al. [1]. Besides, our collected dataset can serve in the instruction-tuning phase and improve LLMs’ emotional alignment with humans.

[1] Whose Emotions and Moral Sentiments Do Language Models Reflect? Zihao He, Siyi Guo, Ashwin Rao, Kristina Lerman.

[2] Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset. Hannah Rashkin, Eric Michael Smith, Margaret Li, Y-Lan Boureau.

Q2: all evaluated LLMs are general-purpose. Are there domain-specific (i.e., emotion-support domain investigated in this paper) finetuned LLMs to be evaluated?

Thank you for your insightful suggestion. There are few open-source LLMs tuned for emotional alignment. However, we can tune such LLMs using our constructed dataset. We conducted an experiment using the GPT-3.5-turbo model. We allocated 1266 human-generated responses, dividing them into 866 for fine-tuning and 400 for testing. The table below presents the performance comparison between the vanilla and fine-tuned models against human norms, based on negative affect scores in the test set:

The data indicates that the fine-tuned model better mirrors human emotional response, especially in representing both default and emotion-evoked states.

Thanks for your further comments! We finetune the LLaMA-3.1-8B and here are the results:

Our dataset can also enhance the emotional alignment with humans for open source models, as expected.

We have uploaded our EmotionBench, including our dataset and collected 1,266 human resposnes, has been pushed to huggingface for easier use of our community. Due to anonymity, we do not append the links, but we have added the links to our paper and can be seen in the updated version.

Thank you once again for your reviewing efforts and your interest in our work!