EIBench: Assessing the Emotion Interpretation ability of Vision Large Language Models

This paper presents several notable limitations that warrant discussion:

1. Limited Practical Impact Due to Absence of Fine-tuning Paradigm

• The work focuses solely on evaluating EI capabilities of existing models without providing methods to enhance their performance
• The lack of fine-tuning guidelines or frameworks significantly limits the practical value for the research community

2. Inconsistent Emotion Classification Taxonomy

• The fine-grained emotion categorization in Table 5 contains logical inconsistencies. For example, 'happy' is inappropriately classified under the 'excite' primary category rather than 'Happy'
• Such classification errors suggest insufficient validation of the emotional taxonomy structure

3. Questionable Value of Fine-grained Classification

• The practical contribution of the fine-grained emotion classification system is unclear. Experimental results are predominantly reported using the four coarse-grained categories. The granular categorization appears superfluous given its limited use in the actual analysis. The work would benefit from either:
  • Demonstrating concrete applications of the fine-grained classification
  • Focusing more deliberately on the core four-category system

These limitations suggest opportunities for future work to enhance the practical impact and theoretical foundation of emotion interpretation tasks.

1. The dataset is relatively small, with only 1,615 images, of which just 50 are complex images. It focuses on four primary emotions (happy, angry, sad, and excited), which may not capture the full range of human emotional experiences. Expanding the dataset to include a broader range of emotions, such as fear, disgust, surprise, and neutral states, would be beneficial.
2. The paper lacks a comprehensive human study to evaluate the identified triggers. Given the subjective nature of emotions, evaluating with only three volunteers seems insufficient, especially if the volunteers share the same demographics. Additionally, it would be helpful to provide the guidelines given to the volunteers.
3. The metrics used in the paper are not very clear and may not accurately capture the correctness of the model's responses.

1. Dataset Size and Composition:

   • 1,615 samples with only 50 classified as complex may not provide sufficient coverage for the number of categories, especially as prior literature [1] considers multiple important factors like colors and objects in emotion elicitation. To ensure robustness, Emotion-Image (EI) samples should cover a more comprehensive combination of these factors.
   • The current set of emotion categories omits primary emotions such as Love and Fear, which are essential in widely accepted emotion taxonomies. Adding these would provide a more complete emotional spectrum.

2. Fine-Grained Emotion Overlap:

   • While the inclusion of fine-grained emotions is appreciated, some labels are inconsistently assigned across categories, resulting in overlap. For example:
     • "Upset," "Frustration," "Displeased," "Enjoying," and "Joyful" appear in multiple categories.
     • Emotions from opposing categories, such as "Surprised," are present in both "Negative Angry" and "Happy."
   • Clarification: It would be helpful to know if these overlaps were intentional and whether samples were included in both classes during dataset construction.

3. Human Evaluation:

   • The evaluation of dataset quality relies on only three annotators, which may not be sufficient. To strengthen the validity:
     • Consider including inter-annotator agreement scores to assess consistency.
     • Provide more details on the annotators' backgrounds and qualifications, as this could impact the dataset quality and the interpretability of the annotations.

4. Dataset Collection Details: Important aspects of dataset construction and filtering are missing, including:

   • How were images selected from CAER and EmoSet?
   • What process was used to obtain the user questions?
   • The use of the feature "X_face" suggests only images with faces were considered—please confirm or clarify.
   • The paper mentions "critical factors impacting human emotion are comprehensively investigated." It would be helpful to specify what these critical factors are.
   • Specify which version of LLaMA-3 was used for summarization in your experiments.
   • In Table 4, was an average applied to minimum and maximum values, or were they rounded? Global minimum and maximum values would provide clarity, and these specifics should be included in the caption.

5. Writing and Clarity:

   • Table Captions: Captions need to clarify the contents better. For example, Table 2 lacks context—is it analysis, input, or output?
   • Metric Definition: "We use Recall as one evaluation metric. If the model’s interpretation overlaps with our ground truth." Please clarify what is meant by "overlap" in this context, as this may not be an ideal metric for a generative task.
   • Model Specifications: For the Recall metric, specify the LLaMA version (e.g., Chat/Base, 7B, 70B) used to maintain clarity.

1. The primary contribution of this paper is proposing a new task on emotion interpretation, claiming it differs from existing mainstream tasks (such as emotion cause extraction) by focusing on interpreting emotional triggers rather than merely classifying emotions. However, this is not a novel task. Emotion-Cause Analysis in [1] and Multimodal Emotion Cause Generation in [2] both consider emotion categories as well as fine-grained cause interpretation (rather than traditional cause extraction), which is almost identical to the formalized definition in the paper. It is recommended that the authors thoroughly review related literature and compare their proposed task with works like Emotion-Cause Analysis and Multimodal Emotion Cause Generation.

• [1] EMO-KNOW: A Large Scale Dataset on Emotion-Cause (EMNLP 2023)
• [2] Observe before Generate: Emotion-Cause Aware Video Caption for Multimodal Emotion Cause Generation in Conversations (MM 2024)

2. The secondary contribution of the paper is introducing a labeled dataset for emotion cause interpretation. However, EIBench has a very small data size (fewer than 2,000 samples). Additionally, the authors have not made the dataset publicly available and have not analyzed or quantified key indicators such as data quality and diversity. Therefore, compared to existing datasets (such as the large and open-source datasets EMO-KNOW [1] and ECGF [2]), the contribution of this dataset is minimal.

3. The experimental conclusions are insufficient. The authors conducted preliminary experiments on some LLMs only in a zero-shot setting without evaluating them in few-shot, supervised fine-tuning, or reinforcement learning (weak supervision) settings. Therefore, the conclusion "Experiments show limited proficiency of existing models in EI" is unreliable. It is recommended that the authors expand the experimental results in few-shot, supervised fine-tuning, and reinforcement learning (weak supervision) settings to verify the accuracy of their conclusions.

4. The authors propose an effective VLLM-assisted data annotation technique for the EIBench dataset, but they have not compared it with existing annotation methods to demonstrate its advantages.

5. The experimental setup in the paper lacks detail. For example, there is no explanation of the rationale behind the selection of model types and sizes used.

Indicated in the summary section above

Reasons to reject

• This is a dataset paper but there is little focus on the annotation process and interpreting the data. It is critical to extend the discussion of the data considerably. This is even more critical since a significant portion of the emotion causes/interpretations are model-generated. Here are important insights that are missing: (1) Are the final responses (i.e., triggers) natural in terms of language? (2) How many changes did the annotators need to make in each stage? How many hallucinations were there? How many times did an extra trigger need to be added? Is there a specific emotion whose triggers were harder to identify? (3) What is the qualification of the annotators? What was the process to train these annotators? How do we know we can trust these annotations? How many annotators were used per example? (4) In L306-L316 there is a human evaluation of EIBench. Why is there such a big variance in scores between the annotators. Are these the same annotators as L299-L305? (5) Why only 4 general questions were used? Have you tried fewer/more? How does this affect the quality of the data? Overall, I think the paper should focus more on the quality of data and the processes used to ensure this quality. Additionally, each step of the annotation process needs a thorough analysis: lexical/language analysis.
• The clarity and quality of the writing could be significantly improved. Grammatical errors and unsuitable phrasings make the paper difficult to read.
• Emotion-cause extraction (ECE) has less real-world applicability compared to emotion-cause pair extraction [1] (ECPE). The evaluation lacks ECPE results.

Minor:

• Size of the data (1615 examples) is somewhat limited.

[1] Xia and Ding, Emotion-Cause Pair Extraction: A New Task to Emotion Analysis in Texts

Overall, I think EIBench can be extremely useful. I am worried about the quality, the coverage of emotion triggers, as well as the "naturalness" of the interpretations. I am willing to raise my scores based on questions 1, 2, 3, 4.