To Err Like Human: Affective Bias-Inspired Measures for Visual Emotion Recognition Evaluation

1 Weakness 2，7:

As shown in Sec.3 of our rebuttal material，Emotional polarity is extremely important for emotion classification tasks. Therefore, we separate emotions of different polarities based on emotional polarity, but we do not shorten the distance between unrelated emotions. For example, before modification, the distance between amusement and fear was 4, and after increasing the distance, it became 8. To prove the effectiveness of our proposed changes, we followed previous work on misclassification[1] to verify the effectiveness of our proposed emotional distances. We converted these distances into ranking and trained with the mixed loss function based on different ranking.

2 Weakness3:

Although the methods in Fig.1 of our manuscript are listed chronologically, the later methods are not necessarily developed based on the earlier ones. For example, when comparing OSANet and ResNet101, although OSANet’s accuracy is much higher than that of ResNet101 and ResNet50, its error severity is also higher. Similarly, even though WSCNet’s accuracy is comparable to ResNet50 and ResNet101, its error severity is very high. With this figure, we want to demonstrate that the current mainstream emotion classification methods do not incorporate the important factor of emotional error severity into the method design and evaluation system.

3 Weakness4:

From the confusion matrix in Fig.1 of our rebuttal material, it can be seen that, in general, misclassification into categories with the same polarity occurs more often than into categories with opposite polarity. Taking OSANet as an example, although there is one more misclassification to emotion categories with opposite polarities (4 categories) than to those with the same polarities (3 categories), the proportion of misclassifications to categories with the same emotion polarity is 22.625%, while it is 13% for opposite polarities.This indicates that images with the same emotional polarity often express similar patterns and have relatively close emotions. This shows that there is a close relationship between emotional polarity and emotional misclassification. We also demonstrated through the validation experiment in Section 4.3 that the emotion rank, derived from the emotion distance we defined, has good properties.

4 Weakness5:

We combined the experiments with fully supervised learning to analyze the emotion retrieval and binary classification tasks. As shown in the table, under the same model architecture, when the ACC is not significantly different, models with higher EMC, due to the incorporation of the concept of label distance, tend to perform better on eight retrieval metrics than models with higher ACC. In the binary classification task, models with higher EMC also show better ACC2 performance.

5 Weakness6:

In semi-supervised training, the quality of pseudo-labels determines the final performance. Although models trained with CE loss have higher ACC, the pseudo-labels may be misclassified into emotion-irrelevant labels. For example, an “amusement” sample might be labeled with a pseudo-label of “fear,” which is disastrous for model training. On the other hand, while models trained with a combined function may have lower ACC, they have higher EMC. This means that although the pseudo-labels may be incorrect, they are likely to be mislabeled into categories close to the true category. Under this positive reinforcement, models with higher EMC are trained with continuously generated high-quality pseudo-labels, ultimately achieving higher ACC than the former.

References

[1].Bertinetto, Luca, et al. "Making better mistakes: Leveraging class hierarchies with deep networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

Weakness:

• First, we have added more extensive semi-supervised comparative experiments in Tab.1 of our rebuttal material, as well as experiments in Tab.2 of our rebuttal material on image retrieval tasks and emotion binary classification tasks.

• Secondly, we strongly agree with your view that “human emotions are not singular,people experience a range of emotions.” Emotions have similarities, so accuracy in traditional classification tasks is not entirely applicable in emotion classification. Our metrics and the loss functions we provided incorporate this design. However, the task you mentioned is essentially a multi-label classification task, similar to emotion distribution learning. Both tasks face challenges in data annotation, and label quality may not always be reliable. It is difficult to determine which specific emotions are present in an image and how the emotional distribution of an image looks.In response to the first point of Reviewer 1, we also addressed this issue. Previous emotion distribution learning has often provided rough labels, and directly using KL loss to force the model’s final distribution to match the ground truth distribution is unreasonable. Therefore, we believe that constraining emotions through emotional ranking at the loss level is more appropriate. This approach helps the model learn which emotions are predominant in an image and which emotions are more prevalent relative to others.

• Discussion of F1-macro and F1-weighted metrics: In fact, these are common metrics in machine learning for misclassification, but they are fundamentally different from our metrics. F1-macro is the average of the F1 scores for each category, while F1-weighted is the weighted sum of the F1 scores for each category. However, the precision and recall in F1 scores are calculated based on correct classifications, which differs from the research significance of our metrics. We aim to highlight the correlation of labels, where each category has a proximity relationship with other categories, but F1 scores cannot reflect this relationship.

• However, we observed many works on facial expression recognition and error severity classification at NeurIPS. Our work could inspire thinking in these two fields and even broader areas. Therefore, we believe that submitting to NeurIPS is still meaningful.

• Finally, we want to discuss the Mikel’s wheel and Ekman models. Any emotion model has varying distances between related emotional labels. Ekman does not have a mature distance definition like Mikel’s wheel, so we did not conduct experiments on Ekman. However, if a reasonable six-category distance for Ekman is defined, or if we define emotional distances based on Mikel’s wheel for the emotions in Ekman, the same metrics can be used. Additionally, Jingyuan Yang and Dongyu She are not a research group. Mikel’s wheel is the foundational model for the eight-category visual emotion model. Almost all visual emotion tasks are now conducted on FI, and further works like EmoGen and EmoEdit, which are based on the eight-category model, are built on this. Therefore, there is no need to worry about the limited scope of researchers using Mikel’s wheel.

Question:

• Yes, we have considered classification based on Plutchik’s model, which is a more fine-grained 24-category model built on the eight primary categories. If using such a model, similar to Ekman’s, it would only require defining a reasonable emotional distance model, such as assigning a weight to each layer of emotions in Plutchik’s model.

• We processed EmoSet in the same way as FI because both are eight-category models based on Mikel’s wheel. Aside from the difference in dataset size, there is no significant difference between the two datasets during training. All our experiments on EmoSet used the same parameter settings as FI.

1 Weakness1:

We have added comparative methods for the semi-supervised experiments in Tab.1 of our rebuttal material.

2 Weakness2:

In fact, it is a semi-supervised learning experiment setting. We divided the training set according to a predetermined number of labeled samples into labeled training samples and samples we consider unlabeled. Then, we trained on the partially labeled training samples and a large number of unlabeled training samples and tested on the official test sets of FI/Emoset. During training, we can assess the model’s ability to label pseudo-labels by obtaining the true labels of the pseudo-labels. In the early stages of using different functions (CE and mixed functions), the model trained with CE loss has higher ACC for labeled pseudo-labels but lower EMC. In contrast, the model trained with mixed loss has lower ACC but higher EMC, which means the model can learn useful knowledge from incorrectly labeled tags that are similar to the true labels, thereby improving the final semi-supervised classification accuracy.

3 Weakness3:

We added a description of the user study in Fig.2 of our rebuttal material. First, we selected samples that both models misclassified. Then, we presented the classification results of the two models, along with ‘Indistinct’ as three options, and asked users to choose which emotion was closer to the true emotion of the image

4 Weakness4:

Our metrics primarily emphasize the similarity between emotional labels, as there are differences in the quality of misclassifications in emotions. However, Mikel’s wheel is off the shelf. Objectively speaking, Ekman also has emotional label similarities. It can be used as long as an appropriate model is defined just like Mikel’s wheel.

5 Question1：

Based on existing semi-supervised work such as S2VER, FixMatch, and FlexMatch, we chose 0.98 and 0.7 as the upper and lower bounds of the threshold, and $\tau$ is 0.95. For e , we derived empirical values for each dataset. For example, in the FI dataset, EMC is usually around 0.5, and in Emoset, EMC is usually around 0.6, so we set them to 0.5 and 0.4, respectively.

6 Question2:

We followed the previous papers FlexMatch and S2VER, conducting the experiments three times under the same experimental settings with different random seeds, and then we calculated the standard deviation.

1 Weakness1: Contribution to visual emotion recognition

• Emotional ambiguity and emotional relevance have always been important issues in the field of emotions, and previous work has been dedicated to solving these problems.Previous works[1] extended single labels into label distributions, or the work of Visual Sentiment Distributions[2][3], Whether it is based on the voted label distribution or the label distribution according to Gaussian fitting, only a rough and inaccurate distribution can be given, and there is a gap between ground truth label distribution and real-world label distribution. It is obviously unreasonable to give the same label distribution to all images of the same category according to Gaussian fitting, while ignoring the differences between the emotions of different images. Conversely, our measure using label rank information can make the ground truth label rank closer to the real-world label rank, or exactly equal, to better guide the model to learn the emotion categories structure.
• From the perspective of misclassification, emotions have a natural adaptability compared to previous work. There is a clear correlation between different emotions, and there is no need for complex category distance definitions, which is of great significance for the field of visual emotion recognition.

2 Weakness2:

As shown in Tab.1 and Tab.2 of our rebuttal material. We conducted supplementary experiments in semi-supervised settings and also tested our metrics on image retrieval and binary classification tasks. In the user study, we primarily analyzed ECC, which is equivalent to analyzing ACC+EMC. Previous work on misclassification has always aimed to reduce the severity of misclassification while ensuring the accuracy of ACC. Indeed, both ACC and EMC are generally used together to evaluate the model.

3 Question1：

From the confusion matrix in Fig.1 of our rebuttal material, we can see that a significant portion of emotions are misclassified into categories with opposite polarity. For example, using OSANet, when the emotional polarity is positive, an average of 6.75% of the samples are misclassified as negative emotions. When the emotional polarity is negative, an average of 19.25% of the samples are misclassified as positive emotions.

4 Question2:

In fact, we reassessed the current mainstream methods in Figure 1 of our manuscripts, where the relationship between ACC and EMC is shown.

References

[1].Yang, Jufeng, et al. "Weakly supervised coupled networks for visual sentiment analysis." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[2].Yang, Jufeng, Dongyu She, and Ming Sun. "Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network." IJCAI. 2017.

[3].Yang, Jufeng, Ming Sun, and Xiaoxiao Sun. "Learning visual sentiment distributions via augmented conditional probability neural network." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 31. No. 1. 2017.