CLIP Facial Expression Recognition: Balancing Precision and Generalization

Here are some questions about this paper:

1. The contribution of this paper is limited. The core idea of this paper is leveraging the rich representation of CLIP feature for expression recognition. However, since the CLIP feature is trained on large-scale open world data, the comparsion with existing SOTA methods are not fair.
2. It seems that the propsoed method can be easily adapted to other tasks such as face attributes or face recognition. More experiments are need to demonastrate the effectiveness of the method.
3. Please describe the baseline in all tables.
4. From Table 2, we can see that the major performance gain is from happiness and neutral class. This is reasonable because of the nature of CLIP features. However, the generability of a method should fouce more on the rare (tail) classes, for which the proposed method is not good at. Besides, the feature distribution of Fig. 4 is not seperatable for the baseline and proposed method.

1. Though the results in cross-dataset settings (Table 1) are slightly better than for competing methods, they are still much worse (5-40%) when compared to training on the same dataset. It seems that the proposed method cannot solve the main issue of existing FER models.
2. The authors do not compare their technique with the state-of-the-art models for each dataset. Just a brief look at, e.g., AffectNet (https://paperswithcode.com/sota/facial-expression-recognition-on-affectnet) clearly shows that the authors’ results are much worse: 7-class accuracy is 57.87%, which is much worse than the top result (67.49%). The same comparison should be made for other datasets. I expect the authors to support their claim “Current facial expression recognition (FER) methods excel in achieving high classification accuracy but often struggle to generalize effectively across various unseen test sets” by showing the results of the state-of-the-art models trained on one dataset and tested on other datasets. The top model might be better on the same dataset but should show much worse results on other datasets. In fact, I personally used several EfficientNet models from hsemotion dataset and noticed that they are quite good in real settings.
3. CLIP model is mainly used due to its zero-shot capabilities caused by multi-modal training and possibility to use textual prompts. It is questionable if the visual part of the CLIP model only is the best choice for the authors’ model. I believe there exist other visual neural networks pre-trained on very large datasets that may be also used here, so the constraint to use CLIP is really required.
4. I cannot find the name of the CLIP model, though there exist several different ViT-based models. I believe it is important to show the experiments with different CLIP models.
5. Mathematical notation is weird. For example, index j is not used in Eq. 8 in F_{max}. \tilde{M}_ i is undefined in Eq.6, M_{sel} is not used in Eq. 5, etc.

• Some other design options should be studied. For example, in the channel separation module, the authors set apart the channels evenly into 7 pieces. The authors should discuss more options as different expression classes have different difficulties. For example, it is well known that the happy expression has the most training sample and it is easy to be captured, while things are different when it comes to the fear or disgust expression as fear is easy to be confused with surprise and disgust is very subtle and they both have very few training samples. Thus, learning to capture fear and disgust is more difficult than learning to capture happy. I am curious what will happen if we assign difficult classes like fear or disgust more channels.

• I noticed the authors study the mask w/wo sigmoid function in the Appendix. However, I think this is a very important experiment as the sigmoid function is vital for the success of the proposed method and I think it is a novel design. Thus, I strongly suggest the authors include the corresponding experiments in the main paper instead of putting them in the Appendix.

• Some experiments are missing. I suggest the authors incorporate experiment results pertaining to the utilization of different channel drop rates.

1. The illustrated channel-separation is confused in Fig.2. It is easy to mis-understand that the channels of the CLIP features are separated and each channel of the features is assigned to one emotion. However, it is difficult to determine which emotion category each channel should be assigned. The main text provides more clear details of how to mask the features and separate the channels. Therefore, please revise Fig. 2 to make it more clear.

2. Eq.7 and Eq.8 are not clear.

   • In Eq.7, \tilda F_max is an output of the max pooling of L max{F_i}. It indicates that each sample has one \tilda F_max instead of L \tilda F_max. However, Eq.8 accumulates N \times L \tilda F_max. Should Eq.8 accumulate by i from 1 to N, without by j from 1 to L?

3. What is the setting for the baseline method in the experiment? Is the baseline the one with the CLIP feature and FC classifier?

4. In the proposed method, the masks are different for different input samples. Does it indicate that, for different samples, the information of the emotion is conveyed in a different position to a different extent? However, the general feature is more likely to encode the same information in the same way, even if the input samples are different. Could the author explain why the same information would be encoded in different ways for different inputs? In my understanding, it is more reasonable to train a set of masks for each class in downstream tasks. The masks should be different for different classes, but be the same for different input faces. Could the author also provide discussion and experimental results in this setting?