Face-Human-Bench: A Comprehensive Benchmark of Face and Human Understanding for Multi-modal Assistants

For Weekness 4: In Appendix B.1, we provide brief descriptions of the actual MLLMs under evaluation, but we do not include details about the data they have been trained on. We list the training data used by the three best-performing open-source models in our experiments as follows.

We discuss whether some models outperform others simply because their training data was closer to that expected for human analysis, under the following 3 scenarios.

(1) The categories of visual-text alignment information included in the MLLMs' training data are closer to those in Face-Human-Bench. This means that the training data includes more images containing faces and humans, along with detailed, accurate, and comprehensive captions for alignment. The MLLMs' abilities to understand faces and humans primarily stem from the alignment training stage (also called the pre-trained stage in some literature about MLLMs), during which visual features are aligned to the LLM's language space using visual-text pair data. During this training process, the model cannot learn language concepts it has not "seen". This "closer" is actually necessary to enhance the MLLMs' ability to understand faces and humans and is harmless to the evaluation.

(2) The distribution of features in the training data about faces and humans is closer to that in the Face-Human-Bench. These features include camera angles, image resolutions, contrast, brightness, and so on. Since the training images used by MLLMs and the images in Face-Human-Bench come from different datasets collected separately, performance gain due to the “closer” distribution of features can be deemed negligible.

(3) The training data for the MLLMs appear in the test set of Face-Human-Bench. This could lead to unfair evaluation results. We have checked the training data sources of the open-source MLLMs we tested and confirm that there is no overlap with the datasets used to construct our Face-Human-Bench. This ensures the reliability of our evaluation.

We express our gratitude for your valuable comments.

For Weakness 1 and the Question:

Over the past two years, LLMs and MLLMs have garnered significant attention from academia and industry. Unlike dedicated models designed to address a specific, well-defined task, LLMs and MLLMs exhibit strong abilities in instruction-following and reasoning, making them general-purpose assistants. An important question that arises is how to define and quantitatively evaluate the capabilities of LLMs and MLLMs. ICLR 2024 has established significant precedence for papers aiming to establish a comprehensive and scientific evaluation framework from a specific dimension for LLMs or MLLMs. Here are some papers accepted at ICLR 2024:

MathVista [1] evaluates MLLMs' abilities in mathematical reasoning, SWE-bench [2] assesses LLMs' capability to resolve real-world GitHub issues, KoLA [3] explores LLMs' grasp of world knowledge, and AgentBench [4] and SmartPlay [5] evaluate LLMs as agents in challenging tasks within interactive environments. These papers, while containing no technical contributions, have enhanced our understanding of MLLM capabilities. Additionally, our paper is submitted under the primary area of “Datasets and Benchmarks”. ICLR encourages the development of high-quality datasets and benchmarks to define problems and evaluate models.

[1] MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts https://openreview.net/forum?id=KUNzEQMWU7

[2] SWE-bench: Can Language Models Resolve Real-world Github Issues? https://openreview.net/forum?id=VTF8yNQM66

[3] KoLA: Carefully Benchmarking World Knowledge of Large Language Models https://openreview.net/forum?id=AqN23oqraW

[4] AgentBench: Evaluating LLMs as Agents https://openreview.net/forum?id=zAdUB0aCTQ

[5] SmartPlay: A Benchmark for LLMs as Intelligent Agents https://openreview.net/forum?id=S2oTVrlcp3

For Weekness 2:

Exploring the "correlation between abilities" aims to reveal whether the enhancement of ability A in a model is accompanied by the enhancement of ability B. Specifically, we represent the scores of all tested MLLMs on ability A and ability B from the Face-Human-Bench as X and Y, respectively. We then calculate the Pearson Correlation Coefficient to indicate the "correlation between A and B" with the following formula:

$r = \frac{{\sum (X_i - \bar{X})(Y_i - \bar{Y})}}{\sqrt{\sum (X_i - \bar{X})^2 \sum (Y_i - \bar{Y})^2}}$.

For example, the correlation coefficient between age estimation and facial expression recognition is 0.88, which is close to +1.00. This indicates that enhancements in age estimation are accompanied by enhancements in facial expression recognition, and vice versa. On the other hand, the correlation coefficient between face recognition and face attack detection is 0.16, which is close to 0. This suggests that improvements in face recognition performance usually do not lead to significant gains in face attack detection. Theoretically, two abilities with high positive correlation might share more activation parameters within the MLLM. Conversely, two abilities with low correlation might share fewer activation parameters. In our updated paper submission, we will provide a clearer definition of the "correlation between abilities".

For Weekness 3:

We also do not consider that RPSS can be counted as a contribution. The focus of Section 3.4 is to uncover the impact of the relative position of targets on performance. When users ask MLLMs questions related to faces and humans, they do not expect the quality of the responses to fluctuate due to changes in the relative position of the targets. Our observation will inspire the MLLM community to train more robust models. RPSS is merely used as a trivial means to quantify performance fluctuations.

We express our gratitude for your valuable comments.

For Weakness 1:

One of the purposes of constructing Face-Human-Bench is to inspire the MLLM community to train models with stronger face and human understanding capabilities. For this reason, we have divided the dataset into a development set and a test set. The experimental results reported in the paper are based on the test set. The development set is intended for the MLLM community to use in evaluating the face and human understanding abilities of MLLMs during training iterations. In our updated paper submission, we will include a description of the roles of these two sets.

For Weakness 2:

We set the weights of all 10 L2 abilities to be equal. For L2 abilities that encompass multiple L3 abilities, each L3 ability equally shares the weight of the corresponding L2 ability. For L3 abilities that encompass multiple image versions, each image version subset equally shares the weight of the corresponding L3 ability. Finally, we obtain the detailed weights of each subset, as shown in Table 12 at Appendix A.2.

This calculation method assumes that the 10 L2 abilities have equal importance. However, in specific application scenarios, it may be more appropriate to assume different importance for each L2 ability. Table 1 provides the scores for each L3 ability, enabling readers to recalculate the overall score under different weight assumptions for the L2 abilities.

For Weakness 3:

As shown in the results in Appendix C.2, for nearly all the models, the test results in English outperform those in Chinese. Only a few models, such as DeepSeek-VL-1.3B-Chat, DeepSeek-VL-7B-Chat, and GLM-4V-9B, exhibit nearly identical performance in both languages.

The variation in training data indeed significantly impacts the language preferences of the models. For instance, consider the models LLaVA-NeXT-7B [1] and GLM-4V-9B [2]. The former demonstrates a substantial performance gap between English and Chinese (with English outperforming Chinese by 10.7), whereas the latter shows almost identical performance in both languages (with English only slightly better by 0.1). LLaVa-NeXT-7B is based on Vicuna-7B-v1.5, which is fine-tuned from LLaMA2-7B [3]. In the pre-training corpus of LLaMA2, 89.70% is English and only 0.13% is Chinese. On the other hand, GLM-4V-9B utilizes GLM-4 [2] as the underlying LLM. Although the specific proportions of Chinese and English data are not provided, the GLM-4 technical report indicates that both Chinese and English are major languages in its pre-training data. This example illustrates that the proportion of different languages in the pre-training corpus of the LLM component directly affects the language preferences of the MLLM.

[1] LLaVA-NeXT: Improved reasoning, OCR, and world knowledge https://llava-vl.github.io/blog/2024-01-30-llava-next/

[2] ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools https://arxiv.org/abs/2406.12793

[3] Llama 2: Open Foundation and Fine-Tuned Chat Models https://arxiv.org/abs/2307.09288

For Weakness 4:

Exploring the "correlation between abilities" aims to reveal whether the enhancement of ability A in a model is accompanied by the enhancement of ability B. Specifically, we represent the scores of all tested MLLMs on ability A and ability B from the Face-Human-Bench as X and Y, respectively. We then calculate the Pearson Correlation Coefficient to indicate the "correlation between A and B" with the following formula.

$r = \frac{{\sum (X_i - \bar{X})(Y_i - \bar{Y})}}{\sqrt{\sum (X_i - \bar{X})^2 \sum (Y_i - \bar{Y})^2}}$

We express our gratitude for your valuable comments.

For Weakness 2:

The purpose of our study is not to use VLLM to complete visual perception tasks. Rather, the aim is to advance the development of VLLM. An excellent VLLM must possess strong face and human understanding capabilities since images containing faces and humans are among the most common inputs for VLLM. As described in the introduction of our paper, "Faces and humans are central to social interaction, a deep understanding of this information can make multi-modal assistants achieve improved response quality and broadened application scope." A comprehensive and scientific evaluation of VLLM's face and human understanding abilities is the foundation for driving VLLM's performance improvements in this area.

Under the zero-shot setting, the best closed-source model, GPT-4o, does not outperform the top-performing open-source models. This is, in fact, a rather surprising result. Our further experiments in Section 3.5 reveal that introducing hints and CoT instructions into the prompts significantly improves the performance of the open-source models but has no effect on the closed-source model. After fully leveraging its potential, GPT-4o does indeed exceed the performance of the best open-source models.

For Weakness 3:

Despite following standard procedures, we have constructed the first comprehensive and scientific benchmark for evaluating MLLM's face and human understanding abilities, featuring a three-level ability taxonomy.

For Weaknesses 1 and 4:

In fact, through a comprehensive evaluation of VLLMs' face and human understanding abilities, we have uncovered several interesting conclusions that enhance our understanding of VLLMs. For instance, for some VLLMs, when the relative position of the targets changes, the quality of the responses also varies. This insight can inspire the MLLM community to train more robust models.

Another surprising result is that GPT-4o, in a zero-shot setting, does not surpass the best-performing open-source VLLMs. This has already been discussed in "For Weakness 2."

Experiments in Section 3.6 suggest that sophisticated pipelines for face analysis (that perform detection, alignment and recognition) , trained on dedicated datasets, generally outperform VLLMs evaluated on corresponding tasks. However, the primary goal of this section is to provide valuable insights for constructing multi-modal agents and enhancing the overall performance of multi-modal assistants. We have comprehensively discussed the abilities in which current MLLMs significantly lag behind specialist models, as well as the abilities in which they are comparable or nearly on par.

It's also worth noting that models with more parameters don't always perform better. As mentioned in Section 3.2, LLaVA-OneVision0.5B, with only 0.5 billion parameters in the LLM, outperforms the earlier InstructBLIP-13B. Additionally, the 13B versions of LLaVA-1.5 and LLaVA-Next perform slightly worse than their 7B counterparts.

Since the rebuttal period is ending soon, we would be deeply appreciative if you could review our response to ensure it addresses your points. Your feedback is invaluable to us.

Thank you so much for your time and consideration.

The authors

We express our gratitude for your valuable comments.

For Question 1: We first utilize an off-the-shelf face detection model to extract faces from the test set of Face-Human-Bench (excluding samples for the crowd counting ability). Then, gender, age, and race labels are predicted for each face with a series of off-the-shelf specialized models. Based on these labels, we can obtain the following statistical results to demonstrate the demographic diversity of the Face-Human-Bench dataset.

To further investigate the impact of dataset biases on the performance of MLLMs, we transform face pairs from the Caucasian, African, Asian, and Indian subsets of the RFW dataset to problems for face recognition similar to those in Face-Human-Bench. The test results of the three best-performing open-source models in our main experiments are presented in the following table, revealing the racial bias of MLLMs in face recognition ability. The performance for Caucasians is the best for each model, significantly surpassing that for other racial groups.

For Question 2:

The methodologies employed to construct the Face-Human-Bench can be summarized as follows:

1. Review existing literature and establish a hierarchical ability taxonomy.
2. For each ability in the hierarchical taxonomy, collect and process relevant data. Construct a new benchmark with standardized test problems.
3. Identify the MLLMs to be tested and conduct testing and analysis on the new benchmark.

These methodologies can be extended to evaluate other capabilities of MLLMs, such as flora and fauna understanding, or multimodal mathematical reasoning.

For Question 3:

In Section 3.5, we analyze the reasons behind the limited effectiveness of CoT prompting on open-source models. Despite incorporating CoT instructions into the prompts, these models fail to provide rationales in their responses. This shortcoming indicates a lack of generalization capabilities, hindering their understanding of CoT instructions. One possible explanation for this deficiency is the models' insufficient exposure to complex reasoning data during training, which leads them to generate direct answers without valid rationales. Another possibility is that these models have inadequate model capacity. The relatively lower number of parameters in their LLM modules constrains their reasoning abilities.