Can MLLMs Understand the Deep Implication Behind Chinese Images?

1. The size of this dataset—698 images and 800 questions—is quite small, which may render the conclusions drawn from the evaluation results non-generalizable.
2. Since all the questions in this benchmark are multiple-choice, the output obtained from MLLM may be biased, as some models tend to favor specific choices. Therefore, the authors are encouraged to use the CircularEval in MMBench[^1] to ensure more robust results.
3. As shown in Table 1, text-only models, such as Qwen2-7B-Instruct, can also answer some questions correctly without referring to images. As a benchmark for evaluating multimodal capabilities, these questions are not appropriate and should be removed.
4. Lacking comparison with CCBench[^2], a benchmark designed to evaluate an MLLM's capability to understand images related to Chinese culture.
5. The benchmark primarily evaluates a model's ability to understand the implications behind Chinese images. However, I find that some images in the appendix, such as Figure G3 and Figure G4, are not particularly representative of Chinese imagery.
6. Since the dataset is divided into six categories, the authors are also expected to explain the rationale behind choosing these specific categories.

[^1] https://arxiv.org/abs/2307.06281 [^2] https://github.com/open-compass/MMBench

• The proposed CII-Bench includes a greater emphasis on understanding the cultural and emotional content behind images. In this context, did the authors design more complex prompts to better guide the model's output? For instance, did they use background information and Chain-of-Thought (CoT) prompting to help the MLLM predict answers from the background context?

• The English images presented in Fig.~1 are not convincing, as there are also complex and suggestive English images. The authors should compare with similar datasets. For example, in the II-Bench work, there is already a significant gap between the performance of existing MLLMs and human results, which is highly consistent with the conclusions of this paper. The authors should provide more rigorous reasons to explain why Chinese images present unique challenges compared to English images.

• Using the II-Bench approach, the authors replicated the entire process on Chinese data. I did not see updated data collection and management content. The authors should distinguish their work from II-Bench in terms of scientific writing and benchmark processes. CII-Bench appears to be a derivative of II-Bench. The authors are advised to clarify additional contributions to enhance the innovation of this paper.

• To ensure the reliability of high-quality evaluation in CII, how did the authors consider the dataset size of less than 1K? Since CII covers multiple aspects, leading to fewer data points in each aspect, how did they address potential biases? The authors should provide statistically significant quantitative results to demonstrate the validity of CII-Bench.

• II-Bench introduced new challenges, and I look forward to seeing the authors' technical innovations to improve the benchmark results. Have the authors conducted relevant technical explorations, and from which perspectives will they address these issues?

Weakness 1 It is mentioned in the paper that “in order to ensure the authenticity of the Chinese context, the pictures in CII-Bench are all from the Chinese Internet and have been manually reviewed, and the corresponding answers are also manually produced.” So the pictures are all from the Internet, and most of them are not real pictures, which greatly limits the development of Chinese language, and it is suggested that Chinese pictures from some real scenarios should be added.

Weakness 2 For some metaphorical work such as FigureG1, these answers are too simple, and the complexity should be increased.

Weakness 3 It is suggested to add some datasets that contain more traditional Chinese culture, such as frescoes and landscape paintings, and experts are needed to judge and calibrate the labels.

Weakness 4 This paper and reference [1] can clearly be combined into a dataset, as the constructed prompts are the same, with at most only some differences in the images. Therefore, for a top conference like ICLR, the contribution is relatively small.

[1] II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models

• The scale of this dataset is a little small. CII-Bench only contains 698 images and 800 questions, which may not be comprehensive enough to evaluate the performance of MLLMs.
• Some detailed information about the dataset should be provided. For example, the ratio of six different types of images.
• The motivation is not strong enough. I think this work is just an extension of II-Bench [1]. So, to demonstrate the necessity of this paper, the authors should discuss or conclude the inconsistencies of the results on II-Bench and CII-Bench. Since this paper is quite similar to II-Bench [1], it is important to analyze the consistencies and inconsistencies of the experimental results. For example, in which scenario do the MLLMs exhibit similar performance for images in two different languages? Meanwhile, under which conditions do the Chinese images present stronger challenges than the English ones?

[1] Liu, Ziqiang, et al. "II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models." arXiv preprint arXiv:2406.05862 (2024).

1. The dataset is small.
2. Multi-choice evaluation may not reveal the real capacity to understand the implications.