Thank you for your insightful suggestions! We appreciate your recognition of our problem formulation as novel and our experimental and evaluation methods as comprehensive. We are also encouraged that you think our work can provide insights for future research. We will revise our paper and incorporate additional discussions in Limitations. Below, we address your questions and concerns:

W1: The dataset size

Please see point 1 in the Overall Response.

W2 & Q4: Bias mitigation in annotation

Thank you for raising this good point! Please see Overall Response point 2.

W3 & Q3: Decomposition does not lead to consistent improvements

Thank you for this good question. Our manual analysis identifies two potential reasons. First, VLMs sometimes misinterpret visual content, leading to incorrect descriptions. This issue is also highlighted in Section 6.3. Such errors can cause cascading errors in the subsequent deep reasoning tasks. Second, the generated descriptions are often lengthy (over 100 words), resulting in more complex input prompts, which can complicate the reasoning process for VLMs.

Additionally, the decomposition is less beneficial for title matching than philosophy. Titles are more abstract and require more in-depth reasoning, whereas decomposition of surface-level description may not suffice. We will update our paper to include examples that better illustrate these potential causes.

W4 & Q6: Concrete suggestions and proposal on future improvements

Thank you for your insightful suggestion! We agree that highlighting directions for future research is crucial. Currently, we aim to uncover potential areas for improvement through our results analyses.

First, our analysis indicates that VLMs often struggle with accurately interpreting image content and may make errors in literal descriptions (Sec. 6.1 & 6.2). This suggests a need for future work to enhance visual interpretation capabilities.

Second, improving the in-depth reasoning ability of VLMs is essential. For instance, LLaVA-1.6 significantly outperforms LLaVA-1.5, likely due to the advancements in reasoning abilities [1]. Future work might incorporate recent advanced reasoning approaches (e.g., multi-agent debate, refinement-based reasoning) to further improve model performance.

Finally, our error analysis reveals that models tend to suffer from hallucination (Line 333), suggesting the need for incorporating external knowledge to enhance human understanding. To mitigate this problem, knowledge augmentation methods can be employed to enhance VLMs performance.

We will revise our paper to include a dedicated section that provides a more detailed discussion on these points and outlines future research directions.

Q1: Different cultural contexts or styles

This is a good question. Currently, we focus on humor understanding based on common interpretation rather than individual preferences with specific contextual information. However, we recognize that understanding humor and accounting for cultural and stylistic variations is crucial. In future work, we plan to delve deeper into these nuances by incorporating more contextual information into the predictions. We will also include a more detailed discussion on this part in our revised version.

Q2: Expanding the benchmark

Yes, we plan to expand our benchmark to include comics with more than two panels. This will allow us to evaluate models' ability to understand more complex narrative structures. Additionally, we will incorporate comics with diverse narrative logic types beyond contradictions. We are committed to continuously updating our benchmark to foster future research in this area.

Q5: Correlation with pre-training

This is a very insightful question! First, we believe that the reasoning ability and world knowledge of VLMs are highly correlated with their performance on our benchmark. Two observations support this hypothesis: (1) Larger models typically outperform smaller models, and it is widely recognized that larger models tend to have better reasoning abilities and world knowledge; (2) LLaVA-1.6 significantly outperforms LLaVA-1.5, likely due to advancements in these aspects [1].

Second, social understanding ability is also highly correlated with model performance. The comics in the YESBUT benchmark mainly focus on daily life concepts, and understanding these nuances requires a deep understanding of human norms.

Therefore, improving models' abilities in reasoning, world knowledge, and social understanding during the pretraining stage will enhance their performance on this task. We will incorporate this discussion into our revised version.

[1] Llava-next: Improved reasoning, ocr, and world knowledge

Suggestions on Limitations

Thank you for your detailed and valuable suggestions! We acknowledge the raised limitations, and we will revise our paper and incorporate your raised points. We will discuss these aspects in greater depth to ensure that our work is aligned with responsible AI practices, respecting cultural and social nuances. Thank you again for the insightful suggestions!

Thank you for your insightful feedback! We are pleased to learn that you consider our benchmark novel, which includes two input images per sample to emphasize the relationship between the images. Additionally, we are grateful for your acknowledgment of our task settings and good analysis. Below, we address each of your concerns and questions in detail:

W1: The dataset size is relatively small, which might lead to instability of the scores

Regarding relative small dataset size

Thanks for raising this comment. Please see point 1 in the Overall Response.

Regarding potential instability of the scores and significance analyses

This is a good point. To enhance the stability and reduce potential bias, we create three distinct prompts for each task and report the average scores from three runs with each prompt (Line 213). Here, we include additional significance analyses for the description and contradiction generation tasks on VLMs. Specifically, we consider the results from all three prompts, resulting in each model producing 1,044 outputs per task. We use t-test [1] for statistical significance analysis:

1. For description generation: GPT4 is significantly better than other baselines on all metrics (p<0.0001). Claude-3 achieves the second best result and is significantly better than all open-sourced VLMs (p<0.0001).

2. For contradiction generation: Similarly, GPT4 output is significantly better than other baselines on all metrics (p<0.01). However, the results between Claude-3 and LLaVA-1.6-13B on ROUGE-2 and between Claude-3 and CogVLM on BERT score are not significant.

We will incorporate the complete results in our revised version.

[1] https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html

W2: The subjectiveness of the task and the noise of automatic metrics.

Subjectiveness of the task

This is a good point! Indeed, humor understanding can be subjective. To address this, we have designed our tasks to minimize subjectivity. Specifically, we formalized literal description and contradiction generation as text generation tasks because they are less subjective and focus on specific descriptions and narrative illustrations of the comics. For the more abstract and subjective components, such as underlying philosophy and title selection, we formulated them as selection tasks (i.e., determining which title is “objectively” better), and evaluated these using straightforward accuracy metrics. We will better clarify this in our revision.

The noise of automatic metrics

Regarding evaluation metrics, we acknowledge that evaluations of text generation tasks remain challenging, and current automatic metrics have limitations. To mitigate this issue: (1) For semantic-based evaluations like BERT score and Rouge, we report recall scores to measure how many key points of the reference are captured by the model, ensuring a more precise assessment of content coverage. (2) To mitigate bias in GPT-based evaluations, we use different GPT variants for different purposes: for experiments, we use the gpt-4-vision-preview version as a baseline, while for evaluation, we employ the gpt-3.5-turbo-0125 version. This approach helps reduce potential bias towards the GPT4 model’s own generation.

We will revise our paper to better clarify these points and discuss the potential limitations of automatic evaluations in more detail. Despite the inherent challenges, we believe our approach offers valuable insights into the model's capabilities.

W3: Human performance

Thank you for this good suggestion! We have included human performance. Due to time constraints, we randomly select 50 samples and ask two participants to perform the underlying philosophy selection and title matching tasks. One participant is male and the other is female, with cultural backgrounds from East Asia and North America, respectively, to ensure a fair and diverse evaluation.

The results are shown as below. These results highlight the significant room for improvement for VLMs on these tasks. We will revise our paper to incorporate these discussions.

Q1: Details of GPT used for evaluation

We employ the gpt-3.5-turbo-0125 variant for the automatic evaluation. We will better clarify this in the paper.

Based on the answer, I have increased the contribution score from 2 to 3. Although GPT4 and gpt-3.5-turbo-0125 are different LLMs, they might be trained on the similar data. If authors have time and budgets, I recommend to try other LLMs (e.g., Claude 3) for evaluating GPT4 and Claude 3 again and see if the results are different from what you got from gpt-3.5-turbo-0125.

Thank you very much for your valuable comments and suggestions! We are pleased to learn that you find our introduction of the YESBUT benchmark is helpful and beneficial to future research, and our rich experimental studies and in-depth analyses offer a comprehensive view. We address your questions and concerns below:

W1: Regarding data size

Thank you for the question! Please see point 1 in the Overall Response.

W2: Regarding model improvements

Thank you for your insightful suggestion! We agree that highlighting directions for future research is crucial. Currently, we aim to uncover potential areas for improvement through results analyses.

First, our analysis indicates that VLMs often struggle with accurately interpreting image content and may make errors in literal descriptions (Sec. 6.1 & 6.2). This suggests a need for future work to enhance visual interpretation capabilities.

Second, improving the in-depth reasoning ability of VLMs is essential for this task. For instance, LLaVA-1.6 significantly outperforms LLaVA-1.5, likely due to the advancements in reasoning abilities [1]. Future work might incorporate recent advanced reasoning approaches (e.g., multi-agent debate, refinement-based reasoning) to further improve model performance.

Finally, our error analysis reveals that models tend to suffer from hallucination (Line 333), suggesting the need for incorporating external knowledge to enhance human understanding. To mitigate this problem, knowledge augmentation methods can be employed to enhance VLMs performance.

We will revise our paper to include a dedicated section that provides a more detailed discussion on these points and outlines future research directions.

[1] Llava-next: Improved reasoning, ocr, and world knowledge

Q1: Why philosophy selection is formulated as the MCQs

This is a good question! Compared to the literal description and contradiction illustration, philosophy and title are more open-ended and subjective, where there might exist multiple valid philosophies and titles for one comic. This characteristic makes the evaluation of these two tasks challenging if formulating them as text generation. Therefore, we follow previous work and formulate them as MCQs [1,2].

[1] Can Large Multimodal Models Uncover Deep Semantics Behind Images?
[2] Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest

Q2: The diversity of the benchmark

Thank you for raising this good question! The comics in our benchmark encompass a diverse range of everyday life scenarios. To ensure and analyze the topic coverage, we prompted ChatGPT to generate topical keywords for each comic based on its description and then clustered these keywords. The complete clusters and their statistics are provided in the one-page PDF of the Overall Response. The results indicate the diversity of our benchmark. We will incorporate the analysis in our revised version.

Q3: Would the prompts play crucial roles in models’ decision making?

To address the potential influence of prompts on model performance, in the experiments we created three distinct prompts for each task and reported the average scores from three runs with each prompt. The specific prompt sets are detailed in Appendix C.3. Our initial observations indicate that different VLMs exhibit varying degrees of sensitivity to prompts (for example, commercial models are less sensitive to the prompts than smaller VLMs); however, the overall performance gap across different prompts is not significant. We will revise the relevant section to clarify this point more effectively.

Q4 : Why GPT-4 can engage in the annotation as well as the model comparison? Will that involve any bias?

This is a good question! To clarify, we use different GPT variants for distinct purposes: For data annotation, we leverage the gpt4-turbo variant; for experiments, we report results using gpt-4-vision-preview; and for GPT-based automatic evaluation on text generation tasks, we employ gpt-3.5-turbo-0125. While we do use different GPT models variants, there might still be some inherent bias since we do not know the backend model details. We will revise our paper to better clarify this distinction and acknowledge the potential for bias.

Thank you for your valuable and constructive comments. We appreciate your recognition of our work's innovative benchmarks, human-AI collaborative annotation, and insightful analysis. Below, we address your concerns and questions individually:

W1 & W5: Regarding Novelty and Suitability

Thank you for the feedback! However, we believe our work is uniquely novel and suitable for this venue. First, our benchmark formulates a new task to address humor understanding through narrative contradictions, a complex and underexplored aspect of AI research. This task challenges models to integrate multiple fundamental abilities including the comprehension of human norms, critical thinking about similarities and differences of elements, and nonlinear reasoning, which are critical for developing socially intelligent systems. Reviewers aFj6 and yTj4 have recognized these contributions, which we will clarify further in our revision. In addition, our work extends beyond a dataset introduction with comprehensive evaluations and detailed analyses, providing valuable insights into AI's interpretive and reasoning capabilities, making it highly relevant for broader AI and NLP communities. As a pilot study on this topic, we believe our work is suitable for the venue and can bring valuable insights for future research.

W2: Limitation of benchmark

juxtaposition may not be broadly applicable

This is a good point. Juxtaposition is a common and sophisticated technique with applications in domains such as arts, literary, and mathematics [1]. As a pilot study, we consider it as a suitable entry point for evaluating complex narrative understanding for AI models.

The task of matching comics with titles involves a high degree of subjectivity

Thank you for this important question. Indeed, humor understanding can be subjective. To address this, we have formalized title matching as a selection task rather than a generation task (i.e., determining which title is "objectively" better). To ensure a "common preference," we conduct multiple quality checks and verifications during annotation, where each sample is reviewed by different annotators to reach an agreement (Line 165). We will clarify this in our revised version.

The dataset consists of only three hundreds of comics

Please see the Overall Response, point 1.

[1] https://en.wikipedia.org/wiki/Juxtaposition

W3 & Q1: Regarding bias and subjectivity in annotation

The annotation process might introduce bias and how did you address the subjectivity

Please see the Overall Response, point 2.

Did you measure inter-annotator agreement

As each component was annotated and verified by at least three annotators to reach agreement, we did not further measure inter-annotator agreement.

W4 & Q4: Regarding the weakness and question of evaluation metrics

Please see the Overall Response, point 3.

Q2: Regarding the quality of negative titles and philosophies

How did you ensure that the negative titles and philosophies were sufficiently challenging yet distinct from the correct ones

Thank you for raising this important question! The negative titles and philosophies are constructed by human annotators. For each sample, we first prompt GPT-4 to generate negative candidate options (as shown in Table 4). The annotators then revise and edit these options to ensure they are sufficiently challenging yet distinct from the correct ones. During the quality check stage, annotators will further verify the quality. As illustrated in Figure 7, all negative options are on-topic and relevant to the comic but may contain incorrect logic or do not accurately reflect the narrative. This process ensures quality of the negative options.

The negative title "Graphs Don't Lie, People Do" in the example of Figure 2

This is a very good point! The comic itself illustrates how the interpretation and presentation of data can be manipulated or selectively used to tell misleading stories, while the graphs and data are factual and real. Therefore, the title implying that people lie is not appropriate.

Q3: Regarding the question of cultural or context bias

This is a very insightful question! In our current study, the annotations are based on the common interpretation of humor, and our evaluations focus on how model performance aligns with such average preferences, without consideration of specific contextual or cultural information. However, we acknowledge the critical importance of considering cultural and contextual variations in humor understanding. We will explore this aspect in our future research. Additionally, we will include a discussion of these considerations in the Limitations section of our revised version.

Regarding the raised ethical consideration

Thank you for highlighting these important concerns. To address them, we have included a comprehensive ethics statement in our paper (Section A in appendix). Below we outline the key points:

Data privacy, copyright, and consent

All data samples are sourced from publicly available content on social media platforms. We strictly adhere to copyright laws by using original links to the comics, thereby avoiding any infringement. We will provide the original links of each image when releasing the benchmark.

Discrimination, bias, and fairness

We have conducted a thorough review and rigorous quality checks of our samples to filter out potentially offensive or harmful content. Our annotators come from diverse cultural and gender backgrounds, and we conduct multiple quality checks and verification to minimize bias.

We will revise our paper to better clarify these points and provide reassurance regarding the ethical considerations of our work. We will also incorporate more detailed instructions to guide users in adhering to copyright regulations, and encourage users to carefully consider the ethical implications of the generated outputs.

Dear Reviewer,

Thank you once again for your valuable review. With the discussion deadline approaching, we would like to ask whether our response has adequately addressed your questions. If there are any outstanding issues, we would like the chance to respond before the discussion period is over.

Thanks again for your thoughtful review!

Best regards,

Authors