MMA: Benchmarking Multi-Modal Large Language Model in Ambiguity Contexts

This paper introduces a benchmark for evaluating multimodal large language models’ (MLLMs) understanding of ambiguous language. The benchmark contains 261 questions and 522 natural or generated images covering lexical, syntactic, and semantic ambiguities. The authors evaluated 16 state-of-the-art MLLMs and found significant performance gaps between MLLMs and human accuracy. While MLLMs handle lexical ambiguities relatively well, they struggle with other types. Detailed evaluation using text-only data reveals that this performance gap is primarily due to the models’ limited ability to effectively integrate multimodal information.

The paper "MMA: Benchmarking Multi-Modal Large Language Models in Ambiguity Contexts" introduces the MMA benchmark, specifically designed to assess how Multi-Modal Large Language Models (MLLMs) handle ambiguous contexts. MMA presents questions associated with pairs of images that imply distinct scenarios, testing MLLMs' ability to resolve ambiguities through visual cues. The benchmark categorizes ambiguities into lexical, syntactic, and semantic types and evaluates 24 MLLMs, finding that while humans achieve an accuracy of nearly 89%, MLLMs struggle, achieving only around 53.22%. The results reveal that MLLMs are particularly challenged by syntactic ambiguities, with open-source models generally performing worse than proprietary ones. This study highlights areas for improvement in MLLMs’ integration of visual and textual information to handle ambiguity effectively.

This paper presents MMA, a benchmark for evaluating Multi-Modal Large Language Models (MLLMs) in ambiguous contexts. It uses a multiple-choice visual question-answering format with 261 questions and associated pairs of images representing different scenarios. The benchmark categorizes ambiguities into lexical, syntactic, and semantic types. Through experiments on 24 MLLMs, it shows that models often overlook image information, perform better on lexical ambiguity, and that proprietary models generally outperform open-source ones.

This paper introduces a designed to evaluate how well MLLMs handle ambiguity in language when provided with visual context. The main distinguished design is to provide the same text with two different visual contexts, and count the models correctly answer only if both answers are correct.

The benchmark consists of 261 (?) questions in total, each paired with two different images that suggest different interpretations of the same ambiguous text, requiring models to leverage visual information to disambiguate meaning. The questions are categorized into three types of ambiguity: lexical, syntactic, and semantic. The authors evaluate various MLLMs, including both proprietary and open-source models, on their ability to correctly interpret ambiguous questions when given different visual contexts.

I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.

Thank you to all reviewers for acknowledging the novelty and interest of our benchmark. We appreciate your feedback and provide the following responses to address your concerns:

Q1: The dataset size is relatively small, particularly for certain ambiguity subcategories.

A：We acknowledge that the limited dataset size could be seen as a potential drawback of our benchmark, but we respectfully believe that it does not significantly impact our main contributions.

1. Highlighting Potential Improvements for MLLMs: Our benchmark is the first to systematically evaluate the ability of MLLMs to leverage visual information to clarify language ambiguities. The comprehensive evaluation clearly demonstrates the significant performance gap between MLLMs and humans (e.g., 3.5 Sonnet achieving 74% vs. human 89%, MLLMs average 53% vs human 89% ) in leveraging multi-modal information to resolve ambiguities. Identifying these areas for improvement is a critical step forward in advancing the field and can inspire more subsequent research efforts.
2. Data Representativeness as a Key Factor: We acknowledge the concern regarding the limited dataset size in certain categories. However, we believe that the representativeness of the data is the key to a meaningful benchmark. Even with a smaller sample size, a benchmark can still serve as a robust standard if the data is well-representative. For example, MM-Vet [1] with only around 200 test samples, remains a widely recognized evaluation benchmark. Before GPT-4o and Claude 3.5, no other MLLMs had achieved over 70% accuracy on the MM-Vet benchmark, even 3 quarters after its release. This underscores the value of well-crafted benchmarks, regardless of dataset size, in driving progress and setting high standards for model performance. In the future, we plan to expand our benchmark, similar to MM-Vet v2, to continue fairly evaluating future state-of-the-art MLLMs.

Reference [1] Weihao Yu, et al. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. ICML 2024.

Q2: Missing the annotation protocol of human baseline. The authors should provide a thorough description since it's the basis of almost all the analysis.

We acknowledge the importance of this protocol as the foundation for our analysis and provide the following detailed explanation:

1. Annotator Recruitment: We recruited five annotators with native or near-native proficiency in English, verified at the CEFR C1 level or above. This ensures a strong command of linguistic nuances, including those relevant to resolving textual ambiguities.
2. Task Overview: Annotators were presented with the complete MMA benchmark, which includes 261 multiple-choice questions, each paired with two images depicting different contexts.For each question, annotators were asked to select the most contextually appropriate answer based on the visual and textual information provided.
3. Guidelines Provided to Annotators: Annotators were given clear instructions emphasizing the need to integrate both visual and textual contexts to resolve ambiguities.They were explicitly instructed to avoid over-relying on either modality and to ensure their answers were consistent with the intended interpretation of the ambiguities.
4. Evaluation Process: Annotators independently completed the task without collaboration or external assistance. Responses were recorded in a standardized answer sheet, ensuring consistency across all participants.

Q3: Table 4: I am unclear about the motivation behind this table. For instance, in the lexical category, items should ideally have two distinct interpretations. Correctly answering one interpretation does not imply the model has accurately handled the ambiguity by answering the other interpretation. Could you elaborate on the intended purpose of this analysis?

A: The text-only evaluation was conducted for evaluating potential bias from human-crafted questions. By examining performance on text-only input, we aimed to investigate whether the performance gap observed in multimodal settings could be attributed to the inherent complexity of the tasks or biases introduced by human-crafted questions. High accuracy in text-only tasks suggests that the textual component alone does not hinder performance.