MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmark

Q5. MMMT-IF Adaption to Single-Modality Tasks

The reviewer asked if the MMMT-IF benchmark can be adapted to fit into single-modality tasks.

The instruction following task can also work for a multi-turn task without image inputs. We include image inputs as we believe this to be an important avenue where there is a lack of instruction following benchmarks. Our approach for extending multi-turn benchmarks may be used to extend other benchmarks not using multi-modal data.

One example would be to have a question answering task, but instead of basing questions on a set of images, you could base the questions on a category, such as questions about a certain country or region.
The multi-turn aspect is important, as one of our main goals is to examine the instruction following capabilities when multiple instructions are given at different places in the input context.

Q6. Impact of Model Performance on Specific Instructions

The reviewer asked how the complexity of individual instructions impact model performance and if categorizing the instructions would lead to better insights.

The complexity of the individual instructions does play a certain role in the model performance. We do categorize the instructions into 6 categories, where instructions in one category will not contradict instructions in a different category. The categories are ‘response length’ (a restriction on the number of sentences), ‘sentence start character’ (a restriction on the start character in each sentence), ‘favorite word’ (a requirement on a specific word being present in the response), ‘sentence length’ (a restriction on the length of each sentence in the response), ‘sentence end character’ (a requirement on the last character in each sentence), and ‘number’ (a requirement that the response should contain either an even or an odd number).

We ensure that only one instruction per category is in each conversation. In Table 6 (page 17) of our submission we show all the given instructions. Figure 10 in the Appendix shows the PIF metric conditional on a given instruction.

Q7. Training Adjustments to Better Handle Instruction Retrieval and Adherence

The reviewer asked what are potential training adjustments needed for models to better handle instruction retrieval and adherence.

This will be the focus of future work. One broad idea is to generate training data with a reward signal corresponding to the programmatic instruction following metric on responses, and then use Reinforcement Learning from Execution Feedback (RLEF) to improve the model performance on the task. This may be less costly than RLHF as we would not need to collect human preference data.

Q8. Architecture Choice Influence on the MMMT-IF Benchmark

The reviewer asked how specific architectural features influence model performance on the MMMT-IF benchmark.

Context window length will matter significantly if the context window is smaller than the input context, as that would truncate the input context. In this work, we have decided to focus on uncovering an important weakness of large language models, and in future work we hope to propose different architecture adjustments and training data curation to improve performance on this task.

Response to Reviewer BesE

Thank you for your thoughtful review and appreciation of our work. We respond to your comments and questions below.

W1/Q1. High Task Difficulty

The reviewer pointed to the significant drops in PIF scores as making the task highly difficult, questioning if the benchmark may currently be too challenging.

We think that a task with somewhat low performance is an advantage, as there will be significant headroom for models to improve on this task, and the task may be relevant for a longer time in the future. We anticipate that models will improve on the task once multi-turn instruction following becomes more prioritized during training.

W2/Q2. Computational Cost of MMMT-IF Benchmark

The reviewer pointed out that the evaluation cost to benchmark the MMMT-IF dataset may be computationally demanding for researchers with limited resources.

We provide an approximate analysis of the API cost to evaluate the OpenAI GPT-4o on our benchmark. We have around 1000 turns, the average input context is around 1500 tokens and the output context is on average less than 200 tokens. This gives a total number of 1.5 million input tokens and 0.2 million output tokens.

For the OpenAI API, using GPT-4o, the cost is 2.5 dollars/1M input tokens, and 10 dollars/1M output tokens, hence an upper bound to evaluate GPT-4o on our benchmark is 6 dollars. We think that this would be affordable by a majority of researchers, but we also recognize that this may be limiting for some. Inherent in long context benchmarks is processing long sequences, which is costly, and this issue will be the same for other benchmarks relying on long context.

Q3/W3. Overfitting on Benchmark Task

The reviewer pointed out that the specific and potentially complex instructions may lead to overfitting on the benchmark tasks, that would perhaps not generalize beyond the benchmark.

By having a diverse set of instruction categories, and within each category having several instructions, we hope to mitigate these issues. Even if overfitting on the specific instructions would happen, the task would still likely remain challenging as to correctly answer a question, all instructions still needs to be retrieved from the input model context.

Q4. PIF Metric Scaling to Larger Datasets/Models

The reviewer asked about how well the PIF metric scale to larger datasets/models, and asked for limitations in applying the PIF metric to real data.

The computation of the PIF metric on an instruction following dataset is linear in the number of samples in the dataset. One limitation in applying the PIF metric to real data is that the dataset needs to contain instructions that are objective, i.e. can be accurately be checked by code execution. For example the instruction: The answer should start with the letter ‘A’ can be objectively checked by code execution, but the instruction ‘Write the answer in a positive tone’ would be hard to objectively verify by code.

Response to Reviewer GPsv

Thank you for your thoughtful review and comments, which have made our work stronger. We respond to your comments and questions below as well as in the common response.

Q1/W1. Instruction Verification Through Code Execution

The reviewer asked for clarification on how instructions are verified through code execution and asked for a concrete example of how the instructions are checked.

A chat can be represented as I1, Q1, A1, …, In, Qn, An, where Q1 is the question 1, I1 is the first instruction and A1 is the first answer. We compute the PIF score for turn n, by using all given instructions: (I1, …, In), and An. We then execute code to first identify each instruction given, and then check if that instruction is followed in An. For instance, suppose I1= “Instruction: Use the word 'like' at least once in all future responses.”, and An = “I like ice cream”, then the following code checks if I1 is followed in A_n:

def check_word_in_answer(word, answer): 
      """ Checks if the specified word is present in the answer. 
      Args: 
         word (str): The word to look for in the answer. 
         answer (str): The response text to check. Returns: bool: True if the word is found in the answer. 
      """
      word = word.lower() 
      answer = answer.lower() 
      return word in answer.split()

We chose all instructions so that the correctness can be objectively checked by code execution.

Q2. Precise Meaning of “Instruction”

The reviewer suggested clarifying the meaning of “instruction”. In this paper, we define an instruction as a constraint on the answer format.

Q3. Inter-Annotator Agreement Statistics

The reviewer asked if we had numbers about the inter-annotator agreement statistics, and asked for more information on the evaluation protocol. Our setup has one annotator rate each dialogue, with different annotators in different dialogues. We are working on collecting multiple annotators per dialogue for a limited subset of the dataset to analyse the inter-annotator agreement in more detail.

We provide the full instructions given to the annotators in appendix G of the paper. If there are some specific aspects of the evaluation protocol that are unclear, we would be happy to elaborate further.

Response to reviewer TUxw

Thank you for your review and thoughtful comments. We respond to your comments and questions below as well as in the common response.

Q1/W1. Concerns About MMMT-IF Being an Augmentation of MMDU

The reviewer noted that MMMT-IF is an augmentation of the MMDU dataset, suggesting that this limits the novelty of the work. We think that the setup where MMMT-IF is created based on an existing dataset is actually a strength rather than a weakness, as MMDU and MMMT-IF are testing different model capabilities, and are hence used for different tasks. MMDU is a question answering dataset, and MMMT-IF is an instruction following benchmark. Another strength with using an augmentation approach is that this augmentation strategy can be used across a range of other datasets (rather than just MMDU) to create additional datasets. Our general response talks in more depth on how the extension is made.

Q2/W2. Lack of Open-Source LLMs

The reviewer suggested that open source LLMs should be included in the benchmark. We thank the reviewer for this suggestion, we will include an experiment comparing with LLaMA in the camera ready version of the paper.

Q3. MMMT-IF Performance for Models of Different Size

The reviewer suggested that we examine the effect of model size on long term instruction following. For the camera ready version, we will include an experiment with different Gemini model sizes to better understand the scaling in the model size.

Example of Extension

Reviewer TUxw asked if we can show a concrete example of the extension, and reviewer GPsv asked us to include a complete example in the main text. We now show the example that we will include in the main text.

Here is a short example (using Gemini 1.5 Pro for the response) that will be added to the main text in the paper: For turn 1 in the chat we have that I1 = “Include at least one even number bigger than 5 in each of your responses.”, Q1 = “What activity are the individuals in the Image1 engaged in, which is characterized by rhythmic and fast-paced footwork?“, and A1 = “The individuals in Image 1 are performing Tinikling, a traditional Philippine folk dance characterized by rhythmic and fast-paced footwork, often using bamboo poles. Dancers need a lot of agility to avoid getting their feet caught between the poles, especially as the pace increases to 10 or 12 beats per second.”

In the original MMDU dataset, Q1 was the same, but it did not include the I1 instruction. At turn 15 in the same chat, all the instructions added up until that point were:

“Instruction: Include at least one even number in the range bigger than 5 in each of your responses. Instruction: End every sentence with a exclamation mark (!). Instruction: Use the word 'itself' at least once in all future responses. Instruction: Only use responses to question where each sentence in the response is at most 18 words in all future responses. Instruction: Make all the following responses no more than 4 sentences. Instruction: Start every sentence with the letter (S).”

Given the input context: I1, Q1, A1, …, I15, Q15, the Gemini model produced the following answer, where Q15=”Identify the possible significance of the props or lack thereof in the settings of Image1 and Image2.”, with answer A15= “Stage props can enhance the narrative or symbolism of a performance. Simple props, like the bamboo poles in Image 1, become integral to the dance itself. Conversely, the absence of props in Image 2 might emphasize the dancers' movements and stage presence. Such choices reflect deliberate artistic decisions in conveying a performance's message.”

Note that the answer at turn 15 needs to follow the instructions from all previous turns.