Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping

W2: Concerns of significant time and computational resources cost when producing new samples.

Our detailed experimental content may have led to your misunderstanding about the time and resources cost of our framework. In reality, our framework is time-friendly and resource-efficient.

(1) About the time consumption: Actually, it is very quick and convenient to generate a new sample via our framework, both visually and linguistically. We hope the specific average time consumption would address your concern. In specific, for the visual modality, as shown in Figure 4, creating a new image mainly requires the GPT and image editing modules. The average time to call the GPT function is 4.3937 seconds, and the average time to generate an image by PowerPaint is 2.4267 seconds. For the language modality, generating a new sample only requires one call to GPT or one rephrasing method, with the average time being 5.1965 seconds. Therefore, our proposed framework does not take a long time, instead, it is very quick and convenient.

(2) About large-scale evaluations: As mentioned in the Section 5.1, we utilize the integrated platform EvalKit for all evaluation, which is a highly convenient and standardized evaluation tool. It integrates various types of benchmarks and mainstream LVLMs. Therefore, once we have prepared the dataset, it only takes a single command line to run the evaluation by EvalKit, requiring little time and resources. Our extensive experiments may lead to your misunderstanding of heavy evaluation, but the entire evaluation process is not cumbersome but rather systematic and well-organized.

(3) About computational resources: Our VLB mainly involves leveraging three tools (i.e., GPT, Powerpaint and SAM. Among these, GPT does not require GPU resources. SAM and PowerPaint can be run on low-specification GPUs. With these powerful tools, our VLB requires low resources to generate high quality new samples. What's more, as mentioned in (1), generating a new sample requires only few seconds.

W3: The VLB relies on PowerPaint, SAM and GPT-4V to generate new samples.

(1) The reason why we chose these models. The utilized tool models like PowerPaint, SAM, and GPT-4V are all currently the undisputed state-of-the-art in their respective fields. Therefore, to ensure the accuracy and consistency of newly generated datas, we integrate these sota and effective models into our unified framework VLB.

(2) These models are replaceable. Any model with similar functions can be substituted. Moreover, as models of this type continue to evolve, our VLB framework can also evolve to become better. and our framework is compatible with models of similar functionality.

(3) To show the compatibility of our framework, we conduct an ablation study, taking image editing model as an example. We selected models embodied similar functionality with PowerPaint, namely Brushnet[3] and Stable Diffusion-outpainting[4], to implement the v1 and v3 strategies for generating image samples. The evaluation results of different image editing models are shown in the table below.

Table A. The evaluation result of our VLB framework utilizing different image editing models on SEEDBench.

As can be seen from the table, although there are minor numerical differences between the results from PowerPaint and other editing models, their trends are similar. Therefore, our framework is verified to be adaptable to many tool models with similar functions.

Q3: sometimes there will be some artifacts after using PowerPaint to remove an object, how does the DME ensure the fairness and accuracy of the evaluation.

(1) First, we want to explain for you the reason why we chose these models. The utilized tool models like PowerPaint, SAM, and GPT-4V are all currently the undisputed state-of-the-art in their respective fields. Therefore, to ensure the accuracy and consistency of newly generated datas, we integrate these sota and effective models into our unified framework VLB.

(2) About some artifacts after using PowerPaint: On one hand, Currently, image editing models have become quite advanced and are widely used. PowerPaint inherently produces fewer artifacts and is among the one of the best-performing models. The original paper of PowerPaint [13] has already demonstrated its reliability in the editing process through comparisons of CLIPScore and Local-FID for its object addition and removal capabilities. On the other hand, during our use of PowerPaint, we continued to uphold its standard for quality control. For strategy v1 and v2, we also have adopted the CLIPScore and Local-FID to set a quality threshold of 25.00 and 13.50 to evaluate the alignment of generated visual content with object name. For strategy v3, we have adopted aesthetic score for selecting outpainting images with pleasing scenery extending content.

(3) About the fairness and accuracy of our evaluation: In our practical experiment, for each image and language strategy, we set up five random seeds, generating five data variants. Therefore, all experimental results are the average of these five variants. Due to page limitations, we did not display the results of all five variants in paper. Below we provide the mean and standard deviation for each strategy across the five variants for you.

Table D.Mean and standard deviation (in parentheses) of accuracy across different strategies and datasets on LVLMs.

As can be seen from the table, the standard deviation in each strategy is slight, thus the variance scale caused by the randomness of GPT-4V and PowerPaint is also minimal. These demonstrate the reliability of the average accuracy result reported in our paper.

(4) As a dynamic evaluation framework, our primary focus is on ensuring the consistency with the original. Therefore, the core issue we address is whether the newly generated samples produce reliable content while maintaining the original answers unchanged. The specific pixel quality of the images generated is not our main concern. Moreover, as shown in W3, we have taken measures to ensure image quality and have selected high-quality images for our final evaluation dataset.

Dear Reviewer jqNx,

Thanks for your thorough review and comments on our work. Below, we will clarify your doubts and address your concerns on each point.

W1: Concerns of the complexity increase of implementation and more unexpected errors or inconsistencies caused by each step, such as GPT-4V, PowerPaint, SAM.

Thanks for your comment, and this point is also one of our key focuses. Therefore, we have taken methodological measures to maintain consistency, and we have conducted extensive experiments to demonstrate the high credibility of generated data. On the one hand, we have indeed taken three mechanisms to ensure the consistency of the generated data, but these measures are not as complex as you imagined. We hope the illustration in W2 would address your concern for the complexity of our framework. On the other hand, as can be seen from our experimental result, our method contains few unexpected errors.

Now we will explain for you about the three mechanisms we take to maintain consistency and avoid unexpected errors:

(1)GPT instruction: For strategies v1 and v2, when utilizing GPT4v for generating bounding-box and mask index, we have made strict and serious prompts for it, as show in Appendix A1. In specific, we input the original image, question, and answer into GPT and emphasize that added the object and removed object must not change the answer-related visual content. The detailed instructions, input content , and output format are also detailed in Table 4. For strategies L3 and L4, when generating context with GPT, we also input the original image, question, answer, and strict instructions. The instructions emphasize GPT to generate relevant captions or irrelevant context, and forbid GPT directly revealing the answer.

(2) Judge module: To further ensure consistency between the original sample and the dynamic variant, we incorporate a judge module as an automated verification technique. As shown in the Section 4.4, the judge module can adversarially select images and questions that meet our modification requirements. Here we designed different judge prompts to check each bootstrapping strategy, detailed prompts are in Appendix A2. Similar to MPA[1], the judge operates in an adversarial manner and returns a 'Yes' or 'No' verdict. If the response is `No', the sample will be regenerated until it passes the judgement. If the new sample does not pass after five attempts, the original sample is used instead. We also provide you a detailed ablation study of judge module in Q2.

(3) Human verification: What’s more, we further conduct a human verification to validate the effectiveness of our judge module, and to ensure that our variants maintain consistent with the original benchmark both visually and linguistically. As shown in Appendix A5, we recruit 20 human experts (with bachelor or higher degree) and randomly selected 100 samples from each strategy for each benchmark, totally 2100 samples. The result in Table 11 showcases a high level of alignment between the judge model and human verification, proving the equivalence and correctness of our dynamic methodology.

Dear Reviewer jqNx,

Thank you for the precious review time and insightful comments on our paper. We have provided corresponding responses and results in great detail. If you have any other concerns, we are more than happy to provide additional clarification as well as experiments at any time. Sincerely looking forward to your reply!

Best,

Authors

Dear Reviewer 7UPF,

Thank you very much for appreciating our paper as clear and well-written. Your constructive suggestions are so valuable to us. We are honored to have the opportunity to clarify the points you raised. Below is our detailed response to answer your concerns.

W1: Concern regarding the veracity of the VLB-modified data, and consider additional automated verification techniques.

We are lucky to have met such a rigorous reviewer like you, this point you raised is also something we consider very important. Therefore, We will now explain from two aspects on how our work ensures if the original test cases have been loyally modified in the way we intended.

（1）In fact, we have implemented three safeguard mechanisms to ensure the consistency and modification loyalty between the origianl sample and our dynamic sample. But sorry for that we may not summarize these three mechanisms together although we have described them in our paper, which might not impress you enough. So below we will exlpain the three safeguard mechanisms for you:

• GPT instruction: For strategies v1 and v2, when utilizing GPT4v for generating bounding-box and mask index, we have made strict and serious prompts for it, as show in Appendix A1. In specific, we input the original image, question, and answer into GPT and emphasize that added the object and removed object must not change the answer-related visual content. The detailed instructions, input content , and output format are also detailed in Table 4. For strategies L3 and L4, when generating context with GPT, we also input the original image, question, answer, and strict instructions. The instructions emphasize GPT to generate relevant captions or irrelevant context, and forbid GPT directly revealing the answer.

• Judge module: To further ensure consistency between the original sample and the dynamic variant, we incorporate a judge module as an automated verification technique. As shown in the Section 4.4, the judge module can adversarially select images and questions that meet our modification requirements. Specifically, The judge module is informed what modification is conducted, and is asked to check whether the answer is still correct for the newly generated image or question. Here we designed different judge prompts to check each bootstrapping strategy, detailed prompts are in Appendix A2. Similar to MPA[1], the judge operates in an adversarial manner and returns a 'Yes' or 'No' verdict. If the response is `No', the sample will be regenerated until it passes the judgement. If the new sample does not pass after five attempts, the original sample is used instead. In practice, we use InternVL-2 pro, an powerful open-source LVLM as a capable and affordable judge.

• Human verification: What’s more, we further conduct a human verification to validate the effectiveness of our judge module, and to ensure that our variants maintain consistent with the original benchmark both visually and linguistically. As shown in Appendix A5, For each dynamic strategy, we recruit 20 human experts (with bachelor or higher degree) and randomly selected 100 samples from each strategy for each benchmark, totally 2100 samples. The specific details are in Appendix A5. And the result in Figure 11 showcases a high level of alignment between the judge model and human verification, proving the equivalence and correctness of our dynamic methodology.

Dear Reviewer 7UPF,

Thanks very much for your careful attention and response! We are lucky to have met such a rigorous reviewer like you! And we take every comment you made very seriously.

As you mentioned, in our human verification process, it is observed that the V1 strategy leads to a much higher fail-to-detect rate before passing the judge module. We thought this is primarily because, in a small portion of generated images, the addition of objects causes partial occlusion of the core objects related to the question. In order to give you a more intuitive understanding, we have updated the revised paper and added visualizations of failed cases in Appendix Figure 15. We selected two fail cases from the MMBench V1 strategy, with the types of being attribute comparison and celebrity recognition, respectively. As can be seen in figure 15 (a), the added laptop covers most of the cat, thus affecting the comparison of animal sizes. In figure 15 (b), the added balloon slightly obscures the person's face, which might impact the recognition of the person. Therefore, images with partial occlusions like those are filtered out by the judge module, which we thought primarily reflects the rigor and effect of the judge module, and may not imply potential biases. Your comments are very revealing, and we will try our best to meet your high standards.

We are deeply grateful for your interest in our work and for your valuable comments, which have been instrumental in refining our paper. It is truly our luck to discuss with such a meticulous and knowledgeable reviewer as you!

Dear Reviewer RXK7,

Thank you for your valuable comments. We appreciate the chance to clarify and address your concerns, which we believe will enhance our paper. Below, we respond in detail to each point you raised:

W1: The scale of the performance variance caused by randomness in GPT-4V and PowerPaint.

Thanks for your deep thought, we also considered it important this and have taken some measures. In our practical experiment, for each image and language strategy, we set up five random seeds, generating five data variants. Therefore, all experimental results are the average of these five variants. Due to page limitations, we did not display the results of all five variants in paper. Below we provide the mean and standard deviation for each strategy across the five variants for you. We value your comment and have included the table in our revised paper.

Table A. Mean and standard deviation (in parentheses) of accuracy across different strategies and datasets on LVLMs.

As can be seen from the table, the standard deviation in each strategy is slight, thus the variance scale caused by the randomness of GPT-4V and PowerPaint is also minimal. These demonstrate the reliability of the average accuracy result reported in our paper.

W2: The consistency remains with composition of multiple strategies.

Thanks you for your careful consideration. We indeed conducted human verification only for single strategy, as we thought that if both images and texts individually ensure consistency with the original one, then the combination of them should also be consistent with the original pair.

However, we value your opinions, so we selected two compositions(V1+L4, V1+V3+L4) of SEEDBench to conduct a same human verification as single strategy. In specific, we randomly select drew 100 samples from each composition, together with the original 100 sample, totaling 400 VQA samples. Similarly as Appendix A5, we recruit 20 human experts(with bachelor or higher degree), presenting them both the multi-strategy and original samples, and asked them (1) whether these two samples maintained consistency, and (2) whether the two samples corresponded to the same correct answer. We required them to respond with only 'yes' or 'no' . Experts independently reviewed samples. Once all experts completed their evaluations, a voting process ensued. A sample was only considered to have passed if more than half of the experts deemed it consistent with the original. The results are displayed in the table below.

Table B. The human verification result of multiple strategies on SEEDBench.

From the table, there is a slight decrease in consistency as the number of compositional strategy increasing. But compared to the considerable decrease in LVLM accuracy shown in figure 6, this decrease is minimal and can be somewhat disregarded(about average 36% decline). Therefore, our conclusion remains reliable that the composition of multi-strategies poses a greater challenge for LVLM.

W2: The difference and overlap of our proposed linguistic rephrasing strategy with those other benchmarks.

Thanks for your valuable comment, we are lucky to have met such a rigorous reviewer like you. We have carefully read the works you mentioned and analyzed the overlap and differences between our rephrasing strategy and these methods one by one. Furthermore, we value your comment and have included these works in the related work of our revised paper.

(1) Compared to the VQA Rephrasings dataset[3]: The VQA Rephrasings dataset is collected by manual rewriting based on the static VQA v2.0 dataset. This manual rewriting process is not only time-consuming and labor-intensive, but is also unable to extend to other benchmarks automatically. In contrast, our rephrasing strategy is a fully automatic process, allowing us to effectively generate new rephrased questions based on various existing multi-modal benchmarks.

(2) Compared to the VQA-LOL dataset[4]: The VQA-LOL dataset can only transform questions in the format of yes/no, because it used logical compositions (negation, disjunction, conjunction, and antonyms) to generate question and answer. Only in yes/no questions can they obtain the correct paired answers through logical compositions. However, yes/no questions only represent one format of multi-modal benchmarks for LVLMs. More formats like multiple-choice and VQA have been proposed for evaluating LVLMs. In contrast, our rephrasing strategy l2 ‘role-playing’ has broader applicability and can be used across various question types including yes/no, multiple-choice, and VQA.

(3) Compared to the VQA-Subquestions dataset[5]: The VQA-Subquestions dataset has the above two limitations. Each sub-question in the VQA-Subquestions dataset is generated by three unique workers, and they hired a total of 463 workers to complete the annotation tasks. This process is time-consuming and labor-intensive, and is hard be applied to other datasets. What's more, the VQA-Subquestions dataset also merely includes yes/no questions, lacking transferability to other question formats. In contrast, our rephrasing strategy can effectively address the above two limitations.

(4) Most importantly: Our rephrasing strategy better reflects our perspective of user interactions. We simulate how different users with distinct identities or educational backgrounds might pose questions when using a VLM. It aligns more closely with real-world scenarios and facilitates the practical application of LVLMs.

W3: Can the framework also handle capabilities that have to be evaluated without VQA？

Thanks for your constructive suggestions. Since our method can dynamically transform both images and text to generate new samples, we believe our framework can also be applied to other tasks that VLMs can perform. Below, we conduct experiments on a common multi-modal task of LVLM: image caption.

Firstly, we select the famous image caption benchmark, COCO Caption[6], and randomly sample a 100 pairs subset. Secondly, we use our three visual strategies to generate new images on COCO Caption: adding new objects(v1), removing existing objects(v2), and outpainting(v3). Next, for each newly generated image, we use GPT-4V to generate corresponding captions, composing new <image, caption> pairs. Finally, we evaluate the original and the three dynamic COCO Caption datasets on 3 popular VLMs. The results are shown in the table below.

Table A. The result of image captioning task about the original, and V1, V2, V3 strategy in COCO-Caption.

As can be seen from the table, our newly generated samples achieve similar results to the original dataset. And similar conclusions can be drawn, namely that strategy V2 is simpler than V1 and V3. This demonstrates that our dynamic framework can also effectively be applied to other multi-modal tasks, allowing a more flexible and comprehensive evaluation of LVLMs.

Dear Reviewer Rays,

Thank you for appreciating our paper as comprehensive and identifing an important research direction, your constructive comments and suggestions are valuable for us. Below is our detailed response to clarify the points and answering the concerns you raised.

W1&Q1: The detailed visual attention and linguistic understanding strategy converted into V1, V2, L1, L2.

Thanks for your interest in the details of our dynamic strategy. I’m honored to explain the conversion details for you. Take Figure 3 as an example, below is how we implement user interaction changes from both visual attention and linguistic understanding perspective. Figure 3(a) has been modified accordingly.

The user interaction with LVLM is defined by the cognitive process through which users comprehend the image and ask questions of their interest in the VQA setting. This process is influenced by visual attention and linguistic understanding.

Specifically, Visual attention affects the way how individuals selectively focus on specific visual elements within an image or scene while filtering out less relevant information. Since different users pose various identities and backgrounds, their levels of visual attention[1] also vary. For instance, children's attention might be more easily distracted by irrelevant objects in images than educated adults. We categorize visual attention with concentration and distraction. The attention concentration is implemented by object removal while distraction is implemented by object addition and image outpainting.

Linguistic understanding indicates how individuals interpret and comprehend questions. This process influences the language proficiency of different users. Specifically, when different users utilize LVLMs, they may have different linguistic expressions towards a same question due to their distinct identities or educational backgrounds. Therefore, as shown in Figure 3(a), our work conducts transformation on the original question from three linguistic levels[2]: word-level (L1), sentence-level (L2), and context-level (L3, L4).

Here we detailedly explain the example in Figure 3(a) for you:

(1) Visually, as described in section 4.2, since different users pose various identities and backgrounds, their levels of visual attention[1] also vary. For instance, children's attention might be more easily distracted by irrelevant objects in images than educated adults. Therefore, as shown in Figure 3(a), we simulate the distraction, simplification, and expansion on visual attention for image bootstrapping, respectively corresponding to our three strategies : adding new objects(V1), removing existing objects(V2), and expanding original images(V3).

Specifically in figure 3(c), compared to the original image, we added a plant, removed a wall, and outpainted the image with a ratio of 1.5, respectively, obtaining three images after visual dynamic strategies V1,V2,V3. **(2) LiLinguisticallyas described in section 4.3, we also simulate the language usage on LVLMs from different users. Specifically, when different users utilize LVLMs, they may have different linguistic expressions towards a same question due to their distinct identities or educational backgrounds. Therefore, as shown in Figure 3(a), our work conducts transfomations on the original question from three linguistic levels[2]: word-level (L1), sentence-level (L2), and context-level (L3, L4) .

For example in figure 3(c), the original question is 'What type of flooring does the kitchen have?'. From the word-level, some users might habitually use the word 'cookroom' instead of 'kitchen'. From the sentence level, more literary users like writers, might phrase it as 'is equipped with what kind of' instead of 'have what type of'. From the context level, more talkative users like teachers pretend to guide into the question by giving some scene description aforehead, which corresponds to our L3 strategy 'adding relevant context'. Meanwhile, children users, who may lack mature logical thinking ability, might first ramble about unrelated gadgets in the image before posing his realistic question to LVLMs. This is corresponds to L4 strategy 'adding irrelevant context'.

Generally, our dynamic strategies effectively simulate various real user interaction scenarios from both visual and linguistic perspectives, which is very significant for the practical application and widespread adoption of LVLMs.

Dear Reviewer Rays,

Thanks very much for your response on Q2!

We are glad that our detailed experiments have addressed your concerns. We also regard table B important, and have been planning to update the content discussed with you into our paper. Due to the page limitations, we now updated this content in Appendix A.12 of the revised paper.

Thanks for your interest in our research and for your constructive comments, which have been instrumental to our work!

Dear Reviewer Rays,

Thanks very much for your response on W2 and W3!

We are glad that our additional experiments have addressed most of your concerns. And about W3, we will conduct a larger scale of experiments these days, and then update the experimental results for you. At that time, we will add this content into a new revised paper in a few days.

We are deeply grateful for your thorough and constructive review!

Dear Reviewer Rays,

Thanks very much for your response on W1&Q1!

We are glad that our detailed explanations have addressed your concerns. We have now updated this content in Appendix A.11 of the revised paper constrained by page limitations. We believe our discussion with you has been invaluable, which will help readers better understand our motivation and contributions.

Thanks for your thorough and constructive review! We are so lucky to have met such a careful and insightful reviewer like you!