ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models

Please find our responses as follows:

Response 1: The scale is too small (corresponding to weakness #1).

• The smaller size is a trade-off for the high-quality manual annotations. Each image’s integration with shifting and zooming tasks adds depth to the analysis, compensating for the scale .

• We contain 314 unique images and 325 unique questions. We further split the images and disrupt the order to enable shifting and zooming tasks. Altogher, 1625 evaluation instances are generated, which is not a small number.

• Also note that FISBe [1] contains 101 images, V* Bench [2] contains 191 images with manual annotated quesitons, ViP-Bench [3] contains 303 images, and [4] 310 instances.

[1] Mais, L., Hirsch, P., Managan, C., et al. FISBe: A real-world benchmark dataset for instance segmentation of long-range thin filamentous structures. CVPR 2024.

[2] Wu, P., & Xie, S. V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs. CVPR 2024.

[3] Cai, M., Liu, H., Mustikovela, S. K., et al. ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts. CVPR 2024.

[4] Tian, S., Finn, C., Wu, J., A Control-Centric Benchmark for Video Prediction. ICLR 2023

Response 2: Categories and novelty (corresponding to weakness #2)

• ActiView is complementary to datasets like object detection and event-centric datasets. It uniquely challenges models to operate under perceptual constraints, such as shifting and zooming, which these datasets lack.

• It is inappropriate to dismiss ActiView’s novelty solely based on the overlap in task categories with existing datasets. While some categories like object relationships and attributes are common, ActiView is unique in its integration of active perception tasks through zooming and shifting. This dynamic evaluation mechanism is not addressed by previous benchmarks, which primarily rely on static perceptual fields. Moreover, ActiView’s carefully designed pipelines and constrained perceptual fields push models to perform reasoning under limited views—replicating real-world scenarios that demand active exploration. This emphasis on active perception and reasoning processes differentiates ActiView as a novel and specialized benchmark.

• Our benchmark shows that current models do not demonstrate practical active perception ability, and aims a few years ahead of current SOTA.

Response 3: InstructBLIP (corresponding to weakness #3)

• As InstructBLIP has been available for over a year, many improved vairent based on this model have been released, driven by the rapid advancements in Artificial Intelligence research. Although InstructBLIP is a very promising model, it lacks the ability to read multiple images simultaneously, and the instruction-following ability also needs improving compared to recent models.

• We evaluate two powerfull varients Brote and MMICL (as we already stated in our paper) that are derived from InstructBLIP (as mention in Line 318). Results of these two models are reported in Table 2, 7, 8, 9, 10, 11, and 12. It is out-dated and improfessional to stick to the past even if numbers of advanced approaches are newly presented, and researcher are no longer including simple MLPs and CNNs as baselines for visual-language tasks.

Response 4: Other evaluations except zooming and shifting

• Firstly, our paper’s focus on zooming and shifting stems from their clear representation of active perception. Secondly, other methods, such as evaluating temporal dynamics or integrating real-time feedback loops, could be explored in future iterations.

• Plus, we are now working on more dynamic evaluations including segmenting regions semantically. This pipeline will be released when more models (not limited to GPT-4o) are able to follow the specific instuctions.

Thanks for your comments, please find our responses as follows:

Response 1: Image splitting strategy for evaluation pipelines (corresponding to weakness #1).

• We aim to address the significance of active perception ability, and introduce a set of splitting strategy for the two fundamental abilities of active perception, i.e., zooming and shifting. Additional experiments on different splitting strategies for both zooming and shifting are provided in Appendix F1. Results suggest that different splitting strategies inconsistently affect the performance, presenting a trade-off between zooming and shifting performances along with the increasing of splits. We propose using the 4-split setting to balance zooming and shifting evaluations. This fixed image splitting strategy already demonstrates discrepancies in active perception ability between models and humans, and among different models as well.

• In addition to the basic settings for zooming and shifting, we also propose a more dynamic and natural setting, the mixed evaluation, to simulate the ideal active perception behavior. This setting incorporates both zooming and shifting without restrictions on how should models behave. Models can autonomously zoom and/or shift, and perform further actions (zoom and/or shift) if deemed necessary. This can be easily adapted to finer granularity evaluation of either zooming or shifting by limiting the type of behavior and combining it with different splitting strategies as in Appendix F1 (such as symmetric splitting, e.g. 2 $\times$ 2 grids, and 3 $\times$ 3 grids, and asymmetric splitting, e.g. 2 $\times$ 3, 2 $\times$ 4).

Response 2: Detailed analysis compared to “Full” image (corresponding to weakness #2)

• First, we must clarify that the “Full Image” setting is not part of the evaluation for active perception ability. It is provided for reference only. Our primary focus is on evaluating the active perception abilities of MLLMs (e.g., zooming and/or shifting), rather than directly measuring question-answering accuracy.

• Models like Gemini-1.5-pro and Claude 3.5 Connect may perform better with “full images” because they can access the complete context, allowing them to process visual information holistically. In contrast, the zooming pipeline introduces constraints on the perceptual fields by requiring models to focus on split views. This challenges their ability to integrate multiple dependent details into broader understandings. And the performance drop of models (compared to the full setting) implies the certain weaknesses in active perception ability for models. The weakness could be (but not restricted to) the ability to integrate multiple dependent details, or recognise key information regarding a given question. This discrepancy highlights the importance of the zooming pipeline in exposing weaknesses in a model’s active perception abilities.

Response 3: Zooming pipeline does not allow for a “none” response. (corresponding to weakness #3)

• We did not strictly specify the number of selected views for the zooming pipeline, meaning a “none” selection is permitted. While we recommend selecting at least one view, this is not a mandatory requirement. Some models respond with empty strings (“”), provide a reason for not selecting a view, or explicitly output “none” when required to output the image splits for zooming. We regard these responses as “none” for the zooming selection, and only the full images are provided instead of zoomed splits in this case.

• For example, 1% of Brote-IM-XL’s responses were “none”, and so were 4.9% of Mantis’s responses.

Because of the 5000 characters limitation per reply, our responses will be split into 4 parts.

Thanks for your insightful comments, please find our responses and additional experiments as follows:

Response 1-4 correspond to detailed questions:

Response 1: The dataset statistic (corresponding to quesiton #1)

• The dataset contains 314 images paired with 325 questions, where some images correspond to more than one question. Each image-question pairs is evaluated across five different initial settings, including one for zooming and mixed pipelines (325 $\times$ 1) and four for shifting pipeline (325 $\times$ 4, varying difficulty levels of the inital view, e.g. random/easy/medium/hard), resulting in a total of 1,625 (i.e. 325 $\times$ 5) evaluation instances. An instance is defined as a single “question-initial view” pair.

Response 2: The settings of views of shifting pipeline (corresponding to quesiton #2, #3)

• Question #2: Is it randomly selected once for each image and the same initial view given to each model?

• Response: Yes, in the Shifting-R setting, a random initial view is selected once per image, it is selected according to a fixed random seed. Thus, the same initial view is given to all models to ensure consistency in evaluation.

• Question #3: How is the adjacent view selected for the Shifting tasks? Is the same order given to all models?

• Response: For the Shifting tasks, adjacent views are selected following a fixed predefined sequential order (upperleft-lowerleft-upperright-lowerright, as mentioned in Appendix I), and it is the same order given to all models.

Response 3: The random choice results in Table 2 and textonly evaluation (corresponding to quesiton #4)

• The “random choice result of 33.95%” is calculated by randomly selecting an answer from the available options for each question, averaged over 10,000 runs to minimize noise from random fluctuations. As this is to randomly generate an answer from the options, no input content is provided. Answering according to the images and questions cannot be regarded as RANDOM. This could also be viewed as the distribution of answers. Our options ranges from 2 to 7 (not always 4 options).

• Thanks for your constructive advise. We employ the prompt template for text-only evaluation as following: “Please answer questions based on you commonsense knowledge. If you are not able to answer, please select a most probable one. {Question} {Options} Your answer:” We found that there are some questions that can be answered via commonsense Here are the resutls of text-only evaluation:

  Model	Claude	GPT-4o	Qwen2-VL	MiniCPM-V 2.6	Idefics3	Brote-IM-XL
  ACC	2.14	2.45	23.38	26.77	44.92	40.00
  ACC(guess)	26.07	37.73	42.77	41.54	47.38	40.00

  • We implement two settings: 1. asking models to answer according to commonsense ONLY and reply “None” for questions cannot be answered via commonsense (results shown in Row ACC), and 2. permitting models to guess according to commonsense (results shown in Row ACC(guess)).
  • The above results (Row ACC) indicate that questions in our benchmark cannot be simply answered via commonsense, where two powerful models GPT-4o and Claude achieves only 2.45% and 2.14%.
  • The row of ACC(guess) presents results of generating the most probable answers. These reflect the bias learnt from the training corpus.
  • The differences between these two results demonstrate the ability of instruction-following. We found that Idefics3 and Brote-IM-XL present weaker instruction-following ability compared to other models in this table, that they still exhibit a behavior of guessing when commonsense cannot be used to answer the questions.
  • The corresponding experiments will present in the Appendix of our paper (as there is no available space in main text).

Response 4: Single view in Table 2 (corresponding to quesiton #5)

• The “Single View” column refers to the results where only one sub-view of the image is provided to the model (only the initial view is available). This functions as a reference for the shifting evaluation when only insufficient perceptual field is available. In Table 2, the “Single View” refers to the initial view of shifting-R.

Response 5-8 correspond to weaknesses concerning our pipelines:

Response 5: It is not clear why the mixed pipeline requires the model to discard images. (corresponding to weakness #1)

• There could be some misunderstandings. It is not a required operation to discard images. The case in Fig 4 (c) is just an example demonstrating that the model can discard an image if it determines that image to be irrelevant to the question after zooming. If it is useful, the image will undoubtedly be retained. This approach is designed to simulate ideal active perception behaviors.

Dear reviewer,

We hope our responses have addressed your concerns and look forward to further discussions.

Thanks for your comments, please find our responses as follows:

Response 1: Discussion over segmentation and other operations (corresponding to weakness #1).

• Intuitively, segmentation is a variant of our zooming setting, allowing models to see on object segments or irregular regions in an image. Although segmentation ensures semantic integrity (compared to regular patches), our approach allows multiple patches to be selected to maintain the completeness of semantic segments, even when they are divided into different patches.

• We did not include this specific setting for the following reasons:

  • Most current models do not support directly reading segments. Some models claim to allow segment inputs by converting them into w $\times$ h (according to the segment size) patches for processing. Our pipeline of splitting the whole image into smaller views already presents significant challenges for these models.
  • While this setting is natural and reasonable for evaluation, our experiments showed that current models often fall short to follow instructions (to output correct segments) or suffer from significant hallucinations (producing incorrect segments). These issues result in near-random performance, even for advanced models like GPT-4.
  • As a compromise, we proposed the zooming setting and conducted experiments with varying numbers of image splits. The results are presented in Appendix F1.

• With improvements in model capabilities, we hope future models will be able to satisfy these requirements. We have been working on and continue to explore this setting both for evaluation purposes and to improve model performance.

• For now we only consider zooming and/or shifting for MLLMs. In the future, fine-grained grounding operations such as iteratively responsing with bounding boxes for the deducted visual clues could be considered. Also, the detection of visual redundency (information do not help with answering) and distractive information (information that doom to failure) could also be considered. Regarding the active perception ability, more modalities should be included beyond vision in the future, and operations such as modality division and key modality determination are also worth investigating.

• For now, we focus on zooming and/or shifting for multimodal language models (MLLMs). In the future, fine-grained grounding operations, such as iteratively responding with bounding boxes for deduced visual clues, could be explored. Also, detecting visual redundancy (information that does not contribute to answers) and distractive information (misleading information that doom to failure) could be considered. Additionally, for active perception abilities, incorporating other modalities beyond vision is a future trend for research and practical applications. In this situation, operations such as modality division and key modality determination, will be worth investigating.

Response 2: The significance of results (corresponding to weakness #2)

• We emphasize that the “Full Image” setting is not part of the evaluation for active perception ability. It rather serves as a reference only. Our focus is on evaluating the active perception abilities of MLLMs (e.g., zooming and/or shifting), rather than directly assessing question-answering accuracy.

• Most current models are fine-tuned for VQA tasks, giving them an inherent ability to “guess” answers. Some models guess the correct answer without truly understanding the underlying reasons or clues. We view our proposed zooming, shifting, and mixed settings as OOD tasks compared to traditional image-question pair evaluations that simply gives an image-question pair and requires models to answer. Considering this feature, even small differences (e.g., 1%) in performance are significant within the context of our benchmark. A few percentage points can represent a consistent inability to reason under our constrained settings.

Response 3: Why just VQA? (corresponding to weakness #3)

• VQA is a suitable format for evaluating various visual-language tasks. For most current MLLMs, tasks such as NLVR2, SNLI, and image retrieval can be adapted to the QA format via prompt design.

• We adopt the VQA format because it provides a easy-to-understand, clear and focused evaluation of active perception abilities. VQA inherently requires models to reason over visual information, making it an ideal task for assessing capabilities including zooming and shifting. By concentrating on this task, our benchmark effectively highlights and evaluates active perception without blurring the focus.

Dear reviewer,

We hope our responses have addressed your concerns and look forward to further discussions.