ExoViP: Step-by-step Verification and Exploration with Exoskeleton Modules for Compositional Visual Reasoning

Thank you for your meticulous review and acknowledgment of our method's effectiveness. We believe your support of our work will help us to make a timely contribution and broader impact on both Computer Vision and compositional reasoning community. We would like to provide further clarification to address your concerns.

W1: Detailed information about different prompts is necessary for a comprehensive comparison

A: In our submission, we have demonstrated the exact prompt for self-correctness for all five tasks, Please refer to Appendix G for details. Specifically, for example, for compositional question answering, the prompt goes like: You are a ranker for a planner who use the candidate modules to answer a question: QUESTION, select the best solutions for answering the question \n candidate modules include: LOC: detection, VQA: visual question answering, EVAL: use logic operation, RESULT: wrap up the final result, CROP/CROP_LEFTOF/CROP_RIGHTOF/CROP_FRONTOF/CROP_INFRONT/CROP_INFRONTOF/CROP_BEHIND/CROP_AHEAD/CROP_BELOW/CROP_ABOVE: crop the image \n Current solutions: \n 0 PLAN \n 1 PLAN \n If the modules in the solutions have better cause-and-effect relations, and more likely to answer the question, please rank it first. If you are unsure, please keep the original rank. Return sequence number of currently best solution, for example 0,1,2,3, DO NOT RETURN ANYTHING ELSE EXCEPT FOR NUMBERS SPLIT by ,

We are willing to append more details if there is any further missed information.

W2: The improvements on the GQA dataset appear to be minor when compared to those achieved by ViperGPT

A: Thank you for bringing this to our attention. We would like to highlight 4 aspects:

• Our test set differs from that of ViperGPT. We used the same sampling strategy as in VisProg. As is claimed in Section 5.1, we randomly sample 5 samples from the balanced val set and 20 samples from the testdev set of each question type. e.g. “weatherVerify” for judging the weather, “twoCmomon” for judging common attributions of two objects. In summary, there are 102 question types and 2327 questions in our test set.
• Regarding the different test sets, we re-executed ViperGPT's experimental results on our test set using their official repository (https://github.com/cvlab-columbia/viper) without any modifications. The results indicate that VisProg+ExoViP (61.49%) improves the performance of ViperGPT (46.47%) by 5.02%.
• Our insights differ from ViperGPT. While ViperGPT has transitioned to Python code for planning, thereby inherently addressing numerous planning errors, our module is designed to enhance existing compositional methods by tackling both planning and module errors.
• As shown in Table 7, ExoViP can be generalized to improve ViperGPT by 1.37%.

Thank you for your constructive comments and acknowledgment of our well-motivated idea. We carefully address each of your concerns as follows.

W1: The initial analysis finds the percentage of errors caused by the two types of problems (module error and planning error). The developed method's intuition is to solve these two errors with the two mentioned strategies. However, the final analysis does not include the kind of errors this method actually ends up fixing and in what ratio. It would be beneficial to show the statistics that demonstrate what proportion of errors of each type were fixed by the proposed methods compared to VISPROG [1].

Q4: Could you report an analysis about what percentage of module errors and planning errors were fixed by your method compared to VISPROG?

A: Thank you for the valuable suggestion. In our original submission, we have manually analyzed 100 randomly sampled failure cases on VisProg. We find that there are three typical reasons for the failures: a) vision module prediction error; b) LLM planning error; c) others. We demonstrate the statistics of the failure cases in Appendix.C. We provide illustrations of two representative failure cases resulting from our methodology in Appendix I.

In our revised paper, following the application of our proposed framework, we re-assessed the same cases. As shown in Appendix C, "we were pleased to discover a reduction in module errors by 28.87%, and a decrease in planning errors by 42.35%. Nevertheless, our current strategy was unable to rectify 69.8% of the errors. When juxtaposed with the data from Table 1, our method has enhanced VISPROG by 7.11%, which is lower than the improvement of the failure cases. This outcome suggests that our approach may give rise to novel challenges. We further demonstrate common errors of our method in Figure 10 and Figure 11, As highlighted in the study of failure cases, the majority of these instances originate from the VQA module."

W2: The method is not evaluated with any open-source Large Language Models (LLaMA, Vicuna, etc.) Restricting evaluation only to GPT-3.5-turbo reduces the generalizability of the approach.

Q3: Both ViperGPT [2] and VISPROG [1] implementations used GPT-3 as the model of choice. Will the advantages of your method transfer to GPT-3 and other Open LLMs as well? Or is GPT-3.5-turbo necessary in order to accomplish Self-Correction and Trace Reasoning?

A: Thank you for the helpful suggestion. In fact, both the official implementations of ViperGPT and VISPROG are supported by the GPT-3.5-turbo. In accordance with this line of research, all the experimental results reported herein are also built on the GPT-3.5-turbo model. In addition, we replaced the GPT-3.5-turbo with LLama2-chat-13b and obtained the following results. We are thrilled to discover that our method can yield significant improvements in open LLM. We have added these results to Section 5.3 in our revised paper.

W3: The performance increment across tasks over VISPROG [1] is still incremental, considering the time cost of using beam search and multiple verification modules.

Q2: What is the inference latency for your approach compared to using VISPROG [1]? The use of beam search and multiple verification modules indicates a significant added overhead.

A: Thank you for the constructive advice. We present the mean inference time on the GQA dataset. Generally, the temporal expenditure of our tree-based search method significantly surpasses that of VisProg. However, the majority of the time is consumed by the call of the OPENAI API, an issue we posit is intrinsic to analogous works [3,4,5]. When compared to Depth First Search/ Breadth First Search [3] or Monte Carlo Tree Search [4,5], we assert that our beam search-based method can achieve an optimal equilibrium between efficiency and effectiveness. We have added this part to Appendix D in our revised paper.

[1] Visual Programming: Compositional Visual Reasoning Without Training

[2] ViperGPT: Visual Inference via Python Execution for Reasoning

[3] Tree of Thoughts: Deliberate Problem Solving with Large Language Models

[4] Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training

[5] Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models

Dear Reviewer Tiuv,

Please kindly let us know if you still need further clarification. We will be more than happy to answer.

We thank you again for your valuable comments and suggestions to help us improve our paper. We have tried our best to address all your concerns. We have also substantially revised our paper by following other valuable suggestions from other reviewers. Please take some time to look at them and see if you have further questions.

Best,

Authors

Thank you for the clarifications and the detailed response. I believe adding error analysis of your method on the selected cases and results on open source LLMs warrants an improved soundness score (I have updated the soundness score accordingly). I will maintain my overall rating as positive feedback.

W3: many recent MLLMs achieve better results on GQA than the baselines used in Table 1, e.g., LLaVA get 63 on it.

A: Thank you for the references. We have included the results of LLava, as an upper bound of non-compositional models, in Table 1.

As claimed in the introduction, "despite their merits, current visual programming methods still suffer from challenges due to the failure of the LLM planning or the visual modules, lagging behind the performance of non-compositional models.", the focus of our work is to mitigate the performance gap between compositional models and non-compositional ones (e.g. LLaVA). Therefore, models like LLaVA can only be considered upper-bound and lead to unfair comparisons to our model. Specifically, the result of 63.0 on GQA ("e.g., LLaVA get 63 on it") is from Llava (Llava-1.5-13B) [9]. However, this result is derived from instructions that were tuned on the GQA training corpus, making it incompatible with the zero-shot setting of our experiments. Additionally, the evaluation scripts for Llava come with an extra prompt, "\nAnswer the question using a single word or phrase.". This particular prompt is not included in the scripts for other baselines.

W3: In Table 3, why is the reported referring expression comprehension results only on RefCOCO and RefCOCO+, why not including RefCOCOg, which is quite a standard comparison for this task.

A: Thank you for your suggestion. Due to limited resources and funds, we are unable to apply our method to all existing benchmarks. However, in response to your request, we will endeavor to report the results on the subset RefCOCOg, which is sampled by the same strategy as RefCOCO and RefCOCO+.

W3: And why only a subset is validated, 2 samples per type from the test set is actually not enough.

A: As previously mentioned, our financial constraints limit us from affording a complete test set. However, the scale of our dataset is comparable to that of VISPROG. We also believe that implementing a balanced sampling strategy is of utmost importance.

W3: And the compared model and proposed model are only 30 IoU, which is far below the decent results in recent MLLM, which all receive 80+ (e.g., Shikra, Kosmos2, etc).

A: It is worth noting that the testing dataset used to report evaluation results of Shikra, Kosmos, etc. is different from the evaluation dataset used in our work (following the sampling strategy as in VisProg; refer to Sec. 5.1 GQA setup for details). In order to stay aligned with the current advancements in MLLM, we have included some preprint models in our experiment, such as QWen-vl-7b-chat. To the best of our knowledge, this model is a state-of-the-art MLLM during our experimental period, and it outperforms Shikra in terms of results on RefCOCO. We have used the model weight from its official code repository to evaluate our dataset (URL: https://github.com/QwenLM/Qwen-VL): as reported in our paper, it achieves a score of 32.54 in the subset of RefCOCO&RefCOCO+. We are willing to include more references and compare with them if there are more advanced models. We have also updated all the evaluation scripts in the anonymous link provided in the submitted paper.

W3: In conclusion, the main concern is the baselines compared in each Table, very confusing, a lot SOTA models are removed out e.g., OFA-large is selected for NLVR, but not included for RefCOCO (which is actually reported in the origin paper).

A: As previously stated, our objective in this work is not to achieve the SOTA performance on all tasks. We have selectively referenced parts of the SOTA models. Since Qwen-vl-7b significantly outperforms OFA-Large on the referring task, we did not report its results.

[1] Tree of Thoughts: Deliberate Problem Solving with Large Language Models

[2] Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training

[3] Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models

[4] Mitigating Lies in Vision-Language Models

[5] Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles

[6] A Multi-dimensional study on Bias in Vision-Language models

[7] Multimodal Explanations by Predicting Counterfactuality in Videos

[8] Towards Counterfactual Image Manipulation via CLIP

[9] Improved Baselines with Visual Instruction Tuning

Thank you for your careful review and insightful comments. We have meticulously discussed the issues at hand and have attempted to dispel any misconceptions through our detailed responses. We trust that our comprehensive, point-by-point rebuttals have satisfactorily addressed all your concerns. Please let us know if there are further questions and we are delighted to have a deeper discussion.

W1: The verifier module is not carefully developed, CLIP for image-text matching is particularly not good at give correct similarity score, especially when the task are for compositional visual reasoning in this paper. In addition BLIP version1 (as cited in Section 4.1) for image caption/VQA is also not SOTA, and no ablation study is explored here to choose the best module for verifiers.

W3: And the proposed method is not carefully developed or verified, as not sufficient ablation for each module is conducted, especially the choice for each verifier.

Q1: For the three proposed verifiers, the CLIP is used for image-text matching, and BLIP is used for image captioning and VQA, how these models are selected, is there any ablation experiments here? E.g., C_img is generated by BLIP, why not using SOTA caption model, at least BLIPv2? Is this a severe bottleneck for the verifier or even bring confusion when the verifier model make errors.

A: Sorry for any confusion. We have carefully considered the selection of models for each verifier module. Specifically,

1. Fair module comparisons. We intentionally selected modules from VisProg's original modules. As noted in the implementation detail section of Appendix E, " In order to validate the effectiveness of our method and eliminate the benefits of external knowledge such as more advanced vision-language models which are trained on larger datasets. Both operation modules and verification modules are selected from the same candidate neural module sets.".
2. Trade-off between latency and performance. As claimed in Section 5.2.2, "while Qwen-VL is built on a LLM with 7 billion parameters, which is trained on trillions of tokens from the corpus, our method assembles a team of experts whose collective parameters total less than 1 billion". Our goal is not only to achieve better performance with heavier and more models, but to deliver a carefully designed system that is both lightweight and easy to plug-and-play, offering superior performance with minimal additional computational expense. Specifically, when we substitute the QA model (blip-vqa-capfilt-large) with BLIPv2 (blip2-flan-t5-xl), we see a performance increase from 67.96 to 68.70 on NLVR. However, this comes with a tenfold increase in parameters.

W2: The introduced verifiers can cost the whole system more computation time, however no efficiency info is added here, it should be more fair to consider this.

A: Thank you for pointing it out. We present the mean inference time on the GQA dataset. Generally, the temporal expenditure of our tree-based search method significantly surpasses that of VisProg. However, the majority of the time is consumed by the call of the OPENAI API, an issue we posit is intrinsic to analogous works [1,2,3]. When compared to Depth First Search/ Breadth First Search [1] or Monte Carlo Tree Search [2,3], we assert that our beam search-based method can achieve an optimal equilibrium between efficiency and effectiveness. We have added this part to Appendix D in our revised paper.

W2: It's not clear when the verification score is more reliable than the confidence score, while in the method, the verification score directly replace it.

Q2: It happens when the confidence score is more accurate than the verification score. However, seems in candidate prediction in Equation 6, is only using the averaged verification score to calibrate, while the confidence score is ignored.

A: Sorry about the confusion. There might be severe misunderstandings regarding our method. we never used the verification score to replace the confidence score. In Equation 6, $p$ represents the confidence score, while $w$ denotes the normalized verification score, also known as the weight. Intuitively, the verification score, or weight, is used to scale (calibrate) the confidence score. We have not ignored the confidence score at any point.

Thank you for your insightful review and kind words regarding our innovative concept of verification modules. We appreciate your recognition of the robustness of our method and your acknowledgment of our efforts to address critical limitations in existing methodologies. We have carefully followed your suggestion to revise our paper. Please let us know if you have any more comments and we are willing to answer more details and further improve our work.

W: The paper's experimental evaluation is somewhat limited, primarily focusing on VISPROG evaluation datasets. Expanding the range of datasets, especially including more varied and challenging ones, would significantly strengthen the demonstration of the method's generalizability. While the paper reports a 4% improvement on the GQA dataset, a broader dataset spectrum could provide a more comprehensive assessment of the method's effectiveness.

Q1: Could you elaborate on the choice of primarily using VISPROG evaluation datasets? Suggestion: Consider applying your method to a wider array of datasets, especially those that are more diverse and challenging. This could help in better demonstrating the method's generalizability and effectiveness across different scenarios.

A: Thank you for your kind advice. We choose similar tasks as in VisProg for fair comparison and better demonstration of the improvements. In addition, we have added an abstract reasoning task KILOGRAM, a dataset featuring extensively annotated tangram puzzles that necessitate the model’s comprehension of abstract tangram shapes and their subsequent classification. This is a challenging task designed to evaluate the model’s capacity to abstractly generalize, utilizing visually ambiguous stimuli to do so. In Section 5.2.4, we demonstrate how our method effectively utilizes conceptual components to enhance abstract image understanding. Furthermore, in Appendix E, we present the details of our implementation of abstract reasoning.

W: The paper lacks an in-depth analysis of its limitations and potential failure cases. For visual language grounding, natural language reasoning, and visual abstract reasoning tasks where the model fails to exceed the performance of SOTA models, no further analysis is provided apart from reporting the results. Although several failure cases are mentioned in the attachment, they are not thoroughly investigated or analyzed. A detailed examination of these cases would be invaluable for identifying areas for further refinement and development of the method.

Q2: Why was there no detailed analysis provided for the failure cases mentioned in the attachment? Suggestion: It would be beneficial to include a thorough investigation and analysis of these failure cases. This analysis could offer insights into the method's limitations and guide future improvements.

A: Thank you for your valuable advice. We have reassessed the failure cases in GQA. Both the statistical results and qualitative results are now presented in Appendix C of our revised paper. Additionally, supplementing the analysis in Section 5.2, we analyze the failure cases of tasks that fail to exceed the performance of SOTA models.

In conclusion, We have the following observations:

1. Regarding the sampled failure cases in GQA, we observed a reduction in module errors by 28.87% and a decrease in planning errors by 42.35%. Our method seems to be more effective in resolving planning errors. We believe this improvement is due to the incorporation of verification and self-correction mechanisms in our tree-based search algorithm. In essence, there is a need to enhance the verification process for candidate predictions.
2. In the GQA task, our method has corrected 31.2% of the failure cases. However, when compared to the general improvements on the test set, we found that our method also introduced new issues. Therefore, we analyzed the failure cases specific to our method and discovered that it tends to introduce a planning trace made of the VQA modules, which often perform poorly. This observation has led us to consider introducing penalties for certain modules that perform poorly.
3. Upon analyzing failure cases in visual language grounding, natural language visual reasoning, and visual abstract reasoning tasks, where the model falls short of outperforming SOTA models, we identified two attributes in these reasoning traces that could contribute to subpar performance: (a) the traces for visual abstract reasoning and visual language grounding are relatively short, making few improvements by our verification modules, and (b) the traces for natural language visual reasoning are primarily composed of VQA modules, which could potentially serve as bottlenecks.

Dear Reviewer 2ndj,

We thank you again for your valuable comments and suggestions to help us improve our paper in the aspects of presentation, experiments, analysis, and discussion. We also appreciate that you noticed our innovative concept in the field of visual programming, the robustness of the proposed methodology is evident through its application to two distinct models, yielding a notable 4% improvement in accuracy for compositional visual question answering tasks, a thorough preliminary study, demonstrating how individual components contribute to enhanced performance, excels in clearly articulating its methodology, detailed explanations, the contributions of this paper are **highly significant** for the field of visual compositional reasoning, **addresses critical limitations** in existing methodologies and offers a **substantial improvement** in prediction accuracy, marking a notable advancement in the field. These compliments have encouraged us a lot and will help us to make a timely contribution and broader impact on both the Computer Vision and the compositional reasoning community.

Our current revised paper has taken into account all your suggestions, please kindly let us know if you still have follow-up questions.

Thanks,

Authors

Hi there,

Thanks for addressing all of my questions. I appreciate your efforts in conducting additional error analysis on the failure cases. It makes sense to me that extra errors would be introduced when aggregating the modules, and the performance could be bottlenecked by one particular one (in your case, seems to be the VQA model). I've updated the soundness score in recognition of your analysis.