Thank you for the positive comments and insightful suggestions.

Q1. Is there anything to prevent overfitting to the small train set of a given task, and only creating specific modules tailored for that domain? I can imagine that these overfitted modules may not be very useful in new tasks.

Q2. Isn’t it possible that the method creates a broad function signature, but during stage 2 verification with the small set of train examples, it overfits to a bad implementation that only works for those examples, and therefore actually harms performance from there on out? I’m mainly concerned about the above two overfitting challenges.

Q1-Q2. About overfitting VS training number. Thanks for concerns on the overfitting issue of train examples. Our GNSVR pipeline do have overfitting issue when the number of the training examples is too small (10), which could be reflected in the ablation study in table 5 of the revised paper. However, as the number of training examples goes up (50 and 100), the performance becomes stable, which is unlikely that the GNSVR framework creates a broad function signature overfitting the training examples. We believe that one reason for the strong few-shot performance of GNSVR is that all our pre-defined APIs are general APIs that work well for vision tasks in the wild. We also provide the performance VS test cases in the following tables for different tasks. As shown in the tables, as the training number is larger than a small number, our GNSVR framework becomes effective and relieve the overfitting issue.

Number of Sampling Examples

Table 1: Number of Sampling Examples on GQA.

Table 2: Number of Sampling Examples on RefCOCO.

Table 3: Number of Sampling Examples on RAVEN.

Table 4: Number of Sampling Examples on MEWL.

Thanks again for your response on the overfitting concern of the GNSVR model. We further explain your concerns as followed. In GQA, we have sampled training examples by question categories. This strategy does require the usage of question categories in the training dataset. In refCOCO, RAVEN and MEWL, we sample examples randomly since there is no sub category by types.

To furtehr investigate our model's ability against overfitting, we have conducted new experiments on Table A below. We randomly sample different training samples from the whole training set 3 times with different random seeds, which we denote it Random Sampling. We also perform another sampling strategy that randomly sample training examples by question types, which we call Random Sampling by types. Based on the results in table A, we can see that our model works in both Random Sampling and Random Sampling by types, achieving reasonable performance although we do observe that Random Sampling strategy has larger variances on GQA. We will add such analysis in the revised paper.

Table A: Ablation on GQA for sampling strategies.

Another way to show our GNSVR model's ability against overfitting is that as shown in Table 2, Fig. 4, and Fig. 7, of the paper, the learned modules learned from GQA and refCOCO can be generalized to new tasks like image editing and knowledge tagging. Note that language instructions and images from image editing and knowledge tagging are quite different from that of the GQA's and RefCOCO's images.

Thank you for the constructive comments and insightful suggestions.

Q1. From my perspective, the biggest current weakness of the paper is that from the experimental design its hard to parse out exactly how the discovered modules affect the system's performance. Ostensibly, this can be gleaned from comparisons between GNSVR and VisProg/ViperGPT, but there are more differences between these systems beyond merging in discovered modules. Specifically, GNSVR uses a "base API" that is a combination of VisProg and ViperGPT, so the "fair" comparison would be against an ablated version of GNSVR that removes steps 1 and 2, and just tries to solve test-problems with the original API functions. This condition is considered in the ablation experiment (GNSVR w/o ML), but only a subset of the RefCOCO test-set. To solidify the claim that the improvement GNSVR observes stems from its discovered modules, this base condition should be added to all of the experimental set-ups (tables 1-5), for example, from Table 2 its unclear how much of the delta improvement between VisProg and GNSVR can be attributed to improvements in the base API versus improvements to the API from the new modules.

Q1. More ablation study on Performance

Thanks for the insightful suggestions. We add more baseline experiments for comparison.

For the GQA and RefCOCO baseline's comparison, please refer to Q2 below.

In Table 2 of the main paper, we showcase the capability of GNSVR's transfer learning. That is to say, the new modules learned from GQA and RefCOCO can be applied to the image editing and knowledge tagging task. Therefore, in the two tasks involved in Table 2, we did not learn any new modules but instead transferred and used new modules learned from other tasks. In Table 2, we can see that in all metrics, we surpassed VisProg. This performance improvement actually stems from the new modules generated from GQA and RefCOCO. It is these new modules that enabled functions that the inherent modules of VisProg couldn't achieve, thus leading to the improved performance of our system.

As for RAVEN and MEWL, we have implemented the ViperGPT and VisProg baseline experiments in the following way. For VisProg, it requires a manual implementation of all modules by making use of the provided APIs. Thus, to enable VisProg handle Raven and MEWL tasks, we manually implement and debug new hand-crafted modules for VisProg to recognize and discover patterns to handle the task. We call this baseline VisProg variant. We also put the training examples in GNSVR' stage 1 into the prompt of VisProg variant for better performance. For ViperGPT, it has no manual modules and ask the LLMs to make use of the APIs to handle the instances. Thus, we manually write solutions for the training examples into the prompt of the ViperGPT to teach ViperGPT to handle the task. We call this approach ViperGPT variant. We have added such analysis into the revised paper. VisProg by itself needs a handcrafted solver module to find the target solution and it would be extremely difficult for ViperGPT to generate a solver from scratch. Thus, we add the solver module learnt from our GNSVR model to pre-defined API pool of VisProg and ViperGPT. As shown in table 1 and table 2, our GNSVR model achieves better performance than these two baselines, showing the great value of module learning for handling new tasks from only a few examples.

Table 1: Compare our GNSVR model with baselines, VisProg and ViperGPT on RAVEN.

Table 2: Compare our GNSVR model with baselines, VisProg and ViperGPT on MEWL.

Q2. Beyond this, I'm also slightly concerned about the design of the GNSVR w/o ML baseline. At inference time, is this baseline allowed to invoke arbitrary python logic in the style of ViperGPT (e.g. standard control flow constructs) or is it restricted to only using API function calls in the style of VisProg. I would imagine that the first condition would be more fair to evaluate GNSVR. Solving some tasks might require simple logic that the LLM knows how to express in python, but might not be directly expressible with a series of API calls. In GNSVR, this logic is incorporated into modules, but in the baseline the LLM should still have the opportunity to invoke similar logic in its test-time solutions (otherwise its impossible to properly evaluate the usefulness of the discovered modules). Please clarify which of these modes the baseline is operating in.

Thank you for the detailed responses! I've read through the author's comments, and after the rebuttal have a more favorable view of the paper. As such, I've updated my review accordingly. The consistent (though not terribly large) improvement over the baselines I suggested is an important mark in favor of the method, and tempers my fears about a fair comparison against ViperGPT/VisProg. The quality of the discovered modules having a reliance on a fairly large number of training examples is still a concern, but overall I would still support the paper's inclusion into the proceedings.

Thank you for the constructive comments.

Q1. While I think the framework overall is useful, the components are not all equally important to solve the reasoning problem and hence it is important to understand for future research on modular reasoning to understand what works and what doesn't, and if so why not. How big of a role does "good initialization" of the neural module operators plays? How important is defining the correct input and output format for a new module? How important is the selection of the few shot samples to evaluate a new module? (the authors say "We extracted 300 examples from GQA, 100 from RefCOCO, 10 from Raven, and 10 from MEWL based on experimental experience." - is the experience here just cherry picking for results or something else?) How important is the LLM capability to learn new modules? What role does the prompt play for the LLM in evaluating existing modules and creating new ones? Without detailed analysis to support answers to all these questions, the paper is limited in terms of explaining the method beyond just presenting a new method and showing empirical results.

Q1. Ablation on each new module.

Thanks for raising these insightful questions, which help in understanding the importance of different components in the GNSVR. We randomly select 800 samples from GQA test-dev split to further investigate the effectiveness of different components of GNSVR. To better present all experimental results, all the mentioned ablation studies are organized into three sections below.

Q1-1: Ablation on Prompt Design.

Table 1: Ablation study of Prompt Design on GQA.

We conducted a series of experiments to observe the impact of prompt design on the overall performance of GNSVR. Firstly, we removed the descriptions of input and output formats from the prompt. After removing these descriptions, the performance of GNSVR dropped by 2.7%. This is because, without clear guidance on input and output formats, the modules might output in the wrong format, leading to errors in subsequent parsing of the results. Furthermore, on top of removing the input and output format, we also removed some of the in-context examples and descriptions about module signatures from the prompt. The performance further declined. Since our method consists of three stages: module initialization, module generation, and module execution, where module initialization is the first step of our method. Without adequate module initialization as a foundation, the subsequent results are largely impacted. Therefore, we can see that without good initialization, our performance drops by 4.1%.

Regarding the use of existing modules and creating new ones, from the table above, we can observe that not using the predefined modules from VisProg results in a 0.9% decrease in our performance. This demonstrates the robust module generation capability of GNSVR. Even without a series of predefined modules, our method can still build modules from scratch, solve problems, and the performance does not drop significantly. If we don't create new modules, then we are merely using the predefined modules. We can see that the result is 44.7%, which is 1.2% lower than our result of 45.9%. This performance gap highlights the effectiveness of the newly generated modules. By generating and using new modules, we can achieve better results.

Q1-2: Ablation on Sampling

In this section, we introduce our sampling strategy at first. Then, we conduct an experiment to showcase how the sampling methods will impact GNSVR performance. Subsequently, we investigate how the number of training samples affects our results of different tasks.

Sampling Strategy

Table 2: Ablation on Sampling Strategy on GQA.

Our sampling strategy: the GQA dataset contains five structural types: choose, logical, compare, verify, and query. These structural types inspired the idea of generating our new modules. Taking COMPARE_COLOR as an example, this newly generated module is generated to address questions related to color within the compare structural type. From the visualization of GQA, it is apparent that the query type can be addressed using existing VQA module from VisProg, and problems in the logical type can be decomposed into sub-problems of choose, compare, and verify types. Therefore, when selecting training samples, we randomly chose 100 samples each from the choose, compare, and verify types. Altogether, these three types comprise 300 samples, all sourced from the GQA train split. Hence, we are not cherry-picking our training samples; rather, we are selecting training samples based on the structural types of GQA.

We appreciate the reviewer for the detailed comments and insightful suggestions.

Q1. The prohibitive cost of using SOTA large language models makes reuse of code appealing (although they explicitly don't use ChatGPT4 "due to the prohibitive cost", so maybe this is less of an argument than it would be otherwise).

Q1. Computational Efficiency. Thanks for mentioning the compute efficiency of our modularized design and module reusage. We have calculated the average token number of our GNSVR model and the ViperGPT that has no module reusage mechanism when calling the LLMs. The averaged generated token number is shown in Table 1. It can be seen that our GNSVR's solutions are shorter and more efficient. This result is updated in the revised paper. This approach is particularly advantageous when calling expensive APIs from the OpenAI GPT family.

Table 1: Average token number of generated solutions.

Q2. My main concern is that, based on the presentation, it seems that the authors took a lot of highly intricate API's for LLM's that large teams may have worked on and cobbled them together to solve a new task. I refer to this section: "The success of our GNSVR relies on a set of pre-defined modules and APIs as the starting point. We utilize handcrafted modules from VisProg (Gupta & Kembhavi, 2022) as our initial components. Additionally, we incorporate several new APIs from ViperGPT to enhance module creation. We also include some new APIs from ViperGPT (Sur´ıs et al., 2023) for making new modules." I appreciate their novelty in how they use these API's, but the ratio of insights of these authors vs of the authors of the API's appears insignificant.

Q2. About Reliance of Existing Modules. We agree that our GNSVR relies on some pre-defined modules as the initial start point to learn to generate new modules. However, we want to highlight that the novelty of GNSVR is not "taking a lot of highly intricate API’s for LLMs to work on and cobble them together to solve a new task". Instead, our novelty lies in growing and reusing the established modules that are learnt from the training set. The modules learnt by GNSVR can be applied to different domains, including 1). other instances of the visual reasoning task; 2). instances of a new reasoning task; and 3). adapting to new reasoning tasks by observing only a few training examples. Please refer to the G2 of the general responses for further explanation.

Q3. I'm also not entirely convinced of the novelty of this paper. I refer to "Iterative Disambiguation: Towards LLM-Supported Programming and System Design" (Pereira and Hartmann) and "Self-planning Code Generation with Large Language Models" (Jiang et al). I don't think the fact that this is in the visual domain is enough to call it "novel", because there is virtually no engagement with visual modalities by the authors - as stated above, according to my understanding, they are using predefined modules which handle the interface between vision and language.

Q3. About the paper's novelty compared with existing works. Thanks for reminding related works[1,2]. We have added such revision in the related work section of the revised paper. In [1], Pereira and Hartmann used LLMs to progressively enhance and specify system subcomponents, empowering users to develop versatile programs through a systematic iterative disambiguation method. In [2], Jiang et al learned- to generate code with LLMs, which involves a planning phase for outlining solution steps and an implementation phase for generating code. Besides the dense engagement with the visual modalities as input, our GNSVR is different from them in modularization of code snippets for better module expansion and module reusage. These unique differences make our GNSVR model to have new capabilities like growing new modules to handle other instances of the visual reasoning tasks VQA, image grounding, RAVEN and MEWL and reusing these new modules in new tasks like image editing and knowledge tagging. Such modularization also offers better computational efficiency (See Q1) and model transparency with high-level module abstraction (See the Generated Program in Fig. 3, 5 and 6 for an example). Please refer to the G2 of the general responses for further explanation. [1]. Iterative Disambiguation: Towards LLM-Supported Programming and System Design (Pereira and Hartmann). [2]. Self-planning Code Generation with Large Language Models (Jiang et al).

G4. Paper Revision.

Besides the experimental results, we have also revised the paper correspondingly (highlighted in blue):

• include experimental results into the revised version of the paper;
• discuss the related work[A,B,C,D,E] and their difference with our GNSVR model;
• revise text description for technique details.

[A]. Iterative Disambiguation: Towards LLM-Supported Programming and System Design". Pereira and Hartmann [B]. Self-planning Code Generation with Large Language Models. Jiang et al. [C]. Dynamic inference with neural interpreters. NeurIPS. 2021. Rahaman et al. [D]. Dataset interfaces: Diagnosing model failures using controllable counterfactual generation. 2023. Arxiv. Vendrow et al. [E]. Adaptive testing of computer vision models. 2023. ICCV. Gao et al.