Learning Embodied Vision-Language Programming From Instruction, Exploration, and Environmental Feedback

W1. The data collection process relies on GPT-4, which takes as input language of systems message + environment message to output the required plan, code, and target state. This relies on the strong assumption that the systems + environment message fully captures the environment state, since the planning must be done without access to the view of the visual scene. I presume this means that the systems prompt must be elaborate, handcrafted, and task specific, such that GPT-4 can plan reasonably in the environment using possible objects at hand. Is there a robust way of designing such prompts for different/new tasks without tuning?

We thank the reviewer for your valuable comments. You are right, we make sure the environment message fully captures the environment state since when we design a task, we manually inspect all the necessary objects that should be in the environment message. An automatic alternative is to put everything, no matter whether necessary or not, into the environmental message. However, we tried but we found that GPT-4 will not act properly if the environmental message is too long. However, since OpenAI keeps enlarging the input length support, maybe this issue could be addressed in the future. Besides, since this work is to have an initial exploration of the vision-language programming, which remains uncharted to date, we think some handcrafted work is required, and support our future work of making the training pipeline more flexible, such as iterative training, i.e., with an Octopus model that is trained with limited half-handcrafted environment message, the model itself can explore in the environment with visual input, and collect data for next training loop. However, we will keep this exploration in our future exploration.

W2. From my understanding, GPT-4 generates the target states, and whether the target states have been met is used as environmental feedback for training Octopus. It seems possible that GTP-4 will generate an incorrect or trivial target state to satisfy the language task goal, and be able to successfully reach that predicted target state, without actually achieving the task goal. Is this understanding correct? In this case, the errors from GPT-4 seem more harmful than having unsuccessful execution from GPT-4 generated code.

Yes, sometimes GTP-4 will generate a trivial target state to satisfy the language task goal, but in this case, the main task will not be met, and according to Task-level judgment in Section 3.3, all the steps will marked as failure once the main task goal is not met, and the RLEF training will use the failure to train the model not to do such trivial plan. However, these trivial steps, as long as they do not make syntax errors, will be used for SFT training, helping the model to write good code.

W3. The results on Table 2 are not particularly convincing of this method’s success. Octopus should indeed outperform blind LLMs that do not take visual input. TAPA outperforms/is of equal performance in 2 of the 5 tasks. The paper lacks analysis on why this is the case. Where does TAPA fail? It is also hard to compare when the vision models are different; does OVD and CLIP-ViT perform similarly in terms of capturing information from the input scene?

Actually, the input of the blind LLMs in Table 2 is the textual representation of the objects (and the relations) that are shown in the vision image, i.e, we put everything that exists in the robot's vision into the text to feed into LLMs, so although it is blind, it can "listen" what the environment has. So the higher task completion rate of Octopus does show the superiority of vision input and vision components in Octopus. In Table 2, Octopus reaches a good result on almost all the metrics except the seen environment, showing that Octopus, with the vision as the input can have a better generalization ability in planning. The pure language models, like TAPA, when given the familiar textual input that reflects the environment, seem to have a better result. But we believe task and environmental generalization is a more important aspect.

Nit: Should add a figure describing Octopus model architecture; notations in methods section are not well defined.

Thank you for pointing this! We draw the training pipeline in Figure 4, as the architecture is simply following the Flamingo framework (since Otter follows Flamingo), which can be accessed here.

The novelty of this paper is limited. Essentially, it uses GPT4 with the prompt engineer to collect data from two embodied environments and then trains a vision-language agent model. Moreover, The agent model does not have a specific framework diagram to show the detailed parts, making it difficult to see which part of the model is its innovation point. I suggest providing a framework diagram to clearly explain where the model is newly proposed in the paper and how it differs from existing methods.

We sincerely thank the reviewer's valuable feedback and we add the diagram in Figure 4. Our core motivation is to develop a domain that remains uncharted to date, i.e., an embodied vision-language model to make reasonable plans and write executable programs. The novel technical part is that (1) it is the first attempt to have a VLM for code writing and (2) it designs a novel solution for planning improvement in the RLEF part, which collects the success/failure information when GPT-4 is collecting the vision-code pairs in the simulator.

This paper appears to employ an intentional use of uncommon or less frequently used words in many sentences, substituting them for simpler, more common terms that could convey the message more clearly. As a result, the reading experience becomes somewhat disjointed, and the text may come across as rather weird to the reader.

Thank you for the suggestion and we have revised the whole manuscript thoroughly.

The reason for using both OmniGibson and GTA-V environments to generate data seems not obvious. A more obvious comparison between the two environments is required, such as the visual comparison of the tasks.

The reason of us building OctoVerse is to support the development of our embodied vision-language programmer. OctoVerse is generally based on the existing, most realistic simulators of GTA and OmniGibson, where we add the necessary adaptation to include the instruction set of callable functions and carefully designed tasks. We include the comparison between OctoVerse with other platforms in appendix C1. The general reason for selecting these two platforms is the consideration of their realistic modeling task designability and scalability. Admittedly, other simulators such as AI2THOR and VirtualHome might also be able to support the Octopus exploration if we also design an instruction set of callable functions and carefully designed tasks, we eventually selected OmniGibson and GTA for realistic modeling and task designability and scalability. The detailed task can be accessed here and in the appendix.

In Section Error Management, when the agent executes the wrong command, how does the method perform "error management" on it? It seems that this section only claims the cases under which a task is defined as failure, and does not show how to manage such failure.

Yes, we only annotate commands with "success" or "failure" tags, and these tags are the supervision of the reward modeling in the RLEF part. According to Figure 4 in the paper, all the vision-code pairs in the (a) data collection pipeline will be used for the SFT training in Figure 4(b) Octopus training pipeline, since the GPT-4 does a good job on formatting, meaning that they can write reasonable code. And the "success" or "failure" tag will be used in the RLEF training.

In Section ENVIRONMENTAL FEEDBACK, what if there are multiple erroneous states in a task sequence? Which are the positive states and which are the error states at this time? And it may not be said that if one of the states is wrong, then its previous states must all be negative.

According to Figure 3, one error state is tagged if, after one step, post-execution states misalign their target states. Alternatively, if a task is not completed as intended, every state within that task is labeled as negative. Sorry for the previous confusion and we have clarified this part in Section 3.3.

More related works in Section 2.1 are needed to help the reviewer identify the paper's contributions, like [1-4].

We sincerely thank the reviewer for pointing out these important works, by comparing them we can better show the accurate standing of the Octopus work in the community. We add all these four works in the Table 1, and we show that the proposed Octopus distinguishes itself from other models as an end-to-end, unified vision-language model for both plan and code generation.

The authors term the simulation they used "OctoGibson" which is built upon OmniGibson. Can the authors give more details to elaborate on the main difference between them, or did they just use that Simulator to collect the dataset?

The OctoVerse is generally based on the existing realistic simulators of GTA and OmniGibson, where we add the necessary adaptation to include the instruction set of callable functions and carefully designed tasks that make the development of the embodied vision-language programmer training possible.

A better format is needed. Some lines need a reformat in the revised version. One example is "3.2 Instructions From Exploration", lines above and below seem to belong to the same paragraph.

Thank you for pointing it out! Yes, we have drastically revised the whole draft several rounds and made sure it is easy to understand with good structure.

More experiments are needed. The author only conducts the experiments on the datasets they collected and lacks a direct comparison with more relative frameworks as discussed in Section 2.1.

Yes, we have added several experiments in Section 5.1 and Table 2, where we add throughout comparison with "Blind LLM" including LLaMA, CodeLLaMA, TAPA, which can only "listen" to what is inside the environment. We also implement the EmbodiedGPT for comparison. These experiments have shown several conclusions including:

• Blind LLMs Struggle with Extended Input Content.
• CodeLLaMA Improves Coding but not Planning.
• Octopus Demonstrates Superior Task Generalization Ability as it integrates the visual input.
• RLEF Enhances Octopus’s Planning Strategy, which is more significant than the components in EmbodiedGPT.

These experimental results show the promising direction of the end-to-end vision language models for planning and programming.

The author uses the GPT-4 to collect the training data, and one implicit assumption is that the performance of the GPT-4 is optimal or near-optimal. A comparison between the GPT-4 generated data sample and the human-collected sample would help. Or, did the author conduct some data quality control before the use of dataset?

Thank you for the great question! Actually, when we use GPT-4 for the automatic data collection pipeline, the "environmental feedback" generally acts like the data quality controller. More specifically, when GPT-4 writes the plan, code, and target state, the code will be run in the simulator and check whether the target state is met and whether the final mission is satisfied. All these results of success/failure will be valuable resources for RLEF training. In fact, we intensively designed a very comprehensive system message for GPT-4 and it is good at following the structure with near-perfect code format (seldom syntax error occurs), and the syntax correctness can be identified by the simulator execution and we will remove these with syntax incorrectness when training SFT.

Section 1 / Throughout — The manuscript forgets to properly motivate its contributions. What problem is this work supposed to be solving? What research questions are examined by this manuscript?

We sincerely appreciate the constructive feedback regarding the need to more clearly articulate the motivation behind our work. We have thoroughly revised our Introduction section (highlighted in the submission paper). Our core motivation is to develop a domain that remains uncharted to date, i.e.,an embodied vision-language models to make reasonable plans and write executable programs”. While there are existing models like ToolFormer, HuggingGPT, ViperGPT, and VisProg that delve into programming with non-visual inputs, Octopus stands out in its endeavor to merge these programming models with visual stimuli. Consequently, recognizing the lack of suitable environments for conducting such innovative experiments, we have developed OctoGTA and OctoGibson, enabling us to explore this new territory of vision-language model programming. We hope this explanation can help clarify the standing of the motivation and the purpose of the two environments.

Section 1 — Most of the Introduction section is unnecessary. The space should instead be used to describe what is added on top of GPT-4 to make the model proposed in this paper a sufficiently distinct contribution. How is OctoVerse different from other EAI simulators? Why does the community need OctoVerse? What problems can be solved in OctoVerse that cannot be solved elsewhere? The manuscript fails both to motivate and explicitly describe its contributions.

Thank you again for the constructive suggestions! We are generally using GPT-4 to prepare for the vision input and executable program pairs, and the core contribution is to illustrate the embodied vision-language models can have better achievement than the unimodal language models, with experimental support in Table 2, where we see (1) the drawbacks of unimodal LLMs: they struggle with extended input content that describes the visual input; and (2) The significance of visual inputs in task performance. The OctoVerse is generally based on the existing, most realistic simulators of GTA and OmniGibson, where we add the necessary adaptation to include the instruction set of callable functions and carefully designed tasks. We include the comparison between OctoVerse with other platforms in Appendix C1. The general reason for selecting these two platforms is the consideration of their realistic modeling task designability and scalability. Admittedly, other simulators such as AI2THOR and VirtualHome might also be able to support the Octopus exploration if we also design an instruction set of callable functions and carefully designed tasks, we eventually selected OmniGibson and GTA for realistic modeling and task designability and scalability. In the revised Appendix C1, we incorporate various attributes that enhance diversity and realism, noting a substantial advancement in both OctoGTA and OctoGibson simulation environments. First, a wide range of well-formulated tasks are established in Octopus's simulator showcasing from fine-grained indoor routine activities to outdoor open-world tasks. This contrasts with other simulation environments that often target a relatively restricted set of activities.