Thank you for your feedback. We would like to offer further clarification beyond the general response above to address the specific concerns and points raised in the review.

Weakness: While the paper is well-written for the scope it sets for itself, I am not sure if the contribution is significant enough. There are many works using LLM to generate data and fine-tune domain-specific models, so the idea behind this paper is not super novel.

Response:

While fine-tuning domain-specific models using techniques such as Self-Instruct is common, obtaining high-quality correct instruction-program pairs remains a challenging problem. Since custom-defined domain-specific APIs may not be present in the training corpus of existing LLMs, methods like Self-Instruct can generate incorrect training data, as shown in Appendix A.2.1. Table 1 in the paper also highlights the performance gap when using the Self-Instruct method.

Verifying the correctness of these generated programs using a simulator is a natural approach. However, creating an appropriate simulation environment for each program can be challenging, often requiring manual specifications of entities, types, and states. In this work, we address this challenge by introducing a novel approach that automates the generation of simulation environments for each generated program. As outlined in the general response, this method is versatile, with the potential to be extensible beyond robotics, capable of handling arbitrary open-world tasks without the need for manual coding of entities, types, or states.

Weakness: The performance gain is also limited by the rather heuristic method considering the gap between the best model presented by the paper and GPT-4 it sought out to beat. In fact, given the 17% performance gap, simple baselines could be using GPT-4 to generate the programs and fine-tuning small models or using GPT-4 as the critic to filter programs. These simpler heuristics may yield better results and prove the Robo-Instruct method (which is also rather heuristic) proposed by the paper unnecessary. For reference consider [1] Improving Small Language Models on PubMedQA via Generative Data Augmentation. Therefore, I am not sure the contribution of this paper is all that significant.

Response:

Our preliminary results show that a simple distillation of GPT-4 is not effective for fine-tuning domain-specific models, and the best pass@1 score was around 64%, which is lower than the score reported in Robo-Instruct. We will add this information to the appendix of the revised paper.

More importantly, we chose to generate data from an open-weight model instead of proprietary models like GPT-4 because open-weight models are often preferred in practice. They are cost-free, can be deployed on local servers for greater flexibility, and help address privacy concerns when dealing with sensitive information related to custom domain-specific APIs.

Question: Could you explain why InstAlign would improve pass@1 success rate by ~5% on top of RoboSIM+RU in table 2? What's the error bar/variance of these statistics? What is the intuition that InstAlign would help and could you give some examples?

Response:

The intuition behind why InstAlign would improve pass@1 accuracy is twofold:

1. Specificity of Aligned Instruction: The aligned instruction is more specific to the generated program, providing clearer guidance on the intended actions. Instead of offering generic instructions, it can communicate what the program aims to accomplish more precisely. For example:

def task_program():
    for person in ["Arjun", "Alice", "Eve"]:
        response = ask(person, “Do you like the weather today?”, [“yes”, “no”])

       Original instruction: Ask everyone in the room if they like the weather today

       Aligned Instruction: Ask Arjun, Alice, and Eve if they like the weather today

2. Consistency with Program Actions: Since the program is generated stochastically, it may modify actions that are misaligned with the original instruction. Aligned instructions can correct these discrepancies, as seen in this example:

def task_program():
    if not is_in_room("pen"): 
        pick("pen")

       Original instruction: Ask if there is a pen here. If so, pick it up.

       Aligned Instruction: Check if there is a pen here. If so, pick it up.

Aligning the instruction with the program may reduce ambiguity during fine-tuning, leading to improved performance by minimizing noise and enhancing the clarity of the tasks being performed.

Question: One claim made early on in the paper is that Self-Instruct gives high-diversity data but lacks correctness, and a simulator can give correctness but lacks diversity. The paper is trying to get the best of both worlds. While the current experiment results show that Robo-Instruct does generate data with high correctness, we don't see evidence that Robo-Instruct preserves diversity. Can you show some evidence (through some metrics) on diversity?

Response:

We analyze the generated dataset in Appendix A.3. Our findings indicate that Robo-Instruct is capable of producing data with a distribution similar to that of Self-Instruct. Therefore, Robo-Instruct maintains the diversity of Self-Instruct.

def task_program():
    start_loc = get_current_location()
    list_of_rooms = ["A", "B", "C", "D", "E", "F", "G"]
    money = 1
    for room in list_of_rooms:
        go_to(room)
        if is_in_room("person"):
            say("I have $" + str(money))
            response = ask("", "Would you like to take the money now or double the amount and give it to the next person?", ["Take now", "Double and give"])
            if response == "Take now":
                go_to(start_loc)
                say("The game is over, I gave the money to " + room)
                return
            else:
                money *= 2
    go_to(start_loc)
    say("No one took the money, I still have $" + str(money))

def task_program():
    start_loc = get_current_location()
    rooms = ["A", "B", "C", "D", "E", "F", "G"]
    money = 1
    for room in rooms:
        go_to(room)
        if is_in_room("person"):
            say("I have " + str(money) + " dollars")
            response = ask("", "Would you like to take the money now or double the amount and give it to the next person?", ["Take it", "Double it"])
            if response == "Take it":
                say("Game over. The person in room " + room + " took the money.")
                go_to(start_loc)
                return
            else:
                money *= 2
    go_to(start_loc)
    say("No one took the money. I still have " + str(money) + " dollars.")

def task_program():
    start_loc = get_current_location()
    go_to("my office")
    table_found = is_in_room("table")
    chair_found = is_in_room("chair")
    monitor_found = is_in_room("monitor")
    if not table_found:
        go_to("Jason's office")
        if is_in_room("Jason"):
            response = ask("Jason", "Can I borrow the table from you?", ["Yes", "No"])
            if response == "Yes":
                go_to("Jason's office")
                pick("table")
                go_to("my office")
                place("table")
            else:
                go_to(start_loc)
                say("Jason said no")
        else:
            go_to(start_loc)
            say("Jason is not in his office")
    if not chair_found:
        go_to("Jason's office")
        if is_in_room("Jason"):
            response = ask("Jason", "Can I borrow the chair from you?", ["Yes", "No"])
            if response == "Yes":
                go_to("Jason's office")
                pick("chair")
                go_to("my office")
                place("chair")
            else:
                go_to(start_loc)
                say("Jason said no")
        else:
            go_to(start_loc)
            say("Jason is not in his office")
    if not monitor_found:
        go_to("Jason's office")
        if is_in_room("Jason"):
            response = ask("Jason", "Can I borrow the monitor from you?", ["Yes", "No"])
            if response == "Yes":
                go_to("Jason's office")
                pick("monitor")
                go_to("my office")
                place("monitor")
            else:
                go_to(start_loc)
                say("Jason said no")
        else:
            go_to(start_loc)
            say("Jason is not in his office")
    go_to(start_loc)
    say("task is completed")

def task_program():
    start_loc = get_current_location()
    go_to("my office")
    missing_items = []
    for item in ["table", "chair", "monitor"]:
        if not is_in_room(item):
            missing_items.append(item)
    if len(missing_items) > 0:
        go_to("Jason's office")
        if is_in_room("Jason"):
            for item in missing_items:
                response = ask("Jason", "Can I borrow your " + item + "?", ["Yes", "No"])
                if response == "Yes":
                    pick(item)              <= BUG: did not return to Jason's office after going back to my office
                    go_to("my office")      <= BUG: did not return to Jason's office after going back to my office
                    place(item)             <= BUG: did not return to Jason's office after going back to my office
                else:
                    go_to(start_loc)
                    say("Jason did not allow me to borrow his " + item)
                    return
        else:
            go_to(start_loc)
            say("Jason is not in his office")
            return
    go_to(start_loc)
    say("Task is completed")

Thank you for your feedback. We would like to offer further clarification beyond the general response above to address the specific concerns and points raised in the review.

Weakness: While ROBO-INSTRUCT offers significant advancements for fine-tuning language models in robot programming, there are some weaknesses to consider. Firstly, it heavily relies on SELF-INSTRUCT for generating initial programs, potentially introducing biases from the base model's training data. This could limit the diversity and quality of the generated programs.

Response:

As discussed in Section 5 (Conclusion, Limitation, and Future Works), we acknowledge that our proposed method, which builds on Self-Instruct, could potentially be influenced by biases in the base model's training data. However, we would like to emphasize that our method is orthogonal to many other popular approaches used in Self-Instruct. Therefore, it is possible to apply Robo-Instruct alongside these methods to mitigate these concerns. For instance, the OSS-Instruct method by Magicoder (Wei et al., 2023) and the Evol-Instruct method by Wizardcoder (Luo et al., 2023) both aim to enhance the quality and diversity of data generated from Self-Instruct. Magicoder has demonstrated that combining these orthogonal methods (OSS-Instruct + Evol-Instruct) can lead to better outcomes. Consequently, we believe that using Robo-Instruct in conjunction with these methods could enhance the performance of fine-tuned models in domain-specific applications. Investigating the integration of these orthogonal methods for generating domain-specific datasets would be a valuable direction for future work.

Weakness: Moreover, while ROBO-INSTRUCT shows promising results on benchmarks like ROBOEVAL, its application to real-world robot programming tasks requires thorough evaluation and validation. Real-world robot environments often present unpredictable challenges that benchmark datasets may not fully capture, necessitating further testing to assess the framework's robustness and generalizability in practical applications.

Response:

We have deployed the Robo-Instruct fine-tuned model on a real robot using edge computing to demonstrate its real-world practicality, as illustrated in the PDF in the general response. This information will be included in the appendix of our revised paper.

Question: What factors do you think contribute to GPT-4's superior performance compared to ROBO-INSTRUCT, despite your method showing competitive results against other models like GPT-3.5-Turbo and Gemini-1.0-Pro

Response:

It is possible that GPT-4 has a significantly larger number of model parameters and is trained with more resources. As a result, GPT-4 has much longer inference times. In contrast, our fine-tuned model can perform inference five times faster, as illustrated in the PDF in the general response.

Thank you for your feedback. We would like to offer further clarification beyond the general response above to address the specific concerns and points raised in the review.

Weakness: RoboSIM can only check for semantically meaningful steps of the code and may not catch lower-level error that requires spatial/geometric reasoning, or even reasoning about physics, including commands that take in numerical parameters e.g. move(0,0.2,0), rotate(0.75). This seem to limit the usefulness of RoboInstruct to certain types of robot APIs.

Response:

RoboSim is designed to validate the correctness of programs given a particular domain --- defined by its API, types, and properties to check. We focus on task-level correctness as the domain of application, as it is of great broad interest as mentioned in the general response. However, as described in the general response, it is extensible to other domains, and we illustrate here how the domain can be easily expanded to check for lower-level actions if that is the desired level of verification.

For example, consider a tabletop manipulation setting. A possible API function is rotate(robot_gripper, radians). From the statement rotate(“left_hand”, pi/2), RoboSim will infer that left_hand is an entity with the type of the robot gripper, and the state of left_hand is its current rotation. If there is a domain-specific constraint on the rotation of the robot gripper, such as the gripper can only rotate between $-\pi/6$ to $\pi/6$ radians, then this statement rotate(“left_hand”, pi/2) becomes invalid because no matter where the state rotation position of the robot gripper is, rotating $\pi/2$ radians will exceed the maximal allowable rotation range of the robot gripper: $\pi/2 > \pi/6 + \pi/6$.

Question: Can the prompt of InstAlign be adapted to the instruction generation step? What information after the fact is being used by instAlign that cannot be useful at initial generation round?

Response:

InstAlign can be applied in the initial generation round, which can occur after the Self-Instruct phase and before sending the program to RoboSim. However, as shown in Table 2, where the results compare "+Reject Unsolvable (RU)" vs. "+INSTALIGN + RU", our experiment indicates that this adaptation is not effective.

The key insight here is that InstAlign proposed in Robo-Instruct will align the instruction with the validated programs, whereas adapting it at the initial generation round, the generated program could be invalid, thus limiting its effectiveness.

I have read other reviewer's feedback and would like to keep my current rating.

• I believe "service mobile robots" is a broad enough scope and this work makes meaningful contribution by grounding code generation for robot-specific tasks with logic. The idea of this work can be broadly applied to other robotic applications.
• I also find it unfair to diminish the contribution of this work if we expect future generations of VLM/GPTs will be better at code generation for robots. Embodied code synthesis faces its own challenges than training generic VLMs and this paper provides a framework for safeguarding code generated automatically.

Thank you for your feedback. We would like to provide further clarification beyond the general response above to address the specific concerns and points raised in the review.

Weakness: Limited novelty: The idea of using a sim/emulator to verify the generated program has already been explored in previous works such as Chain-of-code, which is not mentioned in this work.

Response:

The general response explains the novelty of our approach, and we focus here on the relation between RoboInstruct and Chain-of-Code (CoC), which we will cite in the paper revision. CoC focuses on enhancing LLM's reasoning capabilities. It selectively simulates the interpreter by generating the expected output of certain lines of code, which the default interpreter could not execute. The simulation procedure involves an LLM to determine valid output values that will be saved into a variable in the program. For example, given a line of the program

answer += is_sarcastic(‘you don’t say’)

The LLM simulator is responsible for inferring the output of the function is_sarcastic(string), and determines you don’t say is sarcastic and subsequently returns the value of 1.

However, such methods are unsuitable for the problem this work addresses. For example, consider the API function pick_up, mentioned in the general response. This API function does not return a value; instead, it updates the state of the robot in the simulation environment to indicate that the robot is holding the object. Consequently, using an LLM as the interpreter cannot simulate the state of entities in the simulation environment.

Weakness: Limited scope: The paper focuses on a specific domain (service mobile robots), and it is unclear how well ROBO-INSTRUCT generalizes to other robot domains.

Response:

We would like to highlight that our focus on general-purpose service mobile robots is a widely studied and popular area in the AI community as mentioned in the general response. In addition to the example of how RoboInstruct can be applied to other domains, we provide an additional example here of applying RoboInstruct to different types of robot tasks beyond service mobile robots. For example, consider a tabletop manipulation setting. A possible API function is rotate(robot_gripper, radians). From the statement rotate(“left_hand”, pi/2), RoboSim will infer that left_hand is an entity with the type of the robot gripper, and the state of left_hand is its current rotation. If there is a domain-specific constraint on the rotation of the robot gripper, such as the gripper can only rotate between $-\pi/6$ to $\pi/6$ radians, then this statement rotate(“left_hand”, pi/2) becomes invalid because no matter where the state rotation position of the robot gripper is, rotating $\pi/2$ radians will exceed the maximal allowable rotation range of the robot gripper: $\pi/2 > \pi/6 + \pi/6$.

Weakness: Lack of real-world evaluation: The paper only evaluates ROBO-INSTRUCT on a synthetic benchmark. Real-world deployment and testing are required to further assess its practicality.

Response:

We have deployed the Robo-Instruct fine-tuned model on a real robot using edge computing to demonstrate its real-world practicality, as illustrated in the PDF in the general response. This information will be included in the appendix of our revised paper.

Question: Figure 1 is confusing as the authors put the overview of the proposed method, counter examples from previous methods, as well as the benchmark results all in a single figure.

Response:

Thank you for the feedback. Our intention was to provide a quick summary to the reader to highlight the relation to previous approaches, our solution, and the high-level results. This can be presented with sub-figures to separate them out to make them clearer - we can make this change in the revision.