LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence

We believe the Reviewer has significant misunderstandings of this work. In the following, we address the concerns one by one using more precise explanations and sufficient experiments.

Q1: Inference speed analysis

We did not explicitly compare the speed of our method with the counterparts because many of them are based on LLMs deployed on remote servers like GPT-4. Their speeds are largely influenced by network latency, so it is difficult to compare fairly.

To conduct a comprehensive speed comparison as suggested, we first report the efficiency metrics of our method as follows:

Then, we study how the success rates and inference times are changed by adopting different base LLMs. Due to the reply character limit, please refer to the reply to Q5 of Reviewer 2LZg for results.

For the online inference claim, it is because LARM is for high-level schedule. Each time of high-level schedule corresponds to several seconds of low-level skill execution. The inference speed of the low-level skill policy is more than 1000 FPS. So the LARM speed of 0.58 second per inference meets the online inference requirement. To avoid such misunderstandings, we change the claim in the paper as "meet the speed requirement of online high-level action scheduling".

Q2: Evaluated task number

Actually, we have test our method in other tasks of MineDojo. Our method is still very effective.

We do not report results on them because all previous works in MineDojo choose to select representative tasks to report results, although there are totally 1581 tasks in MineDojo. If we report the results on other tasks, we have no method to compare. The reported tasks in Table 1 of the paper follow previous published works.

To further show the effectiveness of our method, we add experiments based on a household simulator VirtualHome and a real-world robot. Refer to the replies to Q1 and Q2 of Reviewer 4ZZV for details.

Q3: Missing comparison

As reminded by the Reviewer, we will add all the missing references to the paper and discuss the relation with them.

In the following, we compare success rates with them. As their experiment settings are different from ours, we unify all methods using the setting of DEPS. The success rates on key achievements are as follows:

Q4: Significance from previous works

We believe this work shows great difference and advantages over the works mentioned by the Reviewer.

• The works mentioned by the Reviewer are also based on LLMs. This means if the LLM does not have accurate knowledge about this environment, the method fails. By contrast, ours combines LLM and RL. These two parts both contribute to learning new tasks. If LLM does not have accurate knowledge (the referee reward is noisy), the RL still conducts learning through environment reward. To show this, we have added experiments using a household simulator and a real robot. Refer to the replies to Q1 and Q2 of Reviewer 4ZZV for results.

• The methods mentioned by the Reviewer are all based on a stack of heavy models, meaning slow inference and high deployment cost. However, embodied applications mostly require fast response and deployment on local devices. Our real robot experiment shows the trained policy can be deployed using the local resource of a robot to achieve good success rates.

Q5: Unfair comparison

The comparison is absolutely fair. Comparing methods in MineDojo and Mineflayer, our method does not use any extra information than the compared methods. If we call an API, the compared methods also call this API to get information. The inventory list and environment information used by our method are also used in other methods.

Q6: How skills are generated

We use a skill generation pipeline similar to Voyager, but the overall method framework is very different. Voyager completely relies on GPT-4, a remote LLM, to complete tasks in an offline way. Ours supports online learning and is lightweight.

Q7: Skills in MineDojo

Although the basic actions in MineDojo are low-level actions like moving forward a step, almost all previous works in MineDojo use RL policies and rule-based methods to build higher level skills, like search a tree. The Reviewer can refer to the code of Plan4MC to know how these skills are built. Our skills used in MineDojo completely follow them, therefore the comparison is absolutely fair. We do not use any extra information.

Q8: Success Rate 0.65

We test our own method for 30 times to compute the success rate. The 0.65 is the success rate of a compared method and is obtained from its original paper.

We have addressed the concerns of the Reviewer one by one in the following. The paper will be revised accordingly.

Q1: Real-time inference

The speed of 0.58 second per inference is the inference time of the high-level scheduling policy. Each time of high-level schedule corresponds to seconds of low-level skill execution. The inference speed of the low-level skill policy is more than 1000 FPS. Therefore, we claim that our method meets the online inference requirement.

To avoid misunderstanding, we change the claim in the paper as "meet the speed requirement of online high-level action scheduling".

Q2: Similar to two previous works

The two works mentioned by the Reviewer are both about low-level action learning, while ours is about high-level scheduling. The motivation, method, and results are all very different:

• CLIP RL: This work uses CLIP attention map to replace text instruction and provides an invariant representation for the policy to approach different objects. This work does not have significant similarity with ours, which combines LLM and RL.

• Auto MC-Reward: This work relies on LLMs to write code to generate reward functions. Its idea can be applied to learn low-level skills where reward functions can be defined explicitly. However, in many tasks (like the real robot block building task in the Reply to Q2 of the Reviewer 4ZZV), the rewards are too subjective and complex to define explicitly, so this work is inapplicable. By contrast, our method does not have this limitation.

As suggested by the Reviewer, we add comparison to these two works. For fair comparison, all three works adopt the train and test settings of Auto MC-Reward. The success rates on different key achievements are as follows:

Q3: Test in more environments

As suggested by the Reviewer, we have test our method in more environments, including VirtualHome (a household activity simulator) and the Cobot Magic robot (a robot in the real world). Due to the reply character limit, please refer to the replies to Q1 and Q2 of Reviewer 4ZZV for experiment details and results. We believe these experiments sufficiently confirm the practical value of our work.

Q4: How skills are generated

The skills are generated by GPT-4. Our fine-tuned model is used for online high-level action scheduling. It does not serve as the referee model.

Q5: Generalization to more LLMs

Our method can be generalized to different LLMs. As suggested by the Reviewer, we add experiments that test using different LLMs as the referee and LARM base model.

First, we study the success rates adopting different sizes of referee LLMs, including TinyLLaVA-3.1B, Fuyu-8B, Llama3-8B, and Qwen2-14B. Notably, these models do not always provide correct referee reward, meaning the reward quality is different. The success rates are as follows:

According to the results, we can observe that generally LLMs with more parameters lead to better performance. The key factor is the quality of the generated referee reward. Refer to the replies to Q1 and Q4 of Reviewer xu7T for more in-depth analysis on the reward noise tolerance of LARM.

Then, we study replace the LARM based model from TinyLLaVA-3.1B to Fuyu-8B and Qwen2-14B. The success rates and speed are as follows:

According to the results, we can observe that replacing the base model with a larger one generally benefits the performance. However, the inference time is also increased significantly.

Q6: Missing reference

As suggested, we will add all these references to the paper and discuss the relation with them.

Q7: Wiki Pre-train effect

The Wiki pre-train improves model convergence speed and success rates on different tasks significantly. Actually, we have reported the success rate gains caused by Wiki pre-train in Table 4 of the paper. For the convenience of review, we present the results in the following table. Besides success rates, we also report the exploration iteration number for arriving convergence.

The Wiki data preprocessing pipeline is the same as the work MineDojo.

We have test our method in more environments as suggested, and the details are in the following. The paper will be revised accordingly.

Q1: Experiment in a household simulator

We thank the Reviewer for this suggestion. As suggested, we conduct experiments in VirtualHome to further validate the effectiveness of our method. We design 10 tasks that each task requires 4 to 6 steps of low-level skill execution. An example of the designed task is "watch TV". To complete this task, the agent needs to complete the following five steps in sequence: (1) Walk to TV. (2) Turn on TV. (3) Walk to a sofa. (4) Sit on the sofa. (5) Watch TV. An anonymous video link describing how LARM performs this "watch TV" task is provided here: VirtualHome Video.

In this experiment, we remove the Wiki data pre-train step for LARM to show that our algorithm can also work well without Wiki data pre-train. we employ Qwen-VL-Max as the referee model. Notably, Qwen-VL-Max does not always provide correct referee reward (about 10% error rate). Therefore, this experiment shows the robustness of our method based on noisy reward. Refer to the replies to Q1 and Q4 of Reviewer xu7T for more in-depth analysis on the robustness of our method under noisy reward.

In this experiment, we compare our method with classic RL methods including Deep Q-learning, TPRO, and PPO to reveal how much efficiency is improved by combining RL and LLM using our method. The input to policies includes task target description,agent initial state, historical actions. As VirtualHome provides the function to execute each low-level skill, we do not need to implement low-level policies like in Minecraft. For each method, we report the avarage times of explorations the policy needs to complete the designed tasks for the first time (all actions are predicted without random exploration). The results are as follows:

The above results sufficiently show the efficiency advantage of LARM. For tasks needing longer action chains (like tens to thousands of steps are needed for the tasks in Minecraft), classic RL methods like Deep Q-learning cannot complete a task for the first time after millions of exploration times. By contrast, LARM learns to complete the task efficiently.

Q2: Experiment in the real world

As suggested by the Reviewer, we design an experiment in the real world to further validate the practical value of our method. In this experiment, we train a LARM policy to control a robot to build blocks.

Specifically, several blocks in various shapes are randomly placed on a table. The LARM policy is trained based on our proposed referee RL algorithm to stack blocks as a building similar to a house. To complete this task, the policy needs to learn to select suitable blocks and decide their relative positions to place. The basic skills are implemented based on imitation learning, where we adopt the robotic manipulation algorithm VIRT to grasp and place blocks.

We use Qwen-VL-Max as the referee model. It judges whether the building is becoming more like a house based on image observation. Notably, we do not clearly define how a house should look like. Therefore, there are multiple ways of stacking different blocks to be similar to a house. Correspondingly, the referee reward provided by the referee policy is noisy.

Interestingly, after 100 iterations of random exploration, the policy learns to select and stack different blocks as a house. The anonymous videos of two stacking block process examples are provided here: Robot Stack Blocks 1 and Robot Stack Blocks 2.

According to the results of this experiment, we can conclude that (1) Our proposed method can be applied to a real robot task. (2) By combining LLM and RL, our method can derive policies that are able to perform creative tasks.

We have addressed all concerns of the Reviewer in the following. The paper will be revised accordingly.

Q1: Referee RL stable update proof

We thank the Reviewer for this reminder and will add this proof to the paper. Due to the reply character limit, we cannot provide the whole proof here, but we can give a concise description.

PPO can be treated as an extension of TPRO, and the stable update of TPRO has been proved. Thus, to prove the stable update criteria of referee RL, we need to focus on the key difference between referee RL and PPO, the reward noise. To imitate this noise, we replace the correct reward as false reward with a ratio $\sigma$ with respect to a random distribution. Empirically, if $\sigma$ exceeds a threshold $\epsilon$, the policy cannot converge stably. We can analyze the value of $\epsilon$ based on the gradient bias analysis in stochastic optimization theory.

Q2: The impact of referee RL and relation with RL techniques

Referee RL can be applied to various RL algorithms beyond PPO.

Referee RL combines the techniques of RL and LLM to address the key problems in these two communities, therefore benefiting both communities.

The key problem of RL is its exploration inefficiency. LLM has general knowledge. Our work shows that LLM can provide guidance to RL and reduce the exploration cost very significantly (like the experiment results in the Reply to Q1 of Reviewer 4ZZV).

The key problem of LLM is it lacks the knowledge of direct environment interaction. Based on RL exploration, we tune a lightweight LLM into a SOTA embodied agent.

Our method does not conflict with the RL techniques mentioned by the Reviewer, such as hierarchical RL. They handle the long-horizon exploration problem from different perspectives and can be applied simultaneously.

Q3: Exploration on more RL methods

The RL methods mentioned by the Reviewer include hierarchical RL, inverse RL, reward relabling, Decision Transformer, more RL baselines. These methods do not conflict with our method and can be applied simultaneously. We will add discussion on the relation with these methods to the paper.

Specifically, our LARM model is for high level scheduling, and a part of the low-level skills are implemented based on PPO or deep Q-learning. So hierarchical RL has been used in our work.

Inverse RL assumes there are optimal expert demonstrations. However, the demonstrations are unavailable in our studied task. For reward relabling, it assumes the historical actions are optimal under the reward function. However, this assumption is often too strong to meet, and the reward is also difficult to define explicitly.

We have tried the idea of Decision Transformer, which relies on training on expert demonstration data. To realize this idea, we first collect abundant demonstration data, and the method achieves similar performance as referee RL with less computing cost (4 hours of training). However, the expert demonstration data is often unavailable.

We have tried replacing the PPO in referee RL as deep Q-learning, and the result comparison is as follows:

We can see that PPO based referee RL gets better performance.

Q4: Influence of referee reward noise

We have analyzed reward noise influence theoretically in the reply to Q1. This part analyzes it using experiments. We randomly modify the reward generated by GPT-4 with a ratio $\sigma$, and the success rates on different tasks are as follows:

We can observe that (1) Stronger noise deteriortes the sucess rates. (2) Tasks requiring longer execution chains are less noise tolerant.

In addition, we design two new experiments using the VirtualHome simulator and a real robot, where the referee LLM sometimes provide incorrect rewards. Refer to the replies to Q1 and Q2 of Reviewer 4ZZV for experiment details and results.

Q5: Results on smaller referee LLMs

As suggested by the Reviewer, we study the success rates adopting different sizes of referee LLMs. Due to the reply character limit, refer to the replies to Q5 of Reviewer 2LZg for details.

Q6: Computing efficiency analysis

The reason we did not provide quatitative efficiency comparison with previous LLM based methods is these methods are often based on large LLMs deployed on remote servers, like GPT-4. We cannot know their specific parameter numbers, FLOPs, etc. What we can make sure is their computing costs are enormous as publicly known and far more than ours. We report the efficiency metrics of our method as follows: