LLaMA Rider: Spurring Large Language Models to Explore the Open World

My main reasons to recommend rejection lie in the following main factors.

Limited originality

While the whole setup of collecting a dataset for finetuning LLMs for better control is somewhat novel, overall I can not tell what is the contribution the paper provides. Finetuning the LLM on a dataset curated to solve the tasks is expected to provide better plans (which we see), and the collection of the said dataset is LLMs trial-and-error with the environment's feedback is only reasonably novel (and priviledged). The core result, that all this helps to a degree, is positive but I'd expect more contribution from an ICLR paper. As a whole, I believe an ICLR reader would not learn much new information from these results.

Limited results

Given the complexity of the method (feedback system, CoT, subtask-labelling), I was expecting more improvement over baseline than small increases in success rates. I would have expected near-perfect solutions given all these additional helpers.

This is compounded by the fact that the proposed method had many advantages over baselines and related work, e.g., access to the environment description in more symbolic way (nearby items), access to feedback during data collection phase and fine-tuning for the specific tasks. Also, while somewhat difficult, the Plan4MC tasks do not seem especially challenging in the MineDojo action space. For example, crafting a chest after collecting some logs is matter of few craft actions, that the model can directly pick. This would be more impressive if agent had to do this through an UI, much like in OpenAI VPT work.

Limited baselines and experiments

While I appreciate the ablations, the experiments lack in several dimensions:

• Limited baselines: only three baselines were included, one of which is, as authors state, an "upper bound" for the success metrics. Also, compared to the ChatGPT baseline, the proposed method has more priviledged access/setup in terms of the additional steps (feedback, CoT and subtask labelling). The RL baseline used a different language model entirely (T5 vs. LLaMA), which is understandable (compute), but still creates uncertainty around the results. It was also modified to fit MineDojo, as pointed out by the authors. Overall, I do not believe these comparisons to be fair.
• Given the focus of the LLMs, only one base LLM was used and we do not know the effect of changing to another. Ideally, a completely different LLM should be also tested with.
• I agree Minecraft is a complex environment and has challenges to solve, having the paper revolve around it and Plan4MC solutions means the results build on a very specific set of environments and models. This makes me uncertain on if this would work in any other setup/environment. I'd recommend authors think if they could, for example, use Nethack Learning Environment to perform similar tests: https://github.com/facebookresearch/nle .

• The proposed approach is fundamentally not novel. It combines adhoc prompting techniques (self-revising prompting, goal relabeling) with behavior cloning.
• Regardless of the results, the authors do not adequately outline the differences between the approaches. Thus the comparison is too simplistic. These methods leverage different types of feedback and requires different amount of human knowledge. It is unfair to compare them on a single performance scale. For example, RL requires the least amount of human knowledge because it only needs the reward function. Plan4MC requires building the whole task graph. Rider and Plan4MC requires language feedback from the environment. The costs and limitations of these requirements are not compared and discussed in depth.
• Feedback revision requires language feedback, which may be expensive to obtain in other domains. The feedback is essentially instructions, which performs planning for the agent to some degree.
• Subgoal relabeling also raises concerns. See my questions.
• The effectiveness of BC is not convincing, because after BC, the agent seems to perform worse (Rider < Rider Exploration). I understand that BC-fine-tuning may reduce prompting effort but this is not compared. In fact, the authors seem to avoid comparing Rider and Rider exploration directly.

• There is still a performance gap between the proposed agent and SOTA methods on minecraft. I think this is the biggest problem. This can be due to, as the authors said, more knowledge can be utilized from the environment. However, we do want to see an agent that surpasses traditional RL agents with the help of LLMs.
• The proposed agent is actually not going to learn a strategy to help do exploration in the open-world environment or actively collect useful information to help further learning. From my point of view, the exploration stage is more like a guided sampling, which is not that novel to the community.

• The paper contains limited analysis and ablations considering the empirical nature of the work. I believe that the submission would strongly benefit from more technical "substance" and insights beyond pure benchmarking. How do the different design choices such as design of observation and action spaces, prompts, data size, LLM model size, finetuning procedure, etc., impact exploration and downstream task performance of LLMs? Additionally, it would be helpful to understand how the different approaches differ qualitatively, e.g., visualizing how plans and trajectories of the LLM change over the course of the exploration phase.
• Similar to the point above, I feel that the discussion of results and limitations is rather superficial. For example, the authors point out that the skill policies are imperfect and may fail. Table 7 shows rather low success rates for certain skills. How does failure in skill execution affect the high-level LLM planner. Quantitatively and qualitatively? Before and after finetuning? Is there a specific mechanism in place for dealing with skill failure, does it emerge, or is this a limitation of the current approach? I'd like the authors to elaborate on this in the text.