VISION-LANGUAGE MODELS AS TRAINERS FOR INSTRUCTION-FOLLOWING AGENTS

This paper is not very clear to follow. For instance, in the second paragraph of the introduction, the authors introduce RLHF, which feels only tangentially relevant to the actual method, which uses IQL (an offline RL adapted for online RL use, which should be further clarified in line 365, since readers will naturally assume IQL is used for offline RL). I don’t believe V-TIFA’s approach to be analogous to RLHF, except in some ablations, but the authors include many discussions of RLHF which are distracting and irrelevant. For instance, typical LLM RLHF focuses on comparison-based data from human annotators, which then learn a reward model, whereas this paper prompts VLMs for rewards, taking a more similar approach to robotics reward-learning papers such as Cui et al, Adeniji et al, Du et al, Shao et al, and Yang et al. In Section 6, the authors do provide an ablation for comparative feedback, but given how much the paper has focused on RLHF, the unpromising results here feel like a disappointing end to the problem setup.

In addition, the environment consists of primitive actions and a maximum of 50 environment steps. This isn’t inherently an issue, as using VLMs for embodied, higher-level tasks instead of low-level control is still interesting, but it does make the problem statement easier.

Visualizations of rewards or trajectories could be helpful in understanding where the VLMs make mistakes in generating ratings, and where they excel.

Abstract, Intro — The manuscript states, “Developing agents that can understand and follow language instructions is critical for effective and reliable human-AI collaboration” (L11-12) and “Recent approaches train these agents using reinforcement learning with infrequent environment rewards, placing a significant burden on environment designers to create language-conditioned reward functions.” (L12-15) Why would developing language-conditioned reward be the responsibility of the environment designers? The sparse rewards found in instruction-following tasks resembles quite closely the task-execution setting that humans are faced with naturally. Artificially providing agents with dense language-conditioned extrinsic rewards, from the environment, for ‘free’, would not be helpful for pursuing ‘effective and reliable human-AI collaboration.’

Abstract, Intro, Related Work — Many approaches attempt to use large-capacity models to generate rewards for training policies under an RL objective. Here are but a few: https://arxiv.org/pdf/2402.16181, https://arxiv.org/pdf/2402.04210, https://arxiv.org/pdf/2405.10292, https://arxiv.org/pdf/2409.20568, https://openreview.net/pdf?id=uydQ2W41KO. The manuscript should consider defining baselines based on these.

Abstract, Intro, Related Work — This manuscript is trying to propose ‘RL with artificial feedback’ — see “RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback” | https://openreview.net/pdf?id=uydQ2W41KO

Section 1, Section 3 — The manuscript should make a distinction between extrinsic and intrinsic rewards, in instruction-following tasks; furthermore, the manuscript should be clearer about where the responsibility for good reward design lies: for extrinsic (environmental) rewards, the responsibility rests with the environment creators; these rewards are sparse for the purpose of realism (see also my previous comment(s)). The responsibility for intrinsic rewards rests with the agent/algorithm/methodology developers. The manuscript constantly conflates these two, inappropriately.

Section 4, Section 5 — I worry that the environment used by the manuscript for assessing the effectiveness of the approach is not sufficiently complex for serving as a good evaluation benchmark. The aforementioned approaches show good performance in their benchmarks as well. For example, ALFRED (the environment that this manuscript uses) is quite photo-synthetic, the tasks are short-horizon single-room tasks, and the benchmark has the luxury of including tasks with well-articulated implicit pre- and post-conditions and sub-goal segmentation. Any results on more complex, more photo-realistic, long-horizon, and/or less curated environments would be much more informative.

Section 5 — Experiments are insufficient. The choice of baselines illustrate comparisons in terms of the embedding spaces used for encoding the multimodal context, the but the evaluation does not provide comparisons with the prior art in terms of the fundamental methodological approach.

Section 5 — All of the tasks are short-horizon tasks. To illustrate the relevance and generality of the approach to more complex settings, experiments on long-horizon settings are needed (e.g., multi-room, multiple semantically dependent tasks in the same room, or at least multiple semantically independent tasks in the same room).

• Originality: The originality of the proposed method is limited. 1. There has already been significant prior work using VLMs as reward functions for RL, such as MineDojo (Fan et al., 2022), which successfully completes a variety of tasks in the more complex, open-ended environment of Minecraft. 2. Using large VLMs to assess task completion has been widely applied in AI agent research, commonly referred to as success detection.

• Quality: The experiments in this paper are insufficient. 1. Experiments are conducted in only one environment, lacking results across multiple environments, which does not adequately demonstrate the method's effectiveness or broad applicability. 2. The baselines are unfair: V-TIFA is provided with a complete trajectory, while the baselines receive only the first and last frames, which is both simplistic and unfair. In MineDoJo (Fan et al., 2022), MineCLIP processes a continuous trajectory window, producing a dense reward. Even if the goal is to maintain a sparse reward setup, it would be fairer to select the highest score from CLIP or another model over the trajectory.

• Efficiency: V-TIFA is inefficient, as it only provides sparse rewards based on whether the trajectory completes the task. Due to RL’s inherent difficulty in handling sparse rewards, this approach is inefficient. Furthermore, using sparse rewards from the environment as an upper bound is not ideal. In MineDoJo (Fan et al., 2022), MineCLIP provides dense rewards to aid the agent in language-conditioned tasks that are challenging to complete using only sparse rewards from the environment. The primary reason for introducing LLMs or LVLMs into decision-making tasks is to improve sample efficiency and generalization in RL. However, here, the use of large VLMs does not surpass the sparse rewards provided by the environment, which limits the practical value of this approach.

References:

Fan, L., et al. (2022). Minedojo: Building open-ended embodied agents with internet-scale knowledge[J]. Advances in Neural Information Processing Systems, 2022, 35: 18343-18362.

There are two primary weaknesses that I see in the paper: (1) the lack of discussion around potential failures of VLMs to perform zero-shot reward specification, and how these failures could be addressed, and (2) the lack of real-robot results.

(1) As far as the reviewer is aware it is not discussed in the paper how the proposed algorithm will work when the task being learnt is of sufficient complexity that off-the-shelf VLMs can no longer reliably assign correct rewards. The authors motivate the use of frozen VLMs for reward generation with the claim that it eliminates human effort spent towards collecting demonstrations or defining reward functions. However this is only true when the off-the-shelf VLM can indeed reliably specify rewards. If the VLMs' performance is sufficiently low to require SFT/RL finetuning on task-specific data to improve accuracy, then human effort cannot be avoided. Indeed this seems to explain the low performance of the baselines the authors evaluate against; the baselines do poorly because their frozen VLMs do not assign good rewards.

(2) Weakness (1) by itself is not a significant issue for me as there is much recent work that attempts to leverage frozen VLMs for control [1, 2, 3, 4, 5]. However like these works I would like to see real robot results, since these works have set the inclusion of real-robot results as a precedent.

[1] Nasiriany, Soroush, et al. "Pivot: Iterative visual prompting elicits actionable knowledge for vlms." arXiv preprint arXiv:2402.07872 (2024). [2] Liu, Fangchen, et al. "Moka: Open-vocabulary robotic manipulation through mark-based visual prompting." First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024. 2024. [3] Shah, Dhruv, Błażej Osiński, and Sergey Levine. "Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action." Conference on robot learning. PMLR, 2023. [4] Wang, Zidan, Rui Shen, and Bradly Stadie. "Solving Robotics Problems in Zero-Shot with Vision-Language Models." arXiv preprint arXiv:2407.19094 (2024). [5] Hu, Yingdong, et al. "Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning." arXiv preprint arXiv:2311.17842 (2023).