Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models

We thank the reviewer for their in-depth review, and for recognizing the simplicity and effectiveness of our method, the strength of our results, and our efforts in real-world experiments. We address your comments in detail below:

• Q1. Could you elaborate on the criteria for selecting trajectory segments from the replay buffer?
  • A1. For MetaWorld, we randomly sample segments of length one (i.e., individual states) and use them to query the VLM. This approach is sufficient for the single-target nature of the tasks, as shown in Figure 1 at Link. For ALFRED, since tasks are often compositional and critical information may appear at different points in the episode, we randomly sample an entire trajectory and use it to query the VLM to provide sufficient context for accurate ratings, as illustrated in Figures 2-4 at Link.
• Q2. Can you provide additional quantitative analysis on the consistency and reliability of the ratings across different tasks and scenarios?
  • A2. The consistency of the VLM ratings is shown in Figures 9 and 10 at Link. We repeatedly query the VLM along expert trajectories in different tasks. As pointed out by the reviewer, we also observe that identical segments may receive different labels. However, as shown in the results, the ratings along expert trajectories remain reasonable and consistent. For example, by calculating the accuracy of ratings based on task success/failure (for $n=3$ rating levels, we only count Bad and Good), we observe the following rating accuracies: for Sweep Into, $90\pm4.6$; for Drawer Open, $65 \pm 3.8$; and for Soccer, $66.8 \pm 3.7$. The variance across trials is small, and these small inconsistencies can be mitigated through our MAE objective.
• Q3. What is the rationale behind choosing a Likert scale for the ratings?
  • A3. The rationale for selecting a Likert scale is as follows:
  1. Human alignment: Since our reward labels are generated by a VLM designed to emulate human interpretation, a Likert scale aligns well with common human rating systems. This allows the VLM to map qualitative assessments (e.g., Bad, Average, Good) to structured numerical values, which would be more challenging to express meaningfully using an arbitrary continuous scale.
  2. Stability and reliability: It is considerably more stable and reliable to prompt VLMs with structured classification tasks (e.g., “Rate the success of this grasp from 1 to 5”) than to request continuous scalar (e.g., “Give a score between 0 and 1”). Continuous scores often yield unstable or inconsistent outputs, as shown in prior work [1], whereas discrete classes reduce ambiguity, providing clearer grounding for ratings.
  3. Scalability and prior work: Prior research in learning from evaluative feedback [2-5] often discretizes qualitative feedback into ordinal scales for reward modeling. These pipelines report that such formats are easier to scale, annotate, and integrate into training loops.
• Q4: Discussion with some works use LLMs or VLMs as coding models to directly generate reward functions without computing similarity score.
  • A4. The relevant works suggested by the reviewer utilize LLMs/VLMs as coding models that directly generate reward functions by accessing environment source code or privileged state information. In contrast, our method relies solely on visual observations and does not require access to internal environment representations. We will incorporate the works recommended by the reviewer ([6-8]) into our revised manuscript to enrich the related work section and more clearly position our method within the broader literature.

[1] Wang, Y. et al. RL-VLM-F: Reinforcement Learning from Vision-Language Foundation Model Feedback, ICML 2024.

[2] Knox, W.B. et al. Interactively shaping agents via human reinforcement: The tamer framework. 2019.

[3] Warnell, G. et al. Deep tamer: Interactive agent shaping in highdimensional state spaces. 2018.

[4] Arumugam, D. et al. Deep reinforcement learning from policy-dependent human feedback. 2019.

[5] White, D. et al. Rating-based Reinforcement Learning, AAAI 2024.

[6] Wu, Y. et al. Eureka: Human-Level Reward Design via Coding Large Language Models.

[7] Li, X. et al. Self-refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Robotics.

[8] Chen, C. et al. AutoReward: Closed-Loop Reward Design with Large Language Models for Autonomous Driving.

We thank the reviewer for the detailed and thoughtful review, and for recognizing the novelty of our method, the extensiveness of our experiments, and the strength of our results across a wide range of tasks and domains. We respond to your comments in detail below:

• Q1. The method is highly dependent on the performance and accuracy of vision-language models. What if the vision-language model consistently exhibits understanding biases and inaccurate ratings in some complex scenarios?

  • A1. We acknowledge that biases and inaccuracies in VLM ratings can arise, particularly in complex scenarios. To mitigate these issues, we focus on designing effective prompts and selecting appropriate rating levels, which help guide the VLM’s output. Additionally, using higher-performing VLMs can further enhance the quality of the ratings. However, if a VLM consistently produces inaccurate ratings due to being undertrained on specific tasks or domains, the finetuning for the VLMs is required.

• Q2. An accuracy metric for the reward function should be introduced to measure the degree of matching between the learned reward function and the real task objective. In environments with sparse rewards, what specific reward designs does ERL-VLM derive that lead to performance improvements?

  • A2. ERL-VLM performs better in sparse reward tasks because the reward model, learned from ratings, provides denser reward signals at meaningful states, rather than only sparse signals at the final state.

    To measure the alignment between the learned reward function and the real task objective, we compare the reward outputs from ERL-VLM with the ground-truth task progress along expert trajectories. The results are shown in Figures 7 and 8 at Link . Outputs from RL-VLM-F and CLIP are also included for comparison. As shown, ERL-VLM rewards are more consistent and less noisy compared to the other methods. In sparse reward tasks, our approach enables reward shaping by assigning higher values to meaningful states. For instance, in the CoolObject task (Figure 8), ERL-VLM assigns higher rewards at timesteps when the agent interacts with the target object (e.g., the fridge), compared to intermediate navigation steps. This behavior arises from the VLM assigning higher ratings to critical subgoal states, as illustrated in Figures 2–4 at Link.

• Q3. ERL-VLM seems to rely on the task description. How should it handle multiple target tasks?

  • A3: ERL-VLM handles multiple target tasks (i.e., compositional task descriptions) by querying the VLM with a sequence of images rather than a single image, as illustrated in Figures 2-4 at Link. We find that incorporating temporal context helps the VLM better understand the overall task structure, leading to more accurate ratings in compositional task descriptions.

We thank the reviewer for their insightful and constructive feedback, and for recognizing the clear motivation behind ERL-VLM, the clarity of our writing, and the strength of our experimental results. We address your comments in detail below:

• Q1. What do trajectories labeled with different ratings look like in practice?

  • A1. Sixteen examples of states/trajectories labeled with different ratings are shown as follows:

    • MetaWorld tasks (Sweep Into, Drawer Open, and Soccer), where each task includes two or three different states, each leading to different ratings assigned by the VLM. These examples are shown in Figure 1 at Link.

    • ALFRED tasks with three different instructions, each paired with three trajectories leading to varying ratings. These are shown in Figures 2, 3, and 4 at Link.

  The reasoning behind the VLM’s ratings (obtained during the analysis stage of our prompts) is also illustrated in the figures. As shown, the VLM assigns appropriate ratings based on the status of the target object in relation to the task objectives specified in the task description.

• Q2. What is the typical distribution of ratings for the considered tasks?

  • A2. The typical rating distribution for the tasks is shown in Figure 5 at Link. As shown, for the first three MetaWorld tasks, the percentage of Good ratings increases as the timestep progresses. For ALFRED tasks, the percentage of Average ratings increases over time, while the percentage of Good ratings remains almost constant. This is because, in ALFRED tasks, successful states occur at the end of the trajectory, so the VLM tends to assign Good ratings mostly at the final step, resulting in an increase in number of Good ratings, although the increase is relatively small compared to other ratings.

• Q3. What does the output of the reward model look like in practice?

  • A3. The outputs of the reward models from ERL-VLM along expert trajectories in three MetaWorld tasks and four ALFRED tasks are shown in Figures 7 and 8 at Link. We also include outputs from RL-VLM-F and CLIP for comparison. In MetaWorld tasks, CLIP rewards are generally noisy and poorly aligned with task progress. While both ERL-VLM and RL-VLM-F exhibit increasing reward trends along expert trajectories, ERL-VLM aligns more closely with the ground-truth task progress and shows significantly less noise compared to RL-VLM-F. In ALFRED, ERL-VLM produces smoother and more consistent reward signals along expert trajectories than the other methods.

• Q4. The effect of #ratings.

  • A4. As mentioned in Q1, the VLM assigns ratings based on the status of the target object. For many tasks, the appropriate number of ratings can be easily determined (e.g., whether the task has been successfully completed is clearly observable from the visual state). However, for tasks where determining the optimal number of ratings is non-trivial, we use a trial-and-error approach. We demonstrate this process in Figure 6 at Link, where we repeatedly query the VLM along an expert trajectory in the Drawer Open task.

    With $n=3$ ratings levels, the VLM produces more consistent and intuitive feedback, with ratings generally increasing as the trajectory progresses toward task completion. Small inconsistencies are mitigated through our MAE objective. However, with other values (e.g., $n=2$ or $n=4$), the VLM exhibits stronger inconsistency. For example, with n=2, the VLM sometimes assigns Bad ratings even when the state reflects successful task completion. Similarly, with n=4, both Very Bad and Bad ratings are occasionally assigned to successful states. We hypothesize that this behavior stems from inherent biases in the VLM. Based on this observation, we recommend performing a simple consistency check when designing prompts for new tasks to help select an appropriate number of rating levels.