Thanks for taking the time to review our paper. Let us address the two key points: real-world impact and dense vs. sparse rewards.

Real world impact: Although the work shows the potential of VLM-RMs, all conducted experiments have been based on relatively simple and synthetic tasks, rather than from the real world.

While it is true that we focused on synthetic tasks in a standard control environment, we do find evidence that VLM-RMs become significantly better the more visually realistic the environments are. Based on this, we would predict our method works even better in more practical environments. Together with the strong positive scaling with model size that we found, we think this suggests our method is highly scalable. Based on these results, we believe we demonstrate a very high potential for real-world impact, certainly compared to typical ML papers.

Of course it is important to validate this hypothesis in practical environments, and we are planning to address this in future work.

When using the VLMs as the reward function for a task, one may need to consider if one should use it as a dense reward signal or a terminal reward, can the authors elaborate on how they considered this question?

This is a good question and it is indeed not always clear how to make this choice.

The main reason we chose to provide dense rewards is that we found empirically (e.g., in the classic control environments) that the VLM rewards are well shaped, i.e., measure progress towards the goal. If the rewards already have this property, we might as well provide them as dense rewards to make exploration easier.

Another reason to provide dense rewards is that you might want to specify constraints that have to hold during the entire trajectory. For example we could specify that a robot should pick up an object and keep distance from a wall. Then we want to make sure to evaluate every state according to if it reaches the goal but also how close you are to the wall. And we certainly want to provide the latter as a dense reward.

We hope, this clarifies any open questions about our work!

Thanks for this detailed review! Your comments will certainly help to improve our paper!

Before answering specific questions, let us make a general point on novelty. While our paper does not propose an entirely new method, it is the first paper to make this method work for learning complex tasks. This positive result which stands in contrast to previous weaker negative results is an important contribution and might make a difference in whether people actually use this method or not. The goal of empirical research in ML should be to produce evidence on which methods work and not only to develop new methods.

How does the evaluated approach for Table 1 differ from prior works like Mahmoudieh et al that also use 0-shot CLIP embeddings for language-conditioned reward computation?

The primary method proposed by Mahmoudieh et al. uses task-specific finetuning. This makes it much more expensive and less general than the zero-shot method we focus on. They do briefly consider a method they call “Vanilla Dot-product Visuolinguistic Base Reward Model” as a baseline which is similar to ours.

Importantly Mahmoudieh et al. conclude this baseline does not work well, motivating the design of their more task-specific method. For example, in Section 2.1. they say: “ One significant limitation of CLIP is that it cannot distinguish spatial relationship of objects in images. This limits our base reward model from being useful for tasks that have spatial goals. Our full zero-shot reward model remedies this issue by leveraging CLIP in a very different way.”

Unfortunately, Mahmoudieh et al. do not provide code to reproduce the experiments. But from the paper, three key differences stick out:

• Reward formulation: Their base reward model uses an (unnormalized) dot product as a reward, whereas we use a cosine similarity (i.e., a normalized dot product). Cosine similarity is more aligned with the CLIP training objective, so it should work better.
• Model size: They use the RN50 CLIP model, which is the smallest model out of the ones we use. One of our key findings is that model size is very important for CLIP models being useful zero-shot reward models.
• Training: They train for 200k steps, whereas we train for 10M steps. We also use SAC over PPO, and the former is typically more sample efficient.

In Mahmoudieh et al. the non-finetuned model does not work at all as a reward function -- is the fact that the model in the submission works well purely based on the chosen task domain?

We suspect that all of the above mentioned differences are important for our method to work well. Specifically, we show in Figure 4 that model size is critically important, and we observed that for less than 1M training steps we see barely any training progress. While we did not test the (unnormalized) dot-product formulation systematically, early experiments suggested that it might be more noisy. Certainly, it is conceptually less principled because CLIP is trained with a cosine similarity objective.

We would have liked to evaluate our method in the environments studied by Mahmoudieh et al. Unfortunately, they use non-standard tasks and do not provide code, so it would be difficult to reproduce their setup without significant effort. We agree that it is an open question how well VLM-RMs will work in robotics domains where many tasks are of a more “spatial” nature, and we are actively interested in studying this in future work.

In the same vein, the paper is lacking any comparison to prior work. This may be because the method is identical to prior work, just applied on a different domain? I do think it is valuable to show that CLIP-based rewards can work 0-shot in more challenging domains, but without technical novelty this contribution seems insufficient to warrant acceptance.

Our focus is on tasks that do not have a ground truth reward function, and as such we rely on human evaluation. The reason for this is to demonstrate we can learn tasks with VLMs that it would be difficult to impossible to specify reward functions for. As far as we can tell, our method is the first to reliably achieve this goal.

Note that we do show that our method contribution (goal-baseline regularization) helps significantly with reward shaping in classic control tasks (Fig 2) and consistently improves the EPIC distance between the reward model and human labels in the humanoid (Fig 4).

We see our main contribution with this work as demonstrating that a method that was previously thought to perform poorly, actually performs well when implemented carefully and scaled up. We provide a lot of new empirical insights primarily about scalability and evaluating tasks without ground truth rewards. We think such work is of high value to the ML community and should be particularly interesting to the empirically minded ICLR community.

We hope that this clarified some of the key contributions of our paper. Please follow-up if any further questions come up.

Thanks for the review. We will aim to provide some context on the key concerns in the review. It would be valuable, if you could provide more specific criticisms or questions for us to respond to.

The contribution is limited and the motivation is not clear. Intuitively, I can also claim that the background information of the language description is not useless since different instructions may have different meanings in different environments.

Motivation: Training RL agents in environments where reward functions are difficult to obtain is an important and practical research problem, with many applications from language models to real-world robotics. Reward functions may not be available because we are operating in the real world as opposed to a simulation, and many tasks do not have a clear ground truth reward function at all. Learning reward models from human data is common practice to address this problem in many domains, but it requires a large amount of expensive human data. We believe that any way to reduce the amount of data necessary is highly relevant and important to solve practical problems. It is true that language instructions can sometimes be ambiguous, but this is no reason to discard this extremely valuable source of reward information. We believe language specification can heavily reduce the amount of explicit human preference data necessary, providing ample motivation for our work.

Contribution: To the best of our knowledge, our paper is the first to show that VLM-RMs can work reliably over a wider range of tasks in a complex environment, contradicting previous weak evidence of the contrary. We provide multiple technical contributions, including a “method” contribution (goal-baseline regularization) and many novel insights into how well VLMs work as reward models. We show how their rewards are shaped, how they compare to human labels, and, most importantly, how their performance scales with model size (a key insight missing from previous work, which at the time seemed to show the vanilla VLM-RM approach not to work). Could you please clarify why you find the contribution limited?

The experiments are not sufficient. Taking Figure 3 as an example, more seeds and more tasks should be included for baselines.

We focus on learning tasks that do not have a ground truth reward function, so we need human evaluation. While we agree that more seeds and tasks would be desirable, evaluation tends to be quite expensive in our setting.

Specifically, note that Figure 3 is an ablation of some features of our setup but still requires human evaluation. Running this ablation for many tasks and seeds would be very expensive and likely provide limited new insights.

We think we did quite extensive experiments in simple and complex environments. For example, we thoroughly studied effects of different model sizes and goal-baseline regularization and performed different ablations. Could you please clarify specifically which parts of our experiments you find lacking?

We hope the response addresses some of your concerns, and we would appreciate any clarification of your specific concerns, to allow us to respond to them.

Thanks for the valuable review, we will address the remaining open questions one by one.

Risk of bias in human experiments While this is duly acknowledged in the paper, the fact that the human analysis was conducted by one of the authors of the paper is a potentially strong source of bias in the human evaluation.

This is true and a valid criticism. To reduce bias in the evaluation, we will publish the full raw videos we evaluated to produce the numerical results together with the paper (rather than only the samples currently provided on the website).

There is some evidence in the experiments of reward misspecification with the VLM, a theoretical limitation acknowledged by the authors in the conclusion. It is noted that when the regularisation strength is high for realistic MountainCar that the CLIP reward function reflects the shape of the mountain. The fact that going up the small hill to the left gives high CLIP reward is useful because it encourages a policy where the mountain car oscillates in the valley until it has enough momentum to reach the top right (where the ground truth reward lies), but this is somewhat of a fortunate coincidence as there is actually a mismatch here with the ground-truth reward, which only rewards reaching the top right.

We note that in the MountainCar, our goal prompt is “a car at the peak of the mountain, next to the yellow flag”. Given this prompt, it would make sense for reward signals to encode both being on top of a mountain and being close to the flag. While the reward landscape follows the shape of the mountains, the peak on the right mountain is bigger than the left mountain (even adjusting for their relative height). So, the reward does seem to capture both the mountain height and the flag position in this case.

That being said, we don’t yet have a good sense of how problematic misspecification with VLM-RMs is in practice. But it is certainly an important question and we plan to investigate this in more detail in follow-up work.

Could the authors elaborate on the ‘minimal prompt engineering’ required to find the single sentence prompts for each task?

In early experiments we tried a few prompt variations for the “kneeling” tasks and did not find much variation. For the other tasks, we only tried the single prompt that is provided in the paper. Only for the “standing on one leg” task, we tried a few prompts to see if we can get better performance, but saw no improvement. We will clarify this "minimal prompt engineering" in the paper.

Overall, we think it's fair to say our results required essentially zero prompt engineering. The experiments we ran with different prompts further suggest that results are quite insensitive to prompt variations.

Also, are there any results demonstrating the robustness of the method to syntactic variations in the prompt?

We tried averaging the reward over synthetic prompt variations in early experiments for the “kneeling” task but did not find noticeable improvements in robustness, so we went for the simplest version of the algorithm for the final training runs. But it is quite possible that we used prompts with too simple of a syntactic structure and alternative versions of this method would yield more improvements. This would certainly be interesting to explore more in future work!

Suggested citation if authors believe it is relevant (using VLMs to improve exploration in RL): https://openreview.net/forum?id=-NOQJw5z_KY

Thanks for the suggestions. We will add it as useful evidence on VLM representations providing useful information for RL algorithms.

Thanks for the useful and positive feedback. The review raises a couple of great questions that we’ll address one by one.

Why not instead focus on more visually realistic simulation benchmarks and not have to modify the environment to make the rendering more realistic to fit the algorithm?

We found the MuJoCo humanoid environment to be a good starting point because it is a standard control benchmark and relatively inexpensive to set up, run and render. This environment provides a fast feedback loop to test different aspects of the algorithm (like scale and baseline prompts) but it is still complex enough to allow for interesting tasks. We discovered only through our experiments that realism of the visual representation is quite important. Based on our results, we would predict that VLM-RMs work even better in visually more realistic environments and we’re planning to test this in follow-up work.

It's unclear if the goal-baseline regularization is necessary. The primary experiments in Table 1 don't use any goal-baseline regularization. The analysis in Fig. 4a shows the maximum or minimum amount of goal regularization to be best.

Recall that lower EPIC distance indicates a better correlation with the human labels, i.e., a better reward model. Fig. 4a suggests that typically a regularization strength between 0 and 1 is better than the extremes 0 or 1. Only for RN50 (the smallest CLIP model we tried), the maximum amount of regularization seems to be best. Importantly, in none of the cases the minimum amount of regularization seems to be best.

However, the effect of the regularization is weaker for larger models, and the difference in our results is not always significant. We interpret this to mean that goal regularization can help a lot with reward shaping when using smaller models, but might be unnecessary when using large enough models.

Additionally, to what degree can the same effects of the goal-baseline regularization be incorporated in the goal prompt? A more detailed goal prompt can just specify to ignore the irrelevant information.

This is a great question! In principle, we can encode all necessary information in the prompt. In practice, this is difficult with current models. For example, it’s well documented that current CLIP models can struggle with negations which would be necessary to describe the contrast between the goal and the baseline prompt. Hence, it is better to use the embeddings of a goal and baseline prompt and encode the negation as a subtraction as in our goal-baseline regularization. But we do expect future VLMs to get better at handling negations and other complex prompts.

What exact values are selected for in Fig. 4a?

Good point, we’ll add indicators to the figure. The values for alpha are: 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.99.

What is the optional context in Eq. 1 used for?

In general, we define the context as a “catch-all” term containing information provided to the VLM that is not the goal prompt. In our case this only happens via the baseline regularization technique. So, you can think of the context in our implementation as containing the baseline prompt and regularization strength. We’ll clarify this in the paper. In other kinds of VLM-RMs involving vision-augmented language models (not yet tested by us), the context could include a system prompt, used to make the model to act as a reward model.

Can the paper include the RL learning curves for all the experiments showing the number of samples versus either the predicted CLIP return or a ground truth metric, like true reward in the classic control tasks or EPIC in the humanoid tasks? Given there is a ground truth reward in the classic control tasks, can the authors also include that as a reference for the VLM-RM learning curves?

We can plot learning curves of the CLIP rewards, but, as you say, they don’t necessarily show actual performance. While we can evaluate the ground truth reward in the classical control tasks, this is not possible for the humanoid tasks where we specifically chose “semantic” tasks with no programmatic ground truth function.

EPIC distance is only a measure of distance between human reward labels and the CLIP reward labels, but it also does not allow us to show the policies’ learning progress during training.

That being said, we agree it would be good to provide some learning curves in the paper to evaluate training stability, and we will do so (we will add them to the appendix). In brief: we find some variation depending on the initialization of the policy (for some random seeds the policies learn poorly), but overall training is relatively stable and comparable to the ground truth reward for task where we have one.

Thanks for your interest in our paper!

How is the rescaling achieved and is it also used in later Humanoid tasks?

The rescaling is only for visualization purposes in the plot, and is not used at all during training. For each value of $\alpha$, we normalize each data series to be between 0 and 1, i.e. $r_\mathrm{norm} = (r - r_\mathrm{min}) / (r_\mathrm{max} - r_\mathrm{min})$, to make it easier to visually compare the reward shape for different values of alpha.

Which VLM (size) is used for Fig. 2?

This figure uses the largest model we considered: ViT-bigG-14