Few-shot In-context Preference Learning using Large Language Models

We thank the reviewer for highlighting that our work enhances the interoperability and capacity of reward design. The main reason for the reviewer to give a rejection decision is uncertainty as to why we need to involve human feedback in this paper. This appears to stem from a misunderstanding of the problem we aim to tackle. We would like to first clarify the scope of our research:

Our focus is on tasks where humans cannot define any clear reward functions, including both dense and sparse rewards. In these cases, learning a reward model from human feedback becomes a viable solution. Specifically, humans provide preferences on trajectories without the need to explicitly write out a reward function. In other words, our work is positioned within the realm of preference-based learning methods, where human feedback serves as a protocol for deriving rewards. The main contribution of our paper is to significantly enhance the efficiency of human involvement in preference-based learning by leveraging large language models (LLMs).

We will address the reviewer’s detailed questions in the following sections.

Q1: ICPL involves human labor, but does not show any significant gain over Eureka, which doesn't require any human feedback.

As mentioned earlier, involving human feedback is essential for tackling tasks where humans cannot design any clear reward function. EUREKA, however, still relies on human-designed sparse rewards as fitness scores for evolving reward functions. Therefore, EUREKA is NOT a true baseline for our work. We have revised the paper to make this clear. We include it solely to demonstrate an approximate upper bound of performance assuming sparse rewards are available. Our goal is NOT to outperform EUREKA, but to show that our approach, which relies only on human preferences, can achieve surprisingly comparable results.

Q2: For challenging tasks, true human feedback does not work better than proxy human feedback. This undermines the necessity of involving humans.

It is true that real human feedback, which includes noise, can not outperform proxy human feedback, which is noise-free. However, this does not undermine the necessity of involving humans in our tasks. We believe the reviewer’s point is based on the assumption that sparse rewards are accessible. However, our work focuses on tasks where sparse rewards do not exist.

First, there might also be a misunderstanding for the reviewer regarding the use of proxy human feedback in our experiments. Tasks without any reward or task metric make it difficult to assess method performance, and conducting real human experiments is time-consuming and not easily scalable. Using proxy human feedback is a standard practice for evaluating preference-based learning methods as It allows for rapid and quantitative comparisons across different methods. In this paper, we use sparse rewards as a proxy for perfect human preferences, but this way is actually not feasible for tasks without sparse rewards. Therefore, real human feedback is necessary in these cases.

Besides, the proxy human feedback is a simulated perfect human without any noise and may not fully capture the challenges humans may face in providing preferences, we further conduct real human feedback experiments to demonstrate the true effectiveness of our approach. In other words, it is expected that true human feedback doesn’t work better than proxy human feedback, the point we want to show is that our method can still work with true human feedback.

Finally, we want to clarify that our goal is NOT to show that human preference or preference-style signals are inherently better than directly using sparse rewards. Rather, we are addressing tasks that lack sparse rewards by incorporating human feedback to guide the learning process.

Q3: For challenging tasks, like humanoid jump task, ICPL does not have any solid comparisons with other baselines.

EUREKA's reliance on sparse rewards makes it unusable as a baseline in tasks where no explicit reward function is defined, as is the case here: we do not have a sparse reward for this task. Thus, a direct comparison with EUREKA is not feasible. Besides, Table 1 shows that PrefPPO, a preference-based learning baseline, struggles to learn effective behaviors in more challenging tasks, even after 15,000 human queries. Given these limitations, it would be impractical to conduct real human experiments on complex tasks like HumanoidJump based on this performance evaluation.

Thanks for the clarification. It really helps me to understand the position of this paper. However, I would suggest you elaborate the interested problem and the limitation of other methods in the introduction part. For example, it would be better to move some contents in the related work to the introduction and reorganize the introduction and relate work parts. I will increase the rating if the writing has been improved.

In addition, in Line 037, "However, as task complexity increases or tasks are distinct from the training data, the ability of LLMs to directly write a reward function is limited." Do you show this conclusion in your paper or have any reference? Please let the readers know.

Last, I would like to re-summarize the method and contributions of this paper. Correct me if I'm wrong. This paper combines the power of LLMs and human preference to design the reward efficiently. Comparing to zero-shot LLMs, ICPL uses human preference-based feedback to refine the prompt for LLMs to generate reward functions. Comparing to PrefPPO, ICPL replaces the reward model with LLMs generating reward functions.

Based on the aforementioned two major comparisons, I have the following questions.

1. About the reward model in PrefPPO

What size of the reward model do you use in PrefPPO baseline? Would it improve the performance if a larger reward model was employed? I'm curious about this because you use GPT-4o, a very large model, as an (implicit) reward model in some sense, which makes it unfair to compare with a relatively small reward model.

2. About the zero-shot LLMs

I still have doubts on the necessity of involving human preference. Let's break it down into a few questions.

(a) I don't see a significant improvement of ICPL over the OpenLoop given that the variance is large. Besides, the OpenLoop performance could be improved given a carefully designed prompt, which brings the following question.

(b) What prompt do you use in OpenLoop? Do you use any prompts that enable the LLM to obtain (implicit) preferences from the trajectories? An example prompt can be "Compare the trajectories carefully, choose the best trajectory, and rewrite the corresponding reward function to improve the performance." If this can work, then it is not necessary to use human preference. The LLM itself can achieve a good result.

(c) What would the performance be if OpenLoop was employed in the humanoid jump task? This is a follow up question for my previous Q3. I would suggest you to establish a baseline on this task because this is the real problem your method aims to tackle, where neither dense nor sparse reward is available. I would also suggest to involve quantitive evaluation. For example, you can involve humans to rate the performance and report the results using Elo score.

We sincerely appreciate the reviewer's suggestion regarding the reorganization of the introduction and related work sections. We have revised the paper to improve clarity and coherence in these parts. Below, we provide detailed responses to the reviewer’s questions:

Q1: reward model in PrefPPO

In PrefPPO, the network architecture of the reward model is a 3-layer MLP with 256 hidden sizes . A reasonable concern arises regarding the impact of larger reward model sizes on performance. To address this, we conducted experiments with three configurations: Deeper reward model: 5-layer MLP with 256 hidden units and 10-layer MLP with 256 hidden units. Wider reward model: 3-layer MLP with 2048 hidden units. Deeper and wider reward model: 5-layer MLP with 1024 hidden units. These experiments were conducted on the humanoid task, where the objective is to make the humanoid run as fast as possible. The results show that larger reward model sizes achieve comparable performance to the standard configuration but do not yield performance improvements as the size increases. Additionally, we experimented with a much deeper model (10×256), which resulted in worse performance.

Q2: What prompt do you use in OpenLoop?

In the OpenLoop experiment, the LLM does not receive any feedback and only uses a basic prompt containing the task description, observations, and reward-writing tips(Prompt 1 and 3 in Appendix). It generates 30 reward functions across 5 iterations (6 samples per iteration). Additionally, we have tried the example prompt proposed by the reviewer at the beginning of this project, however it is challenging to find a suitable representation of trajectories for the task. We attempted to use VLMs to rank videos, but GPT-4v failed to achieve this effectively. Using state-action pairs as trajectory representations is also problematic, as the LLM struggles to identify the relationship between trajectories and behaviors. Moreover, providing all six reward functions along with their corresponding trajectories to the LLM quickly leads to token limitations, making this approach currently impractical.

Q3: What would the performance be if OpenLoop was employed in the humanoid jump task?

Thank you for the suggestion. We have additionally conducted OpenLoop experiments on the humanoid jump task and included the results in the revised paper. Specifically, we performed 5 experiments using the OpenLoop method, with each experiment consisting of 6 iterations and 6 samples per iteration. Then one volunteer selected the most preferred video as the final result for OpenLoop. For quantitative evaluation, we recruited 20 volunteers, including graduate and undergraduate students not involved in this research, to indicate their preferences between two videos shown in random order: one generated by ICPL and the other by OpenLoop. The results show that 85% (17/20) of participants preferred the ICPL agent, indicating that ICPL produces more human-aligned behaviors. These findings demonstrate ICPL's significant advantage over OpenLoop in complex tasks and highlight the necessity of incorporating human preferences in such scenarios.

We thank the reviewer for highlighting the substantial reduction in the number of human queries, and for noting that the evaluation of our method was conducted using both synthetic data and real humans. We also appreciate the advice on making our paper clearer and more comprehensive. We have revised the paper accordingly, with changes highlighted in red, based on the reviewer’s suggestions. Below are the detailed responses to the reviewer’s questions.

Q1: As with all works that use LLMs to generate reward functions from human feedback I question how well it will perform with more complex tasks which is one of the big reason for using human feedback.

It is a reasonable concern that our method may not scale to more complex tasks. However, we would like to clarify two points: 1) we do not claim that our approach will definitively scale to more complex tasks, but rather provide evidence suggesting that it might, and 2) the tasks we evaluate are already sufficiently complex that humans find it challenging to define reward functions for some of them (as demonstrated in EUREKA).

Q2: The synthetic experiment uses completely noiseless preferences while the standard in these kinds of control environments are typical a noise of let's say 10%. What is the rationale for using noiseless preferences and what would be the effect of noisy preferences for your method?

We use noiseless preferences initially to verify whether our method can effectively work solely with preference signals, allowing us to assess the upper bound of expected performance for all preference-based learning methods. Additionally, this setup enables a comparison with EUREKA, which uses noiseless sparse rewards as fitness scores to enhance reward functions. Subsequently, we conduct real human experiments, incorporating true noisy preferences. As shown in Table 3, noisy preferences (ICPL-real) perform comparably or slightly lower than noiseless preferences (ICPL-proxy) across all five tasks, yet still surpass the performance without any preferences (OpenLoop) in 3 out of 5 tasks.

Q3: While the authors uses B-Pref from Kimin et al for some reason they use only the PPO version even tho the repository is more associated with PEBBLE the SAC version. Why is SAC not used as well?

We adopt PPO as IsaacGym tasks are trained by PPO, which is well-suited to high-parallel, GPU-based simulators. Using different training algorithms, such as PPO and SAC, could result in varying basic performance even with the same human-designed reward function, making it difficult to ensure a fair comparison. Furthermore, this way we can ensure that each algorithm has a well-tuned inner loop in which we train the RL agents.

Q4: It would be nice with some more information about the experiment like demographic data as well as discussing the limitation of a smaller sample size when it comes to generalizability.

Thank you for the suggestion. We have revised the paper to incorporate this information. The participants in our study consisted of six individuals aged 19 to 30, including two women and four men, with educational levels ranging from undergraduate students to graduate students and one postdoc. We acknowledge that the small sample size may limit the generalizability of our findings, as it may not capture the full range of human preferences. Expanding the sample size and diversity is a crucial direction for future work to improve the robustness and applicability of our results.

Q5: It seems like Eureka has very similar performance to ICPL, what would you say is the benefit of your method compared to Eureka

Our focus is on tasks where humans cannot define any clear reward functions, including both dense and sparse rewards. EUREKA, however, still relies on human-designed sparse rewards as fitness scores for evolving reward functions. Therefore, EUREKA is not a true baseline for our work. We include it solely to demonstrate the upper bound of performance assuming sparse rewards are available. Our goal is not to outperform EUREKA, but to show that our approach, which relies only on human preferences, can achieve comparable results. We have revised the paper to make this clear.

Q6: To make the related work more complete, there is another paper using LLMs with preferences.

Thanks for pointing that and we have revised the paper to add it in the related work section.

Q7: Why did you not use PEBBLE as a baseline given that you made use of BPref? Also, did you consider any other baselines as there are more recent works [1,2,3] (To name a few)? It would be great if you discuss how you determined which baseline to use and if you considered any others.

Thank you for the suggestion. We selected the most commonly used preference-based learning method as our baseline, but we are happy to incorporate additional baselines. In the revised version, we will report the performance of PEBBLE and SURF as baselines, which update reward models during training as ICPL to ensure a fair comparison.

Thank you again for the valuable feedback. We hope our response has addressed your concerns. If you have any further questions, we would be happy to provide additional information.

We thank the reviewer for pointing out that we made reasonable efforts to optimize the baseline methods. We also appreciate the feedback on the strong comparison against PrefPPO. Below, we provide detailed responses to the reviewer’s questions:

Q1: ICPL in comparison with EUREKA and human preference comparison with sparse reward

The reviewer’s question regarding the comparison between ICPL and EUREKA, as well as the comparison of human preferences with sparse rewards, suggests there may be a misunderstanding of the paper's scope. Our focus is on tasks where humans cannot define any clear reward signal, including both sparse and shaped rewards for RL training. Preference-based learning offers a solution by learning a reward model based on human preferences across different trajectories. For tasks where a clear sparse reward can be defined, human preference is not necessary, and a sparse reward is a more conventional learning signal. Therefore, comparing human preference to sparse reward is NOT the goal of our research. Instead, our objective is to explore how to effectively use human preferences to address tasks that lack clear reward signals.

EUREKA, while also using LLMs to generate reward functions, still relies on sparse rewards as a fitness score for evolving reward functions. Thus, EUREKA is NOT a direct baseline for our work. It serves as an approximate oracle method to demonstrate the upper bound of performance. We have revised the paper to make this clear. Our aim is not to outperform EUREKA, but to show that our approach, which relies solely on human preferences, can achieve results that are comparable.

Lastly, we want to clarify our use of tasks with sparse rewards in the proxy human preference section. This approach is standard practice for evaluating preference-based learning methods, as it allows for a rapid and quantitative comparison across different methods. Tasks without any reward or task metric make it difficult to assess method performance, and conducting real human experiments can be time-consuming and not easily scalable. Using sparse rewards as a proxy for human preferences to automatically run experiments is a practical and effective evaluation strategy.

Q2: I’m not convinced that this success generalizes to the domain of LLM-generated reward functions.

We do not fully understand the point being made here and would appreciate clarification if possible! Our paper is a demonstration that a specific type of preference based-learning does in fact work for LLM-generated reward functions.

Q3: “ however the authors only offer one case-study as evidence of the efficacy of human preference data in this domain.

First, our goal is NOT to demonstrate that human preferences or preference-based signals are inherently better than directly using sparse rewards. Instead, we focus on addressing tasks that lack clear reward signals by incorporating human feedback to guide the learning process.

Additionally, we conducted real human experiments on six tasks, including five IsaacGym tasks and one humanoid jump task. In IsaacGym experiments, humans were unaware of the true sparse reward, which allows these tasks to also serve as test cases. One key reason for adopting the five IsaacGym tasks is that they can provide quantitative results by using human-designed sparse rewards as task metrics, enabling us to present evidence of ICPL’s efficacy in this domain.

Q4: Evaluation Metric to be very confusing. I wasn’t sure what an “environment instance” was. I also didn’t understand which set of task metric values was used to compute the maximum for the RTS.

We appreciate the reviewer for raising the issue of clarity, and we have revised the paper accordingly. The term 'environment instance' refers to the parallel environments used during RL training. Specifically, we collect data from 10 parallel environments, meaning 10 environment instances. The task metric is the average sparse reward across these parallel environments that return sparse rewards. In each iteration of ICPL, we have six RL training runs. For each RL run, the RTS is the maximum task metric value sampled at fixed intervals. TS represents the maximum of the 30 RTS sets (6 RL runs × 5 iterations).

Thank you to the reviewer for engaging and increasing the score. The points you raised have helped us clarify the paper so far. We would like to make one key clarification about EUREKA and we appreciate that the reviewer pointed it out as it suggests that more rewriting of the introduction is necessary which we will do shortly.

Q1: The difference between our method and EUREKA is that EUREKA is a reward shaping method and ICPL is a preference learning method. In EUREKA the reward is known but sparse and hard to do RL with. Thus, an LLM is used to learn reward functions that are easy to learn on and are aligned with the sparse reward. In ICPL, just as in preference learning generally, the reward is unknown and we must learn a functional form for it. An LLM is used both to write out the functional form (in this case code) for the reward function. It then iteratively updates the reward through rounds of preference learning by storing prior rewards and preferences in context and then asking for rewards that match the preferences. So, in ICPL we are in fact converting the preferences directly into a reward function, a problem that is extremely challenging! As you can see from our PrefPPO experiments, naive preference learning does not match our constructed reward functions even with orders of magnitude more samples. As the reviewer suggests, we could then do EUREKA on top of this to make it easier for the RL agent to learn the policy corresponding to this reward. However, it turns out that we don't need to since we already achieve EUREKA-level performance on this task!

To summarize:

• The LLM is iteratively outputting rewards simply from a task description and a set of preferences. In each round, it puts the previous preferences and rewards in context and outputs 6 new rewards. It turns out the rewards constructed this way match the performance of EUREKA, as measured by the sparse rewards constructed in the EUREKA paper, even though we only have a text description of the task and a small number of preferences.
• EUREKA does reward shaping given a known reward to try to learn a policy that is optimal under the known reward; we attempt to both learn this known reward purely from preferences and output a policy that is optimal under it. Note that the LLM is outputting an explicit reward signal. We are doing exactly what the reviewer suggested and converting the preferences into rewards although the rewards are not necessarily sparse.

Q2-Q3: Hopefully our response to Q1 addresses this!

Q6: Great question. We use a temperature of 1.0 and will add this to the appendix.

We sincerely appreciate the reviewer for acknowledging our hard work on the real human study and for recognizing that the idea is generally interesting and compelling. We value this positive feedback, which motivates us to further improve our work. Below, we provide detailed responses to the reviewer’s questions:

Q1: comparison with EUREKA

We would like to clarify the scope of our research: our focus is on tasks where humans cannot define any clear reward functions—whether dense or sparse. In these cases, learning a reward model from human feedback becomes a viable solution, as humans provide preferences on trajectories without explicitly defining a reward function. Our work, therefore, falls within the domain of preference-based learning, where human feedback serves as the protocol for deriving rewards.

EUREKA and ICPL work under different assumptions. EUREKA’s primary goal is to test whether LLMs can produce better reward functions than humans by leveraging human-designed sparse rewards as fitness scores to evolve reward functions. In contrast, ICPL is designed for tasks even without available sparse rewards and leverages LLM grounding to accelerate learning reward functions directly from human preferences. For this reason, EUREKA is NOT a true baseline for our work, as it cannot address tasks lacking sparse rewards. We include it only to illustrate an approximate performance upper bound assuming sparse rewards were available. Our goal is not to outperform EUREKA but to show that ICPL, which relies solely on human preferences, can achieve comparable results. We have revised the paper to make this clear.

In the EUREKA appendix, an additional experiment uses human text feedback to describe behaviors and improve the initial human-designed reward function. In contrast, our approach relies solely on preferences—yielding higher human-involvement efficiency—and does not rely on an initial human-designed reward function. Note also that in our proxy human preference experiments, EUREKA’s results are based on sparse rewards, not human text feedback.

Q2: it's stated in the intro and conclusion that ICPL surpasses RLHF in efficiency, but RLHF is not mentioned anywhere in the experiments.

In this context, RLHF (reinforcement learning from human feedback) refers to the baseline method PrefPPO. PrefPPO, which learns a reward model from human preferences and subsequently uses this learned reward model to train an RL policy, is a preference-based learning method, i.e., a subset of RLHF.

Q3: Based on 5 iterations, I'm not sure that you can make the claim that it will monotonically improve much past that point. Did authors try past 5 (10, 20).

Thank you for pointing that out. We have not tested additional iterations, so we have revised the paper to remove this claim. Our intention is simply to highlight the method’s effectiveness in refining reward functions over time.

Q4: One sort of undiscussed thing here is that, requiring new models to be trained every iteration does mean that loop is pretty slow. Was 5 iterations chosen for that reason (so it wouldn't take multiple days). This should be maybe discussed as a weakness. E.g. for human studies or using humans, doesn't that mean the humans need to wait hours or else get new humans to provide feedback for every iteration?

Thanks for the suggestion. The overall time-consuming of one experiment is a limitation of our method. We have revised the limitation part to discuss this. This could be resolved by continually training an RL agent under non-stationary reward functions which could make for good future work.

Q5: Why GPT-4o for the human experiment only?

Midway through the paper writing process, we found GPT-4o is cheaper. However, we had already completed most of the proxy human experiments using GPT-4, so we only switched to GPT-4o for the real human experiments.

We hope our response has addressed the reviewer’s concerns. Should there be any additional questions, we would be glad to offer further clarifications.