Online Iterative Reinforcement Learning from Human Feedback with General Preference Model

Thanks for your positive feedback and constructive suggestions! Our responses are as follows.

Weakness 1 Include examples of the prompts used for training the preference model and responses generated by different methods.

Sure. Some example prompts are provided as below

What is the ideal temperature to serve a crisp Chardonnay?

is there xor method in vb?

Define cryptography? Why we need cryptography?

how to use viewdid load in swiftui macos xcode 14

We also provide some example responses as below.

What are the names of some famous actors that started their careers on Broadway?

DPO: 
1. Hugh Jackman
2. Meryl Streep
3. Nathan Lane
4. Angela Lansbury
5. Bernadette Peters
6. Liza Minnelli
7. Audra McDonald
8. Philip Seymour Hoffman
9. Laurence Olivier
10. Kelsey Grammer
11. Richard Gere
12. Liev Schreiber
13. Mark Ruffalo
14. Julie Andrews
15. James Earl Jones
16. Whoopi Goldberg
17. Christopher Plummer
18. Robert Redford
19. Angela Bassett
20. Helen Mirren.

IPO: 
Some famous actors that started their careers on Broadway include:

1. Hugh Jackman
2. Meryl Streep
3. Viola Davis
4. Denzel Washington
5. Mark Ruffalo
6. Matthew Broderick
7. Jennifer Holliday
8. Audra McDonald
9. Nathan Lane
10. Bernadette Peters
11. James Earl Jones
12. Angela Lansbury
13. Blythe Danner
14. Richard Dreyfuss
15. Cicely Tyson


Ours: 
Here are some famous actors who started their careers on Broadway:

1. Meryl Streep - Before becoming one of the most acclaimed actors in Hollywood, Meryl Streep began her career on Broadway in the early 1970s. She made her debut in the play "The Play About the Baby" in 1971 and went on to star in several other productions, including "Trelawny of the 'Wells'" (1975), "Henry IV, Part 1" (1976), and "Sunday in the Park with George" (1984).

2. Hugh Jackman - Hugh Jackman made his Broadway debut in the musical "Oklahoma!" in 1990, playing the role of Curly. He went on to star in several other Broadway productions, including "Carousel" (1994), "The Iceman Cometh" (1999), and "The Boy from Oz" (2003), for which he won a Tony Award for Best Actor in a Musical.

3. Angela Lansbury - Angela Lansbury began her career on Broadway in the 1940s, starring in classic productions such as "Mrs. Warren's Profession" (1947), "The Picture of Dorian Gray" (1948), and "The Three Sisters" (1956). She won her first Tony Award for Best Actress in a Musical in 1966 for her performance in "Mame" and went on to win five more Tonys throughout her career.

4. Audrey Hepburn - Audrey Hepburn made her Broadway debut in the play "The Philadelphia Story" in 1956, playing the role of Tracy Lord. She went on to star in several other productions, including "Ondine" (1954) and "Tea and Sympathy" (1953), before transitioning to film. 

We omit some content for our answer because of the limit of characters in our response. It is clear that our proposed algorithm can enhance the helpfulness and engagement of the responses.

Weakness 2 The reproducibility of the proposed method is not fully clear.

We will make the description more detailed and release codes in the revised version. We have 15K prompts for each iteration and perform 2 iterations for our algorithms. For each iteration, we have 2 epochs to train the policy model. For the exploration enhancer, we use the rejection sampling. The learning rate is 2e-7 with batch size 128.

Question 1 Is Assumption 1 mild in real-world applications?

Yes. This assumption is quite standard in the RL literature. In practice, from Lines 122-123, we can allow an infinite function class as long as it has a finite covering number. Besides, when the capacity of the large language model is sufficiently large, it is very likely to contain the true model $P^*$.

Question 2 How does the preference model outperform BT model, given that their training objectives in (2), (7) appear to be similar?

We want to clarify that the training objectives in Equations 2 and 7 are not similar, despite both maximizing an MLE objective. In (7), we directly learn the general preference $P(x,a^1,a^2)$ without assuming any specific reward structure, such as the BT model. This results in a preference function class with a much larger capacity. In contrast, (2) characterizes each response action pair by the reward $R(x,a)$ and assumes that the preference between two responses follows the BT model. The BT model imposes strong assumptions, which may not fully capture complex human preferences, such as intransitivity (see Lines 44-50).

Overall, our training objective learns a more general preference function class, while Equation 2 only considers a preference function following the BT model. Therefore, our approach applies to more general cases.

Question 3 Why does the proposed method outperform the baseline in the offline setting despite suffering from a severe overoptimization problem?

We reiterate that our algorithm should be less likely to have the overoptimization problem due to the better exploration and preference signal. Given the same data sample pairs, offline datasets only provide signals from the offline distribution, while our algorithm has better exploration strategy (equations (11, 12)) so that the overoptimization problem should be less severe.

Thanks for your great efforts in reviewing our paper and constructive comments!

Question 1 Experiment details:

1. how reward models are trained? Is oracle RM trained through next token prediction and with max length = 1 such that probability of A is the score?

We will add more details to improve the readability. For BT model, we remove the last layer of the LLaMA3-8B-it model and add a linear layer to construct the reward model, and train it by maximizing the log-likelihood on the preference dataset. For the oracle, yes, the training is conducted by next-token prediction. The problem is formulated as an instruction-following task, where we mask the prompt [CONTEXT] {x} [RESPONSE A] {a1} [RESPONSE B] {a2} and only compute the loss on the predicted token A or B. When inferencing, we re-normalize the probability on the two tokens and switch the positions of the two input responses to mitigate position bias.

2. Exact policy optimization method used in experiments (especially for the offline setting)

For the online setting (Lines 234-251), in each iteration, we first learn the main policy by optimizing the self-play ipo loss in equation (13). Then, we learn the enhancer by using rejection sampling to choose the most uncertain policy with respect to the main policy. For the offline setting, when the offline dataset has ideally good coverage (uniform coverage over all policies), we can learn the policy via self-play ipo. However, such coverage is hard to guarantee, so we focus on the online algorithm and validate in Table 3 that the online algorithm performs much better than offline ones.

3. Motivations for the special choices in Section 5

Our current choice matches best with our theoretical insights. The main player in Algorithm 2 is Nash equilibrium from oracle 2. Thus, practically, we apply self-play IPO to approximate the oracle. For the enhancer, we apply rejection sampling because equation (12) indicates that we need to maximize the uncertainty w.r.t. $\hat{\pi}_t^1$ within the confidence set. Since it is unknown how to compute the uncertainty for LLM, we practically use $\hat{\pi}_t^1$ to randomly generate $n$ samples, which is regarded as the confidence set. Then, we choose the best response in the confidence set, which aligns with the intuition of making the main agent and the enhancer more diverse. This process is the rejection sampling.

Question 2 Some concerns regarding the fairness in comparisons for empirical results. We answer this question from different aspects.

• Fairness: Our comparison with offline benchmarks is relatively fair since our online algorithm does NOT require more preference label queries than the offline version. Specifically, we used 30K prompts divided into 2 iterations.
• Focus on Theory: This paper primarily addresses theoretical aspects and proposes a framework to handle general preference. We newly conducted an iterative DPO algorithm (win rate: 14.37) compared to our algorithm's 17.67 win rate. While our algorithm outperforms the DPO, exploring the advantages of general preference over the BT model in practical scenarios (e.g., complex tasks like math and reasoning) has great potential for future work.
• Reproducibility： The reproduction of our online algorithms is reliable and similar to other IPO/DPO algorithms. Some concurrent online DPO/IPO studies mainly emphasize empirical performance. Our paper provides robust theoretical support for these empirical findings. For hyperparam sweeping, our algorithm should also be similar to other online alternatives.

Question 3 Ablation studies.

We will include more ablation studies in our paper. We illustrate some early results below

We highlight that benefits from exploration align with other empirical studies, such as RLHF Workflow, who used rejection sampling as the exploration enhancer.

Question 4 Contrast following literature

We will cite the rest and compare our work with them as follows. We denote these papers as:

• [i] [online RLHF is better than offline]
• [ii] [Nash learning from direct preference]
• [iii] [general preference optimization]
• [iv] [self-play in RLHF]

Compared to these works focusing on planning (computing the optimal policy for a fixed model), our work considers the learning problem, which is on top of these planning algorithms, and further learns the optimal policy for the ground-truth model by interacting with the human and environment via exploration strategies.

[i] shows that online and on-policy variants outperform offline counterparts, but the experiments are conducted with a fixed LLM for AI feedback. Therefore, no learning or exploration is considered in their work. Moreover, the paper is from an empirical side.

[ii] also studies Nash learning and proposes a mirror-descent-based approach. Notably, the mixture of the reference policy and the current policy (see Eq. 3) in [ii] is challenging in practice due to the extremely large response space for LLM. Our algorithm employs a self-play strategy and does not require sampling from the geometric mixture, making it more practical. Further, [ii] focuses on the planning problem, whereas we address the learning problem and use exploration. Thus, the two works are complementary.

[iii] proposes a unified framework with different loss functions and is also consistent with the results in [i] where DPO, IPO, and Slic perform similarly. They do not consider general preference oracle, learning, or exploration.

[iv] studies self-play algorithm under BT model to approximate the distribution of offline dataset. However, in standard benchmarks like alpaca-eval, even the 7B model can beat GPT4 in many cases. In contrast, our work relies on an external general preference oracle and our goal is to learn the Nash policy under this oracle.

Thanks for your great efforts in reviewing our paper and thanks for recognizing our work!

Weakness 1 The paper should also have tested using $\log(p/(1-p))$ as a target. Their current target corresponds to the IPO target.

We use $P$ directly as a target since it is more straightforward to empirically approximate the oracle in Definition 2 by iterative IPO according to [1]. Note that our theoretical framework can also apply to $\log(p/(1-p))$. A concurrent work [2] uses $\log(p/(1-p))$ as the target and empirically approximates the Nash equilibrium by using some additional tricks, such as filtering preference pairs.

[1] Daniele Calandriello, et al. Human alignment of large language models through online preference optimisation. [2] Rosset, Corby, et al. Direct nash optimization: Teaching language models to self-improve with general preferences.

Weakness 2 The practical version of the algorithm seems quite far from the theoretical version. Most strikingly, we use rejection sampling for the enhancer. I mentioned the latter as a strength due to the novel thought needed for designing the practical algorithm, but it is also a weakness because the theoretical guarantees do not signal much about the practical version due to this.

We want to argue about the point that our theoretical and practical versions are disconnected since the online theoretical algorithm exactly provides insights for the empirical version. Specifically, equation (12) in the theory part illustrates that instead of making $\hat{\pi}_t^2$ the same as the main policy $\hat{\pi}_t^1$, we need to maximize the uncertainty with respect to $\hat{\pi}_t^1$ within the confidence set. Since it is unknown how to compute the uncertainty for LLM, in practice, we use $\hat{\pi}_t^1$ to randomly generate $n$ samples, which is regarded as the confidence set. Then, we choose the best response in the confidence set, which aligns with the intuition of making the main agent and the enhancer more diverse. This process exactly is the rejection sampling.

Weakness 3 The iterative method queries the learned preference model adaptively, so it has a clear adaptive advantage over purely offline methods like DPO. The paper should also compare to iterative methods that use reward-based preference models, such as the one in [1].

Thanks for your question. First, to ensure fairness, we guaranteed that the number of the preference queries is less than the number of the offline datasets. We use 2 iterations (15K prompt for each) for our online methods, while offline datasets contain 60K paired responses. Second, we will also add comparison with iterative DPO methods under our computation and query number setting. In our setting, it performs slightly worse than our algorithm (14.37 win rate, compared with ours 17.67). In general, for chat optimization, our algorithm should be similar to or slightly better than iterative DPO since BT model seems to be a reasonable assumption in this case as indicated by the similar chat accuracies of BT reward and preference model.

Besides, this paper mainly focuses on theory and proposes a framework to handle the general preference. To explore more advantages of general preference over BT-model in real practice, we need much work on complicated tasks, like math and reasoning, which should be future work.

Weakness 4 Works such as [2,3] that accommodate partial observability are worth discussing and comparing to in related work.

Thanks for this point! We will add some discussions in the revision. We want to kindly remind that the models with partial observability and our general preference setting are two separate lines of work. Our setting considers a general preference function class when the BT-model does not hold, while the partial observability setting considers that the state features cannot be observed. The combination of the two lines of study would be interesting for future work.

Weakness 5 Some minor typos need to be fixed.

Thanks for pointing out the typos. We will correct them in the revision.

Question 1 Are their other options that the authors had while choosing a practical enhancer? Why they ended up choosing rejection sampling.

Of course, there are other options, but the rejection sampling aligns best with our theoretical insights. The reasons have been stated in Weakness 2.

One reasonable explanation is that if the reward/preference model is well-calibrated, which means that the underlying ground-truth accuracy of judging aligns well with the confidence of the model, then, a large margin between the two responses means a higher accuracy. With rejection sampling, the learning signal is more accurate and is also more consistent.

Question 2 If there is a way to estimate the coverage coefficient too. This is a low priority question, but I am wondering if you have considered computing this to have a more compelling story.

Thanks for this point! The importance sampling ratio is an upper bound of the coverage coefficient and we present some case studies in the paper where it is usually large. An accurate estimation of the coverage coefficient itself can be challenging but our general impression is that we usually cannot expect to have a moderate coverage coefficient even for a single policy. The indirect evidence is when we personally tried to apply the offline algorithms in the literature to real-world RL applications, we eventually found that for almost all the cases, the resulting policy can and can only compete with the behavior policy.

Therefore, the main applicable situation of offline learning is that we have a noisy expert who can frequently make mistakes and the offline learning algorithm can automatically adapt to the successful trajectory.

We also remark that when we are considering the Markov game, a unilateral concentration is required, which can be even more challenging compared to the single-policy coverage.

The authors have addressed my concerns on an already excellently written paper. I am raising my score to a 7. I believe that this paper fully deserves to be accepted.

Thanks for your constructive comments! Our responses are as follows.

Weakness 1 The empirical validation, while promising, is limited in scope. More extensive experiments, including comparisons with a broader range of state-of-the-art RLHF methods, would strengthen the paper.

Thank you for your question. We conducted large-scale experiments demonstrating that our algorithm performs comparably to iterative DPO in LLM chat benchmarks. We observed that the BT model is a reasonable assumption in this context, as indicated by the similar chat accuracies between the BT reward and preference models. However, to fully explore the advantages of a general preference model over the BT model in real-world applications, more work on complex tasks, such as mathematics and reasoning, is required. This will be the focus of our future research.

Question 1 How can we maintain the consistency of the learning process in the absence of the transitive assumption? For instance, when preferences are conflictive.

Thanks for this point! As stated in Section 2, we directly optimize the general preference and do not make any assumptions about the general preference, such as transitivity. Our algorithm aims to learn the Nash equilibrium policy as defined in Definition 2. For the Nash policy, it is consistently preferred by the KL-regularized preference in the face of any competing policy (line 126). Therefore, the Nash policy inherently accounts for intransitivity among different responses.

In our experiments, we use self-play IPO (Lines 234-240) to approximate the Nash equilibrium oracle. Intuitively, due to the symmetry of the preference $P$, the max-player $\pi_1^*$ equals the min-player $\pi_2^*$ when Nash equilibrium is achieved. Hence, instead of solving the min-max problem, we iteratively compute the best response to the last iteration.