PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback

General Response: Thank you for your comprehensive review and positive feedback. We are pleased that our PARL framework approach to policy alignment and the effectiveness of the A-PARL algorithm resonated with you. Your recognition of the clarity of our work and the rigor of our experimental comparisons is greatly appreciated. We provide a detailed response to your comments as follows.

Weakness

Weakness 1 : I am not very familiar with the literature of the benchmarks used in this area but I go through the papers along the line and confirm that the experiments conducted by the authors are commonly adopted in the literature so I assume that the experiments are satisfactory (which may need to be further evaluated by other expert reviewers).

Response to Weakness 1: Thank you for your feedback regarding the benchmarks used in our experiments. We appreciate your efforts in reviewing relevant literature to understand the context of our work. We remark that, for this work, our main contribution (as you mentioned in strengths) is to provide a unified novel theoretical framework (A-PARL) to address the issues of the existing literature. To support that, we did a rigorous theoritical analysis of the policy alignment problem, which is missing from existing literature (Deep RLHF (Christiano et. al 2017), Pebble (Lee et. al 2021), Surf (Park et. al 2022)) etc. Hence, we performed experiments on the three domains as a proof of concept of our algorithm and theoritical analysis.

Weakness 2 : Could you comment on the generality of the proposed framework? In particular, I am interested in whether this framework can be extended to handle the alignment of language model because the reward model quality is the bottleneck in many cases and most of the existing approaches use only a fixed reward model.

Response to Weakness 2: This is an excellent point. Indeed, the proposed framework holds promise for extension to the alignment of large language models (LLMs). A significant benefit in the context of LLMs is the deterministic nature of transition dynamics at the lower level. This occurs because we append the new token with the input segment (current state) to generate the next token. This deterministic aspect simplifies and strengthens the framework's application in LLM alignment. We acknowledge this as a valuable direction for future research.

I have read the responses and have no further question. I keep my score toward acceptance.

General Response: Thank you for your insightful review. We appreciate your recognition of the novel bilevel formulation in addressing the policy alignment problem in reinforcement learning. Your acknowledgment of the clarity of the paper and the thoroughness of our theoretical derivation is greatly encouraging.

Weaknesses:

Weakness 1.1: The objective of the paper is somewhat confusing to me. Would the authors please clarify the cause of the misalignment that is considered? and what role does the human agent play?

Response to Weakness 1 Thanks for this opportunity to clarify. The misalignment primarily arises in RL due to the misspecified reward function because true reward is usually unknown in the majority of the practical scenarios. A human feedback helps to learn the true reward function as also highlighted in (Christiano et. al 2017, Lee et. al 2021, Park et. al 2022). However, interestingly, the existing approaches of learning from human feedback such as (Deep RLHF, PEBBLE, SURF) suffer from misalignment for not considering the interdependence between the reward and policy learning leading to sub-optimality (as highlighted in our paper, Sec 3.1, 3.2, Figure 1). Incorporating this dependence through a bilevel formulation helps in mitigating the performance gap of prior algorithms, as shown in Figure 2.

Hope this clears the confusion. Please let us know if there are any further confusions regarding this point, we are happy to discuss and elaborate further.

Weakness 1.2: Is there a fixed RL problem, but the agent can only get some feedback from a human who partially knows the real reward, and such feedback changes over time as human learns more about the reward? or a dynamically changing RL problem is considered i.e. reward (value function) changes over time, and one possible cause of such change is due to how the reward is generated (feedback produced by a human's preference).

Response to Weakness 2.1: Yes, our formulation considers a fixed RL problem. The problem deals with the setting (similar setting to Deep RLHF, Pebble, Surf etc.) where there is a fixed ground truth reward function $r_{\nu^*}$ ($\nu^*$ is the optimal reward parameter unknown to RL agent). The human feedback corresponds to the optimal reward function which is received by the RL agent as preferences on querying the human with relevant trajectory pairs.

In our work, to learn optimal $\nu^*$ from the human feedback, the existing state-of-the-art methods ignore the dependence on the optimal parameter of the policy $\theta^*$. We highlight this drawback and propose to resolve it via formulating a novel bilevel formulation where we iteratively update the parametrized reward function $\nu_t$ at the outer objective (shown in equation 8, 14) by collecting human feedback on trajectories generated by solving the inner optimal policy (shown in equation 15). This is the exact setting of our formulation and is important to note that the true $\nu^*$ doesn't change over time and is fixed, as also followed in previous baselines.

Weakness 2.2: More precisely, the authors used to parameterize the upper level (evaluating the lower level policy). What does each $\nu$ represent?

Response to Weakness 2.2 In our formulation, $\nu$ represents the parameter for the reward model $r_{\nu}$ and the objective is to learn the ground truth reward parameter $\nu^*$.

Weakness 2.3: My other concern is experiments are not detailed. Descriptions and simulation set up are all missing. For instance, figure 2, what precisely do 'walker' or 'door open' or 'button press' aim to do? how does the misalignment occur? what does oracle represent? what are the upper level objective functions for these tasks?

Response to Weakness 2.3 Thank you for raising this concern. Since we have followed the exact similar enviromental configuration and setup used in previous benchmarks, we cited the reference (Christiano et. al, Lee at. al, Park et. al) without specific descriptions. But we agree with the reviewer that providing a detailed description will definitely be helpful for the readers and hence we have updated in Appendix K of our paper with relevant details.

General Response: Thank you for your insightful review. Your detailed summary accurately captures the essence of our approach in formulating the PARL as a bi-level optimization problem. We are grateful for your recognition of the practicality and theoretical rigor of our algorithm, as well as its performance in the experiments. Your constructive comments are highly valuable, and we address them in detail as follows.

Weaknesses:

Weakness 1: Empirical evaluations could have been more thorough. In Figure 2, only three domains were considered (2 manipulation / 1 locomotion), which is much smaller than the ones in the baseline papers. For example, in the PEBBLE/SURF papers, they conducted experiments in 9 domains (6 manipulation / 3 locomotion). I am curious if A-PARL still outperforms the baselines in other domains.

Response to Weakness 1 : We agree that it is possible to conduct more experiments on the remaining baselines. However, for this work, our primary focus is to provide a unified novel theoretical framework (A-PARL) and a rigorous theoritical analysis of the policy alignment problem, which is missing from existing literature (RLHF (Christiano et. al 2017), Pebble (Lee et. al 2021), SURF (Park et. al 2022) etc.). Hence, we performed experiments on three domains (with different level of hardness) as proof of concept of our algorithm and theoretical analysis.

New Experiment Results: However, we remark that our performance is transferable to other environments as well. As mentioned by the reviewers, we added additional hard environments like Cheetah (DMC) and compared the performance of A-PARL with baselines in Figure 6 in Appendix K (marked in blue). We observe that A-PARL outperforms existing baselines in both environments.

Note: Due to time constraints, we have only added Cheetah (Locomotion, DMSuite) for now, but we are running further experiments on more hard environments (like visual-Walker, Sweep Into) and will try our best to add them in the final revised draft.

Weakness 2: Some part of the algorithm is unclear. For example, it seems that gradient/Jacobian/Hessian estimation (Eq 15-17) requires 'on-policy' samples for their computations. Still, I was first assuming the lower-level policy optimization is done using an off-policy RL algorithm, as in the previous work (e.g. PEBBLE). Then, is my understanding correct that upper-level optimization is done using on-policy samples while the lower-level optimization is done in an off-policy way? Does lower-level optimization also use on-policy updates (Algorithm 1 line 5 seems to perform on-policy updates)? If so, I am curious how the on-policy algorithm could have shown better sample efficiency than the off-policy baselines.

Response to Weakness 2 This is a good point. We note that in our bilevel formulation, the inner level is the standard RL policy optimization problem and can be solved using any standard RL algorithm either on-policy or off-policy. For the sake of analysis and establishing a connection to the most basic setting (as in RLHF (Christiano et. al 2021)), our policy learning update in Algorithm 1 is an on-policy. For the experiments and to have fair comparisons with Pebble and Surf, we use the same backbone (SAC which is off policy) with similar configurations and hyperparameters so that the results are comparable.

We remark that irrespective of the algorithm we use to solve the inner-level RL problem, a key innovation in our approach lies in incorporating the gradient $\nabla_{\nu} \theta^*(\nu)$ (equation 12) in the outer objective (equation 8, 14), which helps to learn the true optimal reward and close the gap in the existing literature.

Weakness 3 : Lack of ablation studies. For example, to see the efficiency of A-PARL more properly, it would be great to see the results using varying the number of feedbacks. In the current form of the paper, it is even unclear how many feedbacks were used or how frequently the feedback was collected in the experiments.

Response to Weakness 3: In the current setting, we compare with the exact number of default feedbacks: Walker (max_feedback = $100$), Door Open (max_feedback = $1000$) and Button Press (max_feedback = $20000$) as typically used in (RLHF, Pebble, SURF etc.) for the three environments mentioned.

New Ablation Study: As requested by the reviewer, we did additional experiments by varying the number of feedbacks as shown in Figure 7 in Appendix K.1 in the updated draft. It shows that more human feedback results in better performance as expected. We will add a much more detailed version of this ablation study with respect to human feedback in the final version of the paper.

General Response: We sincerely appreciate the reviewer's thoughtful feedback on our paper. We would specfically like to acknowledge the reviewer for identifying the major motivation and contribution of our work, which is to characterize the dependence of the policy optimization on the upper-level objective and provide a detailed theoritical analysis of the proposed framework. We are pleased that the reviewer recognized the significance and relevance of this contribution to the field of policy alignment.

Questions:

Questions: What values of K and N were used in Algorithm 1 for the experiments? Could you elaborate on how these parameters were tuned and the impact of selecting different values? Specifically, how does a larger K value, which implies more accurate solving of the lower problem but at the cost of additional environment trajectories, affect the overall outcomes? What trade-offs are observed with varying K values?

Responses: Thank you for this comment. For the experiments we choose the default settings as used in (RLHF (Christiano et. al 2017), Pebble (Lee et. al 2021), SURF (Park et. al 2022) etc.). The exact values are for Walker (N = $100$), Door Open (N = $1000$) and Button Press (N = $20,000$) for the human feedback. As requested by some other reviewers, we have also varied the feedback count and performed additional experiments, results are updated in Appendix K (marked in blue).

Regarding the value of $K$, as you correctly pointed out, it represents the number of gradient updates for the inner policy optimization loop i.e., how much we optimize the policy for the given reward. Interestingly, we are able to derive a precise value of $K = t$ ($t$ : outer iteration count) from our analysis (equations 58, 59 section F in Appendix) and use the same in experiments. As pointed out by the reviewer, it's also very intuitive and represents that as the reward becomes more and more closer towards the optimal, we increase the value of $K$ i.e more accurately solve the inner problem. We remark that selecting any other value of $K$ would result in a tradeoff in terms of optimality of the inner level problem.

Thanks for the insightful questions and we will add the discussion to our revised draft.