Learning from Preferences and Mixed Demonstrations in General Settings

Thank you for your review and for highlighting the importance of this area of research.

We are grateful for you highlighting areas in the paper where we can improve the clarity and quality of our explanations. These suggestions will be addressed in our next draft. Additionally, we will include how LEOPARD performs against another, more modern, preference and demonstration combining algorithm, AILP[1], in both the settings of positive demonstrations, and positive demonstrations + preferences. We will also include results from the Mujoco Ant environment, which is more complex than the others we have tested on.

To clarify the relationship between RRPO and reward-rational choice (RRC)[1], we believe that despite the strong resemblance between them, RRPO is a distinct mathematical and conceptual framework. RRC frames each piece of human feedback as a single choice from a set, whereas RRPO treats it as a partial ordering over this set, which is strictly more expressive. Whilst one can theoretically carefully decompose these partial orderings into sets of single choices, this misses the conceptual power and simplicity of RRPO.

In our opinion, it is much more natural and clear how one could effectively encode different types of feedback under RRPO than RRC, when there’s no obvious single choice. It makes LEOPARD easy to express and understand, rather than obfuscating how it operates (as might happen if it was derived under RRC). This is especially apparent when comparing to the formulation of demonstrative feedback considered in the original RRC paper. Looking at that, it’s difficult to see how one might extend it to cover rankings and negative demonstrations. Our new draft will include an appendix chapter that details two alternative methods that one might straightforwardly derive from RRC to do positive-ranked-demonstration and preference learning, and how these are theoretically deficient.

On the mathematical side, the framing of RRPO made it much easier for us to spot and prove theorem 1 of the paper, and will potentially elucidate further interesting theoretical results.

We will make code available on github shortly after the rebuttal period ends.

[1] Jeon, H. J., Milli, S., & Dragan, A. (2024). Reward-Rational (Implicit) Choice: A Unifying Formalism for Reward Learning. In Advances in Neural Information Processing Systems (NeurIPS)

Thank you for your review and for highlighting some of the strengths of our paper.

To address some of the weaknesses you have raised, as part of our planned re-draft we will also include how it performs against another, more modern, preference and demonstration combining algorithm, AILP[1]. This will be done for both the settings of positive demonstrations, and positive demonstrations + preferences. We will also include results from the Mujoco Ant environment.

For clarification, the intention of LEOPARD is to be a novel algorithm for learning from both preferences and mixed demonstrations, and thus its performance in these settings is of most interest to us. We see its ability to achieve performance comparable to, or better than, existing standard IRL algorithms when only (positive) demonstration data is available as an added bonus.

[1] Taranovic, A., Kupcsik, A. G., Freymuth, N., & Neumann, G. (2023). Adversarial Imitation Learning with Preferences. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR) 2023

Thank you for your clarification. Two quick following-up question.

Re: "Instead, it implements a series of Boltzmann-Rational choices over sets containing many trajectories—note the summation in the right hand term of the denominator in equation 5. "

• In the actual implementation of LEOPARD, do you have more than two terms in the denominator of Eq 5? How many percent of them in the whole dataset, and how significant of this part contribute to the final results?
• Does the D_pos or D_neg overlaps with D_pref in your implementation of LEOPARD?

Thank you for your review and for highlighting the ability of RRPO to faithfully learn partial rankings across many types of feedback.

It appears there is some confusion about the nature of LEOPARD. To clarify, it is not just a simple combination of existing approaches. Whilst it does subsume RLHF and reduce to it in the case of only preference information, when there is only demonstration data LEOPARD implements a novel inverse RL algorithm. Furthermore, it represents a novel way of unifying and combining preference and demonstration data (along with potentially many other data sources).

To be more concrete, in the general case where demonstration data is available, it does not implement trajectory-level pairwise preference rankings. Instead, it implements a series of Boltzmann-Rational choices over sets containing many trajectories—note the summation in the right hand term of the denominator in equation 5. Compared to sampling pairwise preferences based on rank data, RRPO/LEOPARD can capture the full hierarchy of the human’s values over the given trajectories. The reward for each one can be increased/decreased in proportion to where it is in this hierarchy at every reward model optimisation step. For pairwise comparisons, this proportionality only arises in the limit of many optimisation steps, as when evaluating each pairwise preference the algorithm is blind to how these trajectories fit into the bigger picture.

To address your concerns about our evaluation, as part of our planned re-draft we will include results from the Mujoco Ant environment.

Thank you for your review and for highlighting the flexibility of LEOPARD to learn from negative demonstrations in addition to the typical sources of feedback.

To help address your concerns, we will include in the updated draft an appendix chapter with the ground truth rewards of the demonstrations. The preferences are generated over randomly selected trajectories on-policy, predominantly from the last iteration, so this can be used to infer their rough distribution of their rewards.

Whilst SotA on HalfCheetah-v4 is much higher than the reward we get, we believe this is primarily because we use PPO instead of algorithms that perform highly on HalfCheetah like SAC. We did this for two reasons.

1. PPO converges faster (even if to a lower performance) and so it takes less computational effort to differentiate the performance of the different algorithms and data sources.
2. Lunar Lander and Cliff Walking (as well as LLM finetuning) have discrete action spaces and so it is typical to use PPO. By sticking with PPO we make the Half Cheetah results more comparable to these experiments and the inferences that can be made from it stronger.

Testing RLHF followed by DeepIRL is an interesting idea and we will run it on Half Cheetah as an ablation. Originally we neglected it, as the other way round is seen as optimal[1], but it would be a good sanity check.

For the stopping threshold, we use the same value for all environments and found that performance is not sensitive to it. Further details on this can be found in Appendix A.1.3 of the paper.

[1] Erdem Bıyık, Dylan P Losey, Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1): 45–67, 2022.

Dear Reviewers,

Thank you for all your feedback. We have now uploaded the amended draft of our paper, and hope this addresses many of your remaining concerns.

Major changes: Added results on Ant. Added results from AILP baseline on all environments, and RLHF then DeepIRL baseline on Half Cheetah. Adjusted relevant sections based on these results. Added illustrative diagram of overall method. Alternative approaches appendix chapter. Plotted individual settings of reward-model-training-epochs for DeepIRL-based methods in the appendix, and included only the best and average in the main figures.

Minor changes: Included figures of demonstration trajectories in the appendix. Moved tables into the appendix. Provided additional detail to method in the abstract. Provided clarification on what is meant by 'algorithm iterations' in figures 1-3. Better integrated equations 1 and 2 into the text. Made some notation more consistent.