Reward Learning from Multiple Feedback Types

Learning a reward model for each feedback type presents some limitations not considered by the authors.

• If these reward models are large or computationally expensive (e.g. in an foundation model finetuning setting where each one may be foundation model sized), then having one for every feedback type might not be very scalable or practical.
• The resulting reward signal cannot incorporate information only deducible from considering multiple feedback types at once. For example, there may be some patterns or generalisations only apparent when considering all feedback.
• If there's only small amounts of one type of feedback, its corresponding reward model might be very inaccurate, and if the uncertainty for the uncertainty-weighting is not well calibrated, it could lead to that reward model making the overall reward signal worse.

The paper only considers synthetically generated feedback and does not evaluate / verify with real human feedback.

The paper focuses on continuous control tasks, and does not test on any discrete or contextual bandit environments. This is important to test as this more closely corresponds to the LLM fine-tuning setting, a key application of RLHF and related methods.

Interpreting the demonstrations as being preferred only to random trajectories seriously limits the information extracted from them compared to standard demonstration learning approaches like MaxEnt. Despite the authors claims suggesting this was "stronger than sampling against other sub-optimal rollouts", the amount of interesting state space explored by random policies decreases exponentially as the environment gets larger and more complicated. For example, in the language modelling setting, interpreting your demonstrated responses as better than random tokens would probably not improve your pre-trained model compared to supervised fine-tuning.

The feedback methods, except one, all use the same fixed segment length. This may be sub-optimal if different feedback methods work better with different sized segments.

The reward model correlation analysis is only done on one environment and are averaged over the whole trajectory. It's not clear how consistent these correlations are across different environments, and it may be interesting to see how these correlations change over the course of a typical trajectory (E.g. maybe some are well correlated to begin with, and then become de-correlated).

Graphs have no error bars, and it's not clear how many random seeds have been averaged over in the RL experiments. If there are not multiple random seeds, then very little of the results presented can be assumed to be statistically significant.

The reward models are pre-trained before being used to train a policy. This is very atypical for RLHF and limits the conclusions that can be drawn from the results.

In figure 3, not showing the score for learning from the ground truth reward as a function of environment time steps limits comparisons between learning from the feedback modes vs the ground truth itself.

It's not clear how amounts of each type of feedback are controlled to be comparable. It seems the amount of each type available is very dependent and sensitive to parameters of the synthetic generation process.

For uncertainty-weighted ensembles, the uncertainty of an ensemble that has been trained on feedback that only constrains reward differences (e.g. comparative) is not well-calibrated.

I do not believe some of the claims in the discussion and conclusion are well-supported.

• On point (2), line 495/496, combining rewards is only tested on two environments and only performs well in one of them.
• Line 523/524, the comparison of these characteristics has been very limited.
• Line 526 to 528, learning from multiple feedback types being "very effective" is not supported by the evidence presented in figure 6.

Errata:

• Figure 2 caption and subcaptions disagree on what environment the results are from (Swimmer-v5 vs HalfCheetah(-v5)).
• Figure 3 partially covers some text
• The paragraph at the start of section 4.4 conflicts with itself: "instead of continuously adapting ... is continuously updated"
• Line 380, sentence abruptly ends and is unfinished
• The axes and legends for figures 2,3,4, and 5 (especially 3), are too small to clearly read, especially on a printed copy of the paper.
• The lighter shaded lines in figures 4, 5, and 6 are hard to see.
• There is no reference to figure 4 in the main text of the paper, nor analysis of what it shows.
• Line 454, "As stated before, each individual reward model ... is an ensemble in itself". This does not appear to have been stated before.
• Line 467, the text refers to the "Hopper-v5" environment, but the figure under discussion, 6, only contains the HalfCheetahv5 and Walker2d-v5 environments.

High-level overview:

There are two main weaknesses of this paper discussed at a high level here, but more details below. The work is valuable and necessary, but the needed level of rigor isn't there yet. (1) the toolkit it not validated against any studies involving humans, therefore it is impossible to know if the conclusions drawn in the paper reflect characteristics of the feedback or of the implementation (2) the text in the paper, especially in the second half describing experiments and results, contradicts itself and presented results making it difficult to draw conclusions and have take aways or learnings.

The details:

• The way the paper is written suggests that both the multi-feedback toolkit and the method for combing the multiple sources of feedback are both equal contributions. I would push back that the main contribution is the toolkit and the proposed method is rather trivial with no true baselines to show its benefit or contribution. It would have been better to limit the feedback combination methods as a tool for proof of concept showing how the toolkit can be used.
• The authors draw conclusions about the usefulness of different types of feedback (e.g. Figure 2 reward model correlation and demonstrative and corrective having the worst correlation with the ground truth reward). However, in the absence of a study to validate the toolkit against human feedback makes it impossible to validate if this is a function of the feedback, or the choices they have made in how the feedback is implemented.
• Only one environment is evaluated therefore it is not possible to assess how general the proposed toolkit and feedback simulation method are. At least one other should be included. MetaWorld is popular for the preference learning and would be a natural fit as it is included in [BPref] (https://github.com/rll-research/BPref).
• There are multiple places in the paper where the authors seem to contradict either themselves or the presented results:
  • at the start of Section 4.4, the first sentence sounds like there was not online reward model adaptation, but then the last two sentences to that first paragraph make it sound like there was: "...we utilize simple pre-trained reward models instead of continuously adapting the models..." versus "The reward model is continuously updated..."
  • the text in the main body does not mentioned that the reward model correlations for other tasks beyond half cheetah have are weaker with different patterns among the feedback types. The full nuance of the results are not represented, and instead strong language and conclusions are drawn about a single task.
  • the statement on lines 325 - 326 "...all feedback types individually can learn reward functions that closely match the ground-truth reward..." is not true for all feedback types according to the Figure 2 and those in the appendix. In Figure 2, demonstrative and corrective have a correlation of 0.61 and 0.5, which is a medium match. In Figure 25, these feedback types have correlations as low as a 0.18.
  • on lines 410 - 411, it is stated that "...no feedback type is obviously more or less robust to noise." however, then looking at Figure 4(b) and Figure 5(b), the performance gap between learning curves across different amounts of noise varies across feedback types, which suggests that some are more sensitive that. others. To back up the claim, "obviously" needs to be quantified to make it clear the threshold for what qualifies as "more or less robust".
  • section 5 opens by stating that different feedback types struggle in different scenarios, but all of the discussion in Section 4 talks about how there is no clear difference between the different sources of feedback.
• The first half of the paper motivating and describing the multiple feedback toolkit is very well written, methodical, and easy to follow. However, there is a switch part way through the paper when it transitions to talking about experiments with the toolkit. Here the presentation and writing changes drastically with things like:
  • Figure 3 overlapping the text of the main body
  • an incomplete sentence (page 8 line 380)
  • what seems to be misconnected analysis and results figures (e.g. figures 4 and 5 - the text seems to talk about figure 4, but references figure 5 and figure 4 is not mentioned in the paper as far as I can tell)
  • mislabelled and titled figures where the captions disagree with the figure axis/title (e.g. Figure 2 - HalfCheetah-v5 versus Swimmer-v5; Figure 5 - swimmer v5 and walker2d-v5 versus half cheetah-v5; and figure 4 - "reward model validation curves", but it looks like policy returns)
  • Figure 4 left plot, the y-axis scale are on different scales, making it tricky to compare across feedback sources.
  • using both a : and - on page 8 line 429
  • it is stated that results are over 5 random seeds, but there are no standard deviation results
  • there are numerous places with typos (e.g. "as the be considered" line 256) that need to be addressed.
  • not all lines in Figure 3 are labelled nor described.

1. Limited Analysis of Feedback Types and Noise Effects: The paper provides only a shallow analysis of the results across different feedback types and the impact of adding noise. The authors introduce Gaussian noise as a way to simulate realistic inconsistencies in human feedback, which is a valid approach. However, they assume that the added noise will uniformly challenge the agent’s learning process, yet they provide limited empirical support to demonstrate the nuanced effects of this noise. In reinforcement learning, robustness to noise is complex and context-dependent; simply adding noise doesn’t necessarily simulate real-world variability comprehensively. This is particularly relevant in cases where the agent encounters unfamiliar scenarios, as it may lack the generalization needed to adapt successfully, which is not addressed here. It would be beneficial if the authors quantified the noise across feedback types and analyzed how this type-specific noise impacts learning stability and performance. Without such quantification, comparing the robustness of different feedback types remains somewhat speculative and may not provide fair insights. As highlighted by Casper et al. (2023), inconsistencies in human feedback are a fundamental limitation in reinforcement learning from human feedback, underscoring the need for a systematic approach to evaluating robustness in such noisy environments.

Reference: Casper, Stephen, et al. "Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback." Transactions on Machine Learning Research, 2023.

2. Reward Ensemble Approach is Underdeveloped: The paper attempts to improve reward learning through a reward ensemble (combining different feedback types), which is an intriguing direction. However, the ensemble approach lacks depth and does not yield substantial improvements. The authors themselves acknowledge that rewards from different feedback types cannot be averaged simply, and yet, the methods presented for combining them are fairly straightforward. Given the challenging nature of ensemble reward learning, the results are not strong enough to warrant this as a core contribution. As such, this component appears more exploratory than foundational, and it could benefit from either more sophisticated ensemble techniques or a more extensive evaluation to validate its potential utility.

3. Insufficient Details as a Software Library: As a contribution to reinforcement learning tools, the paper lacks critical details expected from a software library description. While the library supports diverse RL environments beyond the basic Gym Mojoco tasks, only these tasks are presented in the main text. Further, there is little mention of essential details such as hardware requirements, training time, or memory usage, which are crucial for reproducibility and practical use by researchers. Although some hyper-parameter details are provided in the appendix, this is insufficient for a full understanding of the library’s operational requirements and expected performance.

4. Typos and Formatting Issues

• Figure 4b title should read “RL Episode Returns.”

• Incomplete sentences in the paragraph near line 380.

• Hidden text in line 356.

1.The workload of this paper is substantial, covering many key points, which results in relatively preliminary research on each type of feedback. The characteristics of different feedback types are not well demonstrated. Can you describe several key feedback types or explain which feedback types are more suitable for specific scenarios?
2.The first half of the paper is well-written, but the experimental organization in the latter half is chaotic, making it difficult to draw clear conclusions.

Overall, I believe this paper is highly valuable, but the current version appears somewhat hasty.