Exploiting Action Distances for Reward Learning from Human Preferences

The key premise — Using policy for reward model may not always work

• In RL, the reward acts as the only learning signal and in PbRL, preference determines the reward function. While improving the reward model training by the use of environment dynamics makes sense, reachability via action distance is an artifact of both the original state distances and also the currently trained policy. This could induce a non-stationarity in the learning process because of the cyclic dependency of the reward on the policy and the policy on the reward. So, I have concerns about the robustness of the heuristic of action distance proposed in this work.
• What is the support for the hypothesis that a policy trained over learned rewards tells more about the human preference. This is quite counter-intuitive because the reward is the only signal to train the policy (it acts like a bottleneck interface between the policy and the human preference), so how can the policy have more information to return to the reward function, about the human preference?
• Can the authors justify why the ADLoss would never hurt reward learning in any environment?

Experiment Concerns

• Inconsistent experiment settings from PEBBLE paper: Is there a reason why the number of feedbacks chosen in this work are different from the base settings provided in the PEBBLE paper? Also, why are the set of 6 Metaworld tasks chosen different?
• Why does PEBBLE converge suboptimally? Are the parameters for PEBBLE well-tuned? What was the scheme of hyperparameter optimization? Can the authors report results on the exact same environment setups used in PEBBLE, so the baseline results can be verified and trusted that a fair amount of hyperparameter tuning has been performed? Then using the same hyperparameters, provide the results of ADLoss + PEBBLE.
• Since the proposed solution is a simple heuristic, it requires more thorough experimental validation than the the current results. It is hard to convincingly believe the experimental results being generally better than the baselines. The evaluation should be done on more Metaworld tasks and at least on the settings that PEBBLE reported if not more (because the effect of auxiliary loss could be negative — so it should be known when it works and when it does not work).
• Can the authors demonstrate some experiments with noisy dynamics, to show the robustness of the action distance Loss in expectation?
• No evaluation curves provided from the human study. At the stopping criterion chosen by humans, is it verified that the performance of PEBBLE and ADLoss are equivalent?

Motivation of the approach and Baselines

• It is not clear why this particular form of ADLoss and this particular way of using it as an auxiliary objective is the best solution to the question of "how to make reward model learning more feedback efficient by incorporating the policy". As of now, it seems like a ad-hoc proposal that just happens to work, but it requires more thorough intuition and experimentation to show that this is the best way to answer this question. For instance,
  • The authors mention that using a scaled linear combination is exteremely non-trivial and challenging. Why is that so? In fact, Equation (8) also has scaling hyperparameters between the loss terms. So, using an additional reward term is a valid baseline, and an important one to justify the need for auxiliary loss.
• The motivation from Cheng et al. (2021) is not applicable because they modify the reward function and the discount factor with pessimistic heuristics, and the action distance being a pessimistic measure of the reachability of two states, and then using it as an auxiliary task, has nothing to do with their results, even in the goal-conditioned case.
• Ideally, SURF + ADLoss v/s SURF should be shown, to show ADLoss is complementary.
• Why would auxiliary loss over the state representation help? It is not like the states are image-based, so representation learning doesn't seem like the most crucial challenge here. It has to be providing some task-specific information, in which case, there must be more direct ways of utilizing this action distance idea.

How about alternative auxiliary losses?

• Motivation for ADLoss is lacking: there can be many more baselines compared against for auxiliary losses, including Inverse Dynamics Prediction (or Curiosity Loss) or Contrastive Loss or Self-Supervised Domain-specific losses or reconstruction.

• While I appreciate that the proposed method is simple and general ("can be used with any existing PbRL technique" in theory), I think it is crucial to demonstrate the effective of the proposed method when combined with different existing PbRL algorithms. Currently, the proposed reward learning objective is only combined with PEBBLE, which is not enough. It should be combined with other existing PbRL methods and evaluated as well.
• The experimental results are not that convincing/competitive in my opinion:
  • Based on Figure 2, I think the proposed method ADLoss does not really perform significantly better than SURF in most tasks. ADLoss and SURF perform similarly in Hammer and Button. Their variances overlap in Door Open and Window Open.
  • In the experiments, the oracle scripted human uses the underlying true reward to provide perfect binary feedback. To demonstrate the robustness of the proposed method, I think it is important to test some more realistic models of human teachers (simulated human with various irrationalities).
  • The human user study only consists of 3 human participants. It is hard to draw any conclusion that is statistically useful.
• Some minor suggestions:
  • I think the method section is unnecessarily long. It includes a long discussion about different methods that could be explored but probably wouldn't work well, which could be moved to the appendix.
  • Calling the proposed method (combination of the proposed reward learning objective with PEBBLE) as ADLoss is misleading since it sounds like only the action distance based loss function is used (while the original cross-entropy loss is actually used).
  • The way the references are cited throughout the paper can be improved.

• The evaluations come from the same domain (meta-world), and in robotic manipulation. The evaluation can benefit greatly from introducing other domains (e.g. quadruped, walker).
• While the authors have conducted a human study, the number of participants (3, with 1 familiar with the domain) is low to evaluate any hypotheses.

• W1: I am very skeptical that the discussion about pessimistic heuristics actually explains the performance improvements of ADLoss. Per Section 4.3, there are several pre-requisites for ADLoss to be considered a pessimistic heuristic: the rewards must be $\le -1$, the discount factor has to be $\gamma = 1$, the environment needs to be strong markovian. There is no attempt in the paper to show that either MetaWorld or DMC follow these strong pre-requisites. In my opinion, the intuitions given earlier in the paper about encoding dynamics, coupled with strong empirical evidence would be sufficient to prove the utility of ADLoss.
• W2: The empirical evidence needs to be strengthened. Current results show ADLoss improving upon SURF on only two environments: "Sweep Into" and "Drawer Open". Ideally, I would have liked to see experiments with DMC tasks (beyond the task in the user study), as well as experiments combining ADLoss with other sample efficiency PbRL algorithms (such as SURF and RUNE [1])

[1] X. Liang, K. Shu, K. Lee, and P. Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. ICLR 2022.