Response to Reviewer #e9yZ

We sincerely appreciate your thoughtful feedback and comments on our paper. Below, we address each of your concerns in detail. All our responses will be incorporated into the final paper.

1. Comparisons with other robust PbRL methods?

Thank you for highlighting this point. Following the reviewer’ and other reviewers’ comments, in this rebuttal we have added the comparison with some other robust reward learning methods, including RIME[1], SURF[2], MAE[3] and t-CE[4]. The comparison is performed on the cartpole-swingup and walker-walk tasks, with 20% and 30% error rate.

For the comparison of RIME [1] and SURF [2], we used the same settings as the original PEBBLE baseline for collecting trajectory segments. In RIME, KL-divergence between predicted preference probabilities and labels filters untrustworthy labels and flips them for improved learning. We set RIME parameters to $\alpha = 0.25$, $\beta_{max} = 3.0$, $\beta_{min} = 1.0$, $\tau_{upper} = -\ln(0.005)$, $k = 1/60$ for cartpole-swingup task and $\alpha = 0.3$, $\beta_{max} = 2.2$, $\beta_{min} = 1.7$, $\tau_{upper} = -\ln(0.005)$, $k = 1/100$ for walker-walk task. For SURF, we changed the length of collected segments to 60 and use temporal data augmentation to crop segments of a fixed-length 50. We choose $\tau = 0.95$, $\mu = 1.0$ and $\lambda=1.0$ for SURF in both tasks. All algorithm parameters are chosen for the best performance of the baseline methods.

In MAE[3], the original loss function is replaced with $L_{MAE} = \mathbb{E}|\hat{y} - P_{\theta}|$ for robust reward learning. In t-CE[4], the loss is replaced with $L_{t-CE} = \mathbb{E}\sum_{i=1}^t \frac{(1-\hat{y}^T P_{\theta})}{i}$. Here, $\hat{y}$ is the one-hot version of noisy label and $P_{\theta}$ as the predicted probability of human preference on trajectory pairs. We choose $t=4$ in t-CE loss of its best performance.

The result (sum of reward) is shown in the table below:

The results show that the proposed HSBC method outperforms baselines in robust learning under high error rates. Among the baselines, RIME excels at handling false preference labels with its label denoising design. In the Walker task, using t-CE loss also achieves robust learning.

We will include the above baseline comparison in the revised version of the paper.

2. Validation in a practical setting?

We appreciate this observation. To address this, we performed HSBC on real-human feedback, please refer to the response of question 5 to reviewer #sofd.

3.The theoretical bounds assume a finite VC-dimension, how does HSBC generalize to high-dimensional neural reward functions?

We thank the reviewer for the helpful comments. The VC-dimension is finite for certain neural network classes, such as multilayer perceptrons (MLPs) with ReLU activations, where it scales with the number of parameters and layers [5]. For such networks, the upper bound of the sample complexity applies. However, the exact VC-dimension of general neural networks, especially deep architectures with complex connectivity and unbounded weight norms, remains an open problem. In these cases, Theorem 4.2 provides a worst-case upper bound, which may be conservative. Tighter complexity bounds are an important direction for future work.

We appreciate your time and effort in reviewing our work. Your feedback has been invaluable in strengthening our manuscript.

Reference

[1]Cheng, et al. "Rime: Robust preference-based reinforcement learning with noisy preferences." arXiv preprint arXiv:2402.17257 (2024).

[2]Park, et al. "Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning." arXiv preprint arXiv:2203.10050 (2022).

[3]Ghosh, , et al. "Robust loss functions under label noise for deep neural networks." Proceedings of the AAAI conference on artificial intelligence. Vol. 31. No. 1. 2017.

[4]Feng, et al. "Can cross entropy loss be robust to label noise?." Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence. 2021.

[5]Bartlett, et al. "Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks." Journal of Machine Learning Research 20.63 (2019): 1-17.

Response to reviewer #GLNZ

We sincerely thank you for your comments. Below, we address each of your comments in detail. All our responses will be incorporated into the final paper.

1. There are multiple ways... What kind of errors can it handle more easily?

Thanks for your question. For different human error types in Bpref, our method cannot handle “Equal” preferences as we do not consider ties. However, our method can naturally handle “Skip” cases, as these can be simply excluded when constructing preference batches. We added evaluations on “Stoc,” “Mistake,” and “Myopic” teachers in the Cartpole task, following BPref hyperparameters: $\beta = 10.0$ for "Stoc," $\epsilon = 0.2$ for "Mistake," and $\gamma = 0.98$ for "Myopic". The conservativeness level was set to 20%, with other settings following the paper. Results confirm our method’s robustness across these feedback types.

The results show our method handles “Mistake” and “Myopic” teachers well but struggles with “Stoc” teachers. This could be because in the late stage of learning, the trajectories are close enough in its rewards and “Stoc” teacher tends to provide very noisy labels, which hinders the convergence of the algorithm.

2. How does voting affect the cuts?

Thanks for the question. We will use Figure 3 in the paper for illustration. The voting function $V_i (\boldsymbol{\theta})$ controls the aggressiveness of hypothesis space cuts. Retaining only the maximum-vote region $V_i (\boldsymbol{\theta})=N$ would remove all hypotheses disagreeing with any preference, assuming perfect feedback accuracy. However, human errors could wrongly eliminate the ground-truth reward (left panel of Figure 3). To address this, we set a mild vote threshold $V_i (\boldsymbol{\theta}) \geq \lfloor(1-\gamma) N\rfloor -0.5$, preserving a broader hypothesis space containing $\boldsymbol{\theta}_H$ (Lemma 5.2), as shown in the right panel of Figure 3.

3. How much does the learned reward deviate from ground truth?

Thanks for the question. To assess consistency, we compute the Pearson correlation between the learned and ground-truth rewards in Cartpole, Walker, and Humanoid. For each task, we generate five 200-step trajectories and report the mean and standard deviation. Results are below:

The above results show that the learned reward and groundtruth reward has good correlation. Also, this correlation is weaker when the error rate increases, which is consistent with the performance results.

4. Comparison with SURF?

Thank you for the comments. In this rebuttal, following the reviewer’s and all other reviewers’ comments, we have added the comparison with some other robust reward learning methods, including SURF. Please refer to the response of question 1 to reviewer #e9yz.

5. Is it necessary to use a sigmoid function in Eq 20...?

No, any smooth differentiable function with a similar shape to sigmoid can be used, for example, the tanh function.

6. Ablation of the ensemble size M

We added an ablation experiment on ensemble size $M$ on the walker-walk task with 20% error rate. The final result is shown below:

With the groundtruth result of $472.9$. It can be seen from the result that increasing the ensemble size $M$ can slightly improve the performance, but not signfiicantly.

7. Better for clarity of the function f

Thanks for the comment. The function $f$ represents the signed reward gap between ${\xi}^0$ and ${\xi}^1$, with the sign determined by human label $y$. Because $y$ is the “index” of the preferred trajectory, we set the sign as $1-2y$ to ensure $+1$ when $y=0$ and $-1$ when $y = 1$.

As shown in Equ. (6), one human preference $({\xi}^0_{i,j}, {\xi}^1_{i,j}, y_{i,j})$, no matter true or false, imposes a constraint on hypothesis space using the inequality $f({\theta}, {\xi}^0_{i,j}, {\xi}^1_{i,j}, y_{i,j}) \geq 0$.

8. Typo of Heaviside function

Thank the reviewer for point this out. This was a typo. The correct notation is $\mathrm{\mathbf{H}}(x) =1$ if $x \ge 0$ and $\mathrm{\mathbf{H}}(x) = 0$ otherwise. We will fix this in the revised version.

Reference

[1]Lee, Kimin, et al. “B-pref: Benchmarking preference-based reinforcement learning.” arXiv preprint arXiv:2111.03026 (2021).

Response to Reviewer #sofd

We sincerely appreciate your thoughtful feedback and constructive comments on our paper. Below, we address each of your concerns in detail. All our responses will be incorporated into the final paper.

1. Does gamma really play an adaptive role?

Thank you for the comments. Currently, $\gamma$ is fixed as a (conservative) estimate of the batch error rate, but our method could be extended to adaptively adjust $\gamma$ if real-time error estimation is available, which we leave for future work.

2. It would be better to compare it with more powerful baselines

Thank you for the comments. In this rebuttal, following the reviewer’s and all other reviewers’ comments, we have added the comparison with some other robust reward learning methods. Please refer to the comparison results in our response of question 1 to reviewer #e9yz.

3. Why are the experiments conducted in a custom environment from dmc? ...

Thank you for the comments. We chose to implement our approach in new benchmark environments because we need a GPU-accelerated simulation to accelerate MPPI policies with parallel sampling. All of our custom testing environments will be released to support future research and standardized benchmarking in this area.

Regarding practicality in other simulated environments, the proposed method is directly applicable. Environments without GPU support or parallel simulation may limit the speed of MPPI-based policies. In addition, as we pointed in Section 6.3, it is possible to use other policies (like RL) to generate trajectories for comparison.

As for real-world transferability, recent works such as [5] have demonstrated that sampling-based control policies with parallelized simulation backends like MJX can be effectively transferred to real-world robotic tasks.

4. Lack of discussion of other RL methods with noisy human feedback in related work and experiments.

Thank you for the comments. Both RIME [1] and CANDERE-COACH [2] improve learning from noisy labels by filtering human feedback—RIME uses KL-divergence to filter and flip corrupted labels, while CANDERE-COACH trains a neural classifier to predict preferences and filters based on discrepancies with real labels. Xue’s work [3] employs an encoder-decoder structure for reward models, estimating confidence levels from latent distributions and performing a weighted average for better predictions. Unlike these methods, ours does not explicitly assess feedback quality but instead updates the hypothesis space conservatively based on entire preference batches. Uni-RLHF [4] introduces an annotation platform and large-scale feedback dataset, using accuracy thresholds and manual verification, which are impractical for online reward learning due to the absence of ground truth and the high cost of manual inspection.

In the revised version, we will include the above discussion.

5. Have you tried using real human feedback?

Thank you for the comments. We conducted a new experiment evaluating HSBC with real human feedback on CartPole and Walker. Four volunteers provided trajectory preferences, which were used for reward learning. HSBC ($\gamma=0.4$) was compared to PEBBLE under the same settings as in the main paper, with a small amount of simulated feedback for pretraining. The results (sum of reward), presented below, show performance across different numbers of human preferences, with the first table for CartPole and the second for Walker.

It can be shown that our method can achieve more stable convergence and superior performance when handling human feedback.

Reference

[1]Cheng, et al. "Rime: Robust preference-based reinforcement learning with noisy preferences." arXiv preprint arXiv:2402.17257 (2024).

[2]Li Y, et al. CANDERE-COACH: Reinforcement Learning from Noisy Feedback[J]. arXiv preprint arXiv:2409.15521, 2024.

[3]Xue W, et al. Reinforcement learning from diverse human preferences[J]. arXiv preprint arXiv:2301.11774, 2023.

[4]Yuan Y, et al. Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback[J]. arXiv preprint arXiv:2402.02423, 2024.

[5]Li, et al. "Drop: Dexterous reorientation via online planning." arXiv preprint arXiv:2409.14562 (2024).