We thank the reviewer for their thoughtful comments. We would like to address all of your concerns and questions below with point responses:

---

Practicality: “I find it hard to believe that such a simple continuous feedback model can be applied to real-world scenarios. For example, the paper states in line 38 that "Current research has demonstrated success primarily in simple, low-dimensional tasks with limited solution spaces." However, the experiments conducted in the paper also involve environments where baseline algorithms like DDPG or SAC can converge with a good reward function after only about 10 minutes of training.”

We respectively disagree with the reviewer on this observation. Fig.5 shows the opposite observation where baseline algorithms did not converge within 10 minutes and showed no sign of convergence in 20 minutes in our new environments, Find Treasure and Hide-and-Seek. Therefore, these tasks remain significantly challenging for the current RL baselines. GUIDE, on the other hand, was able to improve over baselines by a large margin.

“GUIDE, which incurs a high cost of human feedback, does not outperform manually designed simple rewards (such as the distance to the target, I think it is not hard to design it). Therefore, despite the fact that the environments used do have continuous actions and image inputs, I believe these environments are not suitable for validating RLHF algorithms because the reward functions are easy to design and the tasks themselves are simple.”

As discussed in the paper, hand-design dense rewards typically do not exist in real-world scenarios and require extensive experience in reward engineering and a practical understanding of RL training. However, we aim to enable human guidance in RL for a broader audience who most likely do not have RL experience or reward engineering. Moreover, dense reward designs, such as precise distance to the target, as the reviewer suggested, assume extra information that is not available to GUIDE, whom only perceives partial visual observations. Therefore, our environments are strong and challenging testbeds for developing human-guided RL algorithms. Furthermore, as prior works have only focused on simple bowling games and low-dimensional observations, we believe that our environment is a large step forward in investigating the potential and scalability of human-guided RL.

“The core argument of the paper is that continuous real-time feedback is extremely difficult to implement in practice. It requires annotators to constantly provide scalar rewards without pause, and such absolute value annotations are more susceptible to biases from different individuals. Pair-wise annotation is much easier than absolute value annotation and can be conducted asynchronously with the training process. If an AI agent needs to be trained for several days, the cost will be unacceptable.”

Our argument is not that continuous real-time feedback is extremely difficult to implement in practice. Instead, we point out that it provides richer feedback and assignments to every state-action pair than previous discrete feedback without a high cognitive load. Our contribution is to enable grounding such novel feedback modality through GUIDE.

As discussed in our paper, while pair-wise annotation is a popular choice in RLHF, it requires parallel policy rollouts in an asynchronous manner or offline trajectory collections. Our setting is different – we focus on real-time decision-making tasks where no such asynchronous rollouts, offline trajectories, or a simulator that can be run multiple times with policy rollouts and resetting are available. Due to the significant difference in the setting, pair-wise annotation is out of scope for our study.

“Although the paper suggests using model predictions to synthesize feedback, such a simple supervised learning regression objective is unlikely to accurately model the complex reward distribution. My reasoning is that predicting the relative goodness of A and B is easier than predicting scalar reward values, but there will still be many prediction biases.”

As discussed in our paper, while pair-wise annotation is a popular choice in RLHF, such as recent LLM studies, it requires parallel policy rollouts in an asynchronous manner or offline trajectory collections. Our setting is different – we focus on real-time decision-making tasks where no such asynchronous rollouts, offline trajectories, or a simulator that can be run multiple times with policy rollouts and resetting are available. Due to the significant difference in the setting, pair-wise annotation is out of scope for our study.

We agree that more advancements can be made to improve the simulated feedback model. However, prior to our work, there have not been studies enabling continual training in real-time human-guided RL. Our work is the first step towards this with potential future work to improve the optimization designs to handle more complex reward distributions.

“What is the meaning of A(s, a) in Equation 1? Also, A(s) = a in the line 123, is it the same? Difference of Q(s, a) and q? How to get the r^hf?”

Thank you for catching the typos. We will correct both $A(s, a)$ and $q(s, a)$ to $Q(s, a)$. $r^{hf}$ is the human feedback reward provided by the human.

“There is a lack of discussion on recent related works in RLHF, such as: …”

We thank the reviewer for pointing out relevant literature, and we will include these in our revised version.

Can you provide more details about the annotator's annotations, such as the actual interface and the frequency of operations?

Yes. A screenshot of our interface is shown in Fig3 (B). The user hovers their mouse over the window to provide feedback. Moving upwards indicates stronger positive feedback, and downwards indicates stronger negative feedback. The decision frequency of the games is set to 0.5s/step. Hence, human feedback values are collected every 0.5 seconds.

Dear Reviewer,

Thank you again for your detailed review of our paper. We aim to try our best to address all your concerns with our point responses above.

We truly value your feedback. As the end of the rebuttal period is approaching, please feel free to let us know if you have any additional questions or comments. We would be happy to answer them. We look forward to hearing your future thoughts!

Best regards,

Authors

We thank the reviewer for their thoughtful comments. We would like to address all of your concerns and questions below with point responses:

---

“There are three major unstated assumptions: The delay between an event appearing on the screen and the change in human feedback is constant (Question 1). The authors tune this constant for each environment. But, more complex environments might induce different delays as the human observer might need to think about what they saw.” (Question1: “Is it reasonable to assume constant reaction time of the human observer?”)

We agree that human response time can have various delays. In GUIDE, the advantage of using continual feedback is that it can partially alleviate this issue by eliminating the need for time window association for feedback to state-action pairs. Such a time window is required in previous discrete feedback designs. However, modeling more fine-grained response delay can potentially further improve GUIDE. We leave the exploration of this aspect in future work.

“People are able to provide constant feedback (Question 2). This might not be true for more complex environments where certain states might have ambiguous values.”

Our assumption is that humans have a preference for any given state-action pair, though the strength of the preference may vary. Continuous-valued feedback can model the strength of preference, e.g., provide a feedback value with a smaller magnitude to reflect weaker preferences. Our experiments suggest that our method surpasses past work on discrete feedback. One future research direction could be empowering the model to learn adaptive changes in human feedback over the course of trajectories or use multiple feedback modalities. We believe our framework can be a strong starting point for incorporating these improvements. We leave such exploration as future work.

(Question 2: “How would your method deal with games with more ambiguous state values, such as Montezuma's Revenge?”)

Our method of incorporating human feedback can also encourage exploration behaviors which is critical for challenging tasks such as Montezuma’s Revenge. We have observed a similar conclusion in Find Treasure, where human feedback can help explore the environment while the target object is not yet available to the agent, as in Fig. 7. Our method is orthogonal to approaches that aim to reduce the ambiguity of feedback specifications. Exploring methods to reduce such ambiguity can be an interesting future direction.

“The human feedback is Markov (Question 3). This might not be true in more complex games.” (Question3: “The human feedback regressor $\bar{H}(s, a)$ uses the Markov assumption, but the human feedback might be dependent on their memory of prior states. What if the human feedback is non-Markov?”)

We agree that human feedback may not always be Markov for any task. As we mentioned in Section 5.4, for Find Treasure and Hide-and-Seek, we stacked three consecutive frames as input to both the RL backbone and the feedback regressor. We found this to be sufficient to model human feedback. For more complex tasks, it will be interesting to explore whether incorporating more prior history steps or using a memory-based neural network will improve human feedback modeling.

“Minor typo and wording issues.”

We thank the reviewer for pointing out some typos and wording issues. We will improve these in the revised version: 1) change A to Q in Equation 1; 2) use upper-case Q in Equation 2; 3) Define $R_{t+k+1}$; 4) list specific designs in line 125 and be specific for lines 125-127; 5) fix spacing in citations; 6) rename computational framework.

Dear Reviewer,

Thank you again for your detailed review of our paper. We aim to try our best to address all your concerns with our point responses above.

We truly value your feedback. As the end of the rebuttal period is approaching, please feel free to let us know if you have any additional questions or comments. We would be happy to answer them. We look forward to hearing your future thoughts!

Best regards,

Authors

We thank the reviewer for their thoughtful comments. We would like to address all of your concerns and questions below with point responses:

---

“The assumptions regarding what human feedback represents do not seem consistent between section 3 and 4 (see Questions). ”

We would like to clarify that c-DeepTamer and GUIDE use human feedback differently. c-DeepTamer, described in sec 3.2, is a baseline algorithm. It is an upgraded version of the original Deep Tamer[1], and follows the original Deep Tamer formulation to use human feedback as the myopic state-action value function. We did not modify how Deep Tamer uses human reward as a fair comparison. In GUIDE, we instead use human feedback as an additional immediate reward to the sparse environment reward.

“Further, the treatment of the feedback collection is rather simple (added to environment reward function) and, especially if it does represent a signal regarding the future value of state-action pairs, heuristic. Relative to Tamer and Deep Tamer, which treated human feedback more consistently by using it directly as a proper state-action value function, this paper feels like a regression on that front.”

Tamer and Deep Tamer can treat human feedback directly as a state-action value function because they ignore the sparse environment feedback. However, we believe that it is still useful to leverage the sparse environment feedback for RL training. In GUIDE, Our design choice to use human feedback as an additional reward function is based on this intuition to integrate both types of feedback together.

“In 3.2 you propose to extend Deep Tamer by using the human feedback estimator F(s,a) as a critic within an actor-critic framework. Thus, you treat human feedback here as an estimate of the future value of each state-action pair (a Q-value). In 4.3, you propose to use the human feedback signal as a reward, to be added to the environment reward. Does this mean you make a different assumption regarding human feedback in 4.3. than in 3.2? Or do you propose to use feedback on future value as a reward?”

Yes, we use human feedback differently in c-Deep Tamer and GUIDE. In Deep Tamer, human feedback is treated as a myopic state-action value. In GUIDE, we treat human feedback as a reward function, which we believe is a more appropriate and effective approximation, as evidenced by our experimental results. Moreover, this design choice makes it more natural to incorporate sparse environment rewards with the human reward in practice.

“I am confused by your claim that Deep Tamer uses DQN. Looking at algorithm 1 in the Deep Tamer paper, it’s policy greedily chooses the action that maximizes the learnt human value function H(s,a). I do not see any use of DQN.”

Deep Tamer can be seen as DQN with $\gamma=0$, where the target state-value function at the current time step is directly assigned by humans. We realize that this phrase may cause confusion. We will remove the DQN claim in the revised paper and instead describe as Deep Tamer learns a reward model to regress human feedback, which is then used for greedy action selection.

“Did your c-DeepTamer baseline receive the same (sparse) environment reward as GUIDE? And if so, how did you use this reward signal in conjunction with a critic that is trained from human feedback only?”

c-DeepTamer only upgrades Deep Tamer to handle continuous actions, with extensive recent neural network designs and hyperparameter tuning. Following Deep Tamer, it does not use an environment reward. As stated above, Deep Tamer treats human feedback as a myopic state-action value, which is not really a reward, making it impractical to add an environment reward to it. Changing the reward setup in Deep Tamer will make it unfair to compare. One contribution of GUIDE is to provide a formulation that uses both human reward and sparse environment reward.

“Why was the training of c-DeepTamer stopped after 5/10 minutes? Would it not be possible to continue training it using simulated human feedback (similar to GUIDE) or without human feedback?”

We strictly follow the original Deep Tamer implementation which does not handle continual training. In the original Deep Tamer, the model weights are only updated every time when a new human feedback signal is received (see Algorithm 1, line 5 in the Deep Tamer paper[1]). When there is no human feedback available, Deep Tamer will not be improved.

Both c-Deep Tamer and GUIDE are trained with the same amount of human guidance time. In fact, one contribution of GUIDE is to provide a mechanism to simultaneously train a simulated feedback model for continual training once human guidance is not available. We would like to clarify that this is our contribution in GUIDE instead of an inheritance from Deep Tamer.

“Within the user study, did participants get time to familiarize themselves with each environment before data recording began? I imagine participants would have needed some time to figure out how to solve each task.”

Yes. Before each experiment, we showed the participants a short video introducing the goal of each game and a quick demonstration of how each game is played and how the feedback interface can be used.

“Is equation (3) correct? What is the purpose of the additional -F(s, A(s)) term?”

Mathematically, the equation (3) is correct. The additional term -F(s, A(s)) is the actor loss, which tries to maximize the feedback value predicted by the critic network, given an action selected by the actor A(s). We recognize that the presentation may cause confusion since the actor and critic loss only affect their corresponding networks separately. We will clarify this by separating these two terms and indicate model updates in our revised paper.

---

references

[1] Warnell, Garrett, et al. "Deep tamer: Interactive agent shaping in high-dimensional state spaces." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.

Dear Reviewer,

Thank you again for your detailed review of our paper. We aim to try our best to address all your concerns with our point responses above.

We truly value your feedback. As the end of the rebuttal period is approaching, please feel free to let us know if you have any additional questions or comments. We would be happy to answer them. We look forward to hearing your future thoughts!

Best regards,

Authors

Thank you for your detailed response. Unfortunately I continue to see two weaknesses in the paper.

I like the proposed mechanism for giving feedback. It is elegant and to my knowledge novel. Reviewer JtyH has a point by saying that complex RL tasks will need more significant amounts of feedback, but in my view any interactive reward learning method would. This could have been explored more in the paper. The idea of combining environment reward with human feedback, however, cannot be counted as a novel contribution considering earlier work in this area [A,B]. Overall, the technical contribution is thus limited.

A very solid set of human subject experiments would have been able to compensate for this, but in my view the experiments presented at this point are not quite complete enough. The c-DeepTamer baseline is artificially limited. My best understanding is that each bit of feedback was used only once in a gradient update, contrary to standard ML practice (please correct me if I'm wrong). Further, c-DeepTamer is not able to make use of the environment's reward, raising questions about whether it is a fair baseline in the first place. Why not use a baseline like the [A,B] that could use the environment's reward? The paper could also have been made much stronger by comparing to alternative interactive methods for learning from humans, including learning from preferences [C] and learning from demonstrations (e.g. [B]).

I appreciate that the authors have clarified some of my concerns, and will raise my score accordingly, but I still feel that the paper falls short of the acceptance threshold.

[A] Xiao, Baicen, et al. "FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback." Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems. 2020.
[B] Brys, Tim, et al. "Reinforcement learning from demonstration through shaping." Proceedings of the 24th International Conference on Artificial Intelligence. 2015.
[C] Christiano, Paul F., et al. "Deep reinforcement learning from human preferences." Advances in neural information processing systems 30 (2017).

We thank the reviewer for their thoughtful comments. We are glad that the reviewer found our method to be novel, the scale of our human study to be “a great contribution,” and our human analysis to be insightful. We would like to address all of your concerns and questions below with point responses:

---

“The baselines are generally quite weak. Based on the experiment results, it is unclear whether real-time continuous feedback is necessarily the best way for humans to guide the policy learning. There might be intermediate points on the spectrum of conventional discrete feedback and full continuous feedback that provides the best tradeoff between amount of human input and effectiveness of guiding policy learning.”

We agree that it will be interesting to explore the “optimal” feedback mode. Based on feedback from our human participants, the continuous feedback mode was not cognitively demanding, hence a full continuous feedback that enables maximum human input is a straightforward and effective choice. We will leave the exploration of blending discrete and continuous feedback as future work.

“Whether the simulated human feedback is helpful is unclear. In both the "bowling" the "find treasure" task, the score does not increase much after switching to simulated feedback. It might be the case that the simulated human feedback only works for tasks where it is straightforward to model the reward.”

We agree that the effectiveness of simulated human feedback will vary depending on the complexity of the environment and human feedback. The main purpose of the simulated feedback is to allow continual training without the human trainer and minimize the shift in reward distribution, which is a novel algorithm design. We will leave this interesting investigation of the relationship between task complexity and the effectiveness of simulated feedback as future work.

“How does human-in-the-loop compare to BC or reward engineering for the same amount of human input?”

It is unclear how to quantify the amount of human input for reward engineering. One advantage of GUIDE is that we do not require human subjects to have prior knowledge of RL. However, reward engineering typically requires domain expert knowledge and experience in designing rewards for tuning RL policies. On the other hand, BC typically assumes expert demonstrations, which also demand more cognitive load from the subjects and expert-level demonstrations. However, for complex tasks, humans may not easily provide high-quality demonstrations while they can still provide an assessment of the agent’s decisions. Therefore, we believe that BC and reward engineering are not suitable comparisons in our case.

“How are the three tasks for evaluation chosen, and what are the motivations for such choices?”

The bowling task is a classic environment commonly used in prior literature [1, 2, 3]. We included it to compare it with Deep Tamer. Find Treasure is a navigation task motivated by potential real-world applications of human-guided RL, such as search and rescue and disaster response. It also serves as an intermediate task before hide-and-seek, where the target does not move. Hide-and-seek is selected as a representative task involving adversarial competition. Most existing tasks in literature are discrete action tasks with low-dimensional states. We believe that our task selection is a large step forward in investigating the scalability of human-guided RL.

“One motivation for doing human-in-the-loop RL is for tasks where designing a reward function or providing demonstrations are difficult, such as the backflip task. However, for the tasks examined in this paper, there seems to be other simple ways for the RL agent to learn from human input.”

Despite reward engineering that could exist for these tasks, our setting is different. Designing rewards requires expert knowledge and RL training experience. However, our setting does not assume such expert experience in our human subjects. Moreover, designing dense rewards for these tasks requires extra information that is not available to our agents, such as the position of the agent, the target object location (our environment only provides partial visual inputs where target objects may not be visible), and the location of the adversarial agent (our environment only provides partial visual inputs where the other adversarial agent are not always visible). Therefore, our setting does fulfill the suggestions from the reviewer where the reward function is difficult to design without such extra experience or extra exposed information.

“The paper focused on the final success rate of policy learning, but did not provide sufficient data from the user's perspective. For example, the user study failed to include a survey regarding whether the real-time feedback system feels easier to use than the discrete feedback system.”

We would like to clarify that our work also quantifies individual differences and their impact on guided AI performance, as in our cognitive studies in Fig. 5 and 7. Surveys about user experience can be subjective. One possible solution is to quantify physiological signals to measure their cognitive load and stress level, combined with surveys. We leave such exploration in future work. Though we did not have a formal survey for this question, verbal feedback from our subjects suggests that most of them did not find the continuous feedback interface cognitively demanding.

---

references

[1] Warnell, Garrett, et al. "Deep tamer: Interactive agent shaping in high-dimensional state spaces." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.

[2] Park, Sung-Yun, Seung-Jin Hong, and Sang-Kwang Lee. "Human Interactive Learning with Intrinsic Reward." 2023 IEEE Conference on Games (CoG). IEEE, 2023.

[3] Xiao, Baicen, et al. "Fresh: Interactive reward shaping in high-dimensional state spaces using human feedback." arXiv preprint arXiv:2001.06781 (2020).

Dear Reviewer,

Thank you again for your detailed review of our paper. We aim to try our best to address all your concerns with our point responses above.

We truly value your feedback. As the end of the rebuttal period is approaching, please feel free to let us know if you have any additional questions or comments. We would be happy to answer them. We look forward to hearing your future thoughts!

Best regards,

Authors