Enhancing Human Experience in Human-Agent Collaboration: A Human-Centered Modeling Approach Based on Positive Human Gain

Thank you for carefully reviewing our paper! We greatly appreciate your recognition and positive evaluation of our work. We have carefully revised the manuscript according to your suggestions and have rephrased the parts that caused confusion. If you have any other questions or comments, we are more than happy to discuss them further.

Q1: Regarding the technical presentation.

A1: We have carefully revised the notations in the manuscript and rewritten Sec. 2.2, Sec. 3.1 and Sec. 3.2 to make the technical presentation clearer. All modifications we marked in red in the rebuttal version. We briefly clarify your confusion below.

Our work aims to train agents to enhance the human experience while maintaining the agent's original abilities as much as possible. Therefore, our approach is essentially optimizing both $R$ and $R^H$, i.e., $J(\theta) = E_{\pi_{\theta},\pi_H} [G_t + \alpha \cdot G_t^H|s_t=s]$, corresponding to the self-centered optimization objective and the human-centered optimization objective, respectively, with the two objectives weighted by $\alpha$.

During the optimization process, we found that the human-centered advantage $A_H(s, a) = E_{\pi_\theta, \pi_H}[G^H_t|s_t=s,a_t=a] - V_H^{\pi_{\theta}, \pi_H}(s)$ in the policy gradient approximation Eq.3 may result in human-agent credit assignment issues. For instance, rewarding agents for assistance when humans can effortlessly achieve their goals is unnecessary, as this could potentially lead to agents learning incorrect behavior and even losing their autonomy (see experimental results in Sec.4.2).

Our key insight to address this challenge is that the human contribution, termed human primitive value, i.e., $V_H^{\pi_0, \pi_H}(s)$, should be distinguished from human rewards. $V_H^{\pi_0, \pi_H}(s)$ represents the expected human return for a given state $s$ under the setting of collaboration between the human $\pi_H$ and the pre-trained agent $\pi_0$. The residual benefits, representing the agent's actual contribution in enhancing human to achieve goals, are referred to as human gains, i.e., $\Delta(s, a)=E_{\pi_\theta, \pi_H}[G^H_t|s_t=s,a_t=a] - V_H^{\pi_0, \pi_H}(s)$. The agent is solely encouraged to learn effective enhancement behaviors, i.e., $A^{+}(s) =${$a|a \sim \pi_\theta, \Delta(s, a) > 0$}. We propose the RLHG algorithm to actualize our insight. The RLHG algorithm initially learns a human primitive value network $V_{\phi}$ in the Human Primitive Value Estimation stage. In the subsequent Human Enhancement Training stage, the RLHG algorithm encourages the agent to learn effective enhancement behaviors (actions contributing to positive human gains). For those invalid enhancement behaviors, i.e., $A^{-}(s)=A(s) \setminus A^{+}(s) =${$a|a \sim \pi_\theta, \Delta(s, a) \le 0$}, the human-centered advantage in Eq.4 is set to 0 and only the self-centered advantage exists to guide the agent's optimization.

Q2: Regarding other presentations.

"Specify the domain and range of the value functions in question".

"Always condition value functions on their inputs $V(s)$ instead of $V$".

"Clearly distinguishing in the definitions between the AI actions and those taken by the human".

A2: Thank you for your suggestions, we have carefully revised the manuscript according to your suggestions in the rebuttal revision.

Q3: Regarding the description of the HRE.

A3: HRE (Human Reward Enhancement) essentially integrates the human reward directly into the policy gradient, which is Eq. 3. Based on your suggestion, we marked the gradient update of HRE when introducing the HRE agent in the experiment part, and we also supplemented the detailed optimization process of WuKong and HRE in Appendix C.1.

Q4: "Were an experiments conducted using RLHG with the task reward and human reward being equivalent (solely optimizing human agency)?"

A4: We conducted experiments to examine the influence of the balance parameter $\alpha$ ($\alpha$ is set to 0, 1, 2, and 4). Please see Appendix C.2 for detailed experimental results.

Q5: "Do the HRE experiments correspond to maximizing the $\alpha$-weighted mixture of human and task reward?"

A5: Yes, the optimization objective of HRE is the same as RLHG, i.e., $J(\theta) = E_{\pi_{\theta},\pi_H} [G_t + \alpha \cdot G_t^H|s_t=s]$. And the difference between them is the policy gradient update (HRE: Eq.3, RLHG: Eq.4).

Q6: "Was the human-centric reward a pre-existing score in the game environment, or was it derived based on the survey of player preferences?"

A6: Human goals were obtained through participant survey statistics (Figure 4(c)). These human goals can be quantified as human rewards through pre-existing scores in-game, such as MVP score, highlight moment times, etc.

Thank you for carefully reviewing our paper! We have responded to your concerns and made revisions to the manuscript based on your valuable suggestions. If you have any further questions or comments, we would be more than happy to discuss them.

Q1: Clarification on the novelty.

A1: Thank you for your suggestions. We have rewritten Sec. 2.2, Sec. 3.1 and Sec. 3.2 to make the technical presentation clearer and moved a concise version of the algorithm to Sec.3.3 in the rebuttal revision, making it easier for readers to follow our method. Our contributions are not limited to network-level modifications in practical implementation, but also include:

• We proposed a human-centered modeling scheme for guiding human-level agents to enhance the experience of their human partners in collaborative tasks.
• We address the human-agent credit assignment challenge by presenting the RLHG algorithm, along with a detailed implementation framework.
• We conducted human-agent tests in Honor of Kings, and both objective performance and subjective preference results show that the RLHG agent provides participants better gaming experience.

Thank you for carefully reviewing our paper! We greatly appreciate your recognition of the contribution of our work and your positive review. We have responded to your presentation concerns and carefully revised the manuscript. If you have any further questions or comments, we will be happy to discuss them further.

Q1: Questions about the notations.

A1: We have carefully revised the notations in the manuscript and rewritten Sec. 2.2, Sec. 3.1 and Sec. 3.2 to make the technical presentation clearer. All modifications we marked in red in the rebuttal version. Below we provide clarification on the notation-related questions you raised.

Q1.1：Regarding "$V^{\pi_{\theta}}(s)$ in the 2nd paragraph of Sec.2.2".

A1.1：In the 2nd paragraph of Sec 2.2, we initially stated, "In agent-only team setting", which does not include humans. In this setting, the optimization objective of the agent is to maximize the cumulative task reward for a given state $s$. To eliminate possible ambiguities, we have rephrased this paragraph in the rebuttal revision as follows: "Most previous research work focuses on learning self-centered agents. In agent-only team settings, the agent is optimized to complete the task, with the optimization objective being to maximize the value function for a given state $s$, i.e., $V^{\pi_{\theta}}(s) = E_{\pi_{\theta}}[G_t|s_t=s]$, where $G_t=\sum_{k=0}^{\infty}\gamma^{k}R_{t+k+1}$ represents the discounted cumulative task rewards."

Q1.2：Regarding "the total return Gt should depend on the robot policy and the human policy".

A1.2: Thank you for pointing this out, we have revised all equations in the manuscript by replacing $E_{\pi_H}$ with $E_{\pi_{\theta},\pi_H}$.

Q1.3：Regarding "how to get the last inequality in page number 4".

A1.3：We have rewritten Sec. 3.2 to make our insights more readable and understandable. After modification, we removed the complex notation representation as well as the inequality.

Q2: Clarification about the human policy.

A2: We have rephrased the 2nd paragraph of Sec 2.2 and supplemented the description of the human policy to make it clear. "We define a human policy $\pi_H$, to be a mapping from observations $o^H \in \mathcal{O}^H$ to actions $a^H \in \mathcal{A}^H$, which remains fixed during the agent's optimization process and cannot be accessible to the agent."