Learning to Assist Humans without Inferring Rewards

Dear Reviewer,

Thank you for the detailed review, and for all the suggestions for improving the paper. Based on the reviewer's feedback, we have run experiments with additional baselines and clarified several parts of the writing. Together with the discussion below, does this fully address the reviewer's concerns with the paper? We look forward to continuing the discussion!

Kind regards,

The authors

It isn't clear why inferring these approximate representations and seeking to maximize empowerment would work better than reward inference. it would be beneficial to test against a framework that actively infers the user's reward function.

To answer this question, we ran an additional experiment comparing our method (which estimates representations) to a reward learning approach that uses IQ-Learn (Garg et al., 2021) as an IRL algorithm to infer human rewards and then use these to train the assistant, as in the CiRL (Hadfield-Menell et al., 2016) framework. The results (rebuttal PDF Figs. 1 and 3) show that our approach outperforms this baseline.

Is this only true for a two-agent scenario and full observability?

Yes. We have revised the paper to add these assumptions. We believe that handling more than two agents should be feasible (in future work) but handling partial observability may be more challenging. Under certain assumptions (e.g., state-independent noise in observing human actions or bounded divergence between the agent’s belief and the true environment) we can reason about these challenges. But without any assumptions pathological MDPs can be constructed.

Does the proof have any assumptions for the robot and human maintaining the same representation and action space?

No, they do not need to have the same representations or action space. Our implementation re-uses one representation ($\psi(g)$) purely for computationally efficiency, but this can be removed if an algorithm user wanted to use entirely disjoint representations.

Clarify the meaning of high-dimensional

Our states for both environments are images: 5x5 with 3 feature channels for the obstacle gridworld and 4x5/5x5 with 26 feature channels for the Overcooked environments.

Why does AvE perform so poorly on these experiments?

The key limitation of AvE is it tries to approximate the empowerment quantity which they define as $\max_\pi I(s^+;a_t\mid s_t)$ through random rollouts. This is completely intractable in high-dimensional environments (as our larger experiments show), where random rollouts don’t produce meaningful behaviors.

Additionally, since the rollouts select actions uniformly at random, even in the limit of infinite computation, the AvE objective doesn’t actually compute empowerment with respect to meaningful human behaviors (either under their definition or our definition). Intuitively, imagine a human performing some task in a room with thin walls. The AvE objective would incentivize the robot knocking down the walls to maximize the human’s ability to leave the house, even though they have no desire to do so. Our ESR objective would help them with the task at hand.

why ESR was able to outperform AvE by such a wide margin and note any qualitative findings about the learned collaborative behavior?

For the reasons noted above, AvE is unable to perform well on the more complex tasks studied, while ESR can scalably empower the human. Qualitatively, in the obstacle environment ESR learns to remove obstacles along the path the human is moving and in the Overcooked setting it learns collaborative behaviors like moving the plate to places the human can reach and placing onions in the pot. We will revise the paper with sketches of some of these behaviors.

Could you comment on the feasibility of this approach with actual humans?

This paper is primarily mathematical and algorithmic and nature, and so we'll provide an mathematical/algorithmic answer: Eq. (6) says that we can measure the degree of (minimax) compatibility via the mutual information. We have run an additional experiment to plot this mutual information throughout training (see Fix X in rebuttal PDF). Visualizing the learned agent, we see that the agent does indeed become more helpful as this mutual information increases.

Of course, the gold standard in human-AI interaction is human user studies, which are beyond the scope of this mathematical/algorithmic paper.

checking to ensure that all variables are defined, which will help improve clarity.

Thanks for the suggestion – we have done this with our local working copy of the paper.

Figure Descriptions could be improved

We have clarified the figure descriptions, incorporating the feedback from the other reviewers as well.

Du, Yuqing, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. 2020. “AvE: Assistance via Empowerment.” Pp. 4560–71 in Advances in Neural Information Processing Systems. Vol. 33. Curran Associates, Inc.

Garg, Divyansh, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. 2021. “IQ-Learn: Inverse Soft-Q Learning for Imitation.” Pp. 4028–39 in Advances in Neural Information Processing Systems. Vol. 34. Curran Associates, Inc.

Hadfield-Menell, Dylan, Stuart J. Russell, Pieter Abbeel, and Anca Dragan. 2016. “Cooperative Inverse Reinforcement Learning.” Advances in Neural Information Processing Systems 29.

Thank you for your response! We have revised the paper to mention this limitation in the conclusion. We would like to note that numerous papers published at NeurIPS and similar venues also aim to establish the algorithmic and theoretical foundations for human-AI interaction and alignment before proceeding with human studies (Ammanabrolu et al., 2022; Chan et al., 2019; Hadfield-Menell et al., 2016, 2017; He et al., 2023; Ngo et al., 2024; A. Pan et al., 2022; M. Pan et al., 2024; Zhuang & Hadfield-Menell, 2020). We will adjust the introduction to indicate that our paper extends this line of work, providing:

1. an algorithmic and theoretical framework for creating aligned AI agents without the machinery of inferring human values (such as in the CIRL framework of Hadfield-Menell et al., (2016, 2017)), and
2. a proof-of-concept showing scalable contrastive estimators enable improved unsupervised assistance in synthetic benchmarks from past work.

References

Ammanabrolu, Prithviraj, Liwei Jiang, Maarten Sap, Hannaneh Hajishirzi, and Yejin Choi. 2022. “Aligning to Social Norms and Values in Interactive Narratives.” in NAACL-HLT. arXiv.

Chan, Lawrence, Dylan Hadfield-Menell, Siddhartha Srinivasa, and Anca Dragan. 2019. “The Assistive Multi-Armed Bandit.” in ACM/IEEE International Conference on Human-Robot Interaction. arXiv.

Hadfield-Menell, Dylan, Smitha Milli, Pieter Abbeel, Stuart J. Russell, and Anca Dragan. 2017. “Inverse Reward Design.” Advances in Neural Information Processing Systems 30.

Hadfield-Menell, Dylan, Stuart J. Russell, Pieter Abbeel, and Anca Dragan. 2016. “Cooperative Inverse Reinforcement Learning.” Advances in Neural Information Processing Systems 29.

He, Jerry Zhi-Yang, Daniel S. Brown, Zackory Erickson, and Anca Dragan. 2023. “Quantifying Assistive Robustness via the Natural-Adversarial Frontier.” Pp. 1865–86 in Proceedings of The 7th Conference on Robot Learning. PMLR.

Ngo, Richard, Lawrence Chan, and Sören Mindermann. 2024. “The Alignment Problem from a Deep Learning Perspective.” in The twelfth international conference on learning representations.

Pan, Alexander, Kush Bhatia, and Jacob Steinhardt. 2022. “The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models.” in International Conference on Learning Representations. arXiv.

Pan, Michelle, Mariah Schrum, Vivek Myers, Erdem Bıyık, and Anca Dragan. 2024. “Coprocessor Actor Critic: A Model-Based Reinforcement Learning Approach For Adaptive Brain Stimulation.” in International Conference on Machine Learning.

Zhuang, Simon, and Dylan Hadfield-Menell. 2020. “Consequences of Misaligned AI.” Pp. 15763–73 in Advances in Neural Information Processing Systems. Vol. 33. Curran Associates, Inc.

Dear Reviewer,

Thanks for the review, and the suggestions for improving the paper. As suggested by the reviewer, we have evaluated the proposed method on a number of additional tasks, and compared with a number of additional baselines. Together with the responses below, does this fully address the reviewer's concerns? About the paper?

Kind regards,

The authors.

Evaluating on additional Overcooked environments

To expand our experimental results, we have run an additional experiment on the “asymmetric advantage” layout in Overcooked (see Table 1 in the attached pdf). We initially excluded this setting due to a lack of good models for human collaboration to evaluate on: policies that imitate expert data and/or use deep RL with self-play struggle to robustly learn collaborative behavior without additional structure. Our new result shows ESR can outperform baselines when playing with a heuristic-planning human model based on Carroll et al. (2019).

While the method addresses the problem of coordination without knowledge of the human's goal, the experiments demonstrating this are contrived. In Overcooked, the goal is fixed per task setting and is serving a dish.

We will add discussion of more complex settings that would be enabled by our method to the future work section. Examples of realistic settings where empowerment objectives could be effective include copilot-style assistants (Dohmke, 2022) and biomedical devices that improve human agency (Bryan et al., 2023; Pan et al., 2024). We note that the Overcooked environment has been used as a primary evaluation for human-AI collaboration in numerous works (Carroll et al., 2019; Hong et al., 2023; Knott et al., 2021; Laidlaw & Dragan, 2022; Lauffer et al., 2023; Strouse et al., 2021).

The paper does not show results in a setting where the goals and preferences of the human are complex to communicate.

This is a good point, the true reward of the Overcooked setting is easily communicated. However, our focus is on learning a collaborative policy without explicitly communicating the true reward, or the method of collaboration, which is more difficult to express.

The description of AvE lacks sufficient detail given it is the only baseline and addresses the same problem. Why does AvE struggle in more complex environments? How does it differ from ESR?

The key limitation of AvE is it tries to approximate the empowerment quantity which they define as $\max_\pi I(s^+;a_t\mid s_t)$ through random rollouts. This is completely intractable in high-dimensional environments (as our larger experiments show), where random rollouts don’t produce meaningful behaviors.

Additionally, since the rollouts select actions uniformly at random, even in the limit of infinite computation, the AvE objective doesn’t actually compute empowerment with respect to meaningful human behaviors (either under their definition or our definition). Intuitively, imagine a human performing some task in a room with thin walls. The AvE objective would incentivize the robot knocking down the walls to maximize the human’s ability to leave the house, even though they have no desire to do so. Our ESR objective would help them with the task at hand.

The acronym ESR is not defined in the main text and is only defined in the caption of Algorithm 1.

We have fixed this.

L160 refers to "the CIRL setting", but CIRL is never defined. What does CIRL stand for?

CIRL refers to the cooperative inverse reinforcement learning setup from Hadfield-Menell et al. (2016), in which an assistant cooperates with a human without knowing the true human reward.

As stated in the limitations, ESR assumes access to the human actions to learn the empowerment reward. Can ESR work with noisy predictions of human actions?

In general, the ESR objective under an arbitrary (state-independent) noise model becomes a lower bound on the “true” ESR empowerment (applying the data processing inequality to the Markov chain $\hat{a}\to a\to s^+$). This suggests that ESR will be conservative in the presence of noisy action observations.

While ESR outperforms AvE, it greatly underperforms baselines from population-based training in [1].

Methods that assume access to the true reward (i.e., Figure 4 of the Overcooked paper (Carroll et al., 2019)) will in general perform better than methods such as ESR that do not.

Is the assumption that the agents share the same state space central to the empowerment learning objective? Is it possible to implement ESR if both agents operate from separate egocentric views?

While we do make this assumption in our formulation, in many cases it may be possible to use the same algorithm in partially observed settings. If the human has partial observability, this corresponds to maximizing empowerment of a suboptimal human (it throws away information about the state in its observation), which will often nevertheless be desirable. If the agent has partial observability, the empowerment objective will only increase the human’s influence over the parts of the environment the agent can observe. In many cases this may be fine, but in environments where the agent can restrict the information in its own observations, care must be taken to mitigate reward hacking (Amodei et al., 2016)—in this case, situations where the agent can choose to look at only the areas the human has influence over. We will add discussion on these points to the limitations and future work section.

Amodei, Dario, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. “Concrete Problems in AI Safety.”

Bryan, Matthew J., Linxing Preston Jiang, and Rajesh P N Rao. 2023. “Neural Co-Processors for Restoring Brain Function: Results from a Cortical Model of Grasping.” Journal of Neural Engineering 20(3).

Carroll, Micah, Rohin Shah, Mark K. Ho, Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca Dragan. 2019. “On the Utility of Learning about Humans for Human-AI Coordination.” in Conference on Neural Information Processing Systems. arXiv.

Dohmke, Thomas. 2022. “GitHub Copilot Is Generally Available to All Developers.” Retrieved July 25:2023.

Du, Yuqing, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. 2020. “AvE: Assistance via Empowerment.” Pp. 4560–71 in Advances in Neural Information Processing Systems. Vol. 33. Curran Associates, Inc.

Hadfield-Menell, Dylan, Stuart J. Russell, Pieter Abbeel, and Anca Dragan. 2016. “Cooperative Inverse Reinforcement Learning.” Advances in Neural Information Processing Systems 29.

Hong, Joey, Sergey Levine, and Anca Dragan. 2023. “Learning to Influence Human Behavior with Offline Reinforcement Learning.” in Conference on Neural Information Processing Systems. arXiv.

Knott, Paul, Micah Carroll, Sam Devlin, Kamil Ciosek, Katja Hofmann, A. D. Dragan, and Rohin Shah. 2021. “Evaluating the Robustness of Collaborative Agents.” in AAMAS. arXiv.

Laidlaw, Cassidy, and Anca Dragan. 2022. “The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models.” in International Conference on Learning Representations. arXiv.

Lauffer, Niklas, Ameesh Shah, Micah Carroll, Michael D. Dennis, and Stuart Russell. 2023. “Who Needs to Know? Minimal Knowledge for Optimal Coordination.” Pp. 18599–613 in Proceedings of the 40th International Conference on Machine Learning. PMLR.

Pan, Michelle, Mariah Schrum, Vivek Myers, Erdem Bıyık, and Anca Dragan. 2024. “Coprocessor Actor Critic: A Model-Based Reinforcement Learning Approach For Adaptive Brain Stimulation.” in International Conference on Machine Learning.

Strouse, Dj, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. 2021. “Collaborating with Humans without Human Data.” Pp. 14502–15 in Advances in Neural Information Processing Systems. Vol. 34. Curran Associates, Inc.

Dear Reviewer,

Thank you for the detailed review, and for the suggestions for improving the paper. It seems like the reviewer's main concern is about baselines and presentation. We have attempted to address these concerns by adding XX additional baselines, and by significantly revising the paper (incorporating the reviewer's suggestions). Together with the discussion below, does this fully address the reviewer's concerns? If not, we look forward to continuing the discussion!

Kind regards,

The authors.

How does ESR compare against goal inference baselines, e.g. those used in Du et al (2020) or Laidlaw et al (2024)?

We have added goal inference baselines for the obstacle environment, with our ESR method achieving the best performance (see attached PDF). In contrast with Du et al (2020), which assumes a model-based, almost “oracle” goal inference, we implement a model-free baseline for fair comparison. We have revised the paper to describe this.

In the Overcooked environments, there is no clear notion of a goal (the environment returns roughly to the same state it started in after the soup is made and delivered). An advantage of our method over goal inference is that it is well-defined in such settings.

Equation 1: What is the expectation in Eq (1) taken over? Is it over $\pi_H, \pi_R$ ? Is $s^{+}$sampled $K$ steps into the future starting from each $t$ ? Or starting from $t=0$?

This expectation is taken over a trajectory $(s_0, s_1, \cdots)$ sampled when the human ($\pi_H$) and robot ($\pi_R$) interact in the environment. Thus, the random variable $s_t$ corresponds to time step $t$ in an episode. Random variable $s^+$ is sampled from the discounted state occupancy measure conditioned on $s_t$; in other words, the sampling procedure can be written as $K \sim Geom(1 - \gamma)$ and setting $s^+ = s_{t+K}$. We have revised the paper to clarify this.

important quantities such as the skill variable z and state occupancy measure ρ(s) are not properly defined.

We will revise the paper to more clearly connect the notation used for the analysis to the main text.

how the mutual information I(s+;z) . between future states s+ and human skills z is related to the definition of empowerment in Equation (1), which involves the (conditional) mutual information I(s+;atH|st) between futurestates and human actions atH In section 4, where the ESR algorithm is introduced, it's also not explained how or why the contrastive representation learning objective in Eq. (7) should lead to right density ratios to be estimated upon training convergence. One has to read van den Oord (2019) to understand why, so this section is not really accessible to readers not already familiar with contrastive learning methods.

Thank you for pointing this out! We agree that it is difficult to understand the mathematical details without this background. In our revision, we will add a derivation of the solution to our symmetrized infoNCE objective in Appendix D, which follows directly from the analysis by Poole et al. (2019) when adapted to the symmetric version of the loss as in Radford et al. (2021) and Eysenbach et al. (2024).

why aR needs to be part of the successor representation at all. By including aR, won't the resulting mutual information you're estimating end up being I(s+;atH|st,atR), which is conditional on both the current state st and the assistant's action atR ?

We found conditioning on a_R as well helped stabilize training. A challenge with training an algorithm like ESR is that as $\pi_R$ changes during training, it affects the successor features and the computed empowerment reward. By conditioning the representation on the current policy’s actions, we improve the ability of the empowerment bonus to “keep up” with the policy during training. We will revise the main text to indicate this more clearly.

Lemma 1 and Lemma 2: What exactly is the skill $z$, and how is it related to human actions $a^H$ ?

A ``skill'' is purely a mathematical construction used in our analysis. The analysis focuses on the distribution over states visited by the human and the robot. Our analysis represents the robot's distribution over states $\rho(s)$ as a mixture, $\rho(s) = \sum_z \rho(s \miz s) p(z)$. We have revised the paper to clarify this.

Relatedly, how is the mutual information $I(s^+; z)$ related to $I(s^+; a_t^H \mid s_t)$?

All instances of random variable $z$ should be replaced by $a^H$; we apologize for any confusion caused by this typo, which was caused by different prior work using different notation. Our theoretical analysis looks at the mutual information $I(s^+; a_t^H)$, which measures the ability of the human to effect change in their environment. Our practical method says that we should maximize this objective at all states: $I(s^+; a_t^H \mid s_t)$ looks at the ability to effect change starting at state $s_t$, and our practical algorithm aims to maximize this objective over all visited states $s_t$ (see Eq. 1).

In Lemma 2, what does it mean for a human to "adapt to a reward function"? If I'm understanding Eysenbach et al (2021) correctly, is the idea here that the human is modeled as starting from prior state distribution $\rho(s)$, then adapting to $\rho *(s)$ by learning to better optimize for the reward function (by learning new skills)?

Yes, that is correct. In the context of this paper, we look at a restricted set of skills: those defined by (open-loop) sequences of actions, $a_t^H$. This is why we use $a_t^H$ in our mutual information objective, rather than the letter $z$ used in (Eysenbach et al 2021).

Carroll, Micah, Rohin Shah, Mark K. Ho, Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca Dragan. 2019. “On the Utility of Learning about Humans for Human-AI Coordination.” in Conference on Neural Information Processing Systems. arXiv.

Eysenbach, Benjamin, Vivek Myers, Ruslan Salakhutdinov, and Sergey Levine. 2024. “Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference.”

Hadfield-Menell, Dylan, Stuart J. Russell, Pieter Abbeel, and Anca Dragan. 2016. “Cooperative Inverse Reinforcement Learning.” Advances in Neural Information Processing Systems 29.

Poole, Ben, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. 2019. “On Variational Bounds of Mutual Information.” in International Conference on Machine Learning. PMLR.

Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. “Learning Transferable Visual Models From Natural Language Supervision.” in International Conference on Machine Learning. arXiv.

In Lemma 2, I can see that the assistant maximizing the value of $I\left(s^{+} ; z\right)$ leads to lower (regularized) regret for the human. But does maximizing the discounted sum of $I\left(s^{+} ; a_t^H \mid s_t\right)$ also lead to lower regret for the human? It seems like this result isn't shown, and so Lemma 2 doesn't actually apply to the notion of empowerment used in the paper.

This is an excellent point, one that we didn't realize in our original submission. There remains an important connection between Lemma 2 and the notion of empowerment in the paper (details below), but it is not quite as strong as claimed in the original paper. We will revise the paper accordingly.

The expected value in Lemma 2 corresponds to a summation $\sum_{t=0}^\infty \gamma^t x_t$, where $x_t = \log p(s^+ = s_t \mid a^H)$. The method used in the practical algorithm (Eq. 1) corresponds to a double summation: $\sum_{t=0}^\infty \gamma^t \sum_{i=0}^\infty \gamma^i x_i$. This corresponds to doing RL with a discount factor that is not the usual $\gamma^t$, but rather is $(t+1) \gamma^t$. In summary: yes, the theory differs from the practical algorithm, but the difference corresponds to a different choice of discounting function.

Line 159: "We can view our objective as a generalization of the assistance problem beyond the CIRL setting" --- I would be careful about making this claim. The assistance game setting is a (cooperative) Markov game, and the solution concepts in that setting are either optimal policies (Hadfield-Menell et al, 2016) or pragmatic-pedagogical equilibria (Fisac et al, 2017). In contrast, Lemma 2 only shows that maximizing mutual information corresponds to minimizing regularized regret --- which is not the same as finding the optimal joint policy, or finding a pragmatic pedagogical equilibrium.

Thank you for raising this point – we will incorporate this into the discussion, clarifying that there are both similarities and differences between our problem formulation and CIRL.

What embedding functions or neural network did you use to learn the successor features in each benchmark?

In the obstacle grid environment, we used a network with 2 convolutional and 2 fully connected layers and SiLU activations. In Overcooked, we adapted the policy architecture from past work (Carroll et al., 2020), using a 3-layer MLP with tanh activations. We will revise the appendix to clearly describe this.

Figure 2: What exactly is the 'center of the polytope" here?

We have revised the figure to clarify that we are referring to a Barycenter, defined with a KL divergence. In other words, the center is the point that has a minimum KL divergence from the maximally distant point inside the polytope.

Line 197: What is $g$ ? A future state? Why not use $s^{+}$ as before?

Yes, this is a typo. We have replaced $g$ with $s^+$ here.

Line 198: Why do $\phi$ and $\phi'$ each suddenly lose one argument?

This is a typo. This line should read: one that aligns $\psi(s, a^R, a^H) \leftrightarrow \psi(s^+)$ and one that aligns $\psi(s, a^R) \leftrightarrow \psi(s^+)$

Line 201: Please provide some derivation or explanation as to why these representations would encode the stated probability ratios upon convergence.

The citation "[24]" contains a proof of this statement (Poole et al., 2019). We will clarify this in the text. In our revision, we will add a derivation of the solution to our symmetrized infoNCE objective in Appendix D, which follows directly from the analysis by Poole et al. (2019) when adapted to the symmetric version of the loss as in Radford et al. (2021) and Eysenbach et al. (2024).

Equations 8 and 9: Why are the conditional probabilities not conditioned on $a_t^R$ ?

Thanks for catching this typo – the probabilities in the numerator on the right hand side of both equation should also be conditioned on $a^R$.

There are a number of typos here and there ("gradefully", "belore") which should be fixed.

We have fixed these, and run a spelling + grammar checker on the rest of the paper

"While much of the amazing work" --- too subjective for an academic paper.

We have revised this to read "much of the prior work."

Dear Reviewer,

Thanks for the detailed review and suggestions for improving the paper. To address the main concern about baselines, we have added two additional baselines (see below). We have also run additional ablation experiments, and tried to address the other concerns in the discussion below. Does this fully address the reviewer's concerns?

Kind regards,

The authors.

My main concern is the lack of baselines.

We have added a goal inference baseline based on Du et al. (2021) to the obstacle environment and a reward inference baseline using IQ-Learn (Garg et al., 2021) for both the obstacle environments and the overcooked settings. Our method outperforms both of these baselines across the environments studied. These experiments are included in the attached PDF.

Regarding the baselines from Carrol et al. (2019) ([1] here): all of these require the true human reward. In contrast, our method does not assume access to the ground truth reward, and we aim to learn a collaborative policy through our empowerment objective.

provide ablation studies to justify the design choices.

As suggested by the reviewer, we have run additional ablation experiments, studying the effective of the contrastive objective and the empowerment parametrization (see Fig. 1 in the rebuttal PDF).

qualitative analysis

We have added additional qualitative results in the PDF. In the obstacle environment ESR learns to remove obstacles along the path the human is moving and in the Overcooked setting it learns collaborative behaviors like moving the plate to places the human can reach and placing onions in the pot. We will revise the paper with visualizations of some of these behaviors.

Figure 2 confuses me.

We will clarify this figure caption in the revised paper. The goal of this figure is to relate the empowerment objective (Eq. 1) to the analysis of skill discovery (Lemma 2). On the left (a), we visualize how pairs of interacting policies induce a distribution over the state space, seen here as points on an $|\mathcal{S}|$-simplex. In the center (b), the orange polygon depicts the set of state distributions that the human's policy can attain when working with a fixed robot assistant. The black lines correspond to Eq 3 (Lemma 1), which say that empowerment relates to the ``diameter'' of this polygon. To the right (c), we show how our empowerment objective corresponds to maximizing the size of this polytope, i.e., maximizing the human’s ability to control the distribution over state distributions in the environment. We will revise the figure to clarify.

Carroll, Micah, Rohin Shah, Mark K. Ho, Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca Dragan. 2019. “On the Utility of Learning about Humans for Human-AI Coordination.” in Conference on Neural Information Processing Systems. arXiv.

Du, Yuqing, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. 2020. “AvE: Assistance via Empowerment.” Pp. 4560–71 in Advances in Neural Information Processing Systems. Vol. 33. Curran Associates, Inc.

Garg, Divyansh, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. 2021. “IQ-Learn: Inverse Soft-Q Learning for Imitation.” Pp. 4028–39 in Advances in Neural Information Processing Systems. Vol. 34. Curran Associates, Inc.

Dear Reviewer,

We thank the reviewer for the comments and suggestions for improving the paper. It seems like the reviewer's main suggestions were to add additional baselines, which we have done by adding comparisons with a goal inference method and a reward inference method. Together with the discussion below, does this fully address the reviewer's concerns?

Kind regards,

The authors.

The paper only compares to one baseline AvE. As Overcooked is a widely used environment studying human-robot collaboration, more results are welcome.

We have added a goal inference baseline (Du et al., 2021) to the obstacle environment and a reward inference baseline using IQ-Learn (Garg et al., 2021) for both the obstacle environments and the overcooked settings. Our method outperforms both of these baselines across the environments studied. We also add additional ablations of our approach (contrastive loss and parameterization). The two ablations are with the norm and diff variants of our reward. The norm reward computes the reward as the difference of norms between phi(s,a) and phi(s). The diff reward computes the reward as the norm of the difference between phi(s, a) and phi(s). These experiments are included in the attached PDF. Note that a key difference between our evaluations and prior work that studies assistance in Overcooked (e.g., Carroll et al. (2019)) is that we do not assume any access to the human reward or a model of the environment / possible reward structures.

Is there any measurement available to measure the "human-compatibility" during training?

Thanks for this suggestion! This paper is primarily mathematical and algorithmic and nature, and so we'll provide an mathematical/algorithmic answer: Eq. 6 says that we can measure the degree of (minimax) compatibility via the mutual information. We have run an additional experiment to plot this mutual information throughout training (see Fig. 4 in rebuttal PDF). Visualizing the learned agent, we see that the agent does indeed become more helpful as this mutual information increases.

Of course, the gold standard in human-AI interaction is human user studies, which are beyond the scope of this mathematical/algorithmic paper.

IIUC, the learned policy will "overfit" to the human policy which it is trained with? How to scale up the system to utilize the (offline) data from different human policy with different skills / characteristics / optimality?

As with any machine learning algorithm, there is a risk of overfitting when learning from limited data, and learning from a larger quantity of data may mitigate this risk. While our study focuses on the online learning of collaborative policies, the contrastive empowerment objective could also be applied to an offline dataset. In the case where diverse offline data is limited, it may be beneficial to initialize this contrastive policy with a large pretrained model that can impose strong inductive biases on the successor representations. Future work could also explore how multi-modal models (e.g., Chen et al., 2020) combined with our method to further boost performance

What's the motivation by sampling k from a Geometric distribution in Line 136?

The choice of distribution for $k$ dictates the ``horizon'' of the human's empowerment: do we want the human's actions to have a high influence over the outcomes in the next hour, or over the outcomes in the next day. For simplicity, we used a Geometric distribution to be consistent with RL standards (where we care about rewards accumulated under a geometric distribution) and set the parameter to the same value as used by the underlying RL algorithm.

No experiments involving real human subjects are conducted.

We have revised the paper to mention this limitation in the conclusion. Many prior papers also aim to establish the algorithmic foundations for human-AI algorithms before proceeding with human studies (Chan et al., 2019; He et al., 2023; Ngo et al., 2024; Pan et al., 2022; Ratner et al., 2018; Robertson et al., 2023; Zhuang & Hadfield-Menell, 2020).

it's infeasible to train such a policy with a real human subject, especially in the task like overcooked which takes 1.5M steps to train…

Yes, our method needs a large quantity of human data. However, we could apply our method in the offline setting where we have large scale existing datasets for this purpose (see e.g., Xie et al. (2018)). We could then fit the ESR features to the dataset and use the empowerment reward with an offline RL algorithm.

Typo Line 268 "limit’s".

We have fixed this.

No closing parenthesis in Line 218.

We have fixed this.

What is "this set" in Line 142 (if the reader don't read caption of Fig, 2)

We have clarified this.

Chan, L., Hadfield-Menell, D., Srinivasa, S., & Dragan, A. (2019). The assistive multi-armed bandit. ACM/IEEE International Conference on Human-Robot Interaction. HRI.

Chen, L., Paleja, R., Ghuy, M., & Gombolay, M. (2020). Joint goal and strategy inference across heterogeneous demonstrators via reward network distillation. In Proceedings of the 2020 ACM/IEEE international conference on human-robot interaction (pp. 659-668).

Du, Y., Tiomkin, S., Kiciman, E., Polani, D., Abbeel, P., & Dragan, A. (2020). AvE: Assistance via Empowerment. Advances in Neural Information Processing Systems, 33, 4560–4571.

Garg, D., Chakraborty, S., Cundy, C., Song, J., & Ermon, S. (2021). IQ-Learn: Inverse soft-Q Learning for Imitation. Advances in Neural Information Processing Systems, 34, 4028–4039.

Hadfield-Menell, D., Russell, S. J., Abbeel, P., & Dragan, A. (2016). Cooperative inverse reinforcement learning. Advances in Neural Information Processing Systems, 29.

He, J. Z.-Y., Brown, D. S., Erickson, Z., & Dragan, A. (2023). Quantifying assistive robustness via the natural-adversarial frontier. Proceedings of The 7th Conference on Robot Learning, 1865–1886.

Ngo, R., Chan, L., & Mindermann, S. (2024). The alignment problem from a deep learning perspective. The Twelfth International Conference on Learning Representations.

Pan, A., Bhatia, K., & Steinhardt, J. (2022). The effects of reward misspecification: Mapping and mitigating misaligned models. International Conference on Learning Representations. ICLR.

Ratner, E., Hadfield-Menell, D., & Dragan, A. D. (2018). Simplifying reward design through divide-and-conquer. Robotics - Science and Systems. Robotics - Science and Systems.

Robertson, Z., Zhang, H., & Koyejo, S. (2023). Cooperative inverse decision theory for uncertain preferences. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, 5854–5868.

Xie, D., Shu, T., Todorovic, S., & Zhu, S.-C. (2018). Learning and Inferring “Dark Matter” and Predicting Human Intents and Trajectories in Videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(7), 1639–1652.

Zhuang, S., & Hadfield-Menell, D. (2020). Consequences of misaligned AI. Advances in Neural Information Processing Systems, 33, 15763–15773.

1. Apologies for the oversight, we have added the mutual information and accuracy as a table in the official comment above, and will include a line plot of it in the final paper. As the empowerment policy improves, the human's actions take on more influence over the future states, increasing the mutual information.

2. Thank you for the clarification—this is a great question. Learning to empower the human, in contrast to learning the human's reward, is less sensitive to the choice of human policy, and can even benefit from a wide variety of human data to train on. The ESR policy learns to take actions that make the human's choice of action have a high impact on the future state. If the human's policy changes—i.e., it takes a different action in the same state—the empowering policy will still empower the human to reach their goal, irrespective of the human subject. On our new "asymmetric advantage" environment results, we trained the ESR agent against an ensemble of three goal-directed human "personas", each with their own, distinct behavior and method of collaboration. We found that this diversity was important to good performance, and we hypothesize that this is because the assistant must learn to pay attention to the human's action in order to predict the future state well. While these experiments were performed online, we reason that the same principles would apply to offline data. Data from diverse policies should help empowerment.

Do these responses address your concerns?

We have added Table 1 to our revision, comparing (reward-unsupervised) assistants collaborating with an expert human model on an additional Overcooked layout. Our method outperforms the new "reward inference" baseline as well as "AvE" and "random" on all layouts tested. Results are shown with one standard error.

Table 1: Additional evaluation on the “asymmetric advantages” Overcooked layout

We have added Figure 4 to our revision, summarized here as a table, showing the mutual information and accuracy of the model throughout training on the "asymmetric advantages" Overcooked layout. The accuracy measures the model's ability to distinguish a true future state from other, random future states given the current state and action. Mutual information is measured between the human's action and the future state, signifying that as training progresses, the human's action takes on more influence over the future states.