Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification

We thank reviewer CDtC for their review. Our responses to your questions are as follows:

Weaknesses:

1. We appreciate that the transformations described in Proposition 3 are quite difficult to visualise. The proof of Proposition 3 includes a geometric description of these transformations, but they are still not easy to understand intuitively. To some extent, this is just a consequece of the mathematics --- we wanted to derive necessary and sufficient conditions, and in this case, those conditions turned out to be somewhat messy. However, we can certainly include a few examples to make Proposition 3 more intuitive. First of all, Proposition 3 is satisfied by any transformation that can be expressed as a combination of potential shaping, positive linear scaling, and S'-redistribution, and these transformations are reasonably easy to understand (see Section 1.2). More generally, a transformation satisfies Proposition 3 if it never changes the policy order of the reward function by a large amount. For example, suppose we have a Bandit MDP with three actions, and that the reward obtained from these actions is $\langle 1, 1-\epsilon, 0\rangle$. A reward transformation that satisfies Proposition 3 might change this reward to $\langle 1-\epsilon, 1, 0\rangle$, for example. We can include a few examples like this, to help make Proposition 3 more intelligible.
2. The MCE policy and the Boltzmann-rational policy are not identical, though they are similar in some ways. The Boltzmann-rational policy is given by softmaxing the optimal Q-function, whereas the MCE policy is given by softmaxing the soft Q-function. The optimal Q-function and the soft Q-function are in turn not identical, which means that these two behavioural models are slightly different.
3. We agree that more extensive empirical investigation would be interesting, but we consider this to be out of scope for this paper. It would be redundant to verify our positive results (i.e. Theorem 1 and 2) experimentally, as our proofs already tell us what the outcome of such experiments would be. It would be more interesting to verify our negative results (i.e. Theorem 3, 4, and 5). However, the outcome of these experiments would be heavily dependent on the choice of prior probability distribution over the true reward function $R^\star$, and on the choice of inductive bias of the learning algorithm --- our proofs tell us that there for any inductive bias is a prior probability distribution over $R^\star$ under which our bounds are attained with high probability. For Theorem 4 and 5, the converse also holds. Nonetheless, more extensive empirical investigation would certainly be interesting, but we consider this to be out of scope for this paper.

Questions:

1. Yes, it should be $||R - t_n(R)||_2$. In this paper, we have used the convention where $||x, y||_2$ denotes the metric induced by the $\ell_2$-norm, ie $||x - y||_2$. Since several reviewers found this confusing, we will replace it with the more typical notation.
2. The result that all the common behavioural models fail to be $\epsilon/\delta$-separating does depend on the choice of metric(s). Specifically, it is a consequence of the fact that two reward functions can have a small $\ell_2$-norm, even though they have a large STARC-distance (see Appendix D). As such, the proof would not go through if we measure the distance between rewards using the $\ell_2$-norm. However, if we do this, then we would not obtain informative regret bounds (as discussed in Section 2.2). There may be other metrics under which the standard behavioural models are $\epsilon/\delta$-separating, and under which some form of regret bounds or other theoretical guarantees can be obtained. However, no such metric has been proposed in the existing literature. We believe that further exploration of this question is an interesting topic for further work.

We hope that this clarifies our results, and that the reviewer might consider increase their score!

We would like to thank reviewer jn4g for their review and feedback!

We agree that it could be interesting to provide a more detailed analysis of specific IRL algorithms, such as MaxEnt IRL. However, note that all of our results apply to any algorithm that is based on the MCE model, including MaxEnt IRL. In this sense, our analysis is simply more general than an analysis which is restricted to a particular algorithm.

Of course, in general, the inductive bias of an algorithm can be important for understanding its behaviour --- this is discussed in some more detail in Appendix A.4. However, note that Proposition 1 and 2 together imply that if two reward functions have the same MCE policy, then their STARC distance is 0. This means that if an algorithm is based on the MCE model, then its inductive bias will not affect the quality of the inferred reward function in the limit of infinite data. As such, we would not be able to draw stronger conclusions by incorporating more details about the MaxEnt IRL algorithm.

Yes, in Proposition 3, it should be $||R - t_n(R)||_2$. In this paper, we have used the convention where $||x, y||_2$ denotes the metric induced by the $\ell_2$-norm, ie $||x - y||_2$. Since several reviewers found this confusing, we will replace it with the more typical notation.

We hope that this clarifies our results, and that the reviewer might consider increasing their score.

First of all, we would like to thank reviewer Bsty for your thoughts and feedback! Our responses to your concerns and questions are as follows:

Weaknesses:

1. The novelty of our contribution does indeed lie in our analysis (i.e., the results we present in Section 3). While misspecification in IRL has been studied by a handful of recent papers, our analysis is more comprehensive, and is done in a richer formal framework, which lets us infer many novel insights. We have attempted to explain precisely how our work extends earlier work in Section 1.1 and Appendix G, but we can certainly make these sections more detailed. To make this more clear, we have included a more detailed comparison between our paper and earlier work in a separate comment.

2. All proofs are provided in Section F of the appendix, on pages 20-26. We also provide a high-level intuitive explanation of some of these proofs in Section D and E of the appendix, on pages 17-19. There would not be enough space to fit this in the main part of the text. However, note that it is very common to put long proofs in the appendix. As for experiments, we note that this is a theoretical paper, rather than an empirical paper. It would be redundant to verify our positive results (i.e. Theorem 1 and 2) experimentally, as our proofs already tell us what the outcome of such experiments would be. It would be more interesting to verify our negative results (i.e. Theorem 3, 4, and 5). However, the outcome of these experiments would be heavily dependent on the choice of prior probability distribution over the true reward function $R^\star$, and on the choice of inductive bias of the learning algorithm --- our proofs tell us that there for any inductive bias is a prior probability distribution over $R^\star$ under which our bounds are attained with high probability. For Theorem 4 and 5, the converse also holds. Nonetheless, more extensive empirical investigation would certainly be interesting, but we consider this to be out of scope for this paper.

Questions.

1. Yes, there are many potential sources of error in the IRL problem. Our work focuses on misspecification of the behavioural model, as we believe this issue to be particularly important and difficult to overcome. However, IRL certainly has many other difficulties as well.
2. Yes. This is implicit in our formalism, since the behavioural model depends on the transition function. Treating this dependence as being part of the behavioural model makes it easier to reason about the case where the transition dynamics are themselves misspecified.
3. Yes, two behavioural models can produce the same reward function(s) for a given data distribution. However, the aim of our work is not to determine the vailidity of the behavioural model, but rather, to quantify how sensitive IRL is to misspecification of the behavioural model. In other words, we want to know when it is possible to infer a reward function that is "close enough" to the true reward function using a behavioural model that is misspecified.
4. The brackets are indeed incorrect, thank you for spotting this. The correct statement should read $E[R_1(s,a,S')] = E[R_2(s,a,S')]$.
5. $R_1$ and $R_2$ do not have to be different; the purpose of Point 3 is to ensure that $\mathrm{Im}(g) \subseteq \mathrm{Im}(f)$.
6. No, it is the linear ordering induced by the policy evaluation function $J$. That is, $R_1$ and $R_2$ are considered to have the same ordering of policies if $J_1(\pi_1) \leq J_1(\pi_2)$ if and only if $J_2(\pi_1) \leq J_2(\pi_2)$.
7. Any reward function can be viewed as an $|S|^2|A|$-dimensional vector. We use $||x,y||_2$ to denote the metric induced by the $\ell_2$-norm. This means that $||R, t_n(R)||_2$ is the $\ell_2$-distance between the two reward functions $R$ and $t_n(R)$. Since several reviewers found this notation confusing, we will replace it with more typical notation.
8. Cumulative Prospect Theory is not covered by our analysis in this paper, but we consider it (along with other models from the behavioural sciences, such as hyperbolic discounting) to be an important topic for further work. We would however expect the worst-case error to be very large in this case, and proof strategies similar to those used in Section 3.3 could probably be used. Note that Cumulative Prospect Theory can lead to non-stationary policies.

We hope that these points clarify our contributions, and that the reviewer will consider increasing their score!

References:

Joey Hong, Kush Bhatia, and Anca Dragan. On the sensitivity of reward inference to misspecified human models, 2022.

Joar Skalse and Alessandro Abate. Misspecification in inverse reinforcement learning, 2023.

Rachel Freedman, Rohin Shah, and Anca Dragan. Choice set misspecification in reward inference, 2020.

Luca Viano, Yu-Ting Huang, Parameswaran Kamalaruban, Adrian Weller, and Volkan Cevher. Robust inverse reinforcement learning under transition dynamics mismatch, 2021.

We thank reviewer 5VPU for their review and feedback!

We will try to include more examples in the main text. Due to space restrictions, it is difficult to give in-depth explanations without compromising on the technical content. However, since several reviewers have commented on Proposition 3 in particular, we will prioritise more intuition building for this result.

Questions:

1. Our results apply to any algorithm which uses a particular behavioural model, so our results are directly applicable in this case. However, note that if we make additional assumptions about both the true reward function, and about the inductive bias of the learning algorithm, then we may be able to derive stronger results. This is discussed in Appendix A.4.
2. This might affect the results if the reward is parameterised in a way that cannot express all possible reward functions -- this is discussed in the third paragraph of Section 2.1, and in Appendix A.3. If the parameterisation can express all possible reward functions, then it will not affect the results.
3. Yes. Potential shaping, S'-redistribution, and positive linear scaling together completely characterise all ways in which two reward functions can have the same policy order. However, two reward functions can have different policy orderings, and still have the same optimal policy. To see this, consider a Bandit MDP with three actions, and two reward functions $\langle 2, 1, 0\rangle$, $\langle 2, 0, 1\rangle$. These reward functions have the same optimal policy, but different policy orderings (and so they do not differ by potential shaping, S'-redistribution, and positive linear scaling).
4. These types of transformations can be expressed as potential shaping, just let $\Phi(s) = -b/(1-\gamma)$ for all states $s$.
5. Yes, the techniques we have used can also be used to reason about unidentifiability. For example, proposition 1 and 2, together with the properties of STARC metrics, imply that if two reward functions produce the same Boltzmann-rational or MCE policy, then their STARC distance is 0. This means that an IRL algorithm that is based on the Boltzmann-rational or the MCE behavioural model will converge to a reward function that has STARC distance 0 to the true reward function (when there is no misspecification).

We also thank the reviewer for notifying us of the typos!