Randomized algorithms and PAC bounds for inverse reinforcement learning in continuous spaces

We thank the reviewer for the time spent in evaluating our work. In the following, we address your comments.

Weaknesses

The presentation is confusing ...

We would greatly appreciate it if you could specify which parts of the paper you find confusing. We put significant effort into presenting the material in an accessible and comprehensible manner without compromising on mathematical clarity and rigor. Given the constraints of the conference paper format and the substantial amount of theoretical concepts we cover, we aimed to present our findings concisely while maintaining coherence. For the final version of the paper, we will utilize the additional page allowed to further elaborate on certain sections and provide additional intuition and context about the presented theory.

The paper provides mostly negative statistical results ...

Could you please clarify what you mean? Additionally, we respectfully disagree with the comment that our assumptions are too strong. We believe that Ass. 2.1 in the setting of continuous MDPs is standard and, to the best of our knowledge, as general and mild as possible (see also lines 179-187). Ass. 4.1 is easily satisfied if X and A are compact and the sampling distribution continuous. Furthermore, Ass. 4.2 and 4.3 are directly satisfied if a tie-breaking rule is introduced, as noted in the text.

The motivation for some design choices ...

The normalization constraint is motivated by an entire section (Sec. 4.1) and theoretically justified by Prop. 4.1. We are certainly open for suggestions on how to motivate it even better. Also it would be helpful if you could specify what "other design choices" you are referring to.

Questions:

This paper tackles an interesting problem ...

Thank you for the 2024 references, which we will add and discuss in our revised manuscript. We briefly highlight our main contributions and how they differ from recent theoretical works on IRL, underscoring the impact of our paper in this new research wave.

First of all we provide a novel optimization perspective of the IRL problem that allows us to characterize the inverse feasibility set by means of Lagrangian duality. We believe this new formulation, applicable to both discrete and continuous MDPs, is more powerful and independent of the expert's complexity, unlike previous works [48-53] and Lazzati et al. 2024. Most importantly, it erases the problems encountered in the formulations [51-53] and Lazzati et al. 2024. The authors in these works study solely how to deal with the unknown transition law and how to sample efficiently from the expert policy. In their formulation addressing these questions is complicated. However in our formulation this question is easily addressed with efficient expert sample complexity by just using Hoeffding bounds (Prop B.1 in the proof of Thm 4.2).

Second, unlike recent theoretical IRL works [48-53] which avoid discussing this ill-posedness issue altogether, we attempt to address the ambiguity problem and provide theoretical results in this direction, hoping to lay the foundations for overcoming current limitations.

Our result on linear function approximators (Prop. 4.2) is applicable even in the tabular setting considered in [48,49,51-53]. This contribution is crucial and missing from the literature on large-scale finite MDPs.

Finally, we addres the problem of extracting one single cost function with theoretical guarantees. Learning a whole inifinite feasibility set is impossible even for the simplest tabular MDP problem. While constraint sampling has been proposed as a heuristic in the pioneering paper by Abbeel and Ng (2000), we provide explicit performance bounds and sample complexity for this methodology in continuous MDPs. The corresponding sample complexities include expert sample complexity, number of calls to the generator, and number of sampled constraints. The first two complexities scale gracefully with problem parameters, whereas the number of sampled constraints scales exponentially with the state and action space dimensions, making the algorithm particularly suitable for low-dimensional problems.

FUNCTION CLASS:

Our paper tackles the inverse RL problem with minimal assumptions, contributing significantly to the limited theoretical research in this area. While some sample complexity bounds may be impractical, the results are not negative. The reviewer suggested that additional structures like linear MDPs could improve PAC bounds. Prior work supports this idea (see Zhang et al., 2015). We are interested in exploring this in future work, although it will require substantial effort.

INTERPRETABLE RESULTS:

Thank you for your comment. Thm 4.1 ensures that the cost function $\tilde{c}_N$ from the finite-dimensional random convex problem $(SIP_N)$ is $(\tilde{\varepsilon}_N + \epsilon)$-inverse feasible for the expert policy $\pi_E$. Here, $\tilde{\varepsilon}_N$ is the solution to $(SIP_N)$, and $\epsilon$ is an accuracy parameter affecting the sample complexity $N$, which scales with $\{1/\epsilon, \log(1/\delta), n\}$ as described on Line 328. The sample complexity depends on MDP characteristics, such as state-action space dimension (exponentially) and basis functions for cost $n_c$ and value function $n_u$, as noted in Rem. 4.1. Thm 4.2 addresses a more realistic scenario without requiring this knowledge. We will clarify these results further in the additional page we are given.

SOLUTION CONCEPT:

Our approach allows learning an approximately feasible reward in a PAC sense, as demonstrated by Theorems 4.1 and 4.2. While our normalization constraint prevents trivial solutions, our paper does not address selecting the "most informative" reward, which could be motivated by its utility for downstream tasks. Adding regularization constraints, such as an entropy term, could help select such a reward. This integration into our scenario programs is a topic for future research.

Thank you for your prompt response and for providing us with the opportunity to further clarify our points. Below, we address the issues you raised.

CONCERNS: Motivation: The novel optimization perspective looks great, but I would like to understand what kind of benefits it provides on top of prior results. Does it allow for sharper rates in {expert's query complexity, sample complexity} or polynomial rates for previously unaddressed settings? Does it open the door for more scalable algorithms? Something else?

Recent research in IRL with theoretical guarantees has seen significant advancements, as evidenced by works such as [48,49,50,51,52,53], Lazzati et al., "Offline Inverse RL" (ICML 2024), and Lazzati et al., "How to Scale RL to Large State Spaces? A Provably Efficient Approach" (2024). These studies have laid the groundwork for exploring the mathematical foundations of IRL, a complex and challenging topic. In the following, we will clarify the specific benefits our optimization-based perspective and formulation provide on top of prior results.

First, as discussed in our manuscript and demonstrated in Appendix A.2, our formulation—when applied to finite tabular Markov decision processes (MDPs) and a stationary Markov expert policy—simplifies to those considered in [48-53].

Comparison to [48,49,50]

Setting: The formulation underlying the analysis and algorithms in [48, 49, 50] addresses scenarios where the expert policy is known and deterministic. In contrast, our formulation accommodates more general unknown, nonstationary, and randomized expert policies. Additionally, our approach assumes access to an offline, static dataset of expert demonstrations. While the works in [48, 49] focus on tabular MDPs, and [50] addresses MDPs with continuous state and discrete action spaces, we extend these contributions by providing a formulation and formal guarantees for IRL in continuous state and action spaces.

Formal Guarantees: Notably, similar to our Theorems 4.1 and 4.2, the sample complexity in the continuous-state setting, as provided in [50], scales exponentially with the dimension of the state space.

Assumptions: In the tabular MDP setting, our analysis holds without any additional assumptions on the MDP model. This is because we bypass the need for sampling constraints by solving a finite linear program using off-the-shelf solvers. In contrast, [48, 49] assume that the problem is $\beta$-separable and that every state is reachable with probability $\alpha$. For the MDP with a continuous state space in [50], the authors assume that the transition law density has an infinite matrix representation. Our Assumption 2.1 on the MDP is significantly milder, as it encompasses the majority of probability distributions (see lines 179-187) and includes the representation considered in [50] as a special case.

Comparison to [51,52,53]

The characterization of the inverse feasibility set which forms the basis for the analysis in [51,52,53] is the following:

Let $\pi_E$ be stationary Markov. A cost function $c$ is inverse if and only if there exists $u\in\mathbb{R}^{|X||A|}$ and nonnegative $L\in{\mathbb{R}}^{|X||A|}$ such that $c(x,a)-u(x)+\gamma\mathbb{E}_{y\sim P(\cdot|x,a)}[u(y)] = L(x,a)\mathbb{1}[\pi_E(a|x)=0]$, for all $(x,a)\in X\times A$.

This formulation considers a stationary Markov expert, whereas our proposed approach is independent of the complexity of $\pi_{E}$. Their formulation cannot be extended to MDPs with continuous state and action spaces due to the presence of the term $\mathbb{1}[\pi_E(a|x)=0]$. For continuous distributions, $\pi_E(a|x)=0$ always, making the extension infeasible. Additionally, the term $\mathbb{1}[\pi_E(a|x)=0]$ significantly complicates the estimation of the inverse feasibility set. To address this challenge, they assume active querying of $\pi_E$. In contrast, our optimization-based framework formulates the problem using occupancy measures instead of policies. This allows us to tackle the more challenging learning scenario where the learner has access only to a finite set of expert trajectories and is not permitted to query the expert for additional data during training. In our formulation, efficiently estimating the term $\widehat{<\mu^{\pi_{\textup{E}}}_{\nu_0},c>}$ via sample averages is straightforward (see the sampling process in lines 361-367), and the estimation error can be computed using a simple Hoeffding bound.

In [51, 52, 53], the authors focus exclusively on estimating the inverse feasibility set, bypassing the task of learning a single cost with formal guarantees. For the tabular setting with a generative model and an (online) interactive expert, [53] provides a sample complexity lower bound of $\mathcal{O}\left(\frac{|X||A| (|X|+\log(\frac{1}{\delta}))}{(1-\gamma)^3 \varepsilon^2}\right)$ and proposes a sampling strategy that achieves this bound up to logarithmic factors. It is important to note that when estimating the inverse feasibility set, the sample complexities of interest are the number of queries to the generative model and the expert sample complexity. In other words, in our Thm 4.2, the number of sampled constraints required to solve the reduced finite LP for extracting a single cost is not necessary (recall that the curse of dimensionality appears only in the number of sampled constraints).

In our offline setting for tabular MDPs, we require $m = \mathcal{O}\left(\frac{|X||A|(2-\gamma)^2\log(\frac{|X||A|}{\delta})}{(1-\gamma)^4\varepsilon^2}\right)$ expert samples and $K|X||A| = \mathcal{O}\left(\frac{(2-\gamma)^2|X|^2|A|\log(\frac{|X|^2|A|}{\delta})}{(1-\gamma)^2\varepsilon^2}\right)$ calls to the generative model. The factors $\frac{2-\gamma}{1-\gamma}$ arise from Prop. 3.1 and 3.3.

Furthermore, we address the challenge of extracting a single cost with formal guarantees for continuous MDPs by incorporating a normalization constraint, employing function approximation, and using constraint sampling techniques. For tabular MDPs, this task is straightforward, as one only needs to solve a finite MDP using off-the-shelf solvers in polynomial time.

Comparison to "How to Scale IRL to Large State Spaces? A Provably Efficient Approach" (2024 Concurrent)

The authors consider the setting of linear MDPs, where the agent can query the expert in any state and interact online with the environment (forward model). By generalizing the notion of the inverse feasibility set and introducing the cost compatibility framework, they provide a provably efficient algorithm for MDPs with continuous states and discrete actions to estimate the compatibility of all costs with high accuracy.

Interestingly, in our versatile formulation, any cost that is $\varepsilon$-inverse feasible is also $\frac{1-\gamma}{2-\gamma}\varepsilon$-compatible (Prop. 3.1). This key feature, which seems absent and unaddressed in the formulations of [51-53], highlights another significant distinction in our approach. We conjecture that by employing the same Ridge regression estimators as in [Jin et al., "Provably Efficient RL with Linear Function Approximation" (2020)], one could establish similar efficient bounds for continuous $X$ and $A$. We leave this exploration for future work. We also emphasize that, akin to estimating the inverse feasibility set, this new theoretical question does not require sampling constraints.

Comparison to "Offline IRL" (ICML 2024), and Lazzati et al.

This setting contrasts with our assumption of a generative model.

A new perspective

Our optimization-based approach results in problem formulations directly amenable to modern large-scale stochastic optimization methods. Consequently, we anticipate that our techniques will benefit future algorithm designers and establish a foundation for more comprehensive research in this area.

We thank the reviewer for the time spent in evaluating our work. In the following, we address your comments.

Weaknesses

The paper's result lacks benchmark ...

This work focuses on theoretical aspects of inverse RL, specifically on learning a cost function in continuous state and action spaces from observed optimal behavior, and deriving PAC bounds. We do this by approximating the infinite-dimensional problem with a finite convex problem and rigorously quantifying approximation errors.

Most existing inverse RL research is on finite state and action spaces. Current work on continuous spaces either lacks theoretical guarantees or focuses on the feasibility set (see [42, 43, 44, 23, 26, 45, 27], [51, 52, 53]). Our related work section offers a detailed comparison. We also refer the reviewer to our response to Reviewer PJ8y (in section "Questions"), where we briefly highlight our main contributions and how they differ from recent theoretical works on IRL, underscoring the impact of our paper in this new research wave.

While we demonstrate our method with a simple truncated LQR example (see Appendix C), we agree that comparing it with existing heuristic methods in continuous spaces is valuable. Our one-dimensional experiment underscores the need for further investigation into choosing basis functions and sampling informative state-action pairs. We are addressing these questions theoretically and will include a discussion in the revised paper.

Second, the paper doesn't reduce its general result to simpler special cases ...

Thank you for the comment. In the tabular MDP case, our proposed bounds become redundant. If the transition kernel $T$ is known, the $\varepsilon$-inverse feasibility set (Equation 1) is a linear program with finite variables, solvable in polynomial time. Thus, our method for approximating infinite-dimensional LPs with function approximation and constraint sampling is unnecessary, making our error bounds for this approximation irrelevant.

Our approach is intended for continuous MDPs. For large but finite state and action spaces, where direct solution of LPs is impractical, function approximation and constraint sampling are applicable, and our PAC bounds are relevant.

Regarding additional structure, such as a linear MDP, related works suggest that incorporating such structures could improve PAC bounds (see [2]). Investigating how to leverage linear MDP structures in our inverse RL approximation hierarchy is a promising future direction, though it requires considerable effort.

[2] Zhang et. al., On the sample size of random convex programs with structured dependence on the uncertainty, Automatica, Volume 60, 2015

Part of the result is unintuitive ...

We thank the reviewer for this comment. As written, it is indeed unintuitive because there is a small typo causing the confusion. The constant $d$ in Proposition 4.2 should be $d = leb(\mathcal{X} \times \mathcal{A})$, where $leb(\cdot)$ denotes the Lebesgue measure. In the proof, on line 671, it is clear why $leb(\mathcal{X} \times \mathcal{A})$ is the correct constant.

And the bound shows that ...

Indeed, the reviewer is correct. When examining the bound in Proposition 4.2, the resulting error monotonically decreases as $\theta$ increases, suggesting that $\theta = \infty$ would be optimal. However, the sample complexity in Theorem 4.1 increases monotonically with $\theta$. In other words, a larger $\theta$ requires a larger number of constraint samples. Therefore, $\theta$ must be selected to balance these two factors. We will include a remark on this in the revised version.

Questions:

1.For challenge (a), it seems ...

This is an assumption. We assume that the expert policy $\pi_E$ is optimal for a unique true cost function $c^\star$ with $u^\star$ the corresponding optimal value function. Although this true cost is unknown, some prior knowledge about its properties allows us to choose appropriate linear function approximators to make the projection residuals in the theorem sufficiently small. For example, if the true cost function is known to be smooth, Fourier or polynomial basis functions can be used. Essentially, the term $\varepsilon_{\textup{approx}}$ measures the expressiveness of the linear function approximators.

We would like to highlight that, in practice, assuming the expert policy is produced by a unique true cost does not contradict the ill-posed nature of the IRL problem, where the expert policy is optimal for infinitely many cost functions.

A key question is whether, despite assuming a unique true cost, any inverse feasible cost solution can replace the true cost in Proposition 4.2's approximation error expression due to IRL's inherent ambiguity. The answer is yes: if the basis functions approximate the true cost well, it will be recovered accurately. Otherwise, if the approximators represent a different inverse feasible cost, that cost will be recovered instead. This aligns with expectations.

We’ve added this discussion to the revised paper for clarity. Thank you for highlighting this point.

For figure (c) in the experiment ...

Thank you for your comment. Figure (c) shows how sample complexity (x-axis) depends on the confidence level $1-\delta$ (y-axis). As sample size increases, the confidence parameter $\delta$ decreases, which aligns with the logarithmic growth of sample complexity with $\log(1/\delta)$.

Figure (a) evaluates empirical confidence with four different accuracy parameters. As noted, the parameters result in impractically large sample complexity bounds. The proposed bounds have large constants, limiting their practical use compared to the empirical results in figures (a), (b), and (d). The main value of these bounds is in understanding how sample complexity grows with problem parameters like accuracy $\epsilon$ confidence $\delta$ and dimensions.

Additional examples of figure (c) with different parameters are provided in the supplementary PDF.

The paper appears to present a series of theorems without providing sufficient intuitive explanations or justifications. This approach can make the work difficult to understand for reviewer.

We put significant effort into presenting the material in an accessible and comprehensible manner without compromising on mathematical clarity and rigor. Given the constraints of the conference paper format and the substantial amount of theoretical concepts we cover, we aimed to present our findings concisely while maintaining coherence. For the final version of the paper, we will utilize the additional page allowed to further elaborate on certain sections and provide additional intuition and context about the presented theory. In particular, we will make an effort to explain and justify the two theorems of our paper better.

Our first theorem (Theorem 4.1) guarantees that the cost function $\tilde{c}_N$, obtained from the solution of the proposed finite-dimensional random convex counterpart ($SIP_N$), is indeed $(\tilde{\varepsilon}_N + \epsilon)$-inverse feasible for the expert policy $\pi_E$. Here, $\tilde{\varepsilon}_N$ is the solution to $(SIP_N)$ and $\epsilon$ is an accuracy parameter affecting the sample complexity $N$, which scales with $\{1/\epsilon, \log(1/\delta), n\}$ as described on Line 328. The sample complexity depends on MDP characteristics, such as state-action space dimension (exponentially) and basis functions for cost $n_c$ and value function $n_u$, as noted in Rem. 4.1.

Theorem 4.1 is primarily of practical interest because the program $(SIP_N)$ requires knowledge of the transition kernel, which is typically not available in RL. Therefore, our second theorem (Theorem 4.2) considers a more realistic setting where knowledge of the transition kernel is not required. In this setting, the proposed cost $\tilde{c}_w$ for $w=\{N,m,n,k\}$ is derived from the program $(SIP_w)$ for $w=\{N,m,n,k\}$. Essentially, Theorem 4.2 extends the results of Theorem 4.1 to this more realistic scenario.

Despite the theoretical rigor, the paper seems to lack substantial practical demonstrations or empirical results. This absence makes it challenging to assess the real-world applicability.

We illustrate our method with a simple truncated Linear Quadratic Regulator (LQR) example (see App. C) to provide better intuition about the method and the proposed sample complexity bounds. The focus of our paper is on an optimization perspective of IRL and deriving fundamental, provable PAC bounds for continuous inverse RL in a very general setting. Thus, studying the empirical performance beyond the academic LQR problem is out of scope and will be addressed in future work. It is worth noting that the work of Dexter et al. [50], the only theoretical study addressing IRL in continuous state (but discrete action) spaces, also includes simulations for a representative one-dimensional state space MDP.

The algorithms proposed might be theoretically sound but potentially impractical for real-world applications. The paper may not adequately address computational complexity or scalability issues that could arise when applying these methods to large-scale or complex problems.

The theoretical soundness of the proposed algorithm is ensured by our main results (Theorems 4.1 and 4.2). If "large-scale" is interpreted as the number of state/action variables, we address settings with continuous (uncountable) state and action spaces. Thus, even our simple 1-dimensional LQR example might be considered "large-scale." However, if "large-scale" refers to the dimension of the continuous state and action spaces, our bounds indeed suffer from exponential sample complexity in these dimensions. This is a manifestation of the infamous curse of dimensionality, which cannot be avoided. Please refer to our detailed discussion in Remark 4.1 of the paper.

We agree with the reviewer that to apply our bounds to complex and high-dimensional real-world problems, we need to more carefully exploit underlying structures. Otherwise, the sample complexity bounds are too large and of limited practical use. However, we would like to emphasize that these sample complexity bounds are often conservative in practice, as can be seen in our empirical evaluation in the numerical results, see Section C. Experiments conducted on closely related scenarios have shown promising approximation bounds, as detailed in [1,2].

In this work, our aim is to address the most general setting possible with the minimal assumptions required to establish PAC bounds for inverse RL. We plan to explore exploiting additional structures to derive tighter bounds for well-structured subclasses of problems in future work.

Finally, we would like to emphasize that the provided bounds are part of our contributions. We refer the reviewer to our response to Reviewer PJ8y (in section "Questions"), where we briefly highlight our main contributions and how they differ from recent theoretical works on IRL, underscoring the impact of our paper in this new research wave.

[1] Marco Campi and Simone Garatti, Introduction to the Scenario Approach, Society for Industrial and Applied Mathematics, 2018 [2] Xiaojing Zhang, Sergio Grammatico, Georg Schildbach, Paul Goulart, John Lygeros, On the sample size of random convex programs with structured dependence on the uncertainty, Automatica, Volume 60, 2015

We thank the reviewer for the time spent in evaluating our work. In the following, we address your comments.

The paper certainly contains enough material for a substantially longer journal paper - some of the sections could benefit from being longer, trying to convey more intuitions and context about the presented theory.

We thank the reviewer for recognizing the substance and novelty in our paper, which could indeed justify an even longer manuscript. We are delighted by your appreciation of our writing style, as we put significant effort into presenting the material in an accessible and comprehensible manner without compromising on mathematical clarity and rigor. Given the constraints of the conference paper format and the substantial amount of theoretical concepts we cover, we aimed to present our findings concisely while maintaining coherence. For the final version of the paper, we will utilize the additional page allowed to further elaborate on certain sections and provide additional intuition and context about the presented theory.

More thorough empirical evaluation would also be beneficial...

We illustrate our method with a simple truncated Linear Quadratic Regulator (LQR) example (see App. C) to provide better intuition about the method and the proposed sample complexity bounds. Although the focus of this work is theoretical, we will endeavour to provide a more complex numerical result, possibly in the Supplementary Material. It is worth noting that the work of Dexter et al. [50], the only theoretical study addressing IRL in continuous state (but discrete action) spaces, also includes simulations for a representative one-dimensional state space MDP.

The bounds provided by the theorems don't always seem meaningful ...

Theorem 2.1 provides explicit sample complexity bounds for achieving a desired approximation accuracy with our proposed randomized algorithm. The corresponding sample complexities include the expert sample complexity, the number of calls to the generator, and the number of sampled constraints. The first two complexities scale gracefully with respect to the problem parameters, whereas the number of sampled constraints scales exponentially with the dimension of the state and action spaces. This makes the algorithm particularly suitable for low-dimensional problems of practical interest, e.g., pendulum swing-up control, vehicle cruise control, and quadcopter stabilization. We will include these examples of manageable dimensionality and related references in Rem. 4.1. Moreover, in the uploaded supplementary PDF, we show additional instances of figure (c) for different parameter choices, illustrating the significant effect on the resulting sample complexities.

Note that a similar exponential dependence to the dimension of the state space has been established in Dexter et al. [50], the only theoretical work on IRL in continuous states and discrete action spaces.

We intend to provide a more detailed discussion on enhancing sample complexity bounds through the utilization of the underlying problem structure in the paper. In addition, it becomes imperative to gain an understanding regarding the selection of a suitable distribution for drawing samples in the future. Intuitively, it is reasonable to anticipate that certain regions within the state-action space carry more "informative" characteristics than others. One conjecture is that sampling constraints based on the expert occupancy measure could yield a more scalable bound. However, a comprehensive mathematical treatment of these inquiries will be addressed in future research endeavors.

Isn't the identifiability issue a deeper issue than a mere "mathematical artifact"?

The normalization constraint in our formulation primarily aims to address the ill-posedness (or, equivalently, the ambiguity issue) inherent in the IRL problem. In particular we state and prove that the normalization constraint rules out a wide class of trivial solutions, i.e., all constant functions and inverse solutions of the form $c=T_\gamma^*u$, an outcome devoid of physical meaning and a mathematical artifact. While the identifiability problem and the ill-posedness problem are related in IRL, they are not the same. Identifiability deals with the uniqueness of the true cost function, whereas ill-posedness is a broader concept from mathematical and statistical problems and refers to situations where a problem does not satisfy the conditions for being well-posed (e.g., in our case due to infinitely many solutions). Overall, the purpose of the normalization constraint is to exclude a large class of meaningless solutions, rather than to address the identifiability problem. Note that, unlike recent theoretical IRL works [48-53] which avoid discussing this issue altogether, we attempt to address the ambiguity problem and provide theoretical results in this direction, hoping to lay the foundations for overcoming current limitations.

Do you have an intuitive explanation for why the constraint on the epsilon constraint on the consistency of the cost and value function is enforced point-wise from below and in expectation from above?

The characterization of the inverse feasibility set is due to linear duality and complementary slackness conditions (lines 210-211). In particular, the constraint that holds pointwise is due to dual feasibility and and the constraint that holds in expectation is due to strong duality (see proof of Thm. 3.1 lines 611-612). The formulation (IP) is a relaxation of the constraints in the inverse feasibility set and paying a penalty when violated (lines 288-289). We will add this insight to the revised text.

Minor points suggestions

We will follow your suggestions and remove the third motivation as well as the indicated sentence about the recovery of the cost function to avoid confusion.

Minor typos

We thank the reviewer for spoting these typos; We fixed all of them.

I'd like to thank the authors for a thorough response to the points raised in the review. I'll gladly keep the "accept" recommendation.

Regarding the discussion on

Isn't the identifiability issue a deeper issue than a mere "mathematical artifact"?

Just to clarify: I'm happy with your technical solution, and the point I was raising was more of a subtle phrasing issue. What I was objecting to was mainly that in the first paragraph, you mention reward shaping and then you say that "all these examples illustrate ... mathematical artifacts". I'd say your solution correctly filters out mathematical artifacts and meaningless rewards. However, the issue of underdeterminacy (e.g. with respect to reward shaping) can remain (unless we have access to e.g. multiple environments) and is important in IRL, rather than being mere mathematical artifact (which is a label I'm ok with in relation to the rewards you're filtering out). Maybe a slight rephrasing may help not to sound like putting all these transformations (incl. general reward shaping) in a single basket.

Thank you for your valuable feedback. We understand your concern regarding the phrasing in the paragraph, particularly the mention of reward shaping and the reference to "mathematical artifacts.". We will revise the phrasing to ensure that reward shaping and underdeterminacy are not inadvertently grouped with mathematical artifacts, thereby maintaining a clear distinction between them. Indeed, these issues cannot be fully resolved unless, for example, one has access to multiple expert policies or environments for comparison. Thank you again for pointing this out.