Meta Inverse Constrained Reinforcement Learning: Convergence Guarantee and Generalization Analysis

it is more computationally expensive since it needs to solve a regularized optimization problem. Our problem formulation (2)-(3) embeds the spirit of both designs. Moreover, as mentioned in the last paragraph in Section 2.2, our design does not suffer the extra computation usually caused by iMAML because the original supervised learning problem (1) is already bi-level and we need to fully the lower-level problem anyway.

In conclusion, meta-learning (M-ICRL) and supervised learning (ICRL) study totally different problems. Compared to ICRL, M-ICRL requires special designs in order to learn good meta-priors. In our work, we include special designs to solve meta-learning, and thus have a different structure from ICRL.

We now address your comments.

Weakness 1: [D1] has proposed a bi-level optimization objective for ICRL. This paper expands upon that proposal, adapting the objective to fit within a meta-learning context. The primary innovation is the integration of a meta-prior into the initial objective (see objective (2)). The algorithm presented outlines a direct, gradient-based optimization method, which, unfortunately, is computationally infeasible under most circumstances. The core focus of this paper is on mitigating this computational difficulty through the implementation of three levels of approximation. Additionally, it explores the impact of these approximations on both convergence and generalization. However, it should be noted that this study deviates somewhat from the original goal of Meta ICRL. Moreover, the advancements it provides over [D1] could be considered somewhat marginal.

Answer: Thanks for mentioning the focus of this paper and the advancement over ICRL [D1].

For the comment on deviation from meta-learning, we respectfully disagree with it. There are three major reasons.

First, let us assume that we can obtain the exact hyper-gradient and ignore the approximation of the hyper-gradient. The other parts of this paper focus on the original goal of meta-learning. In specific, the original goal of meta-learning [D7,D8,D9], including meta-RL [D11], is to learn good meta-priors such that the corresponding task-specific adaptations can perform well on new tasks even if there is only limited data in new tasks. Along the same line, we extend the problem formulation of meta-learning to meta-ICRL using the bi-level optimization problem (2)-(3). More importantly, we theoretically analyze the convergence and generalization of the algorithm. The convergence and especially the generalization are the focus of meta-learning because the goal of meta-learning is to generalize to new tasks from limited data [D7,D8,D9]. In fact, these two properties have been widely studied in meta-learning [D8,D10,D12,D13], including meta-RL [D11].

Second, the approximation part is essential to solve the problem (2)-(3) which models meta-ICRL. In specific, solving problem (2)-(3) requires to compute the hyper-gradient. However, it is challenging because the hyper-gradient includes the inverse-of-Hessian term and the gradient of the inverse-of-Hessian term (as detailed in Section 3.1). We propose novel approximation methods to address these challenges.

Third, we would like to mention that recent papers [D8,D10,D11,D12,D13] perform theoretic analysis of meta-learning algorithms subject to approximation errors of the hyper-gradient caused by practical implementation. For example, [D10] studies the convergence of MAML [D7] and studies two methods to efficiently approximate the Hessian term in the hyper-gradient. It also quantifies the approximation error and convergence.

For the advancements over ICRL [D1], we would like to notify that M-ICRL studies a totally different problem from ICRL, instead of advancing the methodology of ICRL. The focus of M-ICRL is the meta-learning part instead of the ICRL part. M-ICRL only utilizes ICRL as a task-specific solver, however, the focus of M-ICRL is how to learn good meta-priors and the generalization performance. In order to learn meta-priors, we propose special designs in (2)-(3) and these special designs introduce unique challenges in algorithm design and theoretical analysis. The unique challenges in algorithm design, as mentioned in Section 3.1, are the difficulties of approximating the hyper-gradients. The unique challenges in theoretical analysis have two parts. First, we study generalization while [D1] does not. Second, we need to quantify the approximation errors and their impact on the convergence while [D1] does not.

Thanks for your detailed and constructive reviews. We appreciate that you think this paper well-written and we believe that our discussion can lead to a better paper in general. Before addressing your comments, we would like to first discuss (i) the significant difference between RL and IRL, and thus why regret and sample complexity are not typically studied in IRL; (ii) the fundamental spirit of meta-learning.

RL focuses on policy learning where the agent learns in an online style and requires sampling in the environment to collect the training data $(s,a,r)$. It is typical to use regret and sample complexity as the theoretical metric to analyze RL algorithms. However, IRL focuses on reward learning and the reward learning is a supervised learning problem where the training data is pre-collected and an offline optimization problem is formulated. Therefore, the regret is not a standard metric for IRL.

Indeed, IRL also has a policy learning part, i.e, it usually needs to use RL to compute the policy corresponding to the learned reward function, and analyzing sample complexity is important to RL. However, IRL typically assumes the access to an RL oracle to get the policy corresponding to the learned reward function [D2,D4], so that how the policy is generated by the RL oracle is not the major concern in IRL. Therefore, it is not typical to use regret nor sample complexity to analyze IRL algorithms. In IRL, people focus on the reward learning part, i.e., how the offline optimization problem is solved. Therefore, analyzing the algorithm convergence is standard in IRL, either quantifying the loss function value or distance between the learned parameter and optimal parameter in convex cases [D2,D3] or quantifying the upper bound of the gradient norm in non-convex cases [D1,D5,D6]. We agree that evaluating the regret and sample complexity of the (RL) policy learning part in the IRL framework is interesting and significant, however, at current stage, the IRL theoretical studies focus more on the reward learning part instead of the (RL) policy learning part. In the future, we will explore more on the policy learning part.

Now, we talk about the fundamental spirit of meta-learning. The standard supervised learning aims to learn good parameters from training data such that the model with the learned parameters can perform well on the test data. In contrast, meta-learning proposes to learn the ability to learn. It deals with multiple tasks. In specific, meta-learning wants to learn a meta-prior instead of specific parameters for specific tasks [D7,D8]. The meta-prior is learned from related tasks and is learned in the way that the task-specific adaptations adapted from the meta-prior can have good performance on the corresponding related tasks even if there is only limited data in the corresponding related tasks. Therefore, compared to standard supervised learning, meta-learning studies a different problem and requires special designs in order to learn the meta-priors.

In our case, ICRL [D1] is just a supervised learning problem that optimizes for the reward and cost parameters for a single task. M-ICRL deals with multiple tasks. M-ICRL aims to learn the reward and cost meta-priors such that the task-specific reward and cost adaptations adapted from the reward and cost meta-priors can have good performance on specific tasks. Meta-learning [D7,D8,D9] usually has a bi-level learning structure where the upper level is to learn the meta-prior and the lower level aims to learn the task-specific adaptations. How to design the task-specific adaptation is important in meta-learning. A naive way is just solving the supervised learning problem on a specific task where the initialization is the meta-prior, and then use the result as the task-specific adaptation. However, this kind of design is problematic because we only have limited data to compute the task-specific adaptation, and fully solving the supervised learning problem can lead to overfitting. To solve this issue, the fundamental idea of meta-learning is to use regularization to avoid overfitting. Currently, there are two major types of designs. The first one, called MAML, is to compute the task-specific adaptation via one-step gradient descent [D7,D9]. This is how we compute the task-specific reward adaptation $\varphi_i=\theta-\alpha\frac{\partial}{\partial\theta}L_i(\theta,\eta_i^{\ast}(\theta,\omega))$. This method can be regarded as a regularization [D8] because it restricts the task-specific adaptation $\varphi_i$ from being far from the meta-prior $\theta$. The second one, called iMAML, is to directly use a regularization term [D8,D9]. This is how we compute the task-specific cost adaptation $\eta_i^{\ast}(\varphi_i,\omega)=\mathop{\arg\min}G_i(\eta;\varphi_i)+\frac{\lambda}{2}||\eta-\omega||^2$. Directly adding the regularization term usually has better empirical results than one-step gradient descent [D8], however,

to avoid overfitting using early stop as a regularization (as detailed at the beginning of this response, before answering Weakness 1). This style of computing the task-specific adaptation is called MAML [D7]. It is first introduced in [D7] and then widely adopted in meta-RL [D11] and meta-IRL [D14,D15]. The convergence [D10] and generalization [D12] of MAML are both well studied. It is actually one of the two predominant meta-learning methods. The other one is iMAML [D8].

Weakness 6: My final concern pertains to the intended audience of this paper. The manner in which the content is presented seems to diverge from the primary interests of the mainstream Reinforcement Learning (RL) and Machine Learning (ML) community. This paper primarily focuses on the approximation of computationally intractable gradients and the subsequent implications for convergence and generalization. While these advancements are valid, they may not be entirely consistent with the broader ML community. Furthermore, it's unclear how subsequent work can leverage and benefit from these methods. In my view, the style of this paper aligns more closely with the interests of the optimal control and robotics community. Subjectively speaking, ICLR may not be the most appropriate venue for this paper.

Answer: Thanks for mentioning the intended audience of this paper. We believe that the paper studies the core problems of meta-learning. Please refer to our answer to the comment on deviation from meta-learning in weakness 1. Here we would like to further mention that the approximation of hyper-gradient is consistent with the interest in many other machine learning problems.

In the IRL community, [D1,D5] explicitly formulate the IRL problem as a bi-level optimization problem and they all propose novel approximation methods to efficiently approximate the hyper-gradient (see the "local gradient approximation" in [D1] and Lemma 1 in [D5]). Since IRL usually has a bi-level learning structure where the upper level learns the reward function and the lower level learns the corresponding policy, it is of interest to efficiently compute the hyper-gradient (i.e., gradient of the upper-level problem) to update the learned reward function.

In the bi-level optimization community [D21,D22,D23], approximating the hyper-gradient is one of the fundamental research interests since the hyper-gradient is needed to solve the upper-level problem and is difficult to compute in general. For example, the contribution of [D22] is "comprehensive convergence rate analysis for two popular algorithms respectively based on approximate implicit differentiation (AID) and iterative differentiation (ITD)" where AID and ITD are two methods to approximate the hyper-gradient.

Moreover, other works can build on our paper since they can use some of our proposed approximation methods. For example, in our paper, we propose to solve an optimization problem in order to approximate the inverse-of-Hessian term. iMAML also needs to approximate an inverse-of-Hessian term, however, they just assume that they can approximate the inverse-of-Hessian term with $\delta$-accuracy without theoretical guarantees. In contrast, our work provides a principled way with theoretical guarantees to ensure that the approximation error can be arbitrarily small (proved in Lemma 1).

Weakness 7: The environmental parameters in the MuJoCo experiments raise some questions. The paper states (see Appendix B.2) that "The design of the reward function is such that the robots will receive a reward of +1 if they maintain the target velocity and a reward of 0 otherwise." and "For the constraint, the Swimmer experiment constrains all the states where the front tip angle exceeds $a_0$, where $a_0 \in [0.9, 1.2]$." These constraints do not appear to be significant. In particular, it's unclear why the agent would need to violate the constraint to maximize cumulative rewards, and whether maintaining the correct speed is a suitable reward for a MuJoCo task. More justification for these settings is required.

Answer: Thanks for mentioning the design of reward and constraint. For the design of reward, the motivation of maintaining the target velocity is that it has wide applications in the real world, such as autonomous vehicle driving at a constant speed [D19] and robotic dog skating at a constant velocity [D20]. Moreover, maintaining target velocity in Mujoco is a standard benchmark in meta-RL [D7,D16,D17] and meta-IRL [D18]. Since meta-learning needs multiple related tasks, maintaining different target velocity can be considered as multiple related task.

For the design of the constraint in the Swimmer case, we use this kind of constraints because we empirically find that the angle of Swimmer's front tip will typically exceed 1.2 if there is no constraint. We choose this kind of constraint to show that our algorithm can indeed learn the constraint and avoid violating the constraint.

Weakness 2: Theorem 1 sets an upper bound on the expected gradient of the exact optimal solution. When compared to other more frequently used results, such as the upper bound of regret or the sample complexity in the PAC (Probably Approximately Correct) analysis, this result doesn't appear particularly robust.

Answer: Thanks for mentioning the convergence. We agree that regret and sample complexity are typically analyzed in RL. However, as discussed above, current IRL theoretical studies focus on the reward learning part instead of the (RL) policy learning part. The reward learning is a supervised learning problem where the data is pre-collected and an offline optimization problem needs to be solved. Therefore, analyzing the algorithm convergence is standard in IRL, either quantifying the loss function value or distance between the learned parameter and optimal parameter in convex cases [D2,D3] or quantifying the upper bound of the gradient norm in non-convex cases [D1,D5,D6]. We agree that using PAC to analyze regret and sample complexity is significant and interesting for the (RL) policy learning part in the IRL framework, and we would like to explore more in the future.

Weakness 3: It is surprising to note that the upper bound is raised to the power of $K$ (the number of gradient steps), rather than being linear or sublinear with respect to $K$. This implies that the approximation error could accumulate rapidly with each gradient step. Unfortunately, the paper does not provide a satisfactory explanation for this issue.

Answer: The approximation error exponentially decreases rather than exponentially increases in our case. Note that $\frac{C_{\nabla^2_{\eta\eta}G}}{\lambda}\in (0,1)$, therefore the approximation error in Lemma 1 and Lemma 2 exponentially decreases when the number of $K$ increases, which is a virtue.

Weakness 4: I question the assertion that this upper bound can be regarded as an $\epsilon$-approximate first-order stationary point. While the construction of $\epsilon$ being proportional to $\sqrt{1/N}$ is reasonable, the inclusion of an additional Big O term seems less comprehensible. It is my belief that the approximations exert a significant influence on convergence, which should not be overlooked.

Answer: We agree that the approximation errors can exert a significant influence on convergence, however, this upper bound can be regarded as an $\epsilon$-approximate first-order stationary point for two reasons. First, all the terms in the big O can be arbitrarily small if we choose large enough $K$, $\bar{K}$, $D_i^{\text{tr}}$, and small enough $\delta$. Second, this is a standard practice in meta-learning convergence theory papers [D10,D11]. For example, [D10] also studies $\epsilon$-approximate first-order stationary point where the gradient norm is upper bounded by $\epsilon+O(\cdot)$ and the error terms in the big O can be arbitrarily small.

Weakness 5: In [D1], the bi-level optimization framework of ICRL is structured such that the upper-level problem was focused on reward learning, while the lower-level problem addressed constraint learning. In this current paper, the upper-level problem has been expanded to include the learning of meta-priors for both rewards and constraints ($\theta$ and $\omega$). This shift deviates from the original bi-level optimization framework of ICRL, raising some questions about its consistency. Moreover, the paper proposes that the task-specific reward adaptation is carried out via a gradient update. However, it is unclear whether this is a well-defined adaptation for specific tasks. There is a concern that this adaptation may be inadequate. It would be beneficial to provide additional supporting evidence. This could include referencing relevant papers where similar strategies have been successfully applied.

Answer: Thanks for mentioning the meta-prior and task-specific adaptation. We agree that there is a deviation from the original ICRL problem in terms of learning the cost meta-prior in the upper level, however, we still learn the task-specific cost adaptation $\eta_i^{\ast}$ in the lower level. In fact, this deviation is necessary and standard in meta-learning. Note that in our problem formulation (2)-(3), we learn the reward meta-prior $\theta$ and cost meta-prior $\omega$ in the upper level, and the task-specific cost adaptation $\eta_i^{\ast}$ in the lower-level. This is exactly the standard practice of meta-learning [D7,D8,D9] where the upper level is to learn the meta-prior and the lower level is to learn the task-specific adaptations. Here, we would like to emphasize again that meta-learning (M-ICRL) and supervised learning (ICRL) study different problems and thus it is expected that meta-learning problem has special designs.

The task-specific reward adaptation $\varphi_i$ is computed via one-step gradient descent from the reward meta-prior $\theta$. The intuition behind this is

Thanks for responding. I am indeed surprised by the length of the response. It looks nice, but It takes me a while to read these paragraphs. Some of concerns are resolved, and my further questions are:

• When I mentioned "However, it should be noted that this study deviates somewhat from the original goal of Meta ICRL", when I saw the title, I was expecting to see an advancement of ICRL. but from the paper and the response, most of the advancement has nothing to do with the ICRL, but about how to resolve the meta-prior's computational difficulty with three levels of approximation.

• Some of the concerns (Weakness 4 and 5) about the paper are interpreted as standard practice in Meta-RL. Since my expertise lies in RL rather than meta-learning, I find this somewhat surprising, but I'm unable to comment further on it. From my perspective, this paper primarily focuses on extending the meta-Learning strategies to ICRL, rather than on how to solve ICRL problems within the context of meta-learning.

• I am not sure about the responses "current IRL theoretical studies focus on the reward learning part instead of the (RL) policy learning part" and "The reward learning is a supervised learning problem where the data is pre-collected and an offline optimization problem needs to be solved.". Please refer to the following theoretical works about IRL. This line of works all talk about sample complexity or regrets.

Metelli, A. M., Ramponi, G., Concetti, A., and Restelli, M. "Provably efficient learning of transferable rewards". In International Conference on Machine Learning (ICML), 2021.

Lindner, David, Andreas Krause, and Giorgia Ramponi. "Active exploration for inverse reinforcement learning." Advances in Neural Information Processing Systems (NeurIPS) 2022

Metelli, Alberto Maria, Filippo Lazzati, and Marcello Restelli. "Towards Theoretical Understanding of Inverse Reinforcement Learning." In International Conference on Machine Learning (ICML), 2023.

• I have a further concern regarding the implementation. Notably, the authors of DICRL have decided not to release their code, leading to subsequent works encountering difficulties when comparing with them, which was disappointing to me for a while. I'm interested to know how you managed to compare your work with DICRL. Did you implement their algorithm independently? If so, could you provide guidance on how to locate this implementation?

• Specifically, within the submitted code, it appears that the algorithm depends on SAC, which was not mentioned in the paper. Additionally, the learn.py function does not seem to include the cost inference component.

• Furthermore, do the authors intend to make their code publicly available in the future? I'm raising this question because, once your paper is published, subsequent studies are likely to be required to compare their methods with yours, particularly since your paper introduces an algorithm rather than proposing a theory. Given the complexity of your paper, I foresee that most followers may encounter difficulties in implementing your method.

Thanks for your constructive reviews. We believe that our discussion can improve this paper in general. We address your comments below:

Weakness 1: The paper is an extension of meta IRL and ICRL. Building upon these, the contribution of the study is not significant.

Answer: Thanks for mentioning the novelty. Though our problem combines ICRL in [C1] and meta-learning in [C2,C4], our algorithm design and theoretic analysis are substantially different from those in the three references. We discuss the challenges of algorithm design in Section 3.1 and the challenges of theoretical analysis in the first paragraph of Section 4.1. Here we would like to further discuss our unique challenges and novelties in algorithm design and theoretical analysis compared to [C1,C2,C4].

Algorithm design. [C1,C2,C4] and our work all formulate a bi-level optimization problem. Note that [C2] uses MAML [C3] and MAML can be regarded as a bi-level optimization problem where the lower level is one-step gradient descent from the meta-prior and the upper level aims to optimize for the meta-prior [C4]. A critical step of solving bi-level optimization problems is to compute the hyper-gradient, i.e., the gradient of the upper-level problem. The methods of computing the hyper-gradient in [C1,C2,C4] CANNOT be applied to our case and thus novel algorithm designs for computing the hyper-gradient are needed. In specific, there are two unique challenges in terms of algorithm design.

First, our hyper-gradient includes the term $\frac{\partial^2}{\partial\theta^2}L_i(\theta,\eta_i^{\ast}(\theta,\omega))$ while [C1,C2,C4] do not have this term. As mentioned in challenge (i) in Section 3.1, computing $\frac{\partial^2}{\partial\theta^2}L_i(\theta,\eta_i^{\ast}(\theta,\omega))$ requires to compute the gradient of an inverse-of-Hessian term $[\nabla_{\eta\eta}^2G_i(\eta_i^{\ast}(\varphi_i,\omega);\varphi_i)+\lambda I]^{-1}$. Since we use neural networks as the parameterized model, computing this inverse-of-Hessian term is already challenging enough and now we need to compute the gradient of this inverse-of-Hessian term, which is intractable. To solve this issue, we propose a novel approximation design where we only need to compute first-order gradients (see (6)-(7) and Algorithm 2) and we can prove that the approximation error can be arbitrarily small (Lemma 2).

Second, as mentioned in challenge (ii) in Section 3.1, our hyper-gradient includes the inverse-of-Hessian term $[\nabla_{\eta\eta}^2G_i(\eta_i^{\ast}(\varphi_i,\omega);\varphi_i)+\lambda I]^{-1}$ while [C1,C2] do not have this issue. [C4] also has this issue, however, they just use existing conjugate gradient method and assume that a good approximation can be obtained. In contrast, we propose to solve an optimization problem (10) in Algorithm 3 to approximate and more importantly, we theoretically guarantee in Lemma 1 that the approximation error can be arbitrarily small.

Theoretical analysis. We study BOTH the convergence and generalization of our algorithm while [C2] do not have theoretical analysis and [C1,C4] only study convergence. Therefore, the whole Section 4.2 is unique and novel compared to [C1,C2,C4] because it studies generalization. Even for convergence, our analysis is novel compared to [C1,C4]. In specific, since we have the unique challenges for algorithm design and we provide novel algorithm designs (i.e., approximation methods) to tackle those unique challenges, we need corresponding theoretical analysis to guarantee the performance of our unique algorithm designs, i.e., Lemma 1 and Lemma 2 quantifies the approximation error of our algorithm designs.

Weakness 2: The paper only shows the convergence to $\epsilon$-FOSP, whose optimality is not discussed. Convergence of gradient-based algorithms to stationary point is not a novel contribution.

Answer: Thanks for mentioning the convergence. We agree that convergence to global optimal solutions is very interesting, however, it is quite challenging in our case because our problem is highly non-convex. Note that our problem is still non-convex even if we use linear parameterized models instead of neural networks [C1]. We notice that some bi-level optimization works [C9,C10] have studied the convergence to the global optimal solutions, however, they all impose strong assumptions, e.g., the upper-level objective function is either convex or strongly convex. These assumptions are not satisfied in our case.

In fact, the convergence to $\epsilon$-FOSP is not trivial in our case. Convergence of SGD to stationary points has been widely studied, however, our novelty is beyond the convergence of SGD. As mentioned in the first paragraph of Section 4.1, compared to the standard SGD, the main difficulty of guaranteeing the convergence of Algorithm 1 lies in quantifying the approximation error of the hyper-gradients and how the errors affect the convergence. Note that we do not have the exact hyper-gradient, thus our algorithm is not standard SGD. The most difficult part of solving problem (2)-(3) is how to approximate the hyper-gradient and how to ensure that the approximation error does not ruin the convergence. Therefore, our major contribution of convergence is to design novel approximation methods in Section 3.2 and theoretically guarantee that the proposed approximation method can lead to convergence in Section 4.1. Besides convergence, we also study the generalization in Section 4.2.

Weakness 3: Assumption 1 assumes that the reward function and the cost function are bounded and smooth up to the fourth order gradient, which is a strong assumption, especially for neural networks with ReLU activation and unbounded state-action space.

Answer: Thanks for mentioning the assumption. We agree that these assumptions could be strong when the neural networks use ReLU activation. However, the smoothness assumption can be satisfied by using some smooth activation functions such as tanh and using linear parameterized models. Moreover, as mentioned in the paragraph under Assumption 1, these assumptions are typical assumptions in RL [C5], IRL [C6,C8] and meta-RL [C7]. It is interesting to relax these assumptions, and we will explore it in the future.

References

[C1] Shicheng Liu and Minghui Zhu. "Distributed inverse constrained reinforcement learning for multi-agent systems.“ In Advances in Neural Information Processing Systems, pp. 33444–33456, 2022.

[C2] Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, and Chelsea Finn. "Learning a prior over intent via meta-inverse reinforcement learning.” In International Conference on Machine Learning, pp. 6952–6962, 2019.

[C3] Chelsea Finn, Pieter Abbeel, and Sergey Levine. "Model-agnostic meta-learning for fast adaptation of deep networks." In International Conference on Machine Learning, pp. 1126–1135, 2017.

[C4] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. "Meta-learning with implicit gradients.“ In Advances in Neural Information Processing Systems, pp. 113–124, 2019.

[C5] Kaiqing Zhang ,Alec Koppel, Hao Zhu, and Tamer Basar. "Global convergence of policy gradient methods t o(almost) locally optimal policies." SIAM Journal on Control and Optimization, 58(6): 3586–3612, 2020.

[C6] Ziwei Guan, Tengyu Xu, and Yingbin Liang. "When will generative adversarial imitation learning algorithms attain global convergence." In International Conference on Artificial Intelligence and Statistics, pp. 1117–1125, 2021.

[C7] Alireza Fallah, Kristian Georgiev, Aryan Mokhtari, and Asuman Ozdaglar. "On the convergence theory of debiased model-agnostic meta-reinforcement learning." In Advances in Neural Information Processing Systems, pp. 3096–3107, 2021.

[C8] Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. "Maximum-likelihood inverse reinforcement learning with finite-time guarantees." Advances in Neural Information Processing Systems, pp. 10122-10135, 2022.

[C9] Saeed Ghadimi, and Mengdi Wang. "Approximation methods for bilevel programming." arXiv preprint arXiv:1802.02246, 2018.

[C10] Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. "A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic." SIAM Journal on Optimization 33, no. 1, pp. 147-180, 2023.

Thanks for your insightful reviews. We believe that our discussion can lead to a stronger paper. We address your comments as follows:

Weakness 1: The lower level optimization (as explained in Equation 11, Appendix A.1) maximizes the reward and causal entropy objective while matching the constraint feature expectations. On the other hand, the authors of [B1] formulate the optimization as a maximization of the causal entropy while matching reward feature expectations and expected cumulative costs. The slight difference is that [B1] has an additional Lagrange multiplier term $\lambda$ in the dual formulation (see [B1], page 4, below remark 2, in the definition of $G$). In this work, since the constraint feature expectations are directly matched, the overall form of the lower level objective (Appendix, equation 12) is not exactly the same as [B1]. This is also apparent in the constrained soft Bellman policy definition adapted from [B1] (no $\lambda$ in the constrained policy for this work, whereas there is a $\lambda$ in the constrained policy as described in [B1], page 2, Appendix). The outcome is that the constraint function is just treated as a negative reward term that can be just added to the original reward and thus, constrained RL just amounts to running RL with reward $r-c$ (Appendix A.2 of this work also says this). I am not sure this is representative of typical constrained RL problems, since typically it is not possible to rewrite a constrained RL problem as an unconstrained RL problem with a different reward. (maybe the authors can elaborate on this).

Answer: It is very insightful that you point this out. In fact, the definition of $G$ in Appendix A.1 is equivalent to the one in [B1] even if we do not have $\lambda$, and the definition of $G$ in Appendix A.1 can work for constrained RL in our case. Note that the cost function $c_{\omega}$ in our problem is not given nor fixed, but is actually learned. This point is very important for understanding why the slight difference of $\lambda$ does not matter and why the $r_{\theta}-c_{\omega}$ can work for constrained RL in our case.

First, we would like to explain why the function $G$ in Appendix A.1 is equivalent to the function $G$ in [B1]. The definition of $G$ in Appendix A.1 is $G(\omega;\theta)=H(\pi_{\omega;\theta})+J_{r_{\theta}}(\pi_{\omega;\theta})+\omega^{\top}(\mu_c(\pi_i)-\mu_c(\pi_{\omega;\theta}))$ where $\pi_{\omega;\theta}(a|s)=\frac{\exp(r_{\theta}(s,a)-\omega^{\top}\phi_c(s,a)+\gamma\int_{s'\in\mathcal{S}}P(s'|s,a)V_{\omega;\theta}^{\text{soft}}(s')ds')}{\exp(V_{\omega;\theta}^{\text{soft}}(s))}$. The definition of function G in [B1] (let us call it $\bar{G}$) is $\bar{G}(\bar{\omega};\theta,\lambda)=H(\pi_{\bar{\omega};\theta,\lambda})+J_{r_{\theta}}(\pi_{\bar{\omega};\theta,\lambda})+\lambda\bar{\omega}^{\top}(\mu_c(\pi_i)-\mu_c(\pi_{\bar{\omega};\theta,\lambda}))$ where $\pi_{\bar{\omega};\theta,\lambda}(a|s)=\frac{\exp(r_{\theta}(s,a)-\lambda\bar{\omega}^{\top}\phi_c(s,a)+\gamma\int_{s'\in\mathcal{S}}P(s'|s,a)V_{\bar{\omega};\theta,\lambda}^{\text{soft}}(s')ds')}{\exp(V_{\bar{\omega};\theta,\lambda}^{\text{soft}}(s))}$ (section 8 in Appendix of [B1]). Note that $\lambda$ is just a scalar and $\bar{\omega}$ is a vector ([B1] studies linear cost). In [B1], both $\lambda$ and $\bar{\omega}$ are learned. In our problem, $\omega$ is learned. For any learned $\lambda$ and $\bar{\omega}$, we can always learn an $\omega$ such that $\omega=\lambda\bar{\omega}$. Therefore, $G$ in our case is equivalent to $\bar{G}$ in [B1] as long as we simply replace $\lambda\bar{\omega}$ in $\bar{G}$ with $\omega$. Since the characteristics of $\lambda\bar{\omega}$ can be easily captured by a single $\omega$, there is no need to learn two parameters $\lambda$ and $\bar{\omega}$.

Second, we explain why the $r_{\theta}-c_{\omega}$ can work for the constrained RL in our case. We agree that it is not possible to rewrite a constrained RL problem as an unconstrained RL problem. However, we do not aim to use $r_{\theta}-c_{\omega}$ to solve a constrained RL problem where the reward function is $r_{\theta}$ and the cost function is $c_{\omega}$. What we do is to learn $r_{\theta}$ and $c_{\omega}$ such that the corresponding policy $\pi_{\omega;\theta}$ can imitate the expert and thus solve the original constrained RL problem where the reward function is $r_i$ and cost function is $c_i$. The learned cost function $c_{\omega}$ could be learned as the expert's cost function $c_i$ multiplying a very large positive constant so that the $r_{\theta}-c_{\omega}$ can enable $\pi_{\omega;\theta}$ to approximately satisfy the original constraint which is defined using $c_i$. The situation in [B1]

Thanks for your constructive reviews! We appreciate that you find this work creative and we believe that our discussion can contribute to a better paper in general. We address your comments as follows:

Q1: In terms of novelty, it would be helpful for readers if authors can emphasize the unique novelty and challenges of solving meta inverse constrained RL beyond simply combining the techniques in the inverse constrained RL problem studied in [A1] and the meta RL/ICRL studied in [A2,A3]?

Answer: Thanks for mentioning the novelty. Though our problem combines ICRL in [A1] and meta-learning in [A2,A3], our algorithm design and theoretic analysis are substantially different from those in the three references. We discuss the challenges of algorithm design in Section 3.1 and the challenges of theoretical analysis in the first paragraph of Section 4.1. Here we would like to further discuss our unique challenges and novelties in algorithm design and theoretical analysis compared to [A1,A2,A3].

Algorithm design. [A1,A2,A3] and our work all formulate a bi-level optimization problem. Note that [A2] uses MAML [A7] and MAML can be regarded as a bi-level optimization problem where the lower level is one-step gradient descent from the meta-prior and the upper level aims to optimize for the meta-prior [A3]. A critical step of solving bi-level optimization problems is to compute the hyper-gradient, i.e., the gradient of the upper-level problem. The methods of computing the hyper-gradient in [A1,A2,A3] CANNOT be applied to our case and thus novel algorithm designs for computing the hyper-gradient are needed. In specific, there are two unique challenges in terms of algorithm design.

First, our hyper-gradient includes the term $\frac{\partial^2}{\partial\theta^2}L_i(\theta,\eta_i^{\ast}(\theta,\omega))$ while [A1,A2,A3] do not have this term. As mentioned in challenge (i) in Section 3.1, computing $\frac{\partial^2}{\partial\theta^2}L_i(\theta,\eta_i^{\ast}(\theta,\omega))$ requires to compute the gradient of an inverse-of-Hessian term $[\nabla_{\eta\eta}^2G_i(\eta_i^{\ast}(\varphi_i,\omega);\varphi_i)+\lambda I]^{-1}$. Since we use neural networks as the parameterized model, computing this inverse-of-Hessian term is already challenging enough and now we need to compute the gradient of this inverse-of-Hessian term, which is intractable. To solve this issue, we propose a novel approximation design where we only need to compute first-order gradients (see (6)-(7) and Algorithm 2) and we can prove that the approximation error can be arbitrarily small (Lemma 2).

Second, as mentioned in challenge (ii) in Section 3.1, our hyper-gradient includes the inverse-of-Hessian term $[\nabla_{\eta\eta}^2G_i(\eta_i^{\ast}(\varphi_i,\omega);\varphi_i)+\lambda I]^{-1}$ while [A1,A2] do not have this issue. [A3] also has this issue, however, they just use existing conjugate gradient method and assume that a good approximation can be obtained. In contrast, we propose to solve an optimization problem (10) in Algorithm 3 to approximate and more importantly, we theoretically guarantee in Lemma 1 that the approximation error can be arbitrarily small.

Theoretical analysis. We study BOTH the convergence and generalization of our algorithm while [A2] do not have theoretical analysis and [A1,A3] only study convergence. Therefore, the whole Section 4.2 is unique and novel compared to [A1,A2,A3] because it studies generalization. Even for convergence, our analysis is novel compared to [A1,A3]. In specific, since we have the unique challenges for algorithm design and we provide novel algorithm designs (i.e., approximation methods) to tackle those unique challenges, we need corresponding theoretical analysis to guarantee the performance of our unique algorithm designs, i.e., Lemma 1 and Lemma 2 quantifies the approximation error of our algorithm designs.