Adversarial Inverse Reward-Constraint Learning with Reward-Feasibility Contrast Prior Inspired by Animal Behaviour

1. The main weakness lies in the problem itself, i.e., learning both reward and constraint. This problem seems unnecessary. In specific, problem (6) shows that both reward $r_{\theta}$ and constraint $\bar{\delta}_{\phi}$ are learned by optimizing $E\_{\tau\sim D\_E}[r\_{\theta}(\tau)+\log \bar{\delta}\_{\phi}(\tau)]-\log Z\_{\theta,\phi}$. Why we cannot learn a single neural model $m\_{\psi}$ such that $m\_{\psi}(\tau)=r\_{\theta}(\tau)+\log \bar{\delta}\_{\phi}(\tau)$, and call $m\_{\psi}$ reward or constraint or whatever. The point here is that problem (6) can be captured by standard IRL (that only learns reward) and standard ICRL (that only learns constraint) given that the neural model $m\_{\psi}$ can capture $r\_{\theta}(\tau)+\log \bar{\delta}\_{\phi}(\tau)$ due to the universal approximation property. In that sense, I cannot get a theoretical (or even intuitive) idea of why learning both reward and constraint is necessary. Or in other words, what is the fundamental difference between learning a single $m\_{\psi}$ and learning $r\_{\theta}$ and $\log \bar{\delta}\_{\phi}$. Can the authors provide some evidence, either theoretical or intuitive, to explain this? I acknowledge that there are some papers learning both reward and constraints. However, those papers do not explain the difference between learning a single model and learning both reward and constraint either. Before solving the problem of learning both reward and constraints, the problem itself needs to be first justified.

I would like to increase my rating if the authors can solve weakness 1 because weakness 1 is a very fundamental problem to this paper.

2. To solve the Lagrangian (10), the authors directly set the dual variable $\lambda=1$ for the sake of efficiency and simplification. I agree that ignoring $\lambda$ can make the problem easier to solve. However, this can be potentially incorrect. Usually, we cannot solve a constrained optimization problem by simply penalizing the constraint in the objective function, because the resulting policy may no longer satisfy the constraint. In this case, can the authors guarantee that the learned policy (when $\lambda=1$) satisfy the constraint, i.e., $E\_{\tau\sim \pi_{\omega}}[\bar{\delta}_{\phi}(\tau)]\leq\alpha$?

3. A limitation of this paper is that it can only solve for hard constraints because $\delta(\tau)$ is binary. There are some real-world applications that align with hard constraints, e.g., collision avoidance. However, there are also many soft-constraint scenarios. When the constraints are soft, can we easily revise this paper to solve for soft constraints?

1. Insufficient Comparison and Discussion. References [1,2,3] also infer both reward and cost functions, yet these works are not included in the current discussion. I recommend that the authors provide a more detailed comparison between their approach and the methods described in [1,2,3], highlighting key differences in methodology, performance, and applicability.
2. Lack of Evidence for Main Contributions.

• The authors claim that AIRCL demonstrates strong data and training efficiency; however, the paper lacks evidence to support this claim. Specifically, there is no theoretical complexity analysis or experimental metrics provided to demonstrate faster convergence rates or lower sample complexity. Addressing this gap would strengthen the paper’s argument.
• While the reward-feasibility contrast prior is presented as a central contribution of this work, its impact is not sufficiently demonstrated through the experiments. The current ablation study (with and without the prior) is too simplistic. I suggest the authors include a more thorough analysis, such as a correlation study between rewards and costs, to better validate the prior's effectiveness.

3. Limited Applicability Due to Hard-Constraint Assumptions. The authors’ focus on hard-constraint scenarios may restrict the broader applicability of their method, particularly in practical settings where soft-constraint scenarios are more prevalent. Expanding the framework to handle soft constraints would enhance its relevance to a wider range of real-world problems.

1. The primary concern lies with the reward-feasibility contrast prior, despite it shows effectiveness in the evaluated tasks. It is derived from animal behaviors and has not been tested in more general scenarios where high reward and low constraints align, which frequently happens in certain industrial applications.

2. Additionally, while the experimental evaluation involves simulated robot tasks and a meerkat behavior dataset, the framework’s capacity to manage uncertainties - i.e., an aspect crucial to real-world applications - remains untested. Even if the prior holds, uncertainties could lead to unexpected expert demonstrations and constraint violations, which is a more realistic scenario and may challenge the method's reliability. See Question 2 for further details.

3. It would be great to clarify in theoretical formulation, e.g., a more detailed clarification of the gradients (why can we have Equation 14?). Please also see Question 3.

The attempt to analyze meerkat data and apply insights from the analysis to this algorithm was good. However, the study fails to explain the analysis results sufficiently and logically. Please refer to the questions for detailed weaknesses.