Action-Constrained Imitation Learning

We sincerely appreciate the reviewer’s insightful comments and suggestions. We provide our point-by-point response as follows.

The author should make ablation results about the number of expert demos, and also discuss the different levels of action restrictions for the imitator.

Based on the reviewer’s suggestions, we conducted ablation studies on both the number of expert demonstrations and different levels of action restrictions for the imitator, testing these across two distinct environment settings. The results are shown in the tables below. Our findings demonstrate that our method remains strong under varying constraint levels and can generalize well to datasets with fewer expert demonstrations.

Table 1: Ablation Study on Different Expert Demos

Table 2: Ablation Study on Different Constraint Levels

The experiment on robotics is table-wiping, which is too simple. The authors are encouraged to conduct some manipulation tasks, at least picking and placing tasks.

We appreciate the reviewer's suggestion. However, we have observed that existing and even state-of-the-art ACRL works have not tackled robotic arm tasks either [1-3].Pick-and-place tasks are indeed more complex due to higher-dimensional action spaces and additional constraints such as grasp stability and object interactions[4]. Since this work is the first step in addressing ACIL, we focus on simpler settings and leave the challenge of extending ACIL to more complex robotic manipulation tasks for future work.

[1] Janaka Brahmanage, Jiajing Ling, and Akshat Kumar, “FlowPG: Action-constrained policy gradient with normalizing flows,” NeurIPS 2023.

[2] Kazumi Kasaura, Shuwa Miura, Tadashi Kozuno, Ryo Yonetani, Kenta Hoshino, and Yohei Hosoe, “Benchmarking actor-critic deep reinforcement learning algorithms for robotics control with action constraints,” Robotics and Automation Letters, 2023.

[3] Hung, Wei, Shao-Hua Sun, and Ping-Chun Hsieh. "Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs." arXiv preprint arXiv:2503.12932 (2025).

[4] Lobbezoo, Andrew, Yanjun Qian, and Hyock-Ju Kwon. "Reinforcement learning for pick and place operations in robotics: A survey." Robotics 10.3 (2021): 105.

No generalization experiments are discussed, which is significant for this setting.

We are unsure about the specific meaning of "generalization experiments" mentioned in this comment. However, we believe this may refer to the ablation study on varying numbers of expert demonstrations. In our experiments, we have tested two different environment settings, and the results are presented in Table 2 above.

Why does DTW outperform the l2 distance to a large degree, can you give more proofs in theorems? Is it the optimization issue or the convergence issue?

DTW distance can be viewed as a special case of the Wasserstein distance, as discussed in Sec 2.1 in [5]. Specifically, when applied to discrete measures, the Wasserstein and DTW distances share the same form of optimization problem. In contrast, L2 distance computes element-wise differences without considering the overall alignment. Consequently, in sequence matching tasks, optimizing with L2 distance can lead to issues since it does not align with the underlying objective [6-7].

[5] Samuel Cohen, Giulia Luise, Alexander Terenin, Brandon Amos, and Marc Peter Deisenroth, “Aligning Time Series on Incomparable Spaces,” AISTATS 2021.

[6] David Schultz and Brijnesh Jain, “Nonsmooth Analysis and Subgradient Methods for Averaging in Dynamic Time Warping Spaces,” Pattern Recognition, vol. 74, pp. 340-358, 2018.

[7] Rohit J. Kate, “Using Dynamic Time Warping Distances as Features for Improved Time Series Classification,” Data Mining and Knowledge Discovery, vol. 30, no. 1, pp. 1-22, 2015.

We are grateful for the reviewer’s constructive feedback.

This paper overlooks an important set of baselines—state-based imitation learning [1,2,3,4]. In fact, the problem setting of "Action-Constrained Imitation Learning" is not entirely new and has been extensively discussed in these works. However, this submission does not discuss these papers.

Thank you for your suggestion. We will include a discussion on this line of work in the main text. [1] and [2] focus on handling transition dynamics mismatches in state-only IL. [3] also uses state distribution matching to align the agent’s state trajectories with the expert's, and it’s more specific for dexterous manipulation tasks. [4] proposes a non-adversarial LfO approach. These works align the state-trajectories induced by unconstrained actions—trajectories that the constrained imitator cannot perfectly replicate, thus failing to achieve optimal performance. We found SAIL [1] the most relevant to our problem formulation, so we chose SAIL as a baseline. To adapt SAIL to the ACIL setting, we projected the actions generated by the SAIL policy onto each task’s feasible set before interacting with the environment, like other baselines. The table shows the return of SAIL on benchmark tasks. SAIL is only comparable to DTWIL in the HC+O and W+B task, while in all other tasks, it fails to achieve good returns.

I did not find a detailed explanation of how the action constraints are implemented in the simulator. If the weak imitator outputs an action that violates the action constraints, is it discarded or somehow mapped back to a valid action?

We follow the standard approach in the ACRL literature to enforce action constraints [5-7]. Specifically, we use a projection layer that ensures all output actions are mapped to the feasible set before interacting with the environment. This is mentioned in L364–L366, C1, and the caption of Figure 2. We appreciate the reviewer’s feedback and will add further clarification in the main text.

[5] Janaka Brahmanage, Jiajing Ling, and Akshat Kumar, “FlowPG: Action-constrained policy gradient with normalizing flows,” NeurIPS 2023.

[6] Kasaura, Kazumi, et al. "Benchmarking actor-critic deep reinforcement learning algorithms for robotics control with action constraints." IEEE Robotics and Automation Letters 8.8 (2023): 4449-4456.

[7] Lin, Jyun-Li, et al. "Escaping from zero gradient: Revisiting action-constrained reinforcement learning via Frank-Wolfe policy optimization." Uncertainty in Artificial Intelligence. PMLR, 2021.

The proposed method employs MPC to generate action-constrained surrogate demonstrations. However, MPC relies on the ability to reset the environment to specific states, which is feasible in simulation but not transferable to real-world scenarios.

The reviewer raises a valid concern regarding MPC's reliance on environment resets when using true dynamics for planning. However, our method mitigates this limitation by leveraging a learned dynamics model for planning, as done by various existing MPC methods [8-9]. Instead of requiring resets, we roll out possible trajectories within this model, making our approach more adaptable to real-world scenarios. Further details can be found in PETS [10].

[8] Draeger, Andreas, Sebastian Engell, and Horst Ranke. "Model predictive control using neural networks." IEEE Control Systems Magazine 15.5 (1995): 61-66.

[9] Kamthe, Sanket, and Marc Deisenroth. "Data-efficient reinforcement learning with probabilistic model predictive control." International conference on artificial intelligence and statistics. PMLR, 2018.

[10] Chua, Kurtland, et al. "Deep reinforcement learning in a handful of trials using probabilistic dynamics models." Advances in neural information processing systems 31 (2018).

One implicit assumption of this paper is that the state contains information strongly correlated with task success. However, a more common scenario involves access to only a partial state, meaning the task is a POMDP. It is unclear whether the proposed method would still perform effectively in such cases.

Since even standard MDP settings are not extensively discussed in prior work, we take the first step towards tackling action-constrained imitation learning and leave the challenges of partial observability to future work. One possible direction to address this issue is to learn a latent representation that captures task-relevant information, enabling the agent to infer missing state information.

In Table 1, the meanings of "H+B," "M+O," "W+L2," etc., are not explained. I assume "B" refers to "box constraints," correct?

The reviewer is correct here. "B" refers to "box constraints." We will add explanations to the main text.

We appreciate the reviewer’s time and effort in reviewing our work and raising valuable questions. We provide our point-by-point response as follows.

Analyzing the rationality of constraint design and the strength of constraints is crucial for exploring this new task. Conducting more experiments can contribute to this point.

Thanks for the suggestion. We would like to emphasize that the constraint design follows the standard ACRL literature [1-3], where similar constraints have been considered. Specifically, box and L2 constraints directly limit the numerical range of actions, while state-dependent constraints regulate the agent's power consumption. We provide results on how DTWIL and baseline methods perform under different constraint strengths in the later sections.

[1] Janaka Brahmanage, Jiajing Ling, and Akshat Kumar, “FlowPG: Action-constrained policy gradient with normalizing flows,” NeurIPS 2023.

[2] Kazumi Kasaura, Shuwa Miura, Tadashi Kozuno, Ryo Yonetani, Kenta Hoshino, and Yohei Hosoe, “Benchmarking actor-critic deep einforcement learning algorithms for robotics control with action constraints,” Robotics and Automation Letters, 2023.

[3] Jyun-Li Lin,Wei Hung, Shang-Hsuan Yang, Ping-Chun Hsieh, and Xi Liu, “Escaping from zero gradient: Revisiting action-constrained reinforcement learning via Frank-Wolfe policy optimization,” UAI 2021.

Why are the constraints designed according to the content in Figure 01, and what are the reasons for such a design?

We believe you are asking about the constraint used in the Maze2d toy example in Figure 1. In this case, we use a box constraint, limiting action values within [−0.5, 0.5]. Our goal is to illustrate, in a simple environment with a basic constraint, how standard IL algorithms may cause the learner to deviate from the intended trajectory under action constraints, leading to task failure. We did not delve too deeply into this example, as it primarily serves as an intuitive illustration to highlight the challenges of action-constrained imitation learning.

How does the proposed method perform under different constraints? If the constraints are very weak, can the direct learning still achieve good results? If the constraints are very strong, will the proposed method also be unable to handle them? Is the proposed method only effective within a relatively narrow range of constraints?

If the constraints are loose, their impact on the learner's ability to replicate the expert trajectory is minimal. Therefore, if the constraints are loose enough such that they have no effect on reproducing the expert trajectory, direct learning can still achieve good results. In our experiments, we select moderately tight constraints to ensure they significantly impact learning. Here, we show how DTWIL performs under tighter and looser constraints.

We sincerely appreciate the reviewer’s insightful comments and valuable suggestions. We provide our point-by-point response as follows.

L162-164;C1 — Wouldn’t it be more accurate to say that there exist trajectories within \mathcal{T} that cannot be entirely replicated by the learner, since it contains one or more actions that are outside the learner’s feasible set. For one thing, this doesn’t preclude corresponding states to be inadmissible for the learner. Another thing is that these trajectories only make up a subset of all the trajectories in \mathcal{T}. I don’t know where the intuition behind the use of the term “generally”, here, comes from.

Thanks for the insightful comment. Our use of "generally" was intended to convey that when the action constraints are very tight, it is typically difficult for the learner to replicate the expert trajectory. However, we acknowledge that your suggested phrasing more precisely captures the idea that certain trajectories in $\mathcal{T}$ cannot be entirely replicated due to infeasible actions. We appreciate your suggestion and will revise this part for better clarity.

Why wouldn’t a diffusion policy alleviate all of these issues? The manuscript should compare with Huang et al., 2024: https://arxiv.org/pdf/2410.05429 Huang et al., 2024 was cited briefly, but surprisingly not used as a baseline, since it pursues the same problem and even has the same evaluation. L116-120;C2 — Why does LfO underperform so much, if it’s trying to do the same thing? Why not compare to DIfO?

LfO also aims to imitate the state trajectory, but it tries to mimic the trajectory induced by unconstrained actions—trajectories that the constrained imitator cannot perfectly replicate. This limitation is precisely why we generate surrogate expert demonstrations before applying imitation learning, ensuring that the learner has feasible trajectories to follow. DIFO is also an LfO approach, and suffers from the same problem. We added a projection layer after DIFO's policy output and tested it across various action-constrained tasks. The results below show that across all tasks, DIFO performs poorly and fails to achieve competitive results.

Section 4 — Should specify what dynamics model is chosen, in the main paper content

Thank you for the suggestion. We will move the relevant paragraph from A.2 to the main content.

L192-197;C2 — The manuscript states, "... (iii) assigning a⇤t to be the first action of the selected rollout. This a⇤t would be applied to the environment to obtain the next state. This design of taking only the first action ensures that each step of the alignment process can better adapt to the remaining part of the demonstration." Common heuristic, but costly in planning complexity. What happens if more actions are executed before replanning?

We appreciate the reviewer's suggestion. This approach can indeed improve time complexity. We conducted an ablation study on varying the number of steps executed before replanning. The results show that when MPC executes multiple steps per planning cycle, the alignment performance deteriorates as the step count increases. In the Hopper environment, poor alignment further increases the risk of falling.

General: The manuscript should properly use inline citations

Thank you for the suggestion. We will carefully check the inline citations for the final version.

L115;C1: ", using" —> "uses" Pham et al., 2018b — duplicate entry L175;C1: "describing" —> "describe"; "We start .. and then describe" 184;C2: "help a" —> "help of a" Equation 1 — Notation error: \tao_e has not been defined as the expert trajectory yet; however, \tao was introduced as the expert state-action trajectory on L163

Thank you for catching these typos. We will fix them in the final version.