Action-Constrained Imitation Learning

We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below. To save space, I apologize for reducing your question with "...".

The motivation requires to be ... examples for this point.

In the second paragraph of the introduction, we have added other practical application examples of ACIL, in addition to the typical applications commonly seen in ACRL. For example, if data is collected using a human to perform tasks, the imitator, which may be a robot with hardware limitations, is likely to be unable to replicate the large-scale human actions.

Also, what is the difference ... introduction/related works sections.

Following your suggestion, we have added a section on cross-embodiment imitation learning to the related work paragraph. In this paragraph, we also discussed the differences between ACIL and cross-embodiment imitation learning.

From my point of view, the proposed method ... generate good surrogate demonstrations?

1. Thank you for your thoughtful comments. We acknowledge that the assumption—"the surrogate demonstrations generated by Trajectory Alignment (Section 4.1) must solve the task or achieve high rewards"—is indeed a foundational aspect of our method. However, we would like to emphasize that this assumption is relatively mild and is widely adopted in the Learning from Observation (LfO) literature. In imitation learning, our goal is to mimic the expert's behavior—in other words, to replicate the expert's trajectory. Therefore, if we successfully align with the expert's trajectory, the surrogate trajectory obtained through alignment will naturally resemble the expert's trajectory and, consequently, achieve a high return. This aligns closely with the core idea of LfO, where solving the task or achieving high return is pursued by mimicking the expert's state trajectory.

2. We found that the constraint action<-0.5 and action>0.5 have minimal impact on the agent’s performance and would not harm the policy and alignment process. Therefore, we chose action>-0.1 and action<0.1 as our problem.

Why do the IRL and LfO methods ... action space for IRL/LfO?

1. Directly using the constrained action space actually has the same effect as the +P operation.
2. In real-world tasks, the feasible action space is typically fixed and cannot be changed. However, there are situations where it is necessary to restrict the agent's behavior for safety reasons, as mentioned in the introduction. In such cases, controlling the policy's output through projection is a more preferable approach."

Although the proposed method ... train a successful policy.

In the appendix, we have included the 500k-step training curves for each baseline method as well as DTWIL. This number of steps is sufficient for all the baselines to demonstrate their capabilities and whether they can effectively train the policy.

I doubt if directly ... as the (negative) cost function.

Thank you for your insightful comment. We appreciate the opportunity to address this point.

Directly using MPC with a constrained action space and the proposed DTW metric poses challenges in terms of generalizability and practicality. DTW requires a corresponding expert demonstration for alignment, meaning a tailored expert trajectory is needed for each distinct initial state. In real-world scenarios, where the set of starting points is effectively infinite, providing an expert demonstration for every case is infeasible.

Additionally, inference speed is crucial for real-time robotic systems. Using MPC to compute the DTW metric and select optimal actions introduces significant computational overhead, making it unsuitable for time-sensitive tasks.

In contrast, a behavior cloning (BC) policy trained with our method addresses these issues by generalizing across diverse initial states and enabling real-time inference with reduced computational demands, providing a more practical and scalable solution.

The stability of the proposed method ... as well as the $\beta$ hyperparameter.

Thank you for your observations. We would like to clarify that normalization and time-step alignment are not task-specific components; these designs are applied consistently across all tasks in our experiments. In particular, state normalization is a widely used data preprocessing technique in machine learning and is not tailored to specific environments. Regarding the hyperparameter $\beta_{arc}$​, we acknowledge that it is indeed task-specific and is primarily designed for environments with early termination, such as Hopper. The ARC design stabilizes the MPC-generated actions in Hopper, reducing the likelihood of falling and thereby preventing premature termination. However, the hyperparameter tuning effort for $\beta_{arc}$​​​ is minimal. For instance, a fixed value of $\beta_{arc}$​​=0.05 works effectively across both box-constrained and state-dependent constraint tasks in Hopper, demonstrating its robustness.

We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.

I am a bit confused about the action constrained setting that the paper operates in. For the experiments shown in the paper, the demonstrations are collected with the same agents in the same environments as at test time. So why should additional action constraints be added during deployment when they are not need during demonstration collection?

In the second paragraph of the introduction, we have added practical application examples of ACIL, in addition to the typical applications commonly seen in ACRL. In real-world tasks, we will not let the imitator to do behaviors out of his capability if the imitator is a weaker agent because it will hurt itself. Therefore, in the experiments, though the demonstrator and imitator are originally same agents, we add action constraints to the imitator to simulate a weaker imitator.

I am a little confused about the algorithm. From what I understand, given some expert demonstrations, these demonstrations are modified through online interactions with the environment. Some questions based on this - ...

Sorry for the confusion. Each iteration corresponds to a single trajectory rollout, where the iteration K represents the total number of episodes to be executed. This clarification is now explicitly stated, as highlighted in Algorithm 1 and the newly added Trajectory Alignment pseudocode. Furthermore, it is worth noting that for each episode, PETS updates its dynamics models for approximately 10 epochs — a process that requires minimal time compared to the true computational bottleneck, which lies in the DTW alignment.

There is a missing reference in line 231.

We have fixed the link. Thanks a lot!

In Table 1, for HalfCheetah HC+O, BC+P is the best performing method but DTWIL is still boldened. Similarly, in Table 3 for DTW-S, Hopper Box-Sync outperforms Hopper Box but Hopper Box is boldened. This is confusing to the reader and beats the purpose of boldening the results.

We have revised the paper according to the reviewer's suggestions. Additionally, DTW-S represents the DTW distance between surrogate trajectories and expert trajectories. A lower DTW-S value indicates that the surrogate trajectories are more similar to the expert trajectories, so smaller numbers are better.

We sincerely thank you for the constructive comments. Please refer to the new version of the paper and find the response to your questions below.

Limited Experiment Results

We follow the reviewer's suggestion and conduct experiments in a robot arm environment. We will present the results in the paper once the experiments are complete.

Method Limitations. The ARC (Section 4.2) seems like a hack, and could require per-task tuning, which is undesirable. It is also unclear if it's a good idea to always use blended actions up to a certain timestep, compared to other alternatives for incorporating expert actions, like using the notion of residual additive actions (for example, https://arxiv.org/abs/1812.03201). Including more tasks could help show that one set of parameters works well across multiple settings.

The reviewer is correct that mixing actions in this manner does not always result in a better-aligned trajectory. In fact, ARC was specifically designed for tasks with early termination, such as Hopper. Without ARC, the actions executed by the MPC agent are not stable enough and often lead to early termination due to a single slightly suboptimal action. To address this, we adopted the ARC approach to make the MPC agent's actions more stable. In environments without early termination, mixing the sampled actions and expert actions may result in a very slight performance drop, but the difference is minimal. The residual additive actions approach the reviewer mentioned can indeed achieve similar effects, but it requires learning an additional residual policy. While the overall framework is similar, training the residual policy might slightly reduce sample efficiency. Therefore, we chose the ARC-based approach.

Method Limitations. It also seems like this method is only suitable for imitating a single specific trajectory, in contrast to typical scenarios where an agent must deal with a variety of initial conditions (such as a robot that needs to manipulate objects that start in diverse configurations on a table from episode to episode).

We would like to clarify that our method only requires the initial states to be identical during the alignment phase. Once the surrogate demonstration is used to train the behavior cloning (BC) model, the trained policy generalizes across a wide range of initial conditions. This capability allows our method to effectively handle diverse initial states beyond the alignment phase.

To further improve clarity, we have included a statement in Section 5.3 (line 423) explicitly mentioning that the initial points used for evaluation are sampled randomly.

Method Limitations. Finally, it seems like the method implicitly assumes full state observability (e.g. privileged information) compared to partially observed settings (raw sensor data such as images or depth sensing), which also impedes the practicality of the approach.

Considering that we are taking the first step in ACIL, we would like to focus on the standard setting. Our method is compatible with any MPC algorithm, so this issue can be addressed by replacing the MPC algorithm with one that can handle raw images. For example, TDMPC (https://arxiv.org/abs/2203.04955) is an algorithm capable of tackling partially observable states.

Some Writing Issues. Section 4 does not adequately describe the overall approach of how BC is used with MPC. This needs more details -- I had to figure this out from the Algorithm 1 pseudocode.

We apologize for the confusion caused by the absence of the Trajectory Alignment pseudocode in the initial submission. To clarify, MPC is employed exclusively during the Trajectory Alignment phase, where it generates hundreds of candidate rollouts for alignment with the expert trajectories. This process is distinct from the Behavioral Cloning (BC) component, which is used at the end of the pipeline (Algorithm 2 in the revised version) to train a policy using all aligned surrogate trajectories.

Some Writing Issues. It also isn't clear why BC is needed versus directly using MPC for control -- some further experiments might help point out the value of using BC.

The inclusion of the BC model provides a significant advantage: it enables our method to generalize effectively to unseen initializations and rollouts that were not present in the expert dataset. In contrast, directly using MPC for control inherently relies on having corresponding expert trajectories for alignment, limiting its ability to generalize.

I suggest organizing Related Work further, with a bolded title for each paragraph at least.

Following the reviewer's suggestion, we have added bolded titles to each paragraph and streamlined some of the content.

line 231, undefined Figure reference

We have fixed the link. Thanks a lot!

Section 5 beginning - "both offline baselines online baselines"

We have revised this line to "both offline baselines and online baselines". Please check it out (line 358).

We greatly appreciate the reviewer's acknowledgment of our rebuttal and suggestions for improvement. We are happy to provide further details.

The MPC is, in fact, reset to the initial state of each expert trajectory, which can vary between trajectories. Its role is solely to generate surrogate expert data that aligns with the original expert trajectory. Once this surrogate expert data is obtained, we use BC to train the agent's policy, which can handle different initializations. Therefore, DTWIL's generalization capability is equivalent to that of BC. As long as the diversity of starting points in the expert data is sufficient for BC, the diversity of starting points in the surrogate expert data will also be sufficient for DTWIL.

We sincerely thank the reviewer for the constructive comments. Please refer to the new version of the paper and find the response to your questions below.

At L.231, there is a statement "The pseudo code for trajectory alignment is presented in ??" and the pseudo code for trajectory alignment is missing in the paper.

We have fixed the link. Thanks a lot!

The definition of DTW distance in Eq. (2) is not provided, though its high level idea is stated in page 4.

We have added an explanation of the meaning of DTW distance in the Preliminaries section (line 188) in the new version.

Since trajectory alignment algorithm is missing, it is not clear how the action constraints are handled practically.

We have fixed the link to Trajectory Alignment. Additionally, right after Eq. (2), starting from line 262, we added some explanations, mainly about that we solve this optimization problem approximately, and that we use rejection sampling to ensure actions to conform to the constraints. We hope it is clearer now.

The definition of $\bar S (A, f_{\theta})$ is not concrete. Does this sequence start from $\bar{s}^e_{t_{pg}}$?

We thank the reviewer very much. We removed this ambiguous notation, and now using $s_{t:t+H}$ instead to represent the same meaning.

How $t_{pg}$ is updated in practice? Figure 3 explains only discrete cases. For continuous spaces, I suppose that we need to compute the distance of states by some metric and determine by a threshold, which must be an additional hyper parameter to be tuned.

The table in this figure does not represent environments with discrete state spaces, such as grid worlds. Instead, it is a memoization cost matrix for dynamic programming for DTW computation. It illustrates the warping path that needs to be identified within the DTW matrix during the computation of the distance between two trajectories.

I am not convinced of the validity of Actor Regularized Control (ARC). In my understanding, the expert actions before projection, $a^e$, are sampled from the dataset. On the other hand, $a^{sampled}$ are computed for states in the generated trajectory. Therefore, the states for which $a^e$ and $a^{sampled}$ are sampled are different by construction. Mixing these different-state-dependent actions seems not valid.

Before mixing $a^e$ ​ and $a^{sampled}$, we first project $a^e$​ onto the feasible set that $a^{sampled}$ is required to satisfy. Thus, the mixture of two actions that satisfy the same state-dependent action constraint will also satisfy this state-dependent action constraint.

Ablation study is not comprehensive enough. Since the paper describes the importance of excluding the final expert state in Figure 3, the reader may expect its experimental impact.

We thank the reviewer for your suggestion. We have done this additional ablation study and showed the results at section 5.5. For Maze2d-Medium, excluding the final expert state improves the success rate by about 5%.

The time complexity of the naive DTW algorithm is O(NM), where N and M are the lengths of the two input sequences. I suppose that DTWIL has a drawback in the computational complexity compared to the baselines, which might hinder the applicability of DTWIL to real world applications. I think that it is necessary to compare the computational time with baseline methods.

Though the training time is substantial, there are several strategies available to mitigate this during the training phase. Moreover, in real-time applications, inference time typically takes precedence over training duration. Thus, this aspect would not be a critical limitation, as our method benefits from a relatively short testing time due to the use of a behavioral cloning (BC) model for inference.

For Maze2d, state-dependent constraint is not stated.

We thank the reviewer for your suggestion. We will include the results for the state-dependent Maze2d environment in the experiments chapter. As this was not part of our initial experimental plan, generating these results may require a few additional days. Currently, the experiments are still in progress.

Section 5.6 and Appendix A.3 look exactly the same.

They are fine in this new version. Please check it out.

Reference

[1] Sakoe Hiroaki and Seibi Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1978.

[2] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv:2004.07219, 2020.

I appreciate the authors' extensive effort to revise the paper. I acknowledge that the presentation of the paper improved largely. I raise the score from 3 to 5. However, I believe there are more room of improvements as listed below.

• I believe that the definition of DTW distance must be explicitly provided. The current form of the paper is not self-contained. It still lack the explanation related to the cost matrix, which makes hard the reader to understand what Figure 3 means.
• I understand that, the authors' claim that ERC is valid because "the feasible set that $a^{\rm sampled}$ is required to satisfy", which is state dependent. However, L321 simply states that "onto the constrained action space". I believe this state-dependence must be clearly denoted.
• On computational complexity compared to the baselines: could you explicitly show the computational time required for DTWIL and allowed time in real-time applications? It must help the readers to understand the soundness of the proposed method.

Thank you for the clarifications. My original concerns are mostly resolved. I raise the score to 6.

Minor issues:

• $H$, $h_{erc}$ and $P$ are not used in Algorithm 1; I supposes these values are fed into Trajectory Alignment as well as $\tau^{i}$.
• Eq.(3): ${\rm Proj}(a^e_{t_{pg}+h}|s_h)$ -> ${\rm Proj}(a^e_{t_{pg}+h}|\mathcal{C}(s_h))$