Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation

Thank you for your valuable time and effort in providing detailed feedback on our work. We hope our response will fully address all of your points.

---

How many samples are used as rollouts in the target domain?

Our imitation learning step rolls out data from the target domain every 100 steps of the source domain rollouts, which is 1% of the source domain rollouts.

---

Additional samples from the target domain for DARC.

DARAIL does have more target domain rollouts. However, as mentioned in the general response, more rollouts of the target domain data is not the reason why DARAIL works. More rollouts of the target domain will not improve DARC performance, according to our analysis of why DARC fails in more general settings in the general response. As mentioned in the third bullet point in the general response, we compare DARC and DARAIL with the same amount of rollouts from the target domain in Tables 4 and 5 in the attached PDF, indicating that more target domain rollouts does not yield significant improvement in DARC's performance due to its inherent suboptimality. Notably, DARAIL consistently outperforms DARC even when subjected to comparable levels of target rollouts.

---

Explanation of Figure 3.

Figure 3 in the paper is an ablation study showing that our method works well under different scales of off-dynamics shift. Specifically, we conducted experiments on Ant-v2, and the target environment is broken with probability 0.2,0.5,0.8, respectively. The smaller the broken probability is, the smaller the dynamic shift is.

---

Smaller discriminator update steps are better.

In GAIL or, more generally, GAN, the discriminator serves as the local reward function for the policy training. The smaller discriminator update steps (higher update frequency) probably lead to better discriminator training (better local reward estimation). In our experiments, we did the grid search for the parameters and noticed that further decreasing the update steps (increasing the update frequency) would not improve the performance; it is probably already well-trained, so we stopped that at 50. We will provide a more thorough grid search results to justify this in the future.

---

Presentation comments.

We thank the reviewer for the detailed comments that help us improve the readability of the paper. We will follow your suggestions in our revision. For several questions, we briefly answer as follows:

1. Missing discount factor $\gamma$ in equations. We will state clearly in the revision that we use $\gamma = 1$ in the analysis of the error bound.
2. Discuss of the related work. [1] is most related to our problem. However, they assume that the optimal policy in different domains has similar stationary state distribution, thus, they learn a policy in the source domain and regularize the state distribution in two domains. This method is similar to DARC, which tries to learn a policy that behaves similarly in both domains. However, when the assumption is violated, the learned policy will have performance degradation when deployed to the target domain, similar to DARC. In contrast, our method doesn’t require such an assumption and we propose one more imitation learning step to transfer the policy to avoid performance degradation. [2] and [3] are designed to solve a slightly different problem that transfers prior knowledge to solve a new task instead of just considering the dynamic shift between the tasks. We will add the discussion of these works to the related work in the revision.
3. Discussion of observations vs. states in L90. The observation here represents the state trajectories we obtain from the environment, which is a general terminology from imitation learning from observation. We use imitation learning from observation to distinguish from generative adversarial imitation learning (GAIL) which uses $(s_t,a_t)$ as the expert demonstration. Meanwhile, imitation learning from observation purely learns from the observation (state observation) from the environment, i.e. $(s_t,s_{t+1})$.
4. Right-hand side needs a min for Eq 3.3. The right-hand side of Eq 3.3 is the mathematical expansion of the reverse KL divergence loss. We will add a min to that to make it clearer.
5. Meaning of do not have much access to the target domain. The limited access means that we cannot freely roll out infinite data from the target domain, and we can only roll out a small amount of data from the target domain. This is a standard setup shared by many off-dynamics problems, including DARC. Also, we assume we do not query the target domain reward during the training. As discussed in the first bullet point, we roll out the states and actions from the target domain data every 100 steps of the source domain rollouts. We will make it more clear in the revision.

We appreciate the reviewer's feedback. We hope we have thoroughly addressed your concerns. We are willing to answer any additional questions.

---

References

[1] State Regularized Policy Optimization on Data with Dynamics Shift.

[2] Accelerating Reinforcement Learning with Learned Skill Priors.

[3] Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning.

Thanks for your valuable time and detailed feedback. We hope our response will fully address all of your points.

---

Evaluation on a very specific kind of dynamics change.

1. In the general response, we provide additional experiment results under DARC setting and more general off-dynamics settings to show that our method works well beyond the specific dynamics change. In the second bullet, we clarify why DARC fails in the broken source and intact target settings theoretically and empirically.

2. DARAIL is specifically constructed so that it learns to output 0. This is not true. As mentioned in the general response, Table 1 in the provided PDF shows DARAIL works for the general off-dynamics setting other than broken action. Indeed, if we know the difference in dynamics, we can “simply” learn/set 0 for 0th index. However, we do not assume we know beforehand. We will add this discussion to our revision.

3. DARAIL does not work in the off-dynamics settings considered in DARC. It’s not true that most of the state transitions $(s_t,s_{t+1})$ generated by DARC in the source domain are not possible in the target domain in DARC settings. DARC learns a near 0 value for the 0th index, as mentioned in the second bullet in the general response. Table 3 in the provided PDF shows that DARAIL performs better than the DARC evaluation in the target domain in DARC setting.

---

Method is overly complicated for the problem setup.

First, we do not assume that “when $\pi_{DARC}$ is optimal for the source domain, the trajectories it obtains are optimal in the target domain.” Instead, we notice that $\pi_{DARC}$ generates the trajectories in the source domain that resemble the target optimal ones, as this is the DARC’s learning objective. So we transfer $\pi_{DARC}$ to the target domain instead of mimicking the source optimal policy. We provide additional experiments that mimic the source optimal policy instead of DARC trajectories on two settings in Tables 2 and 3 in the PDF. DARAIL significantly outperforms that, probably because the optimal source domain trajectories might not perform well in the target domain.

Second, we want to clarify further that we do not assume that the source trajectories generated by DARC are exactly the optimal trajectories in the target domain. Instead, we utilize DARC’s learning objective to learn a policy that generates trajectories in the source domain that resemble the target optimal ones. Moreover, we considered DARC's training error in the first term of the error bound in our analysis.

---

Paper presentation questions

1. Performance degradation happens when the source domain is less comprehensive. We want to clarify that less comprehensive means that the support in the source is less than the target, such as our setting. However, we agree that this is not the only case when performance degradation happens, as suggested in Table 1. We will clarify this point in our revision.

2. Why is it a harder problem when the source domain has less support? In the domain adaptation problem, if the source domain/training data has larger support or covers more data distribution than the target, it would be easier to generalize to the target domain because the training data has seen all possible target actions/data. In off-dynamics RL settings, if the target domain has less support, we can restrict the action support in the source domain. For example, in the DARC setting, whose target environment has less support, the DARC policy learns to output near 0 value for the 0th-index, and the problem becomes optimizing the policy in the source domain such that the DARC policy outputs 0 for the 0th-index. From this perspective, DARC is restricting the action support in the source domain to align the target domain action support. However, on the opposite, where the source has less action support, such methods don’t work well, and we need to account for the unseen/unsupported action in the target domain.

3. DARAIL relaxes the assumption made in previous works that the optimal policy will receive a similar reward in both domains. As we discussed in the general response, the assumption made by DARC guarantees that DARC’s performance and generated trajectories in the target are similar to those in the source domain. This assumption may not hold in other off-dynamics settings. We want to relax this assumption by not restricting the performance of target optimal policy in the source domain. Note that the DARC source domain trajectories resemble the optimal target ones, according to DARC’s learning objective. Thus, we use imitation learning to transfer such policy. And the experiment shows the effectiveness of our method when their assumption fails.

4. Two domains have the same reward function. It means that $r_{\text{src}}(s_t,a_t) = r_{\text{trg}}(s_t,a_t)$ not $r(s)$. We will make it clearer in the revision.

5. Clarifications on Figure 1(a) and performance degradation. To show the performance degradation empirically in more general settings, we employ DARC in the target domain and compare it with its performance in the source in Figure 1(a). Here, the motivation is, according to the DARC’s objective, that the DARC policy on source should achieve a similar performance (and a similar trajectory distribution) to the optimal policy on target. But we agree, empirically, it may not be true. So, we followed the reviewer’s suggestion to add the reward for the optimal policy on target to show the performance degradation better in Figure 1, Table 2 and 3. In contrast, little performance degradation happens in the DARC setting. We will update the figures and tables accordingly in our revision.

6. Not trained for the same steps. We train the baselines until convergence. Though we believe the result would not change much with more training steps, we will train the baselines to the same number of steps for fair comparison in the revision.

We first thank the reviewer for their time and comments. We now address your questions point by point.

1, Notation of $\tau$. $\tau$ represents the trajectory sequence of state-action pairs, so it is a notation for the trajectories. For example, we use $\tau_{\pi_{\theta}}^{\text{src}}$ to represent the trajectories generated by the policy $\pi_{\theta}$ in the source domain.

2, Why DARC fails to address dynamics shift. First, as mentioned in the general comment, DARC works well under some dynamic shifts when the assumption about the dynamic shift made in the DARC paper is satisfied, i.e. the optimal policy for the target domain is also a good policy in the source domain. DARC works well in this setting because the objective of DARC can also be viewed as maximizing the cumulative reward in the source domain under a constraint that the learned policy performs similarly in two domains. However, in more general off-dynamics settings. This assumption is not satisfied. DARC will be suboptimal in the target domain as DARC does not perform similarly in the two domains, leading to performance degradation in the target domain.

Our method is motivated by DARC, which generates trajectories in the source domain that resemble the optimal ones in the target domain. We propose to use imitation learning to transfer DARC trajectories to the target domain so that deploying the policy to the target domain will not receive such performance degradation.

3, On page 4, lines 120-122, recover the wrong target policy. We would appreciate any clarifications on the definition of $\pi_{trg}$ in the second part of the question. To answer the first part, the matching of the distribution is the main goal of the DARC’s learning objective. Under the unknown dynamics and unknown dynamic shift, DARC tries to learn a policy that generates trajectory distribution similar to the target trajectories from the target optimal policy. But the policy $\pi_{DARC}$ will not be the same as the target optimal policy due to the dynamics shift. Here, we only use the $\pi^*$, which is the target optimal policy, in our analysis of DARC’s failure.

4, DARC does not need any online learning, while DARAIL does. DARC does need online learning and they sample $(s_t,s_{t+1})$ from the target domain. Similarly, our method requires sampling from the target domain. In both cases, the amount of data rollouts from the target is significantly smaller than the rollouts from the source. The details can be referred to the third bullet point of the general response.

5, Assumption about the shared reward. In the off-dynamic RL setting, we specifically focus on the setting where the two domains only differ in the dynamics but share reward functions and state/action space. If this assumption is not true, this becomes a different domain adaptation problem, and we might need other techniques to adapt to the shift in the reward function.

6, Experiment on other settings instead of Mujoco. We follow the recent work in off-dynamics RL and experiment on the MuJoCo environment, which is easy to access and analyze. We will also add new experiments on other environment settings in our revision.

We appreciate the reviewer's feedback. We hope we have thoroughly addressed your concerns. We are willing to answer any additional questions.

We first thank the reviewer for their time and comments. We now address your concerns point by point.

---

How would DARC trajectory quality influence the overall imitation learning performance in the target domain?

We use trajectories generated by the DARC in the source domain as the expert trajectories for imitation learning, which resemble the optimal trajectories in the target domain, according to the DARC’s objective. In general, the imitation learning performance should be similar to the DARC performance in the source domain, with the latter one expected to perform similarly with the target optimal policy if the DARC learning error is small. The better the DARC trajectory quality, the better performance the resulting policy from imitation learning can achieve. Overall, our analysis in section 4 shows that a good DARC trajectory quality will have a smaller learning error in the first term of the error bound. Further, we want to emphasize that our method is not particularly limited by the performance of DARC because our method can be viewed as a general framework that uses imitation learning to transfer trajectories from the source domain to the target domain as long as we can obtain trajectories that resemble the optimal trajectories on the target.

---

There are no comparison experiments of directly using $(s,s’)$ from the expert trajectories in the source domain for DARAIL.

We assume that by “$(s, s’)$ from the expert trajectory in source,” the reviewer means using the optimal trajectory in the source. We provide such experiment results on HalfCheetah and Reacher in Table 2 and 3 in the attached PDF. We first train a policy in the source domain (do not consider the dynamics shift) and use imitation learning to transfer it to the target domain. The results show that DARAIL outperforms the method that uses optimal trajectories in the source domain as the expert policy for imitation learning because DARC can generate trajectories that resemble the optimal ones in the target domain.

---

Experiments on other off-dynamics shifts.

We provide additional experiment results on HalfCheetah and Reacher in Table 1 in the provided PDF. The off-dynamics shift is constructed by modifying MuJoCo's configuration files. We modify the coefficient of gravity from 1.0 to 0.5 for the target domain. Our results show the effectiveness of our method. In our revision, we will add more experiments on these more general cases to demonstrate the performance of DARAIL.

We thank the reviewer for their detailed suggestions for improving the readability of our paper. We will follow them closely, fix all the typos, and clarify the writing of the paper.