Cross-Domain Offline Policy Adaptation with Optimal Transport and Dataset Constraint

We thank the reviewer for commenting that our work proposes an informative proof and well-designed method and that our method can serve as a well baseline for follow-up works. Please check our clarifications to the concerns below.

Concern 1: $\mu_{t,t^\prime}$ is not defined

Thanks for the question. $\mu\in \mathbb{R}^{n\times n^\prime}$ is the coupling matrix, and $\mu_{t,t^\prime}$ denotes its $t$-th row, $t^\prime$-th column element, $t=1,\ldots,n, t^\prime=1,\ldots,n^\prime$, which corresponds to the coupling with respect to $x\_t, y\_{t^\prime}$. We have made it clearer in the revision.

Concern 2: Can a more complex kinematic shift be proposed?

Yes, it is feasible. The kinematic shift can occur at different parts of the robot.

• for halfcheetah task, we modify the rotation angle of the joint on the thigh of the robot’s back leg in the submission. Furthermore, one can modify the rotation range of the foot joint to fulfill kinematic mismatch
• for hopper task, one can modify the rotation range of either the head joint or the foot joint (we already modified the rotation angle of both the head joint and the foot joint in our paper)
• for walker2d task, we modify the rotation angle of the foot joint on the robot’s right leg in our submission. One can also modify the rotation range of the thigh joint.
• for ant task, we modify the rotation angles of the joints on the hip of two legs in our original submission. We can also modify the rotation range of the ankle joint

Note that the joints can be broken simultaneously, and it depends on the user to design the kinematic shift (e.g., the broken joints occur only in one joint or multiple joints).

To examine the effectiveness of OTDF under possibly more complex kinematic shifts, we consider two domains (walker2d, ant) and modify their XML files. We only evaluate on two domains because we need to train the agent in the modified environment to ensure that the agent can get meaningful performance and modify the XML files if not. We also need to gather offline datasets with different qualities. All these consume a lot of time. For the walker2d task, we modify the rotation range of the thigh joint in the target domain to be 0.2 times of that in the source domain:

<joint axis="0 -1 0" name="thigh_joint" pos="0 0 1.05" range="-30 0" type="hinge"/>
<joint axis="0 -1 0" name="thigh_left_joint" pos="0 0 1.05" range="-30 0" type="hinge"/>

For ant task, we modify the rotation range of the ankle joint of all legs from [30, 70] to [30, 38]:

<joint axis="-1 1 0" name="ankle_1" pos="0.0 0.0 0.0" range="30 38" type="hinge"/>
<joint axis="1 1 0" name="ankle_2" pos="0.0 0.0 0.0" range="-38 -30" type="hinge"/>
<joint axis="-1 1 0" name="ankle_3" pos="0.0 0.0 0.0" range="-38 -30" type="hinge"/>
<joint axis="1 1 0" name="ankle_4" pos="0.0 0.0 0.0" range="30 38" type="hinge"/>

We then first train the SAC agent in the above new kinematic tasks and gather medium, medium-expert, and expert offline datasets following D4RL. All target domain datasets still have a limited budget of 5000 transitions. Due to the limited rebuttal period, we only run OTDF and IQL* (with both domain data) on some modified kinematic tasks (medium/medium-replay/medium-expert source domain dataset and medium/medium-expert target domain dataset). We use the default hyperparameters of OTDF and summarize the results below. All results are averaged across 5 different random seeds, and we report the normalized scores on the target domain.

Table 1. Results on new kinematic tasks. walk=walker2d, m=medium, r=replay, e=expert. We report the mean normalized score with the standard deviations.

It can be seen that OTDF still outperforms IQL* on most of the tasks, further demonstrating the effectiveness of our method. One can also modify the kinematic tasks by simulating broken joints at multiple joints (e.g., simultaneously broken ankle joints and hip joints at the ant task) to construct more diverse and complex kinematic shifts and run OTDF on them.

Hopefully, these can resolve the concerns. If there is still something unclear, please let us know!

We thank the reviewer for the insightful review. We appreciate that the reviewer acknowledges that our work provides a clear foundation for the proposed method and that OTDF significantly outperforms baseline methods. Please find our clarifications to the concerns below

Concern 1: on the computational overhead

We clarify that the CVAE is actually pre-trained before the policy learning stage of OTDF (as stated in Line 281). This indicates that no training costs of CVAE will be included during the training of OTDF. Meanwhile, the computation cost of training CVAE is also tolerable since we only train CVAE for 10000 steps (please see Table 5 for hyperparameter setup). Despite that measuring the log probabilities of the action lying in the span of the target domain dataset can still require some extra computational overhead. We respectfully argue that there is no free lunch. Many cross-domain offline RL methods also suffer from the same issue, e.g., DARA [1] needs to train domain classifiers and construct reward penalties, and IGDF [2] includes the contrastive learning objective for data filtering. These can all introduce additional computational overhead. Based on our experiments, we find that OTDF enjoys a runtime similar to those baseline algorithms

Concern 2: insights on using the small amount of target domain data in OTDF

The insights behind the small amount of target domain data lie in the following aspects:

• under many scenarios, the target domain data is hard to acquire. It can be expensive to gather data in the target domain (e.g., real-world robotics data). It is somewhat unrealistic to train policies using a large amount of target domain data (e.g., 1e5). Nevertheless, a small amount of target domain data (e.g., 5000) is more acceptable and practicable
• human beings are able to quickly adapt to the downstream tasks with only a small amount of data. For example, Alice used to play tennis without any exposure to other ball sports (Line 33 of the main text), and she never played badminton before. However, based on her experience in tennis, she can play badminton after a few rounds of play. We believe such ability is also expected in decision-making agents. Hence, we only provide a limited budget of target domain data to realize efficient offline policy adaptation
• existing cross-domain offline RL papers still rely on a large amount of data (e.g., papers like DARA/IGDF/BOSA rely on 10% D4RL data, which can still result in 1e5 data eventually, which is indeed a comparatively large amount of data). We note that this can downgrade the necessity of a source domain dataset since one can get satisfying performance on them with existing strong offline RL methods. This can be verified in Table 1 in the BOSA paper [3], and Table 1 in the IGDF paper, where offline RL methods like IQL, CQL, and SPOT can achieve quite strong performance with 10% D4RL data. In contrast, we find that these offline RL methods typically fail under the single-domain setting when only 5000 data is available

We are sorry that a strict and formal theoretical analysis of a small amount of data can be difficult. To the best of our knowledge, there are no suitable mathematical tools for connecting the performance bound of the policy and the dataset size in a general manner. There exist some works that try to do so, e.g., [4], but they often rely on some assumptions that deviate far from the practical applications

Concern 3: more explanations on the CVAE

We use CVAE to fulfill the dataset constraint, i.e., using CVAE to estimate the behavior policy of the target domain dataset and maximize the log probability of the current policy lying in the span of the target domain dataset. This can intuitively prevent the learned policy from getting biased towards the source domain data, which is vital since we focus exactly on the agent's performance in the target domain. Theoretically, incorporating the dataset constraint term can better control term (b) in Theorem 3.1. Please refer to more details in Lines 216-233 of the main text.

We use CVAE to model the behavior policy due to the fact that (a) CVAE is a very popular generative model with good theoretical foundations, which can generate in-distribution samples; (b) CVAE is widely adopted and proven to be effective in the offline RL community to estimate the behavior policy, e.g., BCQ [5], SPOT [6], PLAS [7], etc. (c) Using CVAE can incur better performance. It turns out that CVAE is very important to OTDF. In Figure 3 of the main text, we conduct parameter study on the policy regularization coefficient $\beta$, which determines the strengths of the dataset constraint, i.e., a larger $\beta$ indicates a stronger constraint and vice versa. Setting $\beta=0$ usually leads to a significant performance drop. The impact of $\beta$ can be large on some tasks, indicating that CVAE is undoubtedly a critical component of OTDF.

We thank the reviewer for the constructive comments. We provide point-to-point clarification to the concerns of the reviewer. If we are able to resolve some concerns, we hope that the reviewer will be willing to raise the score.

Concern 1: on the experiment setting

Thanks for the question. We use source and target domain data of distinct qualities because (a) we can better capture the generality and effectiveness of the cross-domain offline RL methods and there is no definite rule that one can only conduct cross-domain offline policy adaptation for source domain offline datasets and target domain datasets with the same quality level; (b) in many real-world applications, we often cannot provide exactly the same performance level offline dataset for both domains.

We agree that there is a certain overlap between the source and target domain data to some extent when they are of similar quality (it is hard to quantify that though). Nevertheless, we respectfully argue that they can also have some overlap even when they are of different qualities since they are still striving towards the same goal. For example, the robot aims at walking forward in the walker2d task. The expert-level walker2d dataset with kinematic shift can also share some overlap with the medium-level source domain dataset (e.g., some walking movements and patterns). Furthermore, when one adopts a medium-level target domain dataset, it can at least share overlap with medium/medium-replay/medium-expert source domain datasets since they both contain medium-level data. It also applies to target domain datasets with medium-expert quality and expert quality. Hence, we believe that it is reasonable to adopt a setting like this. Moreover, OTDF still significantly outperforms baseline methods like IGDF and SRPO across various kinds of dynamics shifts when the source domain dataset and target domain dataset have the same performance level (e.g., medium-level source domain data and medium-level target domain data), as depicted in Table 1, Table 2 and Table 6.

We further clarify that both data filtering based on optimal transport and policy regularization indeed play the critical role. We reiterate that our method is fully motivated by the theoretical results in Theorem 3.1 and the experimental setting where we only have a limited budget of target domain data. Data filtering effectively controls the dynamics mismatch term (c) in Theorem 3.1 by keeping source domain data that have similar transition dynamics as the target domain dynamics (as discussed in Lines 160-165). Optimal transport is a principled approach for comparing two distributions and is ideal to be applied for reliable data filtering due to its training-free nature, given that we only have limited target domain data. Furthermore, the dataset constraint objective better controls the target policy deviation term (b) in Theorem 3.1 and ensures that the learned policy would not get biased towards the source domain dataset distribution considering that the target domain dataset size is limited. The parameter study in Figure 2 and Figure 3 show that excluding either data filtering ($\xi=100$) or dataset constraint ($\beta=0$) often incurs a performance drop, indicating the necessity and effectiveness of both components to OTDF. The influence of data filtering can also be large for many tasks. We agree that the introduction of the dataset constraint term is important to OTDF and previous methods like SRPO and IGDF do not include it, but this component is still our novel contribution.

Concern 2: differences between our work and some prior works

Thanks for recommending GWIL [1], which we have actually included in our original submission. Despite that we both utilize optimal transport in the cross-domain setting, our method differs from GWIL [1] in the following aspects:

• the motivations are different. GWIL [1] solves the cross-domain imitation learning problem from a new perspective by directly formalizing it as an optimal transport problem. Our method, instead, utilizes optimal transport (OT) due to (a) OT is a principled and widely used approach for comparing two distributions; (b) we only have a very limited budget of target domain data, and OT can work given such low data regime thanks to its training-free nature.
• the experimental settings are different. GWIL focuses on the cross-domain imitation learning scenario, where expert demonstrations are needed. Our OTDF addresses the cross-domain offline RL problems with different performance level offline source domain or target domain datasets (no expert demonstration is required).
• the core methods are different. GWIL leverages the optimal transport for constructing pseudo-rewards while OTDF utilizes the optimal transport for data filtering (i.e., keeping source domain data that have similar transition dynamics as the target domain dynamics)

[1] Cross-domain imitation learning via optimal transport