Policy Rehearsing: Training Generalizable Policies for Reinforcement Learning

Q1: Without an offline dataset, it seems that there is no idea for the agent to learn about the dynamics.

A1: We would like to clarify that ReDM does NOT learn the target dynamics. ReDM generates diversified and eligible candidate models. ReDM then trains a meta-policy in these models, so that the meta-policy is able to generalize over different dynamics. Without any interaction data, the generated candidate models are expected to represent the whole model space. In such cases, the meta-policy can adapt to any environment including the real environment.

Q2: Relation to unsupervised RL and potential of incorporating fine-tuning?

A2: It is quite novel and interesting to interpret rehearsal as a kind of unsupervised RL without dynamics, and we have added a discussion about the relation in Appendix G. As for fine-tuning, it is worth noting that in Section 4.1 Figure 2, we conduct a fine-tuning experiment on InvertedPendulum, where we fine-tuned the learned meta-policy with online trajectories. Our meta-policy can achieve fast adaptation compared to directly learning from online trajectories.

Q3: Is random sampling inefficient? What is the choice of N?

A3: We agree that a random shooting planner can be less efficient and there are more advanced methods that can be applied instead. However, for the purpose of concept-proof and simplicity, we show that with such a simple method ReDM can also yield good results. N is set to $50$. We have updated the hyper-parameter table in Appendix C.2.

Q4: The performance of ReDM in the D4RL setting but without any interaction data?

A4: As the state dimensionality increases, ReDM requires much more computation power to generate sufficient many models for training an effective meta-policy. We will report the results on D4RL benchmarks. In these cases, we simultaneously explore ways to incorporate more constraints to reduce the model space, such as ReDM-o that can incorporates mismatching data.

We hope the above explanation can answer your questions, but we are willing to follow up if you have further questions.

Thank you for your interest in our paper, we briefly summarized your concerns and questions as follows:

Comment1: The connection between diversity and controlling $\epsilon_a$ and $\epsilon_e$ is unclear.

A1: We did not claim that diversity can control the error term $\epsilon_e$ in the paper, so we conjecture that you were actually questioning the relationship between $\epsilon_a, \epsilon_m$ and diversity. To control the error term $\epsilon_m$, the candidate models should be dispersed throughout the hypothesis space of the dynamics. This ensures that the target environment falls within the vicinity of one of the candidate dynamics with a high likelihood. Conversely, if the candidate dynamics are concentrated within a limited area, the target environment risks being an outlier.

As for $\epsilon_a$, we conjecture that with a diversified candidate model, it should be easy for the meta-policy to differentiate the context. However, this is an intuition without rigorous proof, and we have removed this statement.

Comment2: Can you prove that the objectives can control the error terms?

A2: The error term $\epsilon_e$ requires the candidate dynamics to contain paths toward high rewards. We formulate this as a reinforcement learning task, by treating the dynamics as an agent and optimizing for the reward achieved by a fixed planner. Regarding $\epsilon_m$, after building the connection between controlling this term and enhancing the diversity, we adopted an adversarial methodology which is proven to generate distinguished candidate models (Lemma 3.4). we will continue to explore how to optimize these errors more directly in future work.

Comment3: In experiments, providing the results directly trained in the target environments as the reference will better show the results.

A3: We trained expert policies directly in the three environments using SAC, and the relative performances (the performance of SAC minus that of the random policy) are listed in the following table. Although expert policies achieve higher scores, we find that ReDM still manages to obtain about 30% and 70% of the expert’s performances in MountainCar and Acrobot respectively. We think this is notable considering that ReDM requires no interaction with the environment. More details about this can be found in Appendix F.

Comment4: Related work on utilizing model-based methods for improving generalization and skill discovery in unsupervised RL is missing.

A4: Thanks for pointing out this. We updated Section 2 in the main text to discuss about papers that are most related to ReDM. It's worth noting that a detailed discussion about related works is deferred to Appendix G due to space limits. We also updated this appendix section to include papers about generalization and unsupervised RL based on your recommendation.

We would like to express our gratitude for your constructive feedback on our paper. We have revised our paper based on your suggestions, including:

• Using Figure 15 in Appendix H to explain the idea of policy rehearsing. We do not use the figure to replace the theory part in our main paper currently since there are concerns about the theories from other reviewers. However, we are willing to reorganize the paper structure once these concerns are addressed.
• Adding standard deviation for Table 1. However, we did not add an error bar to Figure 7 since the results are aggregated over different datasets and gravity levels. Instead, we provide the mean and standard deviation of the performance for each task in Appendix E. Furthermore, we have revised our paper here to enhance clarity in our expression.
• Explaining $D_{TV}$ in Theorem 3.3.
• Explaining the meaning of relative performance. Yes, it is calculated by minus the baseline performance.
• Changing the ticks of the x-axis in Figure 3.

For the remaining questions:

Q1: How about replacing the random shooting planner with a human-crafted planner?

A1: Thanks for your suggestions. We agree that a random shooting planner can be less efficient and there are more advanced methods that can be applied instead. However, for the purpose of concept-proof and simplicity, we show that with such a simple method ReDM can also yield good results.

Q2: ReDM's performance in complex MuJoCo environments without offline data.

A2: As the state dimensionality increases, ReDM requires much more computation power to generate sufficient many models for training an effective meta-policy. We will report the results on D4RL benchmarks. In these cases, we simultaneously explore ways to incorporate more constraints to reduce the model space, such as ReDM-o that can incorporates mismatching data.

We hope the explanation provided above addresses your questions. However, we are more than willing to provide further clarification or address any additional inquiries you may have.

Thank you for your suggestions and comments. For those smaller comments you mentioned, we have fixed those issues and updated our paper. In the following, we provide more explanation for the rest questions and concerns.

Comment1: Significant assumptions that may limit the applicability of ReDM: (1) random plans to achieve non-zero data; (2) query-able reward, initial state distribution and parameterized dynamics.

A1: About planners: We agree that a random shooting planner can be less efficient and there are more advanced methods that can be applied instead. However, for the purpose of concept-proof and simplicity, we show that with such a simple method ReDM can also yield good results.

About reward, termination functions, and initial state distribution. In many real-world applications of reinforcement learning, human experts are aware of the basic properties of the task, including the reward function, termination function, and initial states. These properties can often be cheaper to obtain than interaction data. Therefore, assuming known reward and termination functions are acceptable in these situations. Meanwhile, we will continue to explore more methods without these assumptions in the future.

About parameterization of dynamics models. We employ neural networks to represent the dynamics, which can have strong expressive power and have been extensively applied in previous model-based RL studies (e.g., MBPO, MOPO, MuZero, etc). Thus we don't think this is an overly strong assumption.

Comment2: Another concern is the limited scope of the experiments relative to prior work. It would be good to additionally include at least Walker2d. Additionally, it would be good to also show comparisons to prior work with interaction data of varying optimality and amounts.

A2: We actually did the comparison between ReDM-o with other baseline offline RL methods using full D4RL datasets including Walker2D, and the aggregated results are presented in Section 4.3 Figure 7 in the manuscript. Detailed results are listed in Table 5 in Appendix E, where Gravity-1.0 corresponds to standard offline RL tasks, and others mean the dataset and the target environment are mismatched. We also updated our paper and included some D4RL Adroit tasks in the following table, which also shows the effectiveness of our method.

Comment3: MAPLE is very related and deserves discussion in the related work.

A3: Thank you for your suggestions. We discussed with MAPLE in Appendix G, and we have revised to move the discussion into Related Work. We would like to clarify that MAPLE does not focus on model generation, and its models are learned from data with some randomness. ReDM studies how to generate models for training generalizable policies, and shows to work even with zero data or with mismatching data.

Q1: The explanation of the optimal policy gap is confusing. Specifically this sentence: "This discrepancy highlights the candidate model set’s capability to derive a proficient policy in the model itself." Does "model" here mean the true dynamics?

A1: Sorry for the confusing expression. The "model" refers to the learned candidate model for policy optimization instead of the real dynamics. The optimal policy gap refers to the difference in performance between the optimal policy in the true dynamics and the best policy found in each candidate model. A large optimal policy gap indicates that the learned model has a large difference from the real dynamics.

Q2: It's not clear why a more diverse candidate model would lower the adaptation cost.

A2: This statement is only an intuition without proof. We will revise and remove it.

Q3: In Figure 6, what is the shown performance relative to? Is this the performance of the policy at each iteration in the model at that iteration minus the performance of the policy at that iteration in the ground truth model?

A3: The relative performance at iteration $i$ is calculated as (return of the final policy in the $i$-th model) - (return of the final policy in the ground truth model). We will revise to make this clear.

Q4: Figure 7: Are these results averaged over Hopper and HalfCheetah and averaged over each gravity level?

A4: Yes, the results are averaged over Halfcheetah, Hopper and Walker2D and over all gravity levels and datasets in D4RL MuJoCo. The detailed result for each domain, dataset and gravity level is listed in Table 5 in Appendix E.

We hope our explanation and clarification can address your concerns. We are willing to offer further clarification or address any additional questions you may have.

Thanks for your constructive comments. We provide more clarifications and empirical results for your questions and concerns.

Comment 1: The environments used in the experiment are limited and more environments should be included.

A1: Thank you for your suggestions. We have conducted experiments on 6 environments and will include more environments in the revised version. For example, we are conducting more experiments on D4RL Adroit tasks. The results on 4 of the environments are listed below, showing that the proposed method is still effective.

Comment 2: ... needs to train the new dynamics models and meta-policy simultaneously, ... the training stability/convergence are encouraged to be clarified and analyzed.

A2: We would like to clarify that the dynamics models and meta-policy are trained alternately. Such alternating training has been widely employed in solving minimax problems. We will revise to discuss that when there are sufficiently many dynamics models generated, the models can cover the model space, and thus the performance of the meta-policy should converge.

Comment 3: The assumption of access to task reward and initial state distribution is unrealistic.

A3: About reward, termination functions, and initial state distribution. In many real-world applications of reinforcement learning, human experts are aware of the basic properties of the task, including the reward function, termination function, and initial states. These properties can often be cheaper to obtain than interaction data. Therefore, assuming known reward and termination functions are acceptable in these situations. Meanwhile, we will continue to explore more methods without these assumptions in the future.

About parameterization of dynamics models. We employ neural networks to represent the dynamics, which can have strong expressive power and have been extensively applied in previous model-based RL studies (e.g., MBPO, MOPO, MuZero, etc). Thus we don't think this is an overly strong assumption.

Q1: How can the generated dynamics model approximate the real dynamics without any interaction data? Does there exist the probability that the generated dynamics models are far from the dynamics in the target environment?

A1: ReDM does not approximate the real dynamics as we explained in Common Question 1. ReDM generates diverse dynamics models and thus dynamics models far from the real dynamics are often generated. These diverse dynamics models help train strongly generalizable meta-policies. We also present ReDM-o for when some interaction data is available. ReDM-o can generate dynamics models around the real dynamics.

Q2: The D4RL benchmark in your experiments is all Mujoco tasks with low input dimensions. Could you please consider incorporating some more high-dimensional tasks, in which the hypothesis space is too large to narrow down?

A2: Please refer to our response to Comment 1.

Q3: I cannot agree with the statement that the biasedness of the interaction data will hinder policy optimization in traditional offline RL methods.

A3: We would like to clarify that our purpose is to obtain a highly generalizable policy, in particular when the offline data mismatches the real environment. In the case that the pre-collected trajectories are from the expert and match the real environment, our experiments also verify that ReDM does not degrade the performance.

Q4: Learned dynamics models may be unreasonable like violating the physics or economics laws, and eligibility can hardly help alleviate this issue.

A4: We would like to clarify that the key to the success of ReDM is to avoid generating too many unreasonable dynamics models. ReDM has employed eligibility to prune some unreasonable dynamics models, which has already led to good performance in cases. If more unreasonable dynamics models can be pruned, ReDM can surely improve its efficiency. We will also explore other principles to prune more unreasonable dynamics models.

We hope that our explanation and clarification have effectively addressed your concerns. We are readily available to provide further clarification or address any additional questions you may have.

[1]. Rawal Khirodkar et al. Adversarial Domain Randomization.

[2]. Lerrel Pinto et al. Robust Adversarial Reinforcement Learning.

[3]. Marc Rigter et al. RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning.

[4]. Tianhe Yu et al. MOPO: Model-based Offline Policy Optimization.

Thank you for the response. Most of my concerns are addressed. Besides, you are encouraged to further explore more methods without the assumptions mentioned in my Weakness 3. Overall, this paper is still of nice quality and I will raise my score. Good luck :)