Iteratively Refined Behavior Regularization for Offline Reinforcement Learning

Whether the algorithm does what the authors claim it does.

While Proposition 1 establishes that the exact solution of Eq 9 remains within the data support when the actor is initialized as $\pi_\omega=\pi_D$, practical implementations often rely on a limited number of gradient descent steps for optimizing Eq 9, thus could suffer from our-of-support samples. This leads to policy optimization errors which are further exacerbated iteratively. To approximate the in-support learning, we find it is useful to add the original behavior regularization, which further constrains the policy on the support of data to enhance learning stability. As different datasets have different support, it’s necessary to help the policy stay in the support of the dataset by tuning the hyperparameter across different environments. However, it’s not hard to tune the parameter across different environments and datasets. In Section 5.3.4, we provide ablations of the effect of the two hyperparameter and also summarize some empirical experiences in the section to help users find the suitable hyperparameters when encountering new environments.

Results are based on per-environment hyperparameter optimization.

We also provide results of CPI with as little as possible hyper-parameter tuning on the same domain in Table 5 in our appendix. The results show that compared with IQL, CQL and TD3BC, the overall performance of CPI with as little as possible hyper-parameter tuning is still significantly better.

As analyzed before, the main reason to tune the hyper-parameter is to approximate the in-support learning according to the property of different datasets. For example:

• $\lambda$ : When $\lambda=0.1$, the early-stage performance excels, as the behavior policy assists in locating appropriate actions in the dataset. However, this results in suboptimal final convergence performance, attributable to the excessive behavior policy constraint on performance improvement. For larger values, such as 0.9 , the marginal weight of the behavior policy leads to performance increase during training. Unfortunately, the final performance might be poor. This is due to that the policy does not have sufficient behavior cloning guidance, leading to a potential distribution shift during the training process. Consequently, we predominantly select a $\lambda$ value of 0.5 or 0.7 to strike a balance between the reference policy regularization and behavior regularization. In practice, for dataset of higher quality and lower diversity (e.g., expert dataset), we encourage authors to try small $\lambda$. For datasets of lower quality and higher diversity (e.g., medium dataset), larger $\lambda$ should be chosed.
• $\tau$ : The regularization parameter $\tau$ plays a crucial role in determining the weightage of the joint regularization relative to the Q-value component. We find that (Figure 6 b) $\tau$ assigned to dataset of higher quality and lower diversity (e.g., expert dataset) ought to be larger than those associated with datasets of lower quality and higher diversity (e.g., medium dataset).

Anonymous code link is expired

We have modify the link and it could be open now. Please check the anonymous link in the paper.

discrepancy between KL and MSE

Actually $D_{K L}\left(\pi_w(s) \| \bar{\pi}(s)\right)$ was implemented in the experiment as an MSE loss like $1 / N \sum\left(\pi_w(s)-\bar{\pi}(s)\right)^2$. For two gaussians, the KL between them can be calculated as $K L(p, q)=\log \frac{\sigma_2}{\sigma_1}+\frac{\sigma_1^2+\left(\mu_1-\mu_2\right)^2}{2 \sigma_2^2}-\frac{1}{2}$. As $\sigma_1$ equals to $\sigma_2$ in the two gaussians, we can obtain the simplified form of this KL as an MSE loss as you described. For deterministic policies, they could be seen as the mean of a gaussian. Therefore, MSE is the specific implementation of KL in our work, so does in TD3BC. That's also the reason we claim that our algorithm is easy to implement, requiring only a few lines of code modification to existing method, i.e., TD3+BC. In our provided codes, you could find the corresponding implementation details.

the delay between the current policy and older policy, how much does this impact performance of the algorithm

The default update interval (UI) of the reference policy in CPI is set to 2 gradient steps. We ablate the update interval 4 and 8 here on six datasets across three seeds in the below table:

It could be seen that increase the update interval of reference policy in CPI could have overall negative impact on the policy’s performance. While with UI=4 on some datasets (halfcheetah-medium, walker2d-medium) it seems there isn’t significant negative influence, in other situations the performance is severely dropped. This may be attributed to the delayed update causes the performance gap between learning policy and the reference policy become larger, that is, the reference policy will remain worse performance for longer time. Thus the reference policy could be more possible to introduce more instability to the training and drag down the performance of the learning policy.

Thanks for your reply!

Actually, from the results we can see that UI=8 has already severely drag down the algorithm's performance. Thus it's highly likely increasing the update rate further could cause more negative influence. In addition, as far as we know, the mentioned large target updating rate are nomally used in the Q-function updating instead of the policy updating. In the standard TD3 algorithm, the target policy's UI is set to 2. And in the recent SOTA off-policy RL method CrossQ [1], the target policy updating interval is set to 3. Thetefore, from both our empirical results and the setting of other works, it's highly likely the policy updating interval would be better for the performance at a small value.

It's true that there are baselines that don't require tuning in their paper, such as TD3+BC on MuJoCo domain, which is known for its simplicy. However, the unchanged setting could cause the algorthm perform quite horrible on other domains such as AntMaze and Adroit. See CORL code repository for detailed results (https://github.com/tinkoff-ai/CORL/tree/main). In addition, several recent SOTA works such as XQL [2], Diffusion-QL [3], STR [4] and SVR [5] on different domains. Therefore, we believe hyperparameters tuning are neccessary for most algorithms on different domains to achieve satisfying performance. We will also delve into some techniques to automatically optimize the hyperparameters in the future works.

We sincerely hope this could address your concerns. We look forward to receiving your further feedback!

[1] CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity. ICLR 2024.

[2] Extreme Q-Learning: MaxEnt RL without Entropy. ICLR 2023.

[3] Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning. ICLR 2023.

[4] Supported Trust Region Optimization for Offline Reinforcement Learning. ICML 2023.

[5] Supported Value Regularization for Offline Reinforcement Learning .NeurIPS 2023.

more detailed discussion and hyperparameter study results

In Section 5.3.4, we provide ablations of the effect of the two hyperparameters. We also summarize some empirical experiences in the section.

• $\lambda$ : When $\lambda=0.1$, the early-stage performance excels, as the behavior policy assists in locating appropriate actions in the dataset. However, this results in suboptimal final convergence performance, attributable to the excessive behavior policy constraint on performance improvement. For larger values, such as 0.9 , the marginal weight of the behavior policy leads to performance increase during training. Unfortunately, the final performance might be poor. This is due to that the policy does not have sufficient behavior cloning guidance, leading to a potential distribution shift during the training process. Consequently, we predominantly select a $\lambda$ value of 0.5 or 0.7 to strike a balance between the reference policy regularization and behavior regularization. In practice, for dataset of higher quality and lower diversity (e.g., expert dataset), we encourage authors to try small $\lambda$. For datasets of lower quality and higher diversity (e.g., medium dataset), larger $\lambda$ should be chosed.
• $\tau$ : The regularization parameter $\tau$ plays a crucial role in determining the weightage of the joint regularization relative to the Q-value component. We find that (Figure 6 b) $\tau$ assigned to dataset of higher quality and lower diversity (e.g., expert dataset) ought to be larger than those associated with datasets of lower quality and higher diversity (e.g., medium dataset).

In addition, we also provide results of CPI with as little as possible hyper-parameter tuning on the same domain in Table 5 in our appendix. The results show that compared with IQL, CQL and TD3BC, the overall performance of CPI with as little as possible hyper-parameter tuning is still significantly better.

ablation on additional offline datasets

In the PDF of the general response, we provide more hyperparamerts ablations. All the results are averaged across three seeds using the final obtained model after training for 1M gradient steps. The results further prove our empirical experiences of hyperparameter tuning provided above.

detailed experimental setup of Figure 1

In Figure 1 we empirically demonstrate the impact of regularization utilizing distinct policies on the 'hopper-medium-replay' and 'hopper-medium-expert' datasets in D4RL. We leverage Percentile Behavior Cloning (%BC) to generate policies of varied performance \citep{chen2021decision}. Specifically, for behavioral cloning, we filter trajectories of the top 5%, median 5%, and bottom 5% returns. Following this, we modify TD3+BC to develop the TD3+5%BC algorithm by replacing the behavior regularization with the 5%BC policy, and subsequently train TD3+5%BC on the original dataset. These descriptions are also provided in the introduction section. We’ll try to make them clearer to readers.

tasks or environments where CPI may underperform typical behavior regularization method TD3+BC under the same regularization strength

In Table 8 in our appendix, we ablate different variants of TD3+BC and compare CPI with them. TD3+BC is set $\alpha$ to a constant value of 2.5 for each dataset, whereas CPI chooses the appropriate $\tau$ from a set of $\tau$ alternatives. We note that the hyperparameter that plays a role in regulating Q and regularization in CPI is $\tau$, which can essentially be understood as the reciprocal of $\alpha$ in TD3+BC. Therefore, for the convenience of comparison, we rationalize the reciprocal of $\tau$ as the parameter $\alpha$. In this section, we set the $\alpha$ of TD3+BC to be consistent with that of CPI in order to show that the performance improvement of CPI mainly comes from amalgamating the benefits of both behavior-regularized and in-sample algorithms. Further, we also compare CPI with TD3+BC with dynamically changed $\alpha$ and TD3+BC with swept best $\alpha$ in the ranges {0.0001, 0.05, 0.25, 2.5, 25, 36, 50, 100}, which improves TD3+BC by a large margin, to show the superiority of CPI. The selection of parameters is shown in Table 4.

The results for TD3+BC (vanilla), TD3+BC (same $\alpha$ with CPI), TD3+BC (swept best $\alpha$), TD3+BC with dynamically changed $\alpha$ and CPI are shown in Table 8. Comparing the variants of TD3+BC with different $\alpha$ choices, it can be found that changing $\alpha$ can indeed improve the performance of TD3+BC. However, compared with TD3+BC (same $\alpha$) and TD3+BC (swept best $\alpha$), the performance of CPI is significantly better, which proves the effectiveness of the mechanism for iterative refinement of policy for behavior regularization in CPI.

The algorithm is highly dependent on the value of $\tau$

In Section 5.3.4, we provide ablations of the effect of the two hyperparameters. We also summarize some empirical experiences in the section. The regularization parameter $\tau$ plays a crucial role in determining the weightage of the joint regularization relative to the Q-value component. We find that (Figure 6 b) $\tau$ assigned to dataset of higher quality and lower diversity (e.g., expert dataset) ought to be larger than those associated with datasets of lower quality and higher diversity (e.g., medium dataset).

Therefore, in real-world scenario, if you have a impression of the data quality, you could tune the parameters in a confident way. For example, in the recommendation system or the autonomous driving systems, you could directly distinguish and identify the quality of data collected by the previous deployed algorithms via Return on Investment (ROI) or Success Rate. If the ROI of a policy is much more better than the random policy, one can try to $\tau$ to a relative higher value. If the Success Rate of an autonomous driving car is quite low, then the $\tau$ should be set to a lower value.

In addition, we also provide results of CPI with as little as possible hyper-parameter tuning on the same domain in Table 5 in our appendix. The results show that compared with IQL, CQL and TD3BC, the overall performance of CPI with as little as possible hyper-parameter tuning is still significantly better.

BC seems to dominate performance

It’s true that high values of $\tau$ is essential when the dataset is of higher quality (i.e., higher rewards). However, when the dataset is of lower quality (i.e., lower rewards), the Q loss plays more important role. As you suggested, we include different %BC (run behavior cloning on only the top X% of timesteps in the dataset, ordered by episode returns) in our results, citing from the Decision Transformer paper:

It can be observed on these datasets of lower quality, %BC indeed help improve over the original BC (100%BC). However, they still fall behind CPI. We believe this could help you understand $\tau$'s important role in CPI’s performance.

The writing can be improved

We sincerely appreciate your suggestions for improving the readability of our paper. We have modified our paper as your suggestion!

question regarding Proposition 1.

You’re right that E_{a \\sim \\pi^*}\\left[Q^\\pi(s, a)\\right] \\geq E_{a \\sim \\pi}\\left[Q^\\pi(s, a)\\right] is different from the conclusion that V^{\\pi^*} \\geq V^\\pi. However, V^{\\pi^*} \\geq V^\\pi could be proved in a quite simple way using the Policy Improvement Theorem and the corresponding proof.

Theorem: Consider two policy $\\pi(a \\mid s), \\pi^{\\prime}(a \\mid s)$, and define Q^\\pi\\left(s, \\pi^{\\prime}\\right)=\\mathbb{E}_{a \\sim \\pi^{\\prime}(a \\mid s)}\\left[Q^\\pi(s, a)\\right] .

If $\\forall s \\in S$, we have that Q^\\pi\\left(s, \\pi^{\\prime}\\right) \\geq V^\\pi(s), then it holds that V^{\\pi^{\\prime}}(s) \\geq V^\\pi(s), \\forall s \\in S. This means that $\\pi^{\\prime}$ is atleast as good a policy as $\\pi$.

Proof. Note that Q^\\pi\\left(s, \\pi^{\\prime}\\right) \\geq V^\\pi(s). By expanding $Q^\pi$, we can get that $\\forall s \\in S$,

\begin{aligned} & V^\pi(s) \leq Q^\pi\left(s, \pi^{\prime}\right) \\ & =\mathbb{E}_{a \sim \pi^{\prime}(a \mid s), s^{\prime} \sim \mathcal{T}^{\prime}\left(s^{\prime} \mid s, a\right)}\left[r(s, a)+\gamma V^\pi\left(s^{\prime}\right)\right] \\ & \leq \mathbb{E}_{a \sim \pi^{\prime}(a \mid s), s^{\prime} \sim \mathcal{T}^{\prime}\left(s^{\prime} \mid s, a\right)}\left[r(s, a)+\gamma Q^\pi\left(s^{\prime}, \pi^{\prime}\right)\right] \\ & =\mathbb{E}_{a, a^{\prime} \sim \pi^{\prime}}\left[r(s, a)+\gamma r\left(s^{\prime}, a^{\prime}\right)+\gamma^2 V^\pi\left(s^{\prime \prime}\right)\right] \\ & \leq \ldots \\ & \leq \mathbb{E}_{a, a^{\prime}, a^{\prime \prime} \ldots \sim \pi^{\prime}}\left[r(s, a)+\gamma r\left(s^{\prime}, a^{\prime}\right)+\gamma^2 r\left(s^{\prime \prime}, a^{\prime \prime}\right)+\ldots\right] \\ & =V^{\pi^{\prime}}(s) \\ \end{aligned}

This completes the proof that the new policy $\\pi'$ is at least as good as the original policy $\\pi$ in terms of the state-value function for all states $s$. According to this Policy Improvement Theorem and the proof of Proposition 2 in our paper, we can directly obtain that $V^{\pi^*}(s)=E_{a \sim \pi^*}[Q^{\pi^*}(s, a)] \geq E_{a \sim \pi^*}[Q^\pi(s, a)] \geq E_{a \sim \pi}[Q^\pi(s, a)]= V^\pi(s)$.

We sincerely hope this could address your concerns. If you feel that the manuscript meets the criteria for a higher rating, I would be immensely appreciative of any positive adjustments to the score. Your recognition of the merits of this research would encourage us a lot. I look forward to receiving your feedback and am open to any suggestions that may further enhance the quality of this manuscript.

Thank you for your review and comments. We hope that our additional discussion and rebuttal have addressed your primary concerns with our paper. We would really appreciate feedback as to whether there are any (existing or new) points we have not covered, and we would be happy to address/discuss them!