Doubly Mild Generalization for Offline Reinforcement Learning

We appreciate the time and effort you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.

Q1: More random seeds and use of confidence interval in evaluation.

Thanks for the kind suggestions and this nice reference. [1] provides a comprehensive resource for how to do good experiments in RL. We have taken your advice and conducted experiments on 10 random seeds, reporting results based on a 95% confidence interval (CI). The following table shows the comparison between the newly obtained results (10seeds/95%CI) and the previously reported results (5seeds/SD). We also present the new learning curves on all the tasks in Figures 1 and 2 of the PDF (attached to the global response). The results show that our method achieves about the same performance as under the previous evaluation criterion. In the latter revision, we will cite [1] and update the results accordingly.

Table 1: Comparison of DMG under different evaluation criteria.

Q2: How do the authors think the insights in this paper can be used for off-policy evaluation where the behavior policy is unknown and we want to evaluate a fixed target policy?

Thanks for this insightful question. Indeed, the evaluation setting appears trickier because the target policy is fixed. While mild action generalization no longer seems applicable in this scenario, mild generalization propagation still applies. Its direct analogue in the off-policy evaluation scenario is that: the Bellman target uses a mixture of (1) the Q-value of the action outputted by the target policy and (2) the Q-value of the nearest neighbor of that action in the dataset (or the replay buffer). This approach brings bias on the one hand, but can also be expected to have a smaller variance. An in-depth analysis of this would be an interesting direction for future work.

Q3: The method seems to rely on using the empirical behavior policy to constrain the policy improvement.

We apologize for the confusion. Actually, our method does not require training an empirical behavior policy. The implementation of policy improvement is discussed in Section 3.5 of the paper. Here, we first define a reshaped empirical behavior policy $\hat\beta^*$. Then we enforce the proximity between the trained policy and the reshaped behavior policy in Eq. (13). A special design worth noting is that we use the forward KL divergence $\mathrm{KL}(\hat\beta^*(\cdot|s) \| \pi_\phi(\cdot|s))$, which not only allows $\pi$ to select actions outside the support of $\hat\beta^*$ but also eliminate the need for pretraining an empirical behavior policy. Specifically, by substituting the analytical expression of $\hat\beta^*$ into the KL divergence and using importance sampling (the same derivation as in AWR [2]), Eq. (14) eliminates the need for pre-training a behavior model (maybe multi-modal as you mentioned) and enables direct sampling from the dataset for optimization.

Reference

[1] Patterson, Andrew, et al. "Empirical design in reinforcement learning." arXiv preprint arXiv:2304.01315 (2023).

[2] Peng, Xue Bin, et al. "Advantage-weighted regression: Simple and scalable off-policy reinforcement learning." arXiv preprint arXiv:1910.00177 (2019).

We appreciate the time and effort you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.

Q1: Dependence on Continuity Assumptions.

Thanks for the comment. Indeed, our work relies on continuity assumptions about the learned Q function and transition dynamics. For the former, a continuous learned Q function can be relatively easily satisfied and is particularly necessary for the analysis of value function generalization. For the latter, continuous transition dynamics is also a standard assumption in the theoretical studies of RL [1,2,3,4,5]. Several previous works assume the transition to be Lipschitz continuous with respect to (w.r.t) both state and action [1,2]. In our paper, we need the Lipschitz continuity to hold only w.r.t. action.

Q2: Potential Sensitivity to Hyperparameters

Thanks for the comment. We will answer this question from two aspects: the robustness of the DMG principle and the robustness of the specific DMG algorithm. We have conducted an ablation study on the mixture coefficient $\lambda$ and penalty coefficient $\nu$ in Section 5.3. As shown in Figures 1 and 2, DMG generally achieves the best performance with $\lambda \in (0,1)$ and a moderate $\nu$. This demonstrates that our DMG principle generally outperforms previous non/full generalization propagation and non/full action generalization principles, which proves the robustness of the DMG principle. On the other hand, our specific DMG algorithm also involves less hyperparameter tuning overall compared to other algorithms. As stated in Appendix C1, we use $\lambda=0.25$ for all tasks, and use $\nu=0.5$ for Antmaze, $\nu \in \{0.1, 10\}$ for Gym locomotion ($0.1$ for medium, medium-replay, random datasets; $10$ for expert and medium-expert datasets)。This demonstrates the robustness of the specific DMG algorithm.

Q3: Can you explain why using max and min in Eq. (5)?

We apologize for the confusion. Eq. (5) defines mildly generalized policy, where $\tilde \beta$ and $\hat \beta$ can be any policy beyond Gaussian or deterministic policy. The max and min in Eq. (5) mean that for any $a_1 \sim \tilde \beta(\cdot|s)$ (i.e., from the defined mildly generalized policy), we can find $a_2 \sim \hat \beta(\cdot|s)$ (i.e., from the dataset) such that $\|a_1-a_2\| \leq \epsilon_a$. In other words, in a certain state, the distance between actions taken by $\tilde \beta$ and its closest action in the dataset is bounded by $\epsilon_a$.

Q4: What about directly regularizing the KL between  $\pi$ and $\hat \beta$ in Eq. (13)?

Thanks for the question. This is equivalent to setting the inverse temperature $\alpha$ as 0. The comparisons between DMG ($\alpha=0$) and the original DMG on Gym locomotion tasks are shown in the following table. Compared to DMG, DMG ($\alpha=0$) exhibits only a small drop in performance. Therefore, DMG is robust to this hyperparameter, and the choice of aligning $\pi$ and $\hat \beta^*$ in implementation is not a key factor in DMG's good performance.

Table 1: Averaged normalized scores on Gym locomotion tasks over 5 random seeds.

Q4: Comparison with more SOTA algorithms such as STR is suggested.

Thanks for the suggestion. Due to the page limit, we report the overall performance of STR (taken from its paper) and DMG in the following table. We will include the full comparison in our latter revision. Overall, DMG achieves slightly higher performance in both Gym locomotion and Antmaze domains.

Table 2: Total scores on Gym locomotion and Antmaze tasks, averaged over 5 random seeds.

Q5: What does the symbol × mean in Figure 1?

Sorry for the confusion. The crosses (x) in Figure 1 mean that the value functions diverge in several seeds.

Q6: It seems the performance of DMG when $\lambda=1$ in Figure 1 is inferior to that of TD3BC.

When $\lambda=1$, DMG, just like TD3BC, allows full generalization propagation in value training. However, policy training and the specific hyperparameter configuration of DMG still differ from TD3BC. We hypothesize this may cause the discrepancy.

Reference

[1] Dufour, Francois, and Tomas Prieto-Rumeau. "Finite linear programming approximations of constrained discounted Markov decision processes." SIAM Journal on Control and Optimization 51.2 (2013): 1298-1324.

[2] Dufour, Francois, and Tomas Prieto-Rumeau. "Approximation of average cost Markov decision processes using empirical distributions and concentration inequalities." Stochastics An International Journal of Probability and Stochastic Processes 87.2 (2015): 273-307.

[3] Shah, Devavrat, and Qiaomin Xie. "Q-learning with nearest neighbors." Advances in Neural Information Processing Systems 31 (2018).

[4] Xiong, Huaqing, et al. "Deterministic policy gradient: Convergence analysis." Uncertainty in Artificial Intelligence. PMLR, 2022.

[5] Ran, Yuhang, et al. "Policy regularization with dataset constraint for offline reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

We appreciate the time and effort you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.

Q1: Assumption 11 looks very restrictive.

We apologize for the lack of a detailed explanation of this assumption in the paper. To our knowledge, Assumption 11 (i.e., Assumption 3) is common in the theoretical studies of RL [1,2,3,4,5]. Several previous works assume the transition to be Lipschitz continuous with respect to (w.r.t) both state and action [1,2]. In our paper, we need the Lipschitz continuity to hold only w.r.t. action. However, we also acknowledge, as you mentioned, that the assumption reduces the theoretical applicability. Despite this, DMG performs quite well empirically on those robotics tasks with deterministic dynamics. In real-world scenarios, even for deterministic dynamics, there is some noise, resulting in a narrow distribution for state transitions. In this case, the assumption still applies, although the magnitude of the Lipschitz constant may vary depending on the level of noise.

Q2: The idea of generalization to stata-action pairs in the neighborhood of those seen in the dataset has been studied in [6] and I urge the authors to cite and state the differences to the approach in this paper.

Thanks for the reference. The research work [6] is crucial in the analysis of multi-task offline RL, and we will cite it and include the following discussions.

Recently, theoretical advancement [6] has explored multi-task offline RL from the perspective of representation learning and introduced a notion of neighborhood occupancy density. The neighborhood occupancy density at some $(s, a)$ in the dataset for a source task is defined as the fraction of points in the dataset within a certain distance from $(s, a)$ in the representation space. The authors use this concept to bound the representational transfer error in the downstream target task. In contrast, DMG is a wildly compatible idea in offline RL and provides insights into many offline RL methods. DMG balances the need for generalization with the risk of over-generalization in offline RL. Specifically, DMG achieves doubly mild generalization, comprising mild action generalization and mild generalization propagation. Generalization to stata-action pairs in the neighborhood of the dataset is a specific realization of mild action generalization. Additionally, in our work, the neighborhood is defined based on the action space distance.

Reference

[1] Dufour, Francois, and Tomas Prieto-Rumeau. "Finite linear programming approximations of constrained discounted Markov decision processes." SIAM Journal on Control and Optimization 51.2 (2013): 1298-1324.

[2] Dufour, Francois, and Tomas Prieto-Rumeau. "Approximation of average cost Markov decision processes using empirical distributions and concentration inequalities." Stochastics An International Journal of Probability and Stochastic Processes 87.2 (2015): 273-307.

[3] Shah, Devavrat, and Qiaomin Xie. "Q-learning with nearest neighbors." Advances in Neural Information Processing Systems 31 (2018).

[4] Xiong, Huaqing, et al. "Deterministic policy gradient: Convergence analysis." Uncertainty in Artificial Intelligence. PMLR, 2022.

[5] Ran, Yuhang, et al. "Policy regularization with dataset constraint for offline reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

[6] Bose, Avinandan, Simon Shaolei Du, and Maryam Fazel. "Offline multi-task transfer rl with representational penalization." arXiv preprint arXiv:2402.12570 (2024).

We appreciate the time and effort you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.

Q1: The constants $C_1$ and $C_2$ of Theorem 1 are state-dependent.

Indeed, they are state-dependent. It is worth noting that Theorem 1 provides insight and motivates our method.

Q2: How does (4) relate to the generalizability of Q functions?

We apologize for the confusion caused by the lack of sufficient explanation. Eq. (4) is the update of the parametric Q function ($Q_{\theta} \rightarrow Q_{\theta’}$) at state-action pairs $(s,\tilde a) \notin \mathcal D$, which is exclusively caused by generalization. If $\tilde a$ is within a close neighborhood of $a$, then $C_2\|\tilde a-a\|$ is small. Moreover, as $C_1 \in [0,1]$, Eq. (4) approximates an update towards the true objective $\mathcal T_u Q_\theta(s,\tilde a)$, as if $Q_\theta(s,\tilde a)$ is updated by a true gradient step at $(s,\tilde a) \notin \mathcal D$.

We will add these explanations to the latter revision.

Q3: The meaning of Assumption 1 is unclear at first glance. Rephrase the term "well-defined".

Thanks for the kind suggestion. Indeed, the term "well-defined" may not be very suitable in this context. To avoid confusion, we will delete the sentence "In other words, $\mathcal{T}_{\mathrm{DMG}}$ is well defined in the mild generalization area $\tilde \beta(a|s)>0$." in Assumption 1. In addition, we will include more explanations before line 158 as follows.

Because the mild generalization area $\tilde \beta(a|s)>0$ may contain some points outside the offline dataset, $\mathcal{T}_{\mathrm{DMG}}$ might query Q values of such points. In this assumption, we assume that the generalization in the mild generalization area is correct and meaningful. The rationale for such an assumption…

Q4: Minor comments

Special thanks for your careful review and we will update them in the latter revision.