Offline Robustness of Distributional Actor-Critic Ensemble Reinforcement Learning

``What is the average/overall performance of ORDER compared to baselines, since there is only per-task performance in Table 1. ”

Answer. Thanks for your suggestion. We have added the average performance of ORDER in Table 1 in the new version. Meanwhile, the average performance of ORDER surpasses most basic offline RL algorithms.

``There are too many details of the experiments in Section 5.2, which may be better to leave to the appendix and leave space for a more comprehensive analysis/evaluation of the proposed method ORDER. ”

Answer. Thank you for pointing this issue. We will add the results of ORDER's experiments in more environments in Section 5.2 for better analysis.

``The setting of the experiments seems not clear to the reviewer. As the goal of this work is to handle possible state perturbation, should the experiments be conducted in some settings where the state is perturbed? ”

Answer. Thanks for your valuable suggestion. In further work, we will supplement the ablation study of ORDER performance with varying perturbation scale.

`` There are also some typos and notations need to be defined: ”

``The parentheses in the definition of $Q^{\pi}$ are incorrectly positioned. ”

``In the definition of $\hat{\pi}$: Define and unify the notation of the indicator function ${1}$. ”

``Define the notation $\mathcal{P}(\mathbb{R})$ in Section 2.2. ”

``Define the notation $\mathbb{B}_{d}(s,\epsilon)$. ”

Answer. Thanks for your suggestion. We have added the definitions of the indicator function ${1}$ and the notation $\mathcal{P}(\mathbb{R})$. We also have modified the pasentheses in the definition of $Q^{\pi}$ and the inconsistency of the indicator function ${1}$. In fact, the definition of $\mathbb{B}_{d}(s,\epsilon)$ has already been given in Section 3.1.

``While the paper introduces ORDER as a modification of the prior work RORL into a distributional version, it is essential to engage in a comprehensive discussion comparing this work with RORL to underscore its significance and contributions more effectively. ”

``The experiments presented in Table 1 do not convincingly showcase the superiority of ORDER over the previous state-of-the-art offline RL algorithm, RORL. This raises concerns about the relevance and significance of the studied problem: offline distributional RL. ”

``The ablation experiments, considering the number of introduced components, are rather limited. For a more comprehensive understanding, it would be valuable to explore the individual impacts of components like the policy smooth loss, OOD penalty, and state perturbation on the proposed algorithm's performance. ”

Answer. Thank you for pointing this out. You may have some misconceptions about what we do. In this paper, we introduce smoothing techniques from RORL into CODAC to enhance the robustness of offline distributional reinforcement learning. In our experiments, the performance of ORDER surpasses offline distributional reinforcement learning algorithms, such as CODAC, MQN-CQR. Meanwhile, ORDER also outperformed RORL results on datasets such as hopper-medium-replay and walker2d-random.

``The proposed method lacks enough novelty. First, as mentioned by the authors, the techniques of distributional RL have already been introduced into the offline RL settings. Second, adopting an ensemble of value networks is a widely known technique that can improve performance in the RL community. Third, adding smoothness constraints in the offline RL settings has already been explored in the offline RL settings. ”

Answer. First, distributional RL has been applied to offline RL [1,2]. However, it is well known that distributional RL is widely used in online setting while its academic exploration is still incomplete in offline setting. Existing distributional offline RL methods only focus on the safety of the learned policy. These methods leverage a conservative return distribution to impair the robustness. In contrast, our work considers improving the robustness of the learned policies under control of the distribution shift.

Second, adopting an ensemble of value networks is a widely known technique that can improve performance in the RL community. On the one hand, double $Q$ networks are often used in RL algorithms to alleviate the overestimation of $Q$ functions. On the other hand, the number of distributed integrated networks has been explored in online setting [3]. However, offline RL is different from online reinforcement learning in that it has some additional problems, such as distribution shift and OOD problems. Many scholars' hasty use of the double network technique has hindered the exploration of the number of value networks in ensemble in the field of offline RL.

Third, it is known that adding smoothness constraints to offline RL has been explored in the offline settings, such as S4RL [4] and RORL [5]. However, some previous related work enforced smoothness in $Q$ functions, and the smoothing treatment in our work applied to distribution action value functions. We can use the framework of distributional RL to consider risk-sensitive learning and the performance of the algorithm under different risk measures, so as to ensure the security of learned polices. Besides, when the $Q$ value is updated, the loss function takes the form of mean square error. Huber loss is adopted in distributional RL, which can better fit our research. Therefore, the problem we studied is an interesting and meaningful work.

[1]Will Dabney, et al., Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning, pp. 1096-1105, 2018a.

[2] Xiaoteng Ma, et al., DSAC: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint arXiv:2004.14547, 2020.

[3] Lan, Q., et al., Maxmin q-learning: Controlling the estimation bias of q-learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. [4] Sinha et al., S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning in Robotics. 5th Annual Conference on Robot Learning (CoRL).

[5] Rui Yang, et al., RORL: Robust offline reinforcement learning via conservative smoothing. In Advances in Neural Information Processing Systems, volume 35, pp. 23851–23866, 2022.

``Could the authors provide insights into the computational complexity and scalability of the ORDER algorithm? ”

Answer. On the one hand, for some offline reinforcement learning algorithms, such as RORL [1], our algorithm is to model quantile networks of action values which will introduce more network parameters. On the other hand, for offline distributional reinforcement learning algorithms such as CODAC [2], we add a smooth term loss. Although these will increase the computational complexity to some extent, the performance of ORDER is much higher than the current advanced offline distributional reinforcement learning algorithms, such as CODAC, MQN-CQR. Meanwhile, ORDER also outperforms RORL results on datasets such as hopper-medium-replay and walker2d-random.

``It would be beneficial to evaluate the ORDER algorithm on a wider range of tasks and datasets beyond the D4RL benchmark. This would demonstrate the generalizability of ORDER and its effectiveness in different domains. ”

Answer. Thanks for your comment. The wider application of our algorithm in real-world scenarios will be a future work.

[1] Rui Yang, et al., RORL: Robust offline reinforcement learning via conservative smoothing. In Advances in Neural Information Processing Systems, volume 35, pp. 23851-23866, 2022.

[2] Yecheng Ma, et al., Conservative offline distributional reinforcement learning. In Advances in Neural Information Processing Systems, volume 34, pp. 19235-19247, 2021.