Relaxed State-Adversarial Offline Reinforcement Learning: A Leap Towards Robust Model-Free Policies from Historical Data

We thank reviewer 6xeb for the insightful comments. Below, we address the questions raised by the reviewer. We hope the replies could help the reviewer further recognize our contributions. Thank you.

Q: It appears that the method can be applied to different base algorithms besides ReBrac. Incorporating the method with approaches like ATAC and IQL could enhance the comprehensiveness of the experiments and support the claims made in the paper.

A: We are currently conducting experiments to test our method with different base algorithm IQL, to further validate its applicability and effectiveness. The following table are results from integrating the Relaxed Adversarial State technique with IQL are promising. These experiments, run on three different seeds, show notable improvements in performance.

Q: In the state perturbation experiments, a comparison with the prior state adversarial baseline RORL is needed.

A: Our methodology is primarily based on the ReBrac approach, making the performance comparison between our method and RORL particularly crucial. RORL utilizes an ensemble-based model, with "RORL-n" indicating the use of n ensemble models. In the following analysis on Hopper-medium-expert over 4 seeds, we demonstrate that our method achieves comparable results to RORL-10, which involves an ensemble size five times larger than ours. Furthermore, our approach notably outperforms RORL-2, which has the same numbers of critics of our model.

Q: The impact of each hyperparameter, and , on the performance of RAORL remains unclear.

A: In RAORL, we introduce two critical hyperparameters: the radius of the uncertainty set and the relaxed parameter alpha. The radius of the uncertainty set determines its size, directly influencing the range of potential state-action pairs considered during policy training. The relaxed parameter alpha, on the other hand, plays a pivotal role in defining the distribution over the uncertainty set. Varied values of alpha represent different prior assumptions about the environment. For instance, setting alpha to 1 suggests a belief that the nominal MDP is the most likely scenario, indicating confidence in the representativeness of the training data. Conversely, an alpha value of 0 implies a lean towards the worst-case MDP, preparing the policy for the most challenging possible scenarios.

We thank reviewer 6xeb for the insightful comments. Below, we address the questions raised by the reviewer. We hope the replies could help the reviewer further recognize our contributions. Thank you.

Q: The methods section appears somewhat unclear. It is recommended that the authors include a subsection on the implementation of RAORL, detailing the learning objectives for both value functions and policy.

A: Our algorithm can be found in algorithm 1. Specifically, inspired by TD3+BC (Fujimoto & Gu, 2021) and ReBrac (Tarasov et al., 2023), Objective function for the policy:

$$\pi = argmax_{\pi} \mathbb{E}_{(s,a) \sim D} \left[ \lambda Q(s, \pi(s)) - (\pi(s) - a)^2 \right]$$

The default values of the hyper-parameters($\lambda$) were used in the experiments (same as ReBrac (Tarasov et al., 2023)).

Objective function for the average scenario value function is followed the lemma2:

$$L_{TD}(\theta) = E_{(s,a,s') \sim D} \left[ \left( r(s,a) + \gamma \big( \alpha Q_{\bar{\theta}}(s', \pi(s’)) + (1-\alpha) Q_{\bar{\theta}}(adv(s'), \pi(s’)) \big) - Q_{\theta}(s, a) \right)^2 \right],$$

We set perturbation radius from the set {0.03, 0.05, 0.08, 0.1} multiplied by state differences (in Section 4.4). The setting of attack budgets were chosen to reflect the mean magnitude of the state affected by actions taken in each environment.

Q: A thorough discussion and comparison with this work are necessary. Moreover, the proposed method seems quite similar to the adversarial state training in S4RL [2]. A more comprehensive discussion with closely related works is required to emphasize the significance of the method; otherwise, the novelty may be questioned.

A: The main contribution of RAORL over S4RL is a theoretical guarantee for the lower bound of performance, further justifying the use of state-adversaries . Besides, we acknowledge the importance and “relaxed” technique beyond state-adversarial technique. In S4RL, the state-adversarial technique was identified as the most effective augmentation method based on their experiments. However, applying the state-adversarial technique in a straightforward manner can lead to overly cautious results (Lien et al., 2023). This is primarily because the state-adversarial technique inherently prepares for the worst-case scenario. Therefore, in practical applications, particularly in real-world problems, it becomes essential to adopt a "relaxed" version of the state-adversarial technique. The following table shows a comparative study of RAORL and S4RL in various experimental environments.

We thank reviewer K6G5 for the insightful comments. Below, we address the questions raised by the reviewer. We hope the replies could help the reviewer further recognize our contributions. Thank you.

Q: The quality of writing in this paper could benefit from further refinement. Some explanations are unclear. For instance, in Lemma 1, formula 1 and formula 2 look identical, I had thought that formula 1 might have been inadvertently repeated. But it turns out there are slight differences. I think there should be a better presentation such as using the absolute difference is bounded. Furthermore, the content of Lemma 1 raises questions for me. The textual description seems to discuss the relationship of the policy's value learned from two different properties of P_B. However, the formula appears to depict the policy's value learned from U and P_B. I hope the authors recognize this distinction. One refers to the value function learned under a specific P_B, while the other pertains to the average value function learned on U.

A: We revise Lemma 1 in absolute difference and clarify the details in Lemma 1 as follows.

1. Relevance of U and P_B: As shown in Figure 1, U represents the set of all possible real-world transition kernels, encompassing a wide range of scenarios and environmental dynamics. In contrast, P_B is specific to the offline dataset and represents the transitions recorded in that particular dataset.

2. Objective of Lemma 1: The lemma intends to quantify the performance gap between a policy learned exclusively from the offline dataset (P_B) and the policy evaluated against the universal set of transition kernels (U). This gap highlights the potential discrepancies and challenges in applying policies learned in a limited offline context to a broader range of real-world scenarios.

3. Lemma 1 in absolute difference :
   $

\left| J_{\rho_0^B}(\pi, P_B) -E_{P_0 \sim U}(J_{\rho_0}(\pi, P_0)) \right| \leq & \frac{2R_{\text{max}}}{1-\gamma} E_{P_0 \sim U}[D_{\text{TV}}(\rho_0,\rho_0^B)] + \frac{2 \gamma R_{\text{max}}}{(1-\gamma)^2} \beta + \frac{2R_{\text{max}}}{1-\gamma} E_{P_0 \sim U}{E}[\gamma^{\text{T}_{V}^\pi}].

$

Q: While this paper offers a plethora of proofs and new definitions, it lacks a coherent logical thread to effectively connect them. Additionally, the absence of adequate textual explanations for the theorems and lemmas makes the paper hard to follow and comprehend. There is no formal problem formulation for the problem the paper wants to solve. A more structured approach and clear expositions would greatly enhance its readability.

A: In the second paragraph of Section 4, we provided a logical thread of sections of 4.1, 4.2 and 4.3. In Section 4.1, we systematically measure the performance discrepancies between the policy shaped solely from the offline dataset ($P_B$) and its application in the universal set of transition kernels (U). Sections 4.2 and 4.3 further build upon this by demonstrating methods to enhance policy performance.

Q: This paper introduces the concept of the Relaxed State-Adversarial Transition Kernel. However, the name doesn't seem to genuinely capture the essence of this transition kernel. I'm uncertain where the "State-Adversarial" aspect is present. The approach of crafting a new transition kernel using a linear combination appears to be already employed by many robust RL under model uncertainty methods, such as the R-contamination uncertainty set. You may check this paper "Policy Gradient Method For Robust Reinforcement Learning".

A: The term "State-Adversarial" in our Relaxed State-Adversarial Transition Kernel is defined in Definition 2 of our paper. It refers to the process where state transitions are adversarially perturbed towards those with lower value estimates, a concept critical for robustness in continuous state-action spaces. Our kernel thus represents a bridge between standard and adversarial MDPs, a novel contribution particularly tailored for continuous environments. While the "Relaxed" aspect of our kernel does resemble approaches like the R-contamination uncertainty set found in robust RL literature, the "State-Adversarial" facet specifically addresses continuous domains. This differentiates our work from that in "Policy Gradient Method For Robust Reinforcement Learning," which primarily addresses discrete spaces (as their algorithm requires the summation over all possible states and actions) and does not extend to the continuous settings that our paper focuses on.

We thank reviewer YpUx for the insightful comments. Below, we address the questions raised by the reviewer. We hope the replies could help the reviewer further recognize our contributions. Thank you.

Q: As far as I can tell, the RAORL algorithm is not substantially different from RAPPO, other than that (i) it is applied offline, (ii) a different solver is used (ReBrac vs. PPO), although this is necessitated by (i), and (iii) the relaxation parameter is tuned as a hyperparameter rather than being updated as part of the algorithm. It would be helpful for the reader if you describe the distinction more clearly.

A: RAORL is tailored for offline RL, focusing on adaptability across multiple domains using static data, training within State-Adversarial uncertainty sets to ensure robust performance across a range of MDPs. Theorem 3 in our work theoretically validates the efficacy of RAORL in these offline scenarios. On the other hand, RAPPO is concerned with online settings, aiming to enhance both average and worst-case performance concurrently. These distinct goals highlight the fundamental differences in the application and contribution of RAORL compared to RAPPO.

Q: Similarities between this paper’s theoretical results and those of the MoREL paper (Kidambi et al. 2020) are not highlighted. For example, Lemma 1 here looks nearly identical to Theorem 1 in that paper, with the addition of expectation over the uncertainty set. It would be helpful for the reader if you include some comments on the similarities/differences, perhaps in an appendix.

A: Our Lemma 1 and MoREL's Theorem 1 differ primarily in their treatment of transition probabilities. MoREL assumes known transition probabilities, using them to create a pessimistic MDP. In Theorem 1, MoREL work with the constraint that the Total Variation Distance (DTV) between the model transition probability $\hat{P}(\cdot|s,a)$ and the true probability $P(\cdot|s,a)$ is less than or equal to $\alpha$. However, in practice, $P(\cdot|s,a)$ is unknown since we only have access to the transition kernel derived from the offline dataset. In contrast, RAORL operates without this assumption, reflecting real-world scenarios where true dynamics are often unknown.

Q: In Definition 1, is the expectation over taken over all transitions in the offline dataset? (I think the answer is yes, but wanted to confirm. The notation could be modified to clarify this.)

A: In Definition 1, the expectation indeed goes beyond the transitions present in the offline dataset. It encompasses all possible transition kernels within a universal uncertainty set, representing potential dynamics in real-world environments. This approach extends our analysis beyond limited empirical data, preparing the model for a broader range of scenarios. However, it's important to note that while the concept of a universal uncertainty set might seem overly extensive, in practice, we refine this to a more manageable subset, denoted as $U_r$ in section 4.2 and $U_{\epsilon}$ in section 4.3. This refinement process involves constraining the uncertainty set based on realistic assumptions and characteristics observed in the offline data, thereby making the set both practical and relevant to real-world applications.

Q: In Definition 3, do the transition probabilities need to be re-normalized if the argmin defining contains more than one state? If we are assuming the argmin is always unique, perhaps this should be noted in the paper.

A: The problem can be easily solved by setting a rule, such as picking the state that has the smallest value in the first dimension if there are multiple choices. In our implementation, the worst state is determined using the FGSM method. Although there can be multiple states that are equally worse, the FGSM finds only one of them. We will clarify this issue in the revision.

Q: The definition of (in Lemma 3 in Appendix A.2) is given as “the probabitlity [sic] of for every ∈”. I am not sure how to interpret this?

A: In Lemma 3, $p_r$ represents the likelihood of each possible transition kernel $P$ being within a robust subset $U_r$, which is taken into account by the robust policy $\pi_r$.

Q: Could you describe more precisely how the perturbations were chosen in section 5.3?

A: We defined the perturbation magnitude linearly with 20 points in the range $[0, 0.1]$. A perturbation of 0 signifies no perturbation, while a perturbation of 0.1 implies a perturbation strength of $0.1 \times | \text{ActualNextState} - \text{State}|$, thereby ensuring that the norm of the difference between the actual state and the perturbed state is at most $0.1 \times | \text{ActualNextState} - \text{State}|$.

As depicted in Figure 2, we analyzed the effect of different perturbation magnitudes on agent performance. The results demonstrate that RAORL maintains robust performance even under significant state perturbations.

Q: I am seeking clarity regarding the primary advantage of RAORL. Is it intended to be more effective than other offline model-free algorithms, aimed at improving adversarial robustness, or targeting enhanced offline performance? While the paper mentions these advantages, the results do not strongly demonstrate superior performance or robustness with enough evidence and explanation.

A: The primary objective of RAORL is to offer an effective and efficient solution in the realm of offline model-free algorithms. It significantly reduces the computational and memory overhead typically associated with ensemble methods. In terms of performance and robustness, RAORL is designed to be competitive with other state-of-the-art offline model-free algorithms.

We thank reviewer 8FWn for the insightful comments. Below, we address the questions raised by the reviewer. We hope the replies could help the reviewer further recognize our contributions. Thank you.

Q: The paper's structure is not well-organized, making it challenging to grasp the primary contributions and follow the narrative. Furthermore, the writing is convoluted, containing multiple unclear sentences that hinder comprehension.

A: In the second paragraph of Section 4, we provided a logical thread of sections of 4.1, 4.2 and 4.3. In Section 4.1, we systematically measure the performance discrepancies between the policy shaped solely from the offline dataset (P_B) and its application in the universal set of transition kernels (U). Sections 4.2 and 4.3 further build upon this by demonstrating methods to enhance policy performance.

Q: The experimental results lack persuasiveness. Firstly, the authors do not compare their work with state-of-the-art algorithms such as SAC-N, EDAC [1], and LB-SAC [2]. Even when implementing the algorithms based on https://github.com/tinkoff-ai/CORL, this paper's results are inferior to previous methods in most Mujoco and Adroit environments. Hence, I believe there are significant limitations in the paper's empirical contributions.

A: SAC-N, EDAC, and LB-SAC are all ensemble-based algorithms where the ensemble sizes are 500, 50, and 50, respectively. These ensemble-based algorithms often require extensive computational resources due to their reliance on large ensembles for uncertainty estimation (Nikulin et al., 2022). We did not include these algorithms in the comparative study because our method is not only model-free but also ensemble-free.

Q: Figure 2 should clarify the color codes representing different algorithms and indicate the corresponding perturbation magnitudes. The paper should explain how optimal state perturbations were determined in the robustness analysis. The connection between the state perturbations in the experiments and the real-world environments, as claimed in the paper, seems tenuous. I have reservations about the practical significance of this paper.

A: In the caption of Figure 2, we have clarified “The blue and red solid lines depict the average performances of RAORL and ReBrac, respectively, in the presence of state perturbations”.

In offline reinforcement learning (RL), a common and practical challenge arises when data collected from one system (Machine A) is used to train an agent that will be deployed on a different but similar system (Machine B). Even minor differences between these two machines can lead to distinct Markov Decision Processes (MDPs), posing a significant challenge in terms of MDP generalization. This situation underscores the importance of developing RL agents that can generalize effectively across varying MDPs. In essence, the agent must be capable of adapting to the nuances and potential discrepancies between the training environment (Machine A) and the deployment environment (Machine B). The experiments in Figure 2 simulate this situation.

Q: While the paper focuses on model-free offline RL, I find it puzzling that model-based offline RL is included in the related work. Robust RL works like [3, 4] (and others) dealing with state adversaries seem more relevant, yet the authors do not discuss them in the related work section.

A: The inclusion of model-based offline RL in the related work is to present a comprehensive background, as it also addresses offline scenarios. We acknowledge the relevance of robust RL literature, particularly works that handle state adversaries. We will incorporate references [3, 4] into our discussion on robust reinforcement learning in Section 2.3 to provide a more complete overview of the field and its developments.

Q: How does RAORL effectively bridge the performance gap in real-world applications, as claimed in the introduction?

A: RAORL utilizes a strategic training approach that involves the uncertainty set of an offline dataset, as defined in Definition 4 of our paper. This approach ensures that RAORL is not confined to the narrow bounds of the data it was trained on but instead has exposure to a wider array of potential scenarios within its uncertainty set. Moreover, Theorem 3 in our paper underpins this methodology with a solid theoretical foundation, demonstrating how training within this uncertainty set can reduce the discrepancy between the policy's performance in the training data and its expected performance in real-world conditions.

Q: How to determine the attack budget () in RAORL?

A: We set perturbation radius from the set {0.03, 0.05, 0.08, 0.1} multiplied by state differences (in Section 4.4). The setting of attack budgets were chosen to reflect the mean magnitude of the state affected by actions taken in each environment.