We appreciate the reviewer’s insightful and constructive feedback. We have carefully addressed these concerns and accordingly revised the manuscript. These comments have not only facilitated significant improvements in our manuscript but have also inspired us for further in-depth studies in our future research.

• Q1. The authors missed some important references [1,2]. In the future, the authors may also look into [3], which seems like a compelling work for the research venue explored in the paper.

  • A1. Thank you for pointing out these crucial references. The Shannon entropy for the measures of aleatoric and epistemic uncertainties discussed in these works [1,2] provides important insight and support for our method. We will study these references thoroughly and consider incorporating their findings into our future research. We also appreciate the suggestion to explore the recent work in [3], which could further enhance our understanding and methodology. We will include these additions in our revised manuscript.

  [1] Credal Bayesian Deep Learning. 2023.

  [2] A Novel Bayes' Theorem for Upper Probabilities. Epi UAI 2024.

  [3] Distributionally Robust Statistical Verification with Imprecise Neural Networks. 2023.

• Q2. In addition, would it be fair to say that, on page 3, the expectation that finds $\pi^*$ is an integral taken with respect to the joint probability given by all the sources of randomness $P_0$, $\pi (\cdot | s_t)$, $\rho (\cdot | s_t, a_t)$, $P(\cdot |s_t, a_t)$?

  • A2. Yes. On page 3, the expectation that determines $\pi^*$ involves an integral taken with respect to the joint probability distribution influenced by all sources of randomness, including $P_0$, $\pi(\cdot|s_t)$, $\rho(\cdot|s_t,a_t)$, and $P(\cdot|s_t,a_t)$ [1,2].

  [1] Trust Region Policy Optimization. ICML 2015.

  [2] Distributional reinforcement learning with quantile regression. AAAI 2018.

• Q3. In equation (1), the closed parenthesis ) should go after $a_0=a$.

  • A3. Thank you for catching the typo in Eq. (1). We will carefully correct this in our revision.

We appreciate the reviewer's insightful and constructive comments and suggestions. We respond to each comment as follows and sincerely hope that our responses could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to the comments and enhancing our submission.

• Q1. The proposed method is complicated. Can this method be simplified?
  • A1. Thanks for your meaningful comment.
    • On the actor-critic architecture based on IQL, our approach TRACER adds just one ensemble model $(p_{\varphi_a},p_{\varphi_r},p_{\varphi_s})$ to reconstruct the data distribution and repace the function approximation in the critic with a distribution estimation (see Figure G1 in the PDF of our global response). Therefore, the network structure is not complicated, while significantly improving the ability to handle both simultaneous and individual corruptions.
    • It is worth noting that our primary aim is the development of corruption-robust offline RL techniques for diverse corruptions. To the best of our knowledge, this study introduces Bayesian inference into corruption-robust offline RL for the first time, prioritizing novelty and robust performance in challenging scenarios with simultaneous corruptions.
    • We thank you for reminding us of the importance of the balance between complexity and usability. In future work, we plan to refine TRACER by estimating uncertainty directly within the representation space. Thus, we can simply learn action-value functions without using distributional RL. This advancement will potentially broaden the applications of TRACER in real-world scenarios.
• Q2. The results in the main tables (Table 2 and Table 3) appear similar to those of RIQL, raising concerns about the necessity of the included components.
  • A2. Thanks for the insightful comment. Please refer to A3 in our global response.
• Q3. Another issue is the readability and clarity of the writing. For example, Equations 10 and 11 lack necessary explanation and insight. Additionally, the implementation details for $\varphi_a$, $\varphi_r$, $\varphi_s$ are missing, leaving readers unsure of how the method works in practice.
  • A3. Please refer to A2 in our global response.
• Q4. Furthermore, Equation 10 seems to include $\theta$ as an input unnecessarily, as the right side does not depend on $\theta$.
  • A4. Here is our explanation.
    • The goal of Eq. (10) of our main text is to optimize $(\theta, \varphi_a,\varphi_r,\varphi_s)$, thus minimizing the difference between $p_{\varphi_a}(A|D_\theta,S,R,S')$ and $\pi_{\mathcal{B}}(A|S)$, $p_{\varphi_r}(R|D_\theta,S,A)$ and $\rho_{\mathcal{B}}(R|S,A)$, and $p_{\varphi_s}(S|D_\theta,A,R)$ and $p_{\mathcal{B}}(S)$. More details are in A2 of the global response.
    • In Eq. (10), the input of each $(\mu_{\varphi}, \Sigma_{\varphi})$ includes $D_\theta$ with the parameter $\theta$.
• Q5. The analysis of experiments is not thorough. Why does the accuracy of the measurement for corrupted data decrease during training in Figure 3?
  • A5. Thanks for your meaningful comment.

    • We apologize for any confusion caused by the unclear description and presentation of the validation experiments in Figure 3 of our main text. To clearly illustrate the measurement accuracy of agents learned by TRACER with respect to uncertainty during training, we further conduct validation experiments on Walker2d-Medium-Replay-v2.

      Specifically, we evaluate the accuracy every 50 epochs over 3000 epochs. For each evaluation, we sample 500 batches to compute the average entropy of corrupted and clean data. Each batch consists of 32 clean and 32 corrupted data. We illustrate the curves over three seeds in Figure G3 of the PDF of our global response, where each point shows how many of the 500 batches have higher entropy for corrupted data than that of clean data.

    • Figure G3 illustrates an oscillating upward trend of TRACER's measurement accuracy using entropy under simultaneous corruptions, demonstrating that using the entropy-based uncertainty measure can effectively distinguish corrupted data from clean data.

• Q6. Moreover, I suggest the authors provide the computational cost comparison with prior work.
  • A6. Please refer to A1 in the global response.

We appreciate your insightful comments. We respond to each comment as follows and sincerely hope that our responses could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to the comments and enhancing our submission.

• Q1. (1) The paper introduces LQ (θi), LV(ψ), Lπ(ϕ) in Eq 4, 5 and 6 but does not discuss how they relate to Dθ. Is θ for learning Q function and D same? (2) How is the final policy extracted after learning Dθ.

  • A1. We apologize for any confusion caused by unclear descriptions in our main text.
    1. Relation between $Q$, $D$, and $\theta$.

       • We use the same $\theta$ to learn $D$ and then estimate $Q$. Specifically, following [1], we have $Q_{\theta_i}(s,a) = \sum_{n=1}^N D_{\theta_i}^{\tau_n}(s,a,r).$ See Lines 167-172 of the main text for more details regarding the notations.

         Similarly, we also have $V_{\psi}(s) = \sum_{n=1}^N Z_{\psi}^{\tau_n}(s).$

       • With the relation between $Q$ and $D$, $V$ and $Z$, we use Eq. (4) to derive $\delta$ in Eq. (12) and Eq. (5) to derive Eq. (13) in our main text.

    2. Policy Learning.
       • Following the weighted imitation learning in RIQL and IQL, we use Eq. (6) in our main text to learn the policy. Details for notations are shown in Lines 122-123 of the main text.

  [1] Implicit quantile networks for distributional reinforcement learning.

• Q2. The paper relies on many hyperparameters.

  • A2. Compared to RIQL, one of SOTA methods, our approach TRACER only introduces additional hyperparameters associated with the approximation of action-value distributions: (1) the number $N$ of samples $\tau$; (2) a linear decay parameter $\beta$ used to trade off the losses $\mathcal{L}\_{\text{first}}$ and $\mathcal{L}\_{\text{second}}$.

• Q3. What kind of kernel is used in Gaussian distribution of Eq 10?

  • A3. We do not use a Gaussian kernel in Eq. (10). See the explanation of Eq. (10) in A2 of our global response for more details.

• Q4. Is there a upper bound on what amount of corruption the method can handle?

  • A4. Yes. Please refer to A4 in our global response.

• Q5. While authors consider individual corruptions, data in the real world can have simultaneous corruptions.

  • A5. We specifically design TRACER to handle the challenging simultaneous corruptions. Results in Table 1 of the main text show that TRACER outperforms several SOTAs in all tasks under simultaneous corruptions, achieving an average gain of {\bf +21.1\\%}. Note that "mixed corruptions" in the main text refers to simultaneous corruptions (see Line 233 in the main text).

• Q6. The paper introduces $q^{\pi}$ in Eq 18. Is this the same as action value distribution?

  • A6. No. The notation $q^\pi$ in Eq. (18) of the Appendix denotes the value distribution, consistent with $p(\cdot|s)$ used in Line 170 of the main text, which is an expactation of action value distribution $\mathbb{E}_{a\sim \pi, r\sim \rho} [p\_\theta(\cdot|s,a,r)]$.

• Q7. RIQL already proves the robustness of IQL policy against data corruption. What is the objective of Theorem A.3?

  • A7. Please refer to A4 in our global response.

• Q8. Providing a table summarizing the notations in appendix will enhance readability.

  • A8. Thanks for the insightful comment. We will include this table in our revision.

• Q9. How relevant is the reward corruption given offline RL already robust to noise in the reward function?

  • A9. While the offline RL is robust to small-scale random reward corruptions [1], it tends to struggle with large-scale reward corruptions (see Page 21 in RIQL).

  [1] Survival instinct in offline reinforcement learning.

• Q10. Do the authors assume that complete traces of the system are available?

  • A10. No. We employ the commonly used offline RL setting [1], where the offline dataset consists of shuffled tuples and agents only use individual tuples rather than complete traces.

  [1] Off-policy deep reinforcement learning without exploration.

• Q11. In situations of dynamic corruption these traces might be different than the system. For example, $s_t$ in the corrupted dataset may not be observed in the real system. How are such cases handled?

  • A11. Thanks for the insightful comment. In a scenario where an element (e.g., state) may be corrupted, TRACER captures uncertainty by using (1) other elements and (2) correlations between all elements and action values (see Lines 53-60 of the main text).

• Q12. What amount of clean data needs to be present in the dataset for this method to work?

  • A12. Please refer to A4 in our global response. Results in Table G4 show that while TRACER is robust to simultaneous corruptions, its performance depends on the extent of corrupted data it encounters.

• Q13. What is the reason for not using projected gradient descent for reward corruptions?

  • A13. Following RIQL, the objective of adversarial reward corruption is $\hat{r} = \min_{\hat{r} \in \mathbb{B}(r, \epsilon)} \hat{r} + \gamma \mathbb{E}[Q(s', a')]$. Here $\mathbb{B}(r, \epsilon)=\\{\hat{r}\mid |\hat{r} - r| \leq (1+\epsilon)\cdot r_{\max} \\}$ regularizes the maximum distance for rewards. Thus, we can directly compute $\hat{r} =-\epsilon\times r$ without using projected gradient descent.

• Q14. Could the author provide some understanding of why the method outperforms RIQL under certain settings and why it does not in some cases?

  • A14. Please refer to A3 in our global response.

• Q15. Limitations.

  • A15. We look forward to developing corruption-robust offline RL with large language models, introducing the prior knowledge of clean data against large-scale or near-total corrupted data. Moreover, we plan to add potential negative societal impacts in our revision.

Thank you for your valuable comments. We respond to each of your comments as follows.

Q16. (1) How does $\tilde{\pi}_{\text{IQL}}$ help to prove the upper bound for TRACER? (2) How do you equate $Z^\pi(s)$ with $D^\pi (s,a)$ in steps 24, 25, and 26 in Theorem A.3? (3) What is $D^\pi (s,a)$ as previously D is defined as D(s, a, r)?

• A16.
  • For (1), as TRACER directly applies the weighted imitation learning technique from IQL to learn the policy, we can use $\tilde{\pi}_{\text{IQL}}$ as the policy learned by TRACER under data corruptions, akin to RIQL. In Theorem A.3 of the Appendix, the major difference between TRACER and IQL/RIQL is that TRACER uses the action-value and value distributions rather than the action-value and value functions in IQL/RIQL. Therefore, we further prove an upper bound on the difference in value distributions of TRACER to show its robustness.
  • For (2), we apologize for the missing reference in steps 24, 25, and 26 of Theorem A.3 (see Lemma 6.1 of [1]). We follow [1] to provide the detailed derivation below.
    • For any $\pi'$ and $\pi$, we have

    Thus, we can derive the step 25 from the step 24.
- For (3), we apologize for any confusion caused by the notations. We will replace the notation $D^\pi (s,a)$ with $D^\pi (s,a,r)$ in Theorem A.3 of our revision.

[1] Approximately optimal approximate reinforcement learning.

Q17. (1) What is the value of $c$ we use in experiments? (2) What is the effect on performance for varying $c$?

• A17.
  • For (1):

    • For each experiment with a random seed under individual corruptions in Tables 2 and 3 of our main text, we randomly select $c\\% = 30\\%$ of transitions from the offline dataset. Within these selected transitions, we replace one element per transition with corrupted element.

    • In Table 1 with simultaneous corruptions, we also apply $c\\% = 30\\%$ but extend the corruption process across all four elements of the offline dataset. Specifically, we randomly select $30\\%$ of transitions and corrupt one element in each selected transition. Then, we repeat this step four times until all elements are corrupted. Therefore, approximately $76.0\\%$ of data in the offline dataset is corrupted, calculated as $1 - (1 - c)^4$.

      In Table G4 of our attached PDF of the global response, we evaluate TRACER using different $c \\%$, including $10\\%, 20\\%, 30\\%, 40\\%,$ and $50\\%$. These rates correspond to approximately $34.4\\%, 59.0\\%, 76.0\\%, 87.0\\%$, and $93.8\\%$ of the data being corrupted.

  • For (2):
    • Results in Table G4 show that while TRACER is robust to simultaneous corruptions and significantly outperforms RIQL, its performance depends on the extent of corrupted data it encounters, degrading with increased data corruptions.

We hope our responses adequately address your concerns. If you have further concerns, please let us know and we will continue actively responding to your comments, enhancing our submission. We would deeply appreciate it if you could raise your score based on these revisions.

Dear Reviewer QZTE,

Thanks again for your continued support and the further improvements you have suggested for our manuscript. We sincerely appreciate your insightful feedback, which has significantly helped us in refining the explanations and enhancing the theoretical derivations. We remain committed to incorporating all your valuable suggestions into the final version, if accepted.

We are deeply grateful for your thoughtful comments and for the confidence you have shown in our work by raising your score.

Best,

Authors

We appreciate the reviewer's insightful and constructive comments and suggestions. We respond to each comment as follows and sincerely hope that our responses could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to the comments and enhancing our submission.

• Q1. No experiments for simultaneous corruptions.

  • A1. We apologize for the confusion caused by unclear descriptions in our main text. Specifically, the experiments presented in Table 1 on Page 7 use the simultaneous corruptions. The term "mixed corruptions" in the caption of Table 1 refers to the simultaneous corruptions (see the explanation on Line 233 of Section 4 in the main text). In our revision, we will replace "mixed corruptions" with "simultaneous corruptions" to avoid any ambiguity.

• Q2. The contribution compared to other baselines (RIQL) is not really clear.

  • A2. Thanks for the kind and insightful comment.
    • Relation between TRACER and RIQL. Our approach TRACER applies the weighted imitation learning in RIQL for policy improvement (see Eq. (6) on Page 4 of the main text). This allows us to build and expand upon RIQL.
    • Contributions of TRACER. We present TRACER's major contributions compared to corruption-robust offline RL methods [1,2] (especially RIQL):
      1. To the best of our knowledge, TRACER introduces Bayesian inference into corruption-robust offline RL for the first time. Thus, it can capture uncertainty caused by diverse corrupted data to simultaneously handle the corruptions, unlike other corruption-robust offline RL methods that primarily focus on individual corruptions.
      2. TRACER can distinguish corrupted data from clean data using an entropy-based uncertainty measure. Thus, TRACER can regulate the loss associated with corrupted data to reduce its influence. In contrast, existing corruption-robust offline RL methods lack the capability to identify which data is corrupt.
      3. Based on Tables 1, 2, and 3 of the main text, TRACER significantly outperforms existing corruption-robust offline RL methods across a range of both individual and simultaneous data corruptions.

  [1] Corruption-robust offline reinforcement learning with general function approximation. NeurIPS 2023.

  [2] Towards robust offline reinforcement learning under diverse data corruption. ICLR 2023.

• Q3. The experimental settings are fairly limited.

  • A3. Thanks for your insightful comment.
    • Our experiments using Mujoco and CARLA datasets are consistent with standard practices in corruption-robust offline RL. Existing methods [1,2] often select these datasets to assess their effectiveness.
    • Based on the corruption-robust offline RL methods, we further conduct experiments on two AntMaze datasets and two additional Mujoco datasets, presenting results under random simultaneous corruptions in Table G5 of the PDF of our global response.
      • Settings. Each result represents the mean and standard error over four random seeds and 100 episodes in clean environments. For each experiment, the methods train agents using batch sizes of 64 for 3000 epochs. Building upon RIQL, we apply the experiment settings as follows.
        1. For the two Mujoco datasets, we use a corruption rate of $c=0.3$ and scale of $\epsilon=1.0$. Note that simultaneous corruptions with $c=0.3$ implies that approximately $76.0\\%$ of the data is corrupted.
        2. For the two AntMaze datasets, we use the corruption rate of 0.2, corruption scales for observation (0.3), action (1.0), reward (30.0), and dynamics (0.3).
      • Results. The results in Table G5 show that TRACER significantly outperforms other methods in all these tasks with the aforementioned AntMaze and Mujoco datasets.

  [1] Corruption-robust offline reinforcement learning with general function approximation. NeurIPS 2023.

  [2] Towards robust offline reinforcement learning under diverse data corruption. ICLR 2023.

• Q4. Different perturbation/corruption levels are not shown.

  • A4. Thanks for the kind and insightful comment.
    • Setting. Building upon RIQL, we extend our experiments to include Mujoco datasets with different corruption levels, using different corruption rates and scales. We report the average scores and standard errors over four random seeds in Figure G2 of the PDF of our global response, using batch sizes of 256.
    • Results. Figure G2 in the PDF shows that TRACER significantly outperforms other algorithms in all tasks under random simultaneous corruptions, achieving an average score improvement of {\bf +33.6\\%}.

• Q5. Computational cost of TRACER.

  • A5. Please refer to A1 in our global response.

• Q6. Can you show the hyperparameter tuning results?

  • A6. Thank you for the meaningful comment.

    • Setting. We conduct hyperparameter tuning experiments for both TRACER and RIQL, varying values of $\kappa$ in the Huber loss and $\alpha$ for action-value functions. We report results in Tables G6 and G7 of the PDF of our global response. Note that except the first column in Table G6, we use batch sizes of 64 and a learning rate of 0.0003 over four random seeds on Hopper task under adversarial simultaneous corruptions.

    • Results. Tables G6 and G7 reveal that TRACER is more stable than RIQL and consistently outperforms RIQL, achieving an average performance gain of {\bf +43.3\\%} under adversarial simultaneous corruptions.

      Moreover, we find that both TRACER and RIQL reach their respective highest performance at $\kappa=0.1$ and $\alpha=0.25$, and TRACER still achieves a substantial performance gain of {\bf +24.7 \\%} compared to RIQL.