Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling

Thank you for the valuable comments, and we provide clarification to your concerns as follows. We hope these new experimental results and explanations can address your concerns, and we look forward to your further feedback.

1. ‘Figure 1 is not as intuitive as it could be; it might be more effective to directly display the performance decline under various attacks’. (Q1)

   Thank you for your question. In Figure 1, we aim to highlight the significant performance drop that occurs when using limited data (10%) compared to the full dataset (100%). Therefore, we present the performance of different methods using both the 100% and 10% datasets in a single figure to illustrate more clearly the advantage of DT when dealing with limited data.

   To address the reviewer's concern and further illustrate how performance varies under different data corruption scenarios, we have included another version of Figure 1 in Appendix D.15 and Figure 20. These figures depict a more pronounced drop in performance due to state corruption, while the other two types of corruption result in relatively smaller declines, particularly for sequence modeling methods.

2. ‘A more rational method would be to compare the performance of RDT with RDT w/o (ED, GWL and IDC)’. (Q2)

   We conduct the ablation study using DT(RP) w. (ED, GWL and IDC) to better demonstrate the impact of the components while mitigating the influence of other factors. Since two components’ interaction can bring unintended influence.

   To address the reviewer's concerns, we conduct additional comparisons with RDT variants that exclude ED, GWL, and IDC. The results of these new ablation studies can be found in Appendix D.9 and Figure 16. Notably, these results align with the findings in Section 4.4. This consistency reinforces our initial conclusion that integrating all the proposed techniques (ED, GWL, and IDC) is essential for achieving optimal robustness. The removal of any component leads to a performance drop, with GWL and ED being particularly critical components.

We sincerely thank you for your valuable suggestions and time spent on our work. We hope that we have addressed your main questions in the rebuttal. As the due date (November 26 at 11:59pm AoE) for the discussion period is approaching, we hope to receive your further feedback and address your concerns once again. We are committed to addressing any concerns or suggestions you may have to enhance the manuscript.

5. ‘Do you expect your method to work well in non-dense reward environment’. (Q1)

   In our experiments, the Adroit and Kitchen tasks are both examples of sparse reward tasks. Interestingly, as shown in Appendix D.2, DT(PR) consistently improves performance under state and action corruption for these tasks. Although DT(PR) may decrease performance under reward corruption, other components of RDT, such as GWL, ED, and IDC, can enhance performance in these scenarios, which are not dependent on dense reward signals. Therefore, a "dense reward signal" is not a necessary requirement for our RDT.

6. ‘In Section 3.2.2 still, what happens at the beginning of training, when all examples are unknown? How do you differentiate corrupted examples from natural ones, when we expect the model’s prediction to be poor for all inputs?’. (Q2)

   As shown in Figure 3(b), we plot the loss for clean and corrupted data separately for DT and DT w. GWL. We find that the model quickly differentiates between clean and corrupted data, with an evident gap between the two losses from the very first evaluation point. This disparity suggests that applying GWL effectively prevents the model from overfitting to corrupted data without negatively impacting the learning from clean data. Additionally, for Iterative Data Correction, we initiate the process midway through the training to avoid potential non-differentiable cases at the start of training.

7. ‘Section 3.2.3 : Have you tested how frequently your assumption was true? I am curious to know the proportion of times the z score actually detects the perturbed transitions’. (Q3)

   To further investigate the effectiveness of IDC, we record the accuracy of detection via the $z$-score in Appendix D.13 of our revision. Specifically, DT w. IDC can detect $N$ data points as corrupted within a batch, and $M$ out of these $N$ data points are truly corrupted. We record the ratio of $M/N$ under action attack as shown in Figure 18. As observed, the detection accuracy increases with the training process and reaches nearly 80% ~ 90% at the 50th epoch, demonstrating the effectiveness of IDC. It is worth noting that the accuracy is relatively low at the start of training; therefore, we begin Iterative Data Correction at the middle of training in our default implementation.

8. ‘In Figure 3, different environments are chosen to showcase the effects of the elements forming the robust DT method (which are also different from the ones of the ablation in Fig. 6). How were these environments chosen? Why not use the same environment? If there is a difference in behavior of the RDT additions across environments, is there a pattern?’ (Q5)

   In Figure 3, we select representative tasks to provide clearer visualization of how the different components function. To address the reviewer's concerns, we rerun the experiments for Figure 3, ensuring that all three subfigures feature the same tasks (walker2d, kitchen-complete, and relocate) and are consistent with those presented in Figure 6. We find that the observations remain consistent.

9. Minor weaknesses.

   Thank you for pointing out these minor weaknesses in our manuscript. We have made corresponding modifications to address them in our revised manuscript.

Thank you for the valuable comments, and we provide clarification to your concerns as follows. We hope these new experimental results and explanations can address your concerns, and we look forward to your further feedback.

1. ‘Several of the methodological details are pushed to the appendix without being explicitly referred to in the main text…’. (W1)

   Thank you for the suggestion. To enhance readability, we have incorporated the recommended modifications for key hyperparameters in the main text of the revised version. For instance, line 303 now includes the choice of $\zeta$ and refers to the implementation of RDT in Appendix C.3. Additionally, we have provided a brief explanation of ‘adversarial corruption’ on line 133 and the ‘action diff’ attack on line 431.

2. ‘ I would be also curious to know how these ablated methods compare with unmodified DT in Fig. 6…The fact that the different ablations of Fig. 6 and 3 are evaluated on different environments also makes it difficult to get a clear sense of the general behavior of each module’. (W2 and Q6)

   We have included the comparison results between DT(RP) and the original DT in Appendix D.2, where DT(RP) demonstrates significant improvements in handling state and action corruption. For instance, DT(RP) enhances DT's performance from 84.6 and 73.2 to 93.2 and 92.6, respectively, on state and action corruption in the Adroit tasks. This suggests that the improvement of RDT over DT is more substantial. Since RDT is based on DT(RP), we chose it as the baseline for our ablation study to better understand the impact of the other three techniques.

   We agree that some techniques provide significant improvements under specific data corruption conditions. However, in real-world scenarios, we may not know the specific conditions beforehand. Therefore, our goal is to design an algorithm that is robust against various types of data corruption. From Figure 6, it is evident that the three proposed robustness techniques benefit different tasks. Integrating all these techniques for RDT is essential for achieving strong robustness.

   Regarding Figure 3, we select representative tasks to provide clearer visualization of how the different components function. To address the reviewer's concerns, we rerun the experiments for Figure 3, ensuring the same tasks (walker2d, kitchen-complete, and relocate) presented in Figures 6 and 3. We find the observations remain consistent.

3. ‘Is there any hope for sequence models to overcome Q-function models, even with bigger datasets’. (W3)

   In Appendix D.10, we present new experiments conducted with varying dataset sizes, increasing to five and ten times the training data for MuJoCo and Adroit tasks, respectively. Our empirical findings indicate that:

   (1) The overall performance of all methods improves as the dataset size increases.

   (2) Moreover, temporal-difference-based methods like RIQL are comparable to vanilla sequence modeling methods like DT when the dataset is large, but they are less preferable when the dataset size is limited.

   (3) Notably, RDT consistently outperforms other baselines across all dataset sizes, validating the effectiveness and importance of our proposed robustness techniques.

   For your reference, we provide the results for the Adroit task below.

   Results under random data corruption on the Adroit tasks with various dataset scales.

   Attack	Task	DeFog	RIQL	DT	RDT
   State	Adroit(1%)	69.7	38.1	64.9	93.9
   	Adroit(5%)	77.5	72.9	97.3	99.8
   	Adroit(10%)	85.1	80.2	98.0	106.7
   Action	Adroit(1%)	81.8	39.6	73.2	96.5
   	Adroit(5%)	86.5	85.4	76.6	103.7
   	Adroit(10%)	94.4	103.6	101.9	109.7
   Reward	Adroit(1%)	85.7	39.1	83.8	104.1
   	Adroit(5%)	89.5	94.2	92.6	104.9
   	Adroit(10%)	94.6	101.0	96.5	108.1

4. ‘Only RDT has an uncertainty measure in Table 1, and there is no uncertainty measure in Table 2. Moreover, I did not find a description of the nature of the uncertainty measure in the main text’. (W4, Q4)

   We repeat all the experiments using four different random seeds and calculate the standard deviation as a measure of uncertainty. While we choose not to include the uncertainty of baselines in Table 1 to maintain simplicity and readability, we omitted the variance measure in Table 2 because it averages over each task group (MuJoCo, Kitchen, and Adroit), making the variance less meaningful. Readers can refer to Tables 5 and 6 in Appendix D.6 for the raw results with reported variance.

I want to thank the authors for the rebuttal, the modifications already done and the added experiments. Please find below my subsequent comments, with a few additional questions:

Thank you for doing the modifications.

Thanks for the additional explanations. I still think that DT should be the baseline in Fig. 6 rather than DT(RP), given that it is your actual baseline and we would like to see how all modifications compare. For instance, the performance of DT(RP) degrades compared to DT on the reward attack. You explain that the 3 modules improve on it, so I would like to see all the results on the same Figure to be able to appreciate this exactly. In addition, Fig. in Appendix D2 does not include uncertainty measures while Fig. 6 does. Could you please include DT in Fig. 6, including uncertainty measures?

Thank you for running this additional experiment! It is a very positive sign that the performance of RDT consistently improves over the other methods. I wonder how RDT would fare in the 100% dataset regime, which is the one that was used in Fig. 1 to motivate the focus on small datasets. In particular, are the modifications that RDT brinm sufficient to beat Q methods in all settings, even when the dataset is not limited? To be clear, I do not expect the authors to run additional experiments (though additional results are always welcome!), but rather to discuss whether they expect this improved performance of RDT to be sustained to regular sized datasets like the ones usually found in the offline RL literature.

Sorry, but I am not sure I agree with the reasoning. If you deemed these quantities sufficiently similar to average them together in this table, I do not understand why you cannot also compute the uncertainty over this average. Moreover, I do not agree that the information about the uncertainty of the performance estimates is superfluous in the main text. I kindly encourage you to include the full results, including the uncertainty estimates, in the main part of the document.

3. ‘No discussion of the limitations of RDT’. (W3)

   One limitation of RDT is our decision to avoid state prediction and correction within the robust sequence modeling framework. The high dimensionality of states poses significant challenges to achieving accurate predictions. Additionally, data corruption can further complicate state prediction, leading to potential declines in performance. Despite this, we have validated the feasibility and effectiveness of robust sequence modeling in this paper, and we leave state prediction and correction as promising directions for future exploration. We add the limitation discussion to Appendix E.

4. ‘as early in training, the model is likely to be inaccurate, resulting in high loss values…provide more intuition behind the design of these weighted terms’. (Q1)

   Thank you for the insightful question. In our study on learning under data corruption, we find that the most crucial factor affecting performance is not the absolute scale of the loss but the relative difference between losses for clean and corrupted data. In Figure 3(b), we present results for DT and DT w/ GWL, showing the loss for clean data and corrupted data. Notably, even at the first evaluation point during training, there is a clear disparity between the losses associated with clean and corrupted data. This disparity indicates that applying GWL is effective in preventing the model from overfitting to corrupted data while not adversely affecting learning from clean data.

   We think that a better approach might involve setting the temperature hyperparameter $\beta$ dynamically, starting from 0 and increasing to a maximum value during training. This could potentially address the issue highlighted by the reviewer. However, this would add complexity to the method. Therefore, we choose not to include it by default, but it remains a natural and promising extension of RDT.

5. ‘RDT appears to provide greater benefits for Adroit tasks compared to the other two task settings. Could you explain the reason for this difference’. (Q2)

   While RDT achieves the highest absolute score on Adroit tasks, it is important to note that RDT demonstrates relative improvement across all three task settings. Specifically:

   • Under random attack conditions, RDT outperforms DT by:

     32.15% on MuJoCo, 40.49% on Kitchen, 21.86% on Adroit.

   • Under adversarial attack conditions, RDT exceeds DT by:

     27.44% on MuJoCo, 52.19% on Kitchen, 21.74% on Adroit.

   Thus, Adroit does not show the greatest improvement over DT. Our ablation study in Section 4.4 reveals that each component contributes to the enhancement of RDT, with Gaussian Weighted Learning being the most significant, particularly for the Kitchen task. A possible explanation is that long-horizon tasks with sparse rewards, such as Kitchen, require accurate action labels; otherwise, achieving rewards becomes challenging. This is where RDT proves to be more advantageous.

6. ‘the full dataset results in RIQL, and they seem to conflict with the scores listed in the paper’s appendix’. (Q3)

   Thanks for the question. We would like to highlight a difference in settings between the original RIQL and ours, as emphasized in line 118. In our unified data corruption, only three independent elements—states, actions, and rewards—are considered under a trajectory-based storage framework. Therefore, our definition of state corruption includes both state and dynamic corruption as defined in RIQL. The original RIQL implementation requires separate parameter tuning for state and dynamics corruption, which is not feasible in our context, potentially leading to a performance drop.

   Additionally, we present an analysis of the RIQL learning curve in Appendix D.7. When comparing this curve to that in the RIQL paper, we find that RIQL requires more epochs (3,000 epochs) to converge in the full dataset setting, while it converges faster (500 epochs) with limited data (10% MuJoCo). Given our focus on the limited data setting, we observe that 1,000 epochs are sufficient for both RDT and baselines like RIQL to converge, as detailed in Appendix D.5. Therefore, we use 1,000 training epochs by default. In the revision, we will include results for the full dataset using three times the training epochs for both RDT and RIQL.

7. Typos

   Thank you for pointing out those typos and minor issues. We have fixed them in the revised version.

2. ‘A discussion on the impact of corrupted data with varying quality levels is needed’. (W2)

   We agree that data quality is a critical factor in our benchmark design, and we already consider it in our evaluation. Since we investigate data corruption across three data components—state, action, and reward, and two types of corruption (random and adversarial), the experimental workload is substantial. Therefore, we select one representative dataset for each MuJoCo and Adroit task and all datasets from Kitchen. The medium-replay dataset offers a wide range of data quality, from random to medium, reflecting conditions close to real-world scenarios. In a word, our experiments already cover diverse data quality for comprehensive analysis, including random to medium-quality data in MuJoCo, varying quality of human demonstrations in Kitchen, and expert data in Adroit.

   To further evaluate the robustness across different data quality, we conduct an additional evaluation using expert and medium datasets from MuJoCo tasks during the rebuttal. We randomly sample a similar size of approximately $2 \times 10^4$ transitions to match the dataset size of our main experiments. These results are detailed in Appendix D.11 of our revision. Notably, RDT consistently outperforms other baselines across different quality level datasets. This superior performance reinforces RDT's capability to handle a range of dataset qualities efficiently.

   We provide the following tables from Appendix D.11 for your convenient reference.

Results on "medium-v2" dataset under random data corruption.

Results on "expert-v2" dataset under random data corruption.

Thank you for the valuable comments, and we provide clarification on your concerns as follows. We hope these new experimental results and explanations can address your concerns, and we look forward to your further feedback.

1. ‘An analysis of how dataset scale affects the challenges of data corruption would be beneficial’. (W1)

   Thank you for the insightful question. To provide an intuitive explanation: with larger datasets, a given transition is likely present in the dataset multiple times, increasing the probability that not all instances are corrupted. This makes it easier to learn from clean data. In contrast, in smaller datasets, a transition may exist only once, and if corrupted, it poses a greater challenge for offline RL methods to learn a correct policy. Our results in Figure 1 support this intuition, showing significant performance drops for all methods when limited to 10% of the datasets. Therefore, we primarily focus on the more challenging, limited data setting.

   To address the reviewer’s concern, we conduct new experiments with various corrupted dataset sizes, detailed in Appendix D.10, to better understand the impact of dataset size on the performance of RDT and other methods. Our empirical findings reveal the following:

   1. The overall performance of all methods improves as the dataset size increases.
   2. Temporal-difference-based methods like RIQL are comparable to vanilla sequence modeling methods like DT when dataset is large, but are less preferable when the dataset size is limited.
   3. RDT consistently outperforms other baselines across all dataset sizes, validating the effectiveness and importance of our proposed robustness techniques.

   To provide a convenient reference, we format the following tables extracted from Figure 17.

Results under random data corruption on the MuJoCo tasks with various dataset scales.

Results under random data corruption on the Adroit tasks with various dataset scales.

3. ‘a detailed comparison with other robust sequence modeling approaches outside of decision transformers’. (W3)

   We have included several offline RL algorithms in our main results that incorporate regularization or robustness techniques. Specifically, UWMSG [5] and RIQL [6] are methods that enhance value functions with additional robust mechanisms like uncertainty weighting and ensemble learning. In addition, DeFog[7] is a robust sequence modeling algorithm robust to frame-dropping in offline RL.

   Moreover, we also combine robust offline RL (e.g., UWMSG and RIQL) methods with transformer policy backbone in Section 4.5. These two variants simultaneously incorporate value functions and sequence modeling, serving as a strong baseline. Our RDT outperforms all these methods under various data corruption settings, demonstrating its superior ability to handle data corruption effectively.

4. ‘more clarity on the choice of dropout probabilities $p$’. (Q1)

   As stated on line 248 at the end of Section 3.2.1, we use the same dropout probabilities $p$ for different element (state, action, and reward) embeddings. We also conduct an ablation study on these probabilities in Appendix D.3. Setting dropout probabilities $p$ within a moderate range (0.1 to 0.3) can enhance performance. However, higher dropout probabilities can degrade performance, likely due to excessive loss of information from the input.

[1] D4rl: Datasets for deep data-driven reinforcement learning[J]. arXiv preprint arXiv:2004.07219, 2020.

[2] NeoRL: A near real-world benchmark for offline reinforcement learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 24753-24765.

[3] Neural discrete representation learning[J]. Advances in neural information processing systems, 2017, 30.

[4] Contrastive learning as goal-conditioned reinforcement learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 35603-35620.

[5] Corruption-robust offline reinforcement learning with general function approximation[J]. Advances in Neural Information Processing Systems, 2024, 36.

[6] Towards robust offline reinforcement learning under diverse data corruption[J]. arXiv preprint arXiv:2310.12955, 2023.

[7] Decision transformer under random frame dropping[J]. arXiv preprint arXiv:2303.03391, 2023.

Thank you for the valuable comments. We provide clarification on your concerns as follows. We hope these new experimental results and explanations can address your concerns, and we look forward to your further feedback.

1. ‘Incorporating more complex, real-world datasets could provide stronger support for the robustness claims in real-world applications’. (W1)

   Thank you for the valuable suggestion. Our primary goal is to demonstrate the effectiveness of robust sequence modeling method on popular offline RL benchmarks [1], including MuJoCo, Kitchen, and Adroit. Specifically, the Kitchen and Adroit tasks are two challenging and relatively realistic robotic manipulation tasks.

   To further demonstrate the robustness of RDT in real-world applications, we evaluate it on a near real-world benchmark, the NeoRL benchmark [2], with the citylearn-medium and finance-medium datasets. The Finance dataset simulates real stock market trading, while CityLearn focuses on optimizing energy storage across various buildings to reshape electricity demand. We randomly sample 20% and 10% of the Finance and CityLearn datasets, respectively, resulting in both datasets having a similar size of approximately $2 \times 10^4$ transitions.

   We compare our method against baselines such as RIQL, DeFog, and DT. We have included these experiments in Appendix D.12 and in the table below for your reference. Notably, RDT achieves the best average performance compared to the other baselines, outperforming DT by 12.13% under Finance dataset. This result further demonstrates its robustness and promise for real-world applications.

   Task	Attack	DeFog	RIQL	DT	RDT
   citylearn (20%)	State	39138.9	30235.3	37878.9	39124.2
   	Action	38970.6	29093.2	35589.0	39066.4
   	Reward	38156.4	29634.8	37626.9	38909.8
   Average		38755.3	29654.4	37031.6	39033.5
   finance (10%)	State	92.4	89.4	85.7	104.5
   	Action	96.4	100.6	91.4	99.8
   	Reward	90.2	89.2	89.7	95.1
   Average		93.0	93.1	89.0	99.8

2. ‘What challenges or limitations prevented state correction’. (W2,Q2)

   We have included a discussion on the state prediction and correction mechanisms in Appendix D.14. We compare two variants: DT(SP), which predicts states based on the original DT, and DT(SP) w. IDC, which applies iterative state correction to DT(SP). These variants are evaluated under state corruption.

   Figure 19 shows that DT(SP) does not consistently offer benefits over the original DT, and it sometimes even results in a performance drop. Additionally, DT(SP) w. IDC fails to achieve satisfactory performance across different $\zeta$ values. To provide a convenient reference, we format the following table extracted from Figure 19.

   Task	DT	DT(SP)	DP(SP) w. IDC ($\zeta$=1.0)	DP(SP) w. IDC ($\zeta$=3.0)	DP(SP) w. IDC ($\zeta$=5.0)
   MuJoCo (10%)	20.1	23.2	23.1	24.5	22.8
   KitChen	33.2	25.2	14.5	23.4	24.0
   Adorit (1%)	84.6	81.9	76.3	77.2	79.1

   Several factors may contribute to the challenges of state prediction and correction:

   1. High Dimensionality of States: The high dimensionality of states makes it difficult to predict states accurately. For example, in KitChen tasks, the state space dimension (60) is much higher compared to the action space dimension (9) and reward dimension (1).
   2. Data Corruption: In our setting, data corruption can exacerbate the challenge of predicting states, resulting in further declines in performance.

   Therefore, we leave the state prediction and correction as promising future work. Advanced state representation techniques, such as discrete representation learning through VQVAE [3] and contrastive learning [4], could potentially be beneficial in addressing this issue.

We sincerely thank you for your valuable suggestions and time spent on our work. We hope that we have addressed your main questions in the rebuttal. As the due date (November 26 at 11:59pm AoE) for the discussion period is approaching, we hope to receive your further feedback and address your concerns once again. We are committed to addressing any concerns or suggestions you may have to enhance the manuscript.