Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning

We appreciate the reviewer’s thoughtful and constructive feedback. We revised multiple text issues that the reviewer pointed out. Due to space limits, tables were deferred to our response to Reviewer HX7o or can be found in the paper.

Training time for world model and for algorithms with and without COCOA

Learning dynamics is not a significant burden compared to policy training. The time taken to train dynamics models for each algorithm is about 5-10 percent of the policy training time. For IQL as an example, it takes about 1-2 hours on training dynamics, just a few minutes for anchor seeker training and 13 hours for policy training on TITAN RTX GPU. The training time per epoch increased from 11 seconds to 24 seconds as of now.

We mainly focused on the performance improvement that COCOA can bring to existing offline RL algorithms rather than optimizing the running time of our implementation. The majority of the additional training time by COCOA involves frequent calls to the pretrained anchor seeker policy, the dynamics model and the main policy. We expect that training time with COCOA can be greatly reduced through the operator fusion using a toolbox like TorchInductor.

Overall performance with COCOA

Upon investigation following the reviewer's suggestion, we present our updated experimental results with the layer normalization for the anchor seeking policy and the transductive predictor, which turned out to be the missing component for stable training of COCOA. The results are shown in Table 1 in the paper. Comparison with reproduced IQL performance on Gym-MuJoCo random tasks are also included in Appendix. Note that we add the layer normalization only to the COCOA-specific components for a fair comparison with the baselines. As a result, COCOA increases the average performance of all base algorithms evaluated on the D4RL Gym-MuJoCo benchmark. The winning task ratio also becomes 63%. This suggests that with a proper training stabilization, adding COCOA can be beneficial across diverse base algorithms.

Other benchmarks for COCOA

Following the reviewer’s suggestion, we experiment with two more benchmarks - D4RL Adroit and NeoRL [5]. The outcomes of these experiments are summarized in Table 2, 3 in our response to Reviewer HX7o. This broader analysis reveals that COCOA significantly enhances the performance of IQL and MOBILE in most tasks. For a detailed insight into the experimental setup, please refer to the updated paper.

Our method consistently enhances the performance across six tested D4RL Adroit tasks, showcasing its robustness and adaptability. In the difficult tasks like the door and hammer due to sparse rewards, the baselines work poorly, and our COCOA can still improve them. On the other hand, in the pen task, our method achieves significant performance improvements.

We also verify the effectiveness of our algorithm in the NeoRL benchmark, known for its conservative, less exploratory data generation compared to the D4RL Gym-MuJoCo. Although its state-action space coverage is highly limited, our preliminary experiments, conducted using MOPO, show significant performance improvement, suggesting potential benefits for other algorithms as well, which will be done for the camera-ready.

Due to the time constraint of the rebuttal period, we test COCOA on six D4RL Adroit tasks on top of IQL and MOBILE, with plans to expand to CQL, MOPO, and ReBRAC in the camera-ready, and initially tested only with MOPO for NeoRL, intending to examine more algorithms later.

Comparison with ReBRAC

Comparing with offline RL methods that use policy regularization would make our results more comprehensive. However, due to the limited time frame, our work on incorporating ReBRAC with COCOA into a single framework is still in progress. We are hoping to include the results in the camera-ready version.

Hyperparameter sensitivity

Regarding MOPO and MOBILE's hyperparameter sensitivity, due to the time and resource constraints, we could not perform exhaustive analysis like EOP. We found that the rollout length and penalty coefficient need to be slightly larger than those used in the existing MOPO and MOBILE methods. While MOPO and MOBILE conduct the searches for rollout length and penalty coefficient with 2 and 4, or 2 and 5 values respectively, we use our search with 3 and 5 values.

We appreciate the reviewer’s time and effort put into improving our paper.

[1] Kumar, Aviral, et al. (2020). Conservative q-learning for offline reinforcement learning. NeurIPS. [2] Kostrikov, Ilya, et al. (2022). Offline Reinforcement Learning with Implicit Q-Learning. ICLR. [3] Yu, Tianhe, et al. (2020). Mopo: Model-based offline policy optimization. NeurIPS. [4] Sun, Yihao, et al. (2023). Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning. ICML. [5] Rongjun Qin, et al. (2022). NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning. NeurIPS

We appreciate the reviewer’s thoughtful and constructive feedback.

Comparison with COMBO algorithm

CQL+COCOA and COMBO exhibit some similarities, notably in their use of dynamics and a less conservative approach towards state-action space. However, their methodologies in pursuing conservatism differ significantly: COCOA focuses on conservatism in the compositional input space, whereas COMBO emphasizes regularizing the values for unfamiliar actions. Thus, COCOA and COMBO are orthogonal, it would be an insightful comparison to integrate COCOA with COMBO, as COCOA can be an add-on to any algorithm.

Similar to COMBO, MBPO-based methods like MOPO and RAMBO also show a tendency to outperform model-free methods in random and medium settings [1,2]. It seems that data augmentation through MBPO is particularly beneficial in these tasks. It would be interesting to theoretically or empirically compare the state-specific value functions among CQL, CQL+COCOA, and COMBO for further analysis.

Table 4: Comparison of CQL+COCOA and COMBO on D4RL Gym-MuJoCo benchmark.

Comparison with synthetic experience replay (SynthER)

Thank you for proposing an interesting perspective on the utilization of the dataset. The dataset augmentation using dynamic models could be substituted with that using synthetic experience replay. While the selection of the augmentation strategy is orthogonal to COCOA. It seems extremely interesting to investigate how the synthetic experience replay with a generative model synergizes with our algorithm.

We appreciate the reviewer’s time and effort put into improving our paper.

[1] Yu, Tianhe, et al. (2020). Mopo: Model-based offline policy optimization. NeurIPS. [2] Sun, Yihao, et al. (2023). Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning. ICML. [3] Marc Rigter, Bruno Lacerda, & Nick Hawes (2022). RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning. NeurIPS.

We appreciate the reviewer’s thoughtful and constructive feedback.

Choice of the reverse policy

An effective reverse rollout policy should diverge from an offline dataset and encompass various terminal states. We avoided training a reverse policy because defining a precise loss function for it would involve quantifying (1) the state's distance from the dataset distribution and (2) the terminal state diversity, which is computationally intensive.

Hence, we employed a random divergent reverse policy while comparing COCOA with trained reverse policy as an ablation study. Similar to [1], we employed a state-conditioned conditional variational autoencoder[2] that mimics the offline dataset's behavior in reverse. The performance of MOPO+COCOA on two D4RL Gym-MuJoCo tasks showed that a random diverse policy is superior for high performance.

Table 1: Ablation study on the reverse policy.

Other benchmarks for COCOA

Following the reviewer’s suggestion, we experimented with two more benchmarks - D4RL Adroit and NeoRL [3]. The outcomes of these experiments are summarized in the tables below. This broader analysis reveals that COCOA enhances the performance of IQL and MOBILE in most tasks. For a detail about experimental setup, refer to the updated paper.

Our method demonstrated consistent performance enhancements across six D4RL Adroit tasks we tested, showcasing its robustness and adaptability. While COCOA encountered challenges in complex tasks like the door and hammer, akin to its original algorithm, this reflects the inherent difficulty of these tasks due to sparse rewards. Notably, in tasks like the pen, our method achieved noticeable performance improvements.

We also verified the effectiveness of our algorithm in the NeoRL benchmark, which is known to be more difficult than the traditional D4RL Gym-MuJoCo benchmark as it generates the data in a more conservative and less exploratory manner. The data collected in this manner have less coverage in the state-action space, posing the need for more fine-grained stitching. In this perspective, we can see that the dataset's challenges align with our algorithm. To check whether our algorithm also performs well in NeoRL, we firstly conducted experiments with MOPO which demonstrates an impressive performance improvement. Based on these enhancements, we anticipate that other algorithms would also have a big advantage with our algorithm. Under time constraints, we tested COCOA on six D4RL Adroit tasks on top of IQL and MOBILE, with plans to expand to CQL, MOPO, and ReBRAC in the final paper, and initially tested only with MOPO for NeoRL, intending to examine more algorithms later.

Table 2: D4RL Adroit benchmark results.

Table 3: NeoRL benchmark results.

The performance of IQL

Especially in the context of IQL, we observed that certain tasks occasionally exhibited a sudden drop in performance. Based on the guidance you provided, and considering that model-free algorithms are typically more challenging to train compared to model-based algorithms (as they do not augment the dataset with a dynamics model), we finalized our network design to employ the layer normalization technique. This approach is particularly much more effective in tasks especially such as 'hopper-medium-replay', 'walker 2d-medium-replay' and so forth.

We appreciate the reviewer’s time and effort put into improving our paper.

[1] Wang, Jianhao, et al. (2021). Offline Reinforcement Learning with Reverse Model-Based Imagination. NeurIPS. [2] Sohn, Kihyuk, et al. (2015). Learning structured output representation using deep conditional generative models. NeurIPS. [3] Rongjun Qin, et al. (2022). NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning. NeurIPS