Adaptive $Q$-Aid for Conditional Supervised Learning in Offline Reinforcement Learning

Thank you for your positive comments on the strengths of our work. Your valuable suggestions will help us further improve and clarify our work.

W1. The definition of degree of Q-aid.

Thank you for pointing out a good point. We have dealt with environments where the deviations in the initial state are marginal to affect the scale of the returns significantly, but in practical scenarios, differences may arise. In such cases, a higher trajectory return may not always indicate optimality. Considering how to set the weights in a more practical setting with significant differences in the initial state could be a good follow-up work.

W2. Regarding writing.

As the reviewer mentioned, the focus of our work is combining Q-learning with RCSL. To develop an algorithm for this purpose, we first (1) experimentally demonstrated that the Q-greedy policy and RCSL can be complementary depending on the quality of the dataset, and (2) analyzed the reasons for this through an examination of Q generalization. Based on this analysis, we determined the QCS weight and developed the QCS algorithm. Therefore, we believe that discussing the Q-greedy policy is necessary to explain the need for and development process of our algorithm. However, we will improve the writing to make this point brief and clear. However, we will improve the writing to make this point brief and clear. Additionally, we will refine our manuscript to include more experimental details in the main section.

Thank you for the detailed comments that help clarify our algorithm and for the positive feedback on our work. As suggested, we have included further explanations and experimental results to enhance the significance of our study. Due to the rebuttal word limit, we have addressed the response regarding context length in the global response G5.

W1. Regarding complexity.

We compare QCS training time complexity with IQL, CQL, and QDT [Ref.8]. QCS requires a pre-trained Q learned using the IQL method, while QDT requires a pre-trained Q learned using the CQL method. Therefore, the time was measured by incorporating the Q pretraining time for both algorithms.

The training times for IQL, CQL, QDT, and QCS are as follows: IQL - 80 min, CQL - 220 min, QDT - 400 min, and QCS - 215 min.

The results show that QCS takes longer than IQL but has a total time similar to CQL. Notably, compared to QDT, which requires CQL pretraining, QCS can be trained in nearly half the time but demonstrates superior performance to QDT as shown in our main results in Table 2.

[Ref.8] Yamagata, Taku, et al. "Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modeling in offline rl." ICML 2023.

W2/Q1. Regarding calculating R*

The detailed explanation of how to obtain R$^*$ is as follows. We propose two ways of calculating R$^*$, and we will add this explanation to our manuscript.

• (1) Set R$^*$ with the optimal return for the environment.

For our main experiments, we set R$^*$ for the environments with the optimal return for the environment as follows: Hopper - 3500, Walker2d - 5000, Halfcheetah - 11000, and Antmaze - 1. Note that prior RCSL algorithms, such as Decision Transformer [Ref.8] or RvS [Ref.7], used predefined R$^*$ for target RTG conditioning at the inference stage. Therefore, setting R$^*$ with optimal return requires no further assumptions compared to prior RCSL works. As mentioned in Appendix J.2 of our manuscript, for target RTG conditioning, QCS does not use R$^*$ but instead uses the maximum trajectory return, only requiring one R$^*$ per algorithm.

• Set R$^*$ to the maximum trajectory return within the dataset.

Another way of setting R$^*$ is to use the maximum trajectory return within the dataset. In situations where obtaining the optimal environment return is difficult, we can infer the optimal return using the maximum trajectory return. Therefore, in Table R4 we provide additional results using the maximum trajectory return as R$^*$.

[Table R4. QCS performance when R* is the maximum dataset return and the optimal environment return.]

As shown in Table R4, setting R$^*$ with the optimal environment return is slightly better than setting it with the maximum dataset return, but setting it with the maximum dataset return still outperforms the baselines. Therefore, we propose using the optimal environment return for R$^*$; however, when it is hard to determine, using the maximum dataset return can be a good alternative.

[Ref.7] Emmons, Scott, et al. "RvS: What is Essential for Offline RL via Supervised Learning?." ICLR 2022.

[Ref.8] Chen, Lili, et al. "Decision transformer: Reinforcement learning via sequence modeling." NeurIPS 2022.

W3/Q3. Impact of the QCS weight $\lambda$.

We wish to direct the reviewer's attention to Table 9 in Appendix H.2 of our original manuscript, which investigates the effect of $\lambda$ by varying it from 0.2 to 1.5. We additionally test the QCS with $\lambda$ 0.5 and 1 with MuJoCo medium-expert datasets and the result can be seen in Table R5.

As shown in Table 9 and R5, except for the walker2d-medium and halfcheetah-medium-expert, we found that even the smallest values achieved with changing $\lambda$ either matched or surpassed the performance of existing value-based methods and RCSL's representative methods, including IQL, CQL, DT, DC, and RvS. This demonstrates QCS's relative robustness regarding hyperparameters. For walker2d-medium and halfcheetah-medium-expert, we found that when $\lambda$ exceeds the initial setting of 0.5, performance begins to decrease. We anticipate that further research will address these issues.

Overall, setting $\lambda$ to 0.5 consistently produces good performance, even though it results in performance degradation of around 2 points in some environments. Compared to many offline RL algorithms that require tuning more than 10 sets of hyperparameters depending on the environment, achieving stable performance with only one or two hyperparameter adjustments is a major strength of QCS.

[Table R5. Impact of the QCS weight $\lambda$ in MuJoCo medium-expert datasets.]

We appreciate the reviewer's efforts in providing constructive feedback to improve our work. Our detailed response to the reviewer's comments is posted below and also in the global response G1-G4.

W1. Comparison with HIQL.

Thank you for introducing the highly effective algorithm HIQL [Ref.2] for the goal conditioning task. RL tasks can be divided into two categories: return-maximizing tasks, which aim to earn the maximum return, like MuJoCo, and goal-reaching tasks, which aim to reach the goal with a higher success ratio, like Antmaze. HIQL specializes in goal-reaching tasks as it uses a hierarchical approach that generates high-valued subgoals with a higher-level policy and generates actions with a lower-level policy conditioned on those generated subgoals. HIQL achieves superior performance on Antmaze tasks, but due to the nature of the algorithm, which involves generating and conditioning on subgoals, it has limitations when applied to tasks like MuJoCo.

In Table R2, we compare QCS-R and QCS-G with HIQL in the Antmaze medium and large datasets. QCS previously used a hidden dimension of 256 for training the Q-function, and we observed that HIQL uses a hidden dimension of 512. Therefore, we re-trained the Q-function with an increased hidden dimension of 512 and confirmed that this setting is beneficial for QCS. In Table R2, we mark the original QCS-R/QCS-G score as QCS-R(256)/QCS-G(256) and the new score as QCS-R(512)/QCS-G(512).

As shown in Table R2, our new QCS score is comparable to HIQL, which is specialized for goal conditioning tasks, and particularly, QCS-R performs even better than HIQL. This new result reaffirms that our algorithm, despite being general, is effective even when compared to specialized algorithms.

[Table R2. Performance comparison between QCS and HIQL.]

[Ref.2] Park, Seohong, et al. "Hiql: Offline goal-conditioned rl with latent states as actions." NeurIPS 2023.

W2. Recentness of Baselines.

We want to note that we compared baselines including very recent works such as ACT (AAAI 2024), CGDT (AAAI 2024), DC (ICLR 2024), FamO2O (NeurIPS 2023), EDT (NeurIPS 2023), QDT (ICML 2023), SQL (ICLR 2023). Especially since the NeurIPS 2024 submission deadline was May 22, considering this date, the works for AAAI 2024 and ICLR 2024 can be regarded as very recent, having been published less than three months ago.

We continuously track the latest works, and after the NeurIPS submission deadline, we found a new work, Q-value Regularized Transformer(QT) [Ref.6] that proposes a new way of combining RCSL and Q-learning, published at ICML 2024. Note that according to NeurIPS regulations, QT is considered `Contemporaneous Work' because it appeared online after the submission. Since the work appeared after our submission, we are not obligated to compare it, but to help the reviewer better appreciate the quality of our work, we additionally compare QCS with QT in Table R3 in response W3 using the same evaluation metrics as QCS.

[Ref.6] Hu, Shengchao, et al. "Q-value Regularized Transformer for Offline Reinforcement Learning." ICML 2024.

W3. Comparison with QT and POR with the same evaluation metric of QCS.

As mentioned in our manuscript Section 6.1, we report QCS as the last running average score. However, QT selects the best score throughout the whole training process (as confirmed through email with the QT author), which is not suitable for offline RL settings. Offline RL assumes a situation with limited online interaction, but finding the best score relies heavily on a significant amount of online interaction. Therefore, we re-ran the QT through the official QT code with author-recommended hyperparameters and verified the last running average score.

Regarding POR, mentioned by the reviewer for the antmaze large score, we previously reported POR scores from the POR paper. We were unable to confirm the exact evaluation metric, but the authors shared the training curve of POR on the POR github. From this, we verified the last running average score of POR.

To summarize, in Table R3 and Table R9 in the global response PDF, we compare QCS with QT and POR using the last running average score. As can be seen in Tables, QCS outperforms both POR and QT in MuJoCo and Antmaze domains, with POR's Antmaze large scores being below 70, demonstrating that QCS is a robust and high-performing algorithm.

[Table R3. Performance comparison between QCS (ours), POR, and QT. We evaluate each algorithm based on the last running average score.]

Observation of Q Overgeneralization is one of our contributions.

Starting from the 164th sentence in our manuscript (slightly below the sentence mentioned by the reviewer)—"We present a simple experiment to verify that learning $Q_\theta$ indeed induces over-generalization when trained on optimal trajectories."—this phenomenon is illustrated throughout Fig. 4 and Fig. 5 in Section 3.2.

The topic 'Why Does the Q-Greedy Policy Struggle with Optimal Datasets?' is addressed in Section 3.2 by demonstrating the phenomenon of Q generalization and validating it through various methods across different domains.

Section 3.2 explains that the Q function tends to over-generalize when trained on an optimal dataset, which is why the Q-Greedy Policy struggles with optimal datasets. Throughout the section, we demonstrate that this overgeneralization phenomenon can particularly occur when training with an optimal dataset, using various domains including toy environment, Gym Inverted Double Pendulum, and MuJoCo Hopper, as well as various methods such as Q-value analysis and NTK analysis (please see Figures 4 and 5 in our manuscript). The appendix provides an in-depth analysis that couldn’t be fully covered in the main text due to space constraints, but it is already addressed in the main body.

The results reported in Appendix E (the MuJoCo Hopper results) are additional analyses that connect to the Hopper results shown in Figure 5 of the main paper.

In Section 3.2, Figure 4, we present the results of a toy experiment, and immediately afterward, in Figure 5, we extend this experiment to demonstrate that a similar phenomenon occurs in both the Gym Inverted Double Pendulum and MuJoCo Hopper environments. We would like to note that the main paper includes not only the results of the toy experiment but also the results for Hopper. Therefore, the results in Appendix E are not a sudden comparison involving a change of domain and an expansion of the dimension from the toy example, but rather a more detailed analysis of the results presented for Hopper in Figure 5.

We appreciate the reviewer's additional comments aimed at further clarifying our work.

POR changed their official score after their camera-ready version.

In the POR GitHub, around Nov 2022, questions were raised regarding the reproduction of POR results. Since the POR authors could not access their previous code due to the copyright issue, they re-implemented the work and presented the results anew. While most of the results matched the previous ones, they did not match the hopper-medium-v2 dataset. As a result, they replaced the score with the new results and updated the arxiv version accordingly. Through a comment on GitHub, the author recommended using the new results instead of the conference version, which the reviewer had checked. The exact author's comment is "We have updated our paper in the arxiv to reflect that and the mismatch results in the hopper-medium-v2 dataset (lower it from 98.2 to 78.6). If you want to compare POR with your work, you can refer to this paper version.". Due to NeurIPS rebuttal policies, we cannot include external links, but we provide the address of this comment: github.com/ryanxhr/POR/issues/2. Therefore, our POR score reporting is accurate and aligns with the author's intent.

Comparison with QT in Table R9.

Surpassing QT in 13 out of the 15 datasets we compared, with the remaining 2 scores being similar to QT within the range of standard deviation, cannot be a reason for rejection but rather demonstrates the superiority of QCS. (QCS mean 86.3 > QT mean 60.9)

We want to emphasize that the offline RL benchmark features a variety of state and action dimensions, as well as different goals, making it a diverse and challenging benchmark where it is difficult for a single algorithm to perform well across all datasets compared to all baselines. For example, both POR and QT have instances where their performance is similar to or even lower than that of previous studies when examined closely at each dataset. For example, POR's performance of 76.6 on walker2d-medium-replay is lower than TD3+BC's 81.8, and QT's reported performance of 59.3 on antmaze-medium-diverse is lower than IQL's 70.0. However, POR and QT are recognized for their sufficiently good results due to their generally strong performance. Slightly lower or similar scores to baselines on a few datasets cannot be grounds for rejection. We believe that what truly matters is how robustly the performance has improved compared to previous works.

Comparison of QCS to the Best Score

As mentioned in the previous response, we were unable to confirm the exact evaluation metric for POR. Since we have already compared the scores from the POR paper provided by the POR author in Tables 2 and 3 of our original manuscript, we conduct an additional comparison for QT with the best score. Table R10 shows the results of the comparison using the best score. As can be seen from the table, the QCS score has generally increased compared to the running average score that we previously reported. Compared to QT, it also shows superior results on average in MuJoCo (six datasets are better, two are similar, and one is worse), and especially demonstrates better performance with a large gap in Antmaze. While we knew that reporting the best score would naturally result in better performance, we want to emphasize again, as mentioned in our previous response, that we believe the last running average score is a more appropriate metric for offline settings, which is why we reported using this metric. We have demonstrated better performance than QT, but we would also like to clarify once again that QT should not be a reason for rejection, as it qualifies as concurrent work under NeurIPS policies.

Table R10. Performance comparison between QCS (ours) and QT. We evaluate each algorithm based on the best score.

Thank you for providing comments and positive feedback on our work. We hope that our response below will clarify our algorithm and innovations.

W1. The innovation of QCS.

We summarize the innovations of QCS as follows:

• We first analyze the strengths and weaknesses of RCSL and Q-learning in various dataset qualities and discover that RCSL and Q-learning can be complementary depending on the quality of the dataset.
• Especially, we identify the Q-function over-generalization problem when performing Q-learning with an optimal quality dataset.
• Based on the analysis of Q-function over-generalization, we propose a novel algorithm that combines RCSL and Q-learning, called QCS. QCS demonstrates superior performance compared to various recent and SOTA baselines.

We want to emphasize that our analysis of the complementary conditions of RCSL and Q-learning, as well as the Q-function over-generalization problem, is novel and has not been addressed in previous works. The superiority of the QCS algorithm, which is based on these analyses, further confirms that our analyses are necessary and can lead to a deeper understanding within the offline RL community.

Q1. Regarding the subgoal selection.

For QCS-G, we follow the subgoal selection as proposed by RvS [Ref.7]. As per RvS, during the training phase, we randomly select a subgoal from among the states between the next state and the final state. For the evaluation phase, we prefix the subgoal as the goal state. We select subgoals based on the states present in the dataset, and there is no need for a special algorithm; we simply choose randomly among the next states, incurring no additional computational cost or difficulty.

Moreover, as can be seen in Table 3 of our manuscript, the QCS-R score, which only utilizes reward information, shows performance comparable to QCS-G. This demonstrates that the QCS algorithm can achieve good performance even without subgoal selection.

[Ref.7] Emmons, Scott, et al. "RvS: What is Essential for Offline RL via Supervised Learning?." ICLR 2022.

Thank you for bringing up a good point. As we mentioned in our previous response, subgoal conditioning follows the prior work [Ref.7], and our explanation of the effect of the subgoal is summarized below.

Let's first reconsider return conditioned supervised learning. In the case of return conditioning, the agent basically learns $a = f_\theta(s, \hat{R})$, where $\hat{R}$ is the return-to-go, i.e., sum of rewards received from the current timestep until the end of each episode. During training, the joint function $f_\theta$ of $(s, \hat{R})$ is learned with various $\hat{R}$ values from good or bad trajectories in the dataset. Basically we learn the function $f_\theta$ on the $(s, \hat{R})$ plane hoping for generalization over a wide region of $(s, \hat{R})$ with various $(s,\hat{R})$ data points. Later during the test phase, we set a high target return-to-go as $\hat{R}$ so that the model will output an action $a$ for each $s$ with high return-to-go. That is, we use $a=f_\theta(s, \hat{R}\_{target})$, a function from $s$ to $a$, where $\hat{R}\_{target}$ is a fixed high target value.

Similarly, subgoal conditioning case can be considered to learn a function $a = f_\psi(s, \hat{s}\_{goal})$, where the subgoal $\hat{s}\_{goal}$ is a randomly-picked state between the next step and the episode end in each episode.
If $\hat{s}\_{goal}$ is from a bad episode, the produced action will be bad. If $\hat{s}\_{goal}$ is from a good episode, the produced action will be good. But, again, our goal is to learn an entire function $a = f_\psi(s, \hat{s}\_{goal})$ generalized over a wide joint region of $(s, \hat{s}\_{goal})$, where $\hat{s}\_{goal}$ ranges from good to bad (hoping for interpolation and extrapolation) so that in the later test phase, we use this function $a = f_\psi(s, \hat{s}\_{goal})$ generalized over a wide joint region of $(s, \hat{s}\_{goal})$ by fixing $\hat{s}\_{goal}$ to be the ultimate goal of the task, which is known for each task.

We hope our additional response has addressed all your concerns. If your concerns have been resolved or if you have any further questions, please feel free to let us know.

[Ref.7] Emmons, Scott, et al. "RvS: What is Essential for Offline RL via Supervised Learning?." ICLR 2022.