Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network

Thank you for your insightful review and valuable suggestions. We appreciate your careful reading and constructive feedback. Supplementary Material for our response is at THIS LINK.

Below we address your specific questions and concerns one by one and provide clarifications and additional results as requested.

1. Clarifications on Theoretical Claims (Eq. 3 and Eq. 10)

We are greatly thankful for your careful reading and for pointing out the ambiguity in the theoretical claims.

Indeed, the exponential of advantages is inherently normalized due to its definition. In Soft Q Learning[1], the soft value function is defined using the soft Q function (Eq. 4)

$V^*_{\text{soft}}(s_t) = \alpha \log \int_{A} \exp ( \frac{1}{\alpha} Q^*_{\text{soft}}(s_t, a') ) da'$

Note that the soft value is a softmax of Q, and is different from the common concept of value function in RL. By construction, this directly implies:

$\int_{A} \exp ( \frac{1}{\alpha} ( Q^*_{\text{soft}}(s_t, a_t) - V^*_{\text{soft}}(s_t) )) da_t = 1$

which means that the exponentiated soft advantages are already normalized. This equality, crucial to our theoretical claims (Eq. 3 and Eq. 10 in our manuscript), has been rigorously proven in Theorem 1 of the original Soft Q-Learning paper [1].

To address the reviewer’s valuable comment, We will explicitly clarify this point in the revised paper to strengthen the theoretical justification.

[1] Haarnoja et al. (2017). Reinforcement learning with deep energy-based policies. ICML.

2. Clarification on Theorem 4.3

Thank you for highlighting the complexity of Theorem 4.3.

Theorem 4.3, which exactly means the exponential of the dimensional soft advantage represents a valid probability distribution, allows us to express the overall soft advantage function as the sum of the dimensional soft advantages. In this formulation, the dimensional soft advantage serves as a critical link between auto-regressive policy representation and Q-value prediction.

We agree with the reviewer that the conditions of Theorem 4.3 are non-trivial and may not be satisfied naturally. To address this, we enforce a hard constraint by normalizing the output of the dimensional advantage prediction network. Specifically, we apply a log-sum-exp subtraction, as defined in Eq. (16). This normalization ensures that Theorem 4.3 holds, thereby validating the correctness of all subsequent derivations.

Additionally, we identified a typo in Eq. (16), missing temperature parameter $\alpha$. The corrected equation is:

$A^d(\mathbf{s}_t, \mathbf{a}^{-d}, a^d) = u^d(\mathbf{s}_t, \mathbf{a}^{-d}, a^d) - \alpha log \sum _{a^{d'}} \exp ( \frac{1}{\alpha} u^d(\mathbf{s}_t, \mathbf{a}^{-d}, a^{d'}) )$

We will rearrange the theoretical claims and correct this typo accordingly in the revised manuscript.

3. Evaluation on Fully Offline and Fully Online Settings

We agree with the reviewer that evaluating ARSQ in both fully offline and fully online settings is essential for completeness.

In the fully offline setting, we have experiments in Tab. 1 in Supplementary Material, comparing ARSQ against representative offline RL and imitation learning methods. ARSQ demonstrates superior overall performance across tasks, highlighting its effectiveness on suboptimal offline data.

In the fully online setting, we compare ARSQ against online CQN and PPO[2] in Fig. 1 in Supplementary Material. Online ARSQ achieves better sample efficiency than CQN and PPO, underscoring its potential as a general-purpose reinforcement learning algorithm. However, we observe that although online ARSQ eventually matches the converged performance of the standard ARSQ, it requires approximately $4\times$ more environment interactions to reach comparable performance, highlighting the importance of using offline datasets to enhance sample efficiency.

[2] Schulman et al. (2017). Proximal policy optimization algorithms. arXiv:1707.06347.

4. Training Curves for D4RL Main Results (Fig. 4)

Thank you for pointing out the absence of training curves. We have added detailed training curves for each task in Fig. 2 in Supplementary Material, clearly showing the number of online environment steps (approximately 25k–50k steps until convergence).

We will also include these training curves in the revised manuscript to enhance clarity.

5. Tasks Evaluated in Figure 7

Thank you for pointing out the lack of clarity regarding the tasks evaluated. Figure 7 reports results on two D4RL tasks (hopper-medium-expert and hopper-medium-replay), and one RLBench task (Open Oven).

We will clarify this in the revised manuscript.

We sincerely appreciate your constructive feedback, which has significantly improved the quality and clarity of our paper. We hope these clarifications and additional results address your concerns, and we are happy to further discuss any remaining questions.

We sincerely thank the reviewer for their thoughtful and constructive feedback. Supplementary Material for our response is at THIS LINK.

Below, we address each point raised:

1. Analysis of the Nature of Suboptimality

We agree that explicitly analyzing the nature of suboptimality is important, and provide two additional analyses to better illustrate suboptimality of datasets:

• Trajectory Reward Analysis (Fig. 1 in Supplementary Material): We visualize histograms of trajectory rewards in D4RL datasets, intuitively showing varying data quality.

• Case Study (Fig. 2 in Supplementary Material): We introduce a simplified environment and compare ARSQ against action dimension independent Q decomposition methods. The learned Q landscape demonstrates ARSQ's ability to learn accurate Q-values despite suboptimal data.

2. Comparison to Alternative Discretization Strategies

We agree that ablations of alternative discretization strategies are important. In fact, we have provided some analyses along these lines in Section 5.3, where we evaluate several alternative discretization variants, including variants without hierarchical coarse-to-fine discretization (w/o CF Cond., w/o CF), variants that generate actions independently for each dimension (w/o Dim Cond., Plain), and a variant that swaps the conditioning order (Swap).

The results show that ARSQ consistently outperforms all other variants, highlighting the effectiveness of our discretization strategy.

3. Comparison to Offline RL and Offline IL Methods

We appreciate this valuable suggestion. In response, we have included additional comparisons between ARSQ and several popular offline RL and IL methods, in Table 1 of Supplementary Material. Regarding mentioned baselines:

• SafeDICE primarily targets scenarios with labeled non-preferred trajectories. Thus, we adopt its preliminary version, DWBC, which is more applicable to our setting.
• EDAC is orthogonal and can be integrated to ARSQ in principle. However, due to time constraints, we leave this integration for future work.
• IQL and other offline RL/IL baselines are included in our revised paper.

These comparisons further support the effectiveness of ARSQ in learning from suboptimal data.

4. Analysis of Theorem 4.3

The normalization condition in Theorem 4.3 (Eq. 12) is indeed essential for the conclusion that the soft advantage can be decomposed into the summation of dimensional soft advantages (Eq. 13). Without this assumption, dimensional decomposition does not hold. To ensure this normalization constraint in practice, we apply a hard constraint during training via log-sum-exp subtraction (Eq. 16), ensuring consistency and theoretical validity.

We also identified a typo in Eq. 16, where the temperature parameter $\alpha$ was inadvertently omitted. The corrected equation is:

$A^d(\mathbf{s}_t, \mathbf{a}^{-d}, a^d) = u^d(\mathbf{s}_t, \mathbf{a}^{-d}, a^d) - \alpha log \sum _{a^{d'}} \exp ( \frac{1}{\alpha} u^d(\mathbf{s}_t, \mathbf{a}^{-d}, a^{d'}) )$

We will correct this typo in the revised manuscript.

5. Convergence to Optimal Q-function under Approximation Errors

Our algorithm updates value and advantage functions via soft Bellman iteration (Eq. 14). Haarnoja et al. [1][2] have demonstrated that, in the absence of approximation errors, the optimal soft Bellman operator possesses a unique fixed point and is a contraction mapping in the infinity norm with contraction factor $\gamma < 1$. Thus, we expect our value iteration to converge close to the optimal solution if approximation errors remain bounded.

[1] Haarnoja et al. (2017). Reinforcement learning with deep energy-based policies. ICML.
[2] Haarnoja et al. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ICML.

6. Hierarchical Discretization Bias and Error Analysis with Suboptimal Data

We conducted an error analysis in a simplified 2D environment, in Fig. 2 & 3 and Tab. 2 in Supplementary Material.

The results demonstrate that both dimensional conditioning and hierarchical coarse-to-fine (CF) discretization significantly reduce Q-value prediction bias, further highlighting the necessity of our action discretization strategy.

7. Generalization to Unseen Tasks or Out-of-Distribution Environments

We agree that analyzing ARSQ's generalization to unseen tasks or out-of-distribution environments is highly relevant. However, such analysis is beyond our current scope. We acknowledge this as an important future research direction.

We thank the reviewer again for their valuable feedback, which significantly improves the clarity and rigor of our paper.

We sincerely thank the reviewer for their detailed and constructive feedback. We appreciate the positive remarks about the motivation, clarity, originality, and empirical results of our work.

Below, we respond to the reviewer’s concerns point-by-point:

1. Additional Baseline Results on D4RL

We agree with the reviewer that additional baseline comparisons on the well-established D4RL benchmark would strengthen our evaluation. Following the reviewer’s suggestion, we have included results from several offline RL methods (CQL[1], IQL[2], TD3+BC[3], Onestep RL[4], RvS-R[5]) and offline imitation methods (Filtered BC[5], Decision Transformer(DT)[6], DWBC[7]).

We have included the results in Table 1 of Supplementary Material. All data are sourced from the respective papers, and we re-evaluate DWBC with extending datasets (marked by "*"). These results are summarized below for convenience:

The above results demonstrate that ARSQ achieves competitive performance with existing offline RL/IL algorithms, further demonstrating its potential as a versatile RL algorithm.

[1] Kumar et al. (2020). Conservative q-learning for offline reinforcement learning. NeurIPS 33.
[2] Kostrikov et al. (2021). Offline reinforcement learning with implicit q-learning. arXiv:2110.06169.
[3] Fujimoto et al. (2021). A minimalist approach to offline reinforcement learning. NeurIPS 34.
[4] Brandfonbrener et al. (2021). Offline rl without off-policy evaluation. NeurIPS 34.
[5] Emmons et al. (2021). Rvs: What is essential for offline rl via supervised learning? arXiv:2112.10751.
[6] Chen et al. (2021). Decision transformer: Reinforcement learning via sequence modeling. NeurIPS 34.
[7] Xu et al. (2022). Discriminator-weighted offline imitation learning from suboptimal demonstrations. ICML.

2. Clarification on Number of Seeds

We apologize for the previous lack of clarity regarding the number of random seeds used. To clarify, all experiments were conducted using three different random seeds.

We will ensure this information is clearly stated in the revised version of the paper.

3. Details on the DrQ-v2+ Baseline

We thank the reviewer for pointing out this omission. DrQ-v2+ is an enhanced variant of DrQ-v2, proposed and open-sourced by Seo et al[8]. It incorporates several key improvements, including a distributional critic, an exploration strategy using small Gaussian noise, and optimized hyperparameters. These enhancements make DrQ-v2+ a more competitive baseline compared to the original DrQ-v2.

We will include a detailed description of the DrQ-v2+ baseline in the revised paper.

[8] Seo et al. Continuous Control with Coarse-to-fine Reinforcement Learning. CoRL.

4. Clarification on Target Value Computation

Specifically, we train two separate value networks and maintain two corresponding target networks, which are updated using exponential moving averages of the online value networks. The value targets are computed as the minimum of the two target network outputs.

We will clarify this in the revised paper.

5. Improved Figure Readability

We thank the reviewer for highlighting the readability issues in some figures. We will improve this in the revised paper.

We greatly appreciate the reviewer's insightful feedback, which has helped us significantly improve the manuscript. We would be happy to incorporate any additional suggestions from the reviewer.