Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL

Thanks for your thorough review and positive recognition of our work. We are glad that you consider our work "well-supported, simple yet effective, logically rigorous and coherently structured." We are glad to answer all your questions.

Q1:More comparative results

A1: We compare our approach with two strong baselines, MCQ and MISA. MCQ is a CQL variant achieving mild conservatism, while MISA unifies CQL and TD3+BC via mutual information. The results below demonstrates our competitive performance.

Since we do not employ any specialized techniques in the online phase, we compare our approach with some other advanced unified methods across O2O phases, as shown below. Our method still leads in performance.

Q2: Experiments on hyperparameter sensitivity

A2: We conduct hyperparameter sensitivity experiments on $n_{end}$​, as shown below. The results demonstrate the robustness of our method, as the early stopping mechanism in the update of $n$ ensures that $n$ increases to an appropriate value and then stabilizes.

Q3:The meaning of $\mu$ in Eq.(6)

A3: Apologies for the omission. $\mu$ represents the mean of the stochastic policy (i.e., the learned policy), and $\sigma$ denotes its standard deviation.

Q4:Explanation regarding threshold

A4: The threshold $n$ in Eq.(5) is initially set to a small value (1.0 in our experiments), which results in a large $C_n(s)$. At this stage, the loss value of Eq.(5) is negative, leading the regularization coefficient to increase and encouraging the learned policy to closely imitate dataset actions during the early learning phase. As $n$ gradually increases, the loss value of Eq.(5) eventually becomes positive, indicating that the learned policy has sufficiently approximated the high-value dataset actions in most states. In this trust region, certain state constraints can be relaxed while ensuring that a substantial portion of constraints remain enforced.

Q5:Fine-tuning coefficients online

A5: Since a single online step cannot directly determine action value, directly fine-tuning regularization online is difficult.

Q6:Computational cost

A6: Our method introduces a lightweight network with minimal overhead. Below is the training time on a 2080 GPU, excluding IQL-style pretraining, showing a slight increase in time cost.

Q6:Comparison with FamO2O

A6: Both methods adjust state-wise regularization, but key differences include:

1. Regularization Parameterization: FamO2O employs a hierarchical architecture, using an intermediate variable to learn policies under different constraints. In contrast, our method directly parameterizes the regularization coefficients with a neural network, explicitly modeling constraint strength across different states.
2. Policy Representation: FamO2O learns a family of policies, where the optimal policy is selected by maximizing the Q-value with respect to the intermediate variable. This requires passing through two networks to obtain the final policy. In contrast, our method updates the constraints based on the relationship between the learned policy and dataset actions, directly learning the final policy without additional selection steps.
3. Interpretability: The intermediate variable in FamO2O lacks interpretability, meaning its value does not directly indicate constraint strength. In our approach, the parameterized regularization coefficients directly reflect the constraint strength, providing better interpretability.

Thank you for your insightful review again. We hope we have resolved your concerns. We are always willing to answer any of your further concerns.

Thanks for your insightful review and positive recognition of our paper. We appreciate the questions you raised and are committed to delivering a comprehensive response to address the issues.

Q1: Questions about claims

A1: Our main claim is derived from Proposition 3.1 and can be succinctly stated as balancing pessimism and optimism by constraining the divergence between the likelihood of high-quality dataset actions under the learned policy and a predefined threshold. This motivates the development of our state-adaptive mechanism, which adjusts regularization dynamically as divergence varies across different actions. Since the learned policy's distribution is directly used to update the regularization coefficients, we propose distinct methods tailored for stochastic and deterministic policies. However, the core insight remains the same for both, as detailed in the derivation of Eq. (12) in Appendix C. Additionally, due to the variability in dataset distributions and differences in reward scales, some manual hyperparameter tuning is unavoidable.

Q2: Questions about methods

A2: 1.) For the use of Proposition 3.1, as discussed in A1, it reveals that the regularizer in CQL inherently increases the likelihood of dataset actions under the learned policy. Building on this insight, we formulated Eq. (5) to strike a balance between pessimism and optimism. 2.) For $G_T$ and the schedule used to update $n$, a pre-defined metric is essential to select high-value actions. In contrast, the schedule is not strictly necessary. If training time is sufficient, $n$ can be updated based on some metrics, such as the loss. Such pre-defined hyperparameters are common in RL, and the adaptivity in our approach is primarily reflected in the state-dependent regularization coefficients across varying states. 3.) For the different objectives in our approach, as shown in Fig. 2, a sub-dataset of high-return trajectories may not cover most of the offline dataset, leading to over-optimism in uncovered regions. To address this, we pre-train IQL critics (independent of the learned policy) to filter a reliable high-value sub-dataset that covers most of the offline dataset. 4.) Lastly, for the "speculative lanugage", our claims are empirically supported by the ablation study and the experimental results presented in Table 4.

Q3: Weakness 1 about the separate learnable parametrizations

A3: As discussed in A1, Eq. (5) applies to the stochastic policy, while Eq. (12) is for the deterministic policy (since the deterministic policy lacks an explicit policy distribution); however, both share the same key insight.

Q4: Weakness 2 about construction of the paper

A4: We introduced the technique of IQL to address the sub-dataset selection problem, which is independent of CQL. Regarding the paper structure, our goal is to make adaptive adjustments to the constraints across different states in order to maximize the potential benefits of Bellman updates. We begin by proposing a state-level regularization updating mechanism in Section 3.1, stemmed from Proposition 3.1. To automatically select the appropriate threshold for this mechanism, we introduce a method based on distributed perception in Section 3.2. To further ensure that constraints are valid and to fully harness the benifit of the RL form, we propose selective regularization in Section 3.3. Additionally, we extend the method to deterministic policy algorithms that lack explicit policy distributions. We will provide a summary at the end of each sub-section and emphasize that Eq. (5) and Eq. (12) are fundamentally the same but apply to different policy formulations. Thank you for your constructive suggestions.

Q5: The definition of $\mu$ and $\sigma$ in Eq.(6)

A5: $\mu$ is the mean of a stochastic policy and $\sigma$ is the standard deviation.

Q6: Details of Fig.1

A6: We trained the policy both with and without selective regularization, and then used the trained policy along with all data to compute the values presented in Fig. 1. In Fig. 1, we highlight the advantage of selective regularization: it helps to avoid the imitation of low-value actions.

Q7: Selection of $G_T$

A7: The selection of $G_T$ depends on the dataset, similar to previous works like DT and RvS, as reward scales vary significantly. Since the dataset's reward information is available, it's easy to determine a proper value.

Q8: Two-phase training paradigm

A8: Yes, we first pre-select the high-value sub-dataset before initiating offline training.

Q9: $n$ used in Eq.(12)

A9: Yes, details are in Appendix C.

Q10: The region of policy learning

A10: The policy is updated across all states, but regularization related to the policy (if applied) is restricted to the sub-dataset.

Thanks for your review again. We hope our response sufficiently addresses your concerns, and we remain available for any further clarifications.

Thanks you for the high praise and the comprehensive review of our paper. We appreciate the questions you raised and are committed to delivering a comprehensive response to address the issues.

Q1: Lack of statistical significance

A1: To ensure statistical rigor, we have now included 95% confidence intervals (CIs) for all domain tasks in Tables 1 and 2, consistent with prior works. The revised tables are as follows:

Table 1 (offline performance)

Table 2 (offline-to-online performance)

These results confirm that our method achieves statistically significant performance improvements across different algorithms and task domains.

Q2: About hyperparameters

A2: The hyperparameters used in our implementation are provided in Appendix D. For fair comparisons, we adopt the same hyperparameter configurations for CQL and TD3+BC as in the CORL benchmark [1]. Additionally, for the distribution-aware threshold, we set $n_{start}$ to 1 and $n_{end}$ primarily to 3 (1.5 for expert datasets to better mimic high-quality actions). We will include a detailed listing of these parameters in the revised version for greater clarity.

We also deeply appreciate your constructive suggestions, including pointing out typographical errors, which we have now corrected. Thank you again for your thoughtful review and for helping us improve our work. We hope our responses sufficiently address your concerns, and we remain open to any further questions or clarifications.

[1] Tarasov, Denis, Alexander, Nikulin, Dmitry, Akimov, Vladislav, Kurenkov, Sergey, Kolesnikov. "CORL: Research-oriented deep offline reinforcement learning library". Advances in Neural Information Processing Systems 36. (2023): 30997–31020.

Thank you for your thorough review and positive recognition of our work. We appreciate your thoughtful feedback and are pleased that you found our paper to be "well-written, methodologically sound, and of practical value." We also appreciate the questions you raised and are committed to delivering a comprehensive response to address your concerns in detail.

Q1: Code provision

A1: We have carefully verified that the code has been included in the supplementary material.

Q2: The collection of offline data

A2: Our approach is evaluated on the publicly available D4RL benchmark [1], which provides diverse offline datasets. These datasets are collected using various strategies, including hand-designed controllers, human demonstrators, multi-task agents performing different tasks in the same environment, and policy mixtures. Given that prior works on offline RL have typically used these datasets without explicitly describing their collection process, we followed the same convention. Further details regarding dataset collection can be found in the original D4RL paper.

Q3: Relatively incremental contribution and lack of theoretical contribution

A3: Existing approaches employ global regularization to constrain policy updates across the entire dataset. However, this can lead to excessive pessimism, limiting the agent's ability to harness the full benefits of Bellman updates. Building on the core insight of pessimism in offline RL (Proposition 3.1), we propose a selective state-adaptive regularization mechanism that dynamically adjusts regularization coefficients based on the likelihood of high-quality dataset actions under the learned policy. This allows the agent to assess Bellman update confidence at the state level. Notably, the update mechanism is algorithm-agnostic, making it a broadly applicable enhancement that can be integrated with various offline RL algorithms.Another key contribution of our approach is demonstrating that state-adaptive regularization mitigates the excessive pessimism of fixed regularization, enabling RL agents to better exploit Bellman updates. Given the heterogeneous quality distribution in offline datasets, this state-wise adaptation offers a promising direction for improving offline RL performance. Although our paper does not provide a rigorous theoretical guarantee for the proposed method, given that the regularization encourages the learned policy to mimic dataset actions, the performance lower bound of our method can be approximately ensured by the behavior policy. Furthermore, as our method solely modifies the regularization coefficient, it does not compromise the convergence guarantees of base algorithms such as CQL under standard assumptions.

Thank you for your review again. We hope our response sufficiently addresses your concerns, and we remain available for any further clarifications.

[1] Fu, Justin, Aviral, Kumar, Ofir, Nachum, George, Tucker, Sergey, Levine. "D4rl: Datasets for deep data-driven reinforcement learning". arXiv preprint arXiv:2004.07219. (2020).