From Static to Dynamic: Leveraging Implicit Behavioral Models to Facilitate Transition in Offline-to-Online Reinforcement Learning

In my view, this paper is rather rudimentary, as the following weakness is obvious:

1. In the abstract, the authors claim that BAQ could reduce Q-value estimation errors and improve overall learning efficiency. However, their proposed method does not have any theoretical evidence to show how much learning efficiency can be improved or how many estimation errors can be reduced. If taking this as an empirical paper, then empirical contributions in and off BAQ should be strong enough, which seems to not be the case in the submission e.g. the results are only averaged by 4 seeds, and the experimental results are limited in MuJoCo.
2. The proposed method BAQ is simple and straightforward. However, the details of why it works are missing, adding more interpretation on both theoretical and experimental would be appreciated.
3. Imprecise Terminology: the bootstrap error does not only mean inaccurate Q-value estimates. Please refer to [1] for some details that the bootstrap error also covers the Bellman backup process, namely error accumulating. Only referring to the misguide but not referring to the bootstrapping process is imprecise.
4. Lack of references: the method [2] is also free from constraining the policy shift and balancing the sample replay based on the estimation of distribution divergence or density ratio. It would be appreciated if the authors could take it into discussion and comparison.
5. The OOD NATURE in Section 4.1 is unnecessary as this problem is not proposed the first time. Some similar interpretations can be found in [2].
6. "Advanced methods such as FamO2O are excluded ...". This is not an excuse for excluding the comparison as all the methods have access to the offline training dataset. If having the offline training datasets, the accessibility to the offline training phase is natural.
7. In Section 5.3, it seems that the BC-divergence priority sampling is of little use as the relative performance improvement is limited, e.g. only 1 or 2.
8. The experiments are averaged by multiple seeds, hence please report the deviation in Table 2, Figure 2 and Figure 3.
9. In Fig. 2, it seems that all the methods suffer a big performance drop during the online fine-tuning stage but why? As far as I know, the online fine-tuning stage should improve the asymptotic performance.

[1] Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. Aviral Kumar. et al [2] Actor-Critic Alignment for Offline-to-Online Reinforcement Learning. Zishun Yu & Xinhua Zhang

Misattribution and citation errors

• L32-33 discusses about offline RL, but both of the cited works for this discussion are not focusing purely offline RL. (Zhang & Zanette, 2024) focuses on the setting where additional online data can be collected (informed by the offline data) and (Xie et al, 2021) focuses on offline-to-online RL.
• L34-35 discusses about benchmarks, but two of three papers cited did not first introduce the benchmarks they tested on (e.g., Zhao et al. (2023) and Rafailov et al. (2023). These papers are also about offline-to-online RL which is not suitable in the context of the offline RL discussion.

Poorly justified claims

• L239-240 “As depicted in …, demonstrating its effectiveness in stabilizing the learning process during the offline-to-online transition and thus mitigating the OOD issue” — The Figure referenced shows nothing about how the learning process is being stabilized. The only thing that the figure shows is the difference (MSE) between the model prediction and the offline data is reduced when the model is a BC model. This is also not surprising given that the BC model is directly being optimized to minimize the prediction error on the offline data.

Missing important baselines comparisons. While the proposed technique helps existing offline RL methods to fine-tune better, the authors seem to neglect a few important baselines in the same problem settings. For example,

• RLPD (Ball et al., 2023) uses online RL directly and learn from scratch using a combination of offline data and online data.
• CalQL (Nakamoto et al, 2024) improves CQL so that it fine-tunes better. For offline RL methods, ReBRAC [1] also experimented with offline to online fine-tuning with strong performance.

Some results are missing confidence intervals.

• Table 2 and Figure 2 do not have confidence intervals, which make it hard to tell how significant the results are.

[1] Tarasov, Denis, et al. "Revisiting the minimalist approach to offline reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).

• A significant drawback of the paper is the omission of important baselines such as RLPD [1] and BOORL [2], among many others. Including these baselines would have provided a more comprehensive comparison and a better understanding of how BAQ compares against state-of-the-art methods.

• The paper has some issues in writing and formatting. For example, there is a misuse between \citet and \citep starting from Line 348.

• The proposed method only applies to CQL and IQL. It is unclear how the proposed method applies to general offline RL methods.

References

[1] Ball, Philip J., et al. "Efficient online reinforcement learning with offline data." International Conference on Machine Learning. PMLR, 2023.

[2] Hu, Hao, et al. "Bayesian Design Principles for Offline-to-Online Reinforcement Learning." arXiv preprint arXiv:2405.20984 (2024).

The logic of this paper is difficult to follow, and it is unclear how BAQ addresses the offline-to-online RL issue.

From the experimental results, BAQ combined with CQL or IQL does not outperform the original CQL. While the authors emphasize the total normalized score, this metric is less meaningful since CQL outperforms BAQ in 7 out of 9 tasks. Many claims in the paper lack sufficient evidence and clarity. For example, the statement in Line 375, "Overall, our approach not only surpasses all baseline methods in total scores for both IQL and CQL but also demonstrates strong stability and generalization across diverse tasks," and the claim in Line 398, "In contrast, our method has a robust performance from the beginning without relying on such additional mechanisms," appear overstated or even incorrect based on the experimental results.

Some implementation details are unclear. For instance, the value of $\alpha$ in Eq. (10) is not specified in the main text or appendix (though it seems to be $\alpha = 0.6$ based on the code). Additionally, details regarding the online fine-tuning of CQL and IQL are missing.

The hyperparameters $k_q$ and $k_\rho$ are manually set for each task, but there is no discussion on whether these hyperparameters are sensitive or how they impact the results.

Minor:

Repetition: Line 252, "both offline and online interactions"

The evaluations are weak

1. The algorithm is evaluated only on MuJoCo environments from D4RL. The authors should also evaluate the algorithm on harder domains such as AntMaze, Franka-Kitchen and Adroit.

2. Even on MuJoCo, the reported performances are those after 30,000 online fine-tuning steps, and not after running the online fine-tuning to completion. As such, the numbers reported here seem significantly lower than those of running existing fine-tuning methods like Cal-QL to completion. For eg. on the medium environments, Cal-QL reaches normalized score of ~ 100 after online fine-tuning (see Table 5 - https://arxiv.org/pdf/2303.05479) whereas the numbers reported in Table 1 and Table 2 for BAQ are significantly lower.

This makes it hard to compare BAQ against existing online fine-tuning algorithms.

3. Comparisons against fine-tuning methods like Cal-QL are missing, and it is unclear how BAQ helps with the distribution shift problem in offline-to-online RL.