Uni-RL: Unifying Online and Offline RL via Implicit Value Regularization

We thank the reviewer for their thoughful review of our work. We address the reviewers questions and concerns below:

The related-work section cites recent algorithms such as Uni-O4 and DICE-based methods, yet Figure 5 benchmarks Uni-IVR only against Cal-QL, AWAC, RLPD, and IQL—methods published no later than 2023. To the best of my knowledge, BOORL [3], WSRL [4], and O2SAC [5] are SOTA algorithms. Could you clarify whether Uni-IVR is expected to deliver superior sample efficiency, early-phase stability, and final performance when evaluated against these baselines?.

To further compare the performance of Uni-IVR, we evaluate it against BOORL, WSRL, and O2SAC on AntMaze tasks, as BOORL and O2SAC did not conduct experiments on the Adroit datasets. Uni-IVR continues to demonstrate strong performance compared to these baselines.

Tables 4 and 5 list different hyperparameters for the Antmaze tasks: alpha = 0.1 or 2.0 in the pure-offline experiments, but alpha=0.5 in the offline-to-online setting. Does this difference imply that a policy trained with the offline hyperparameter cannot be carried over directly into the online fine-tuning phase without re-tuning alpha?

There is no guarantee that the best hyperparameters for the offline setting will be optimal for the off-to-online setting. Empirically, we find that using a less optimistic $\alpha$ leads to more stable value learning, which in turn benefits online fine-tuning.

Lemma 1 appears to reproduce a result that was already proved in the original IVR paper (Xu et al., 2023). Please add an explicit citation and indicate the exact place in that paper where the derivation can be found.

Thank you for pointing that out. We would like to clarify that Lemma 1 is a result from the IVR paper and is not claimed as a novel contribution in our work (as indicated by labeling it as a lemma and explicitly stating it is from IVR). It is included solely for completeness and consistency in the presentation. The detailed derivation is provided in Appendix C in the IVR paper.

In the Limitations section, the authors state only that Uni-IVR is restricted to the model-free setting. A more comprehensive ablation study is warranted. For instance, they could remove InPG (replacing it with vanilla BC or MaxQ + BC) and demonstrate robustness across a range of hyperparameter settings.

Due to space limit, please see the response to Reviewer HJMu for a comprehensive ablation study including different $\lambda$, using InPG vs weighted BC vs MaxQ, using Uni-IVR vs IVR.

Uni-IVR is closely related to IVR (Xu et al., 2023). The main difference in offline RL settings is the addition of the InPG method. In Table 1, I think IVR's performance is comparable to that of Uni-IVR (Gaussian). Additionally, baselines (BCQ, TD3+BC, CQL, and IQL) are not SOTA algorithms in offline RL. To make the empirical evaluation more convincing, Uni-IVR should be compared with more recent methods, such as ReBRAC [1] and QDQ [2].

We compare Uni-IVR with ReBRAC and QDQ, and find that Uni-IVR achieves comparable performance to these advanced methods. However, we would like to emphasize that the key contribution of our work is not to outperform methods specifically designed for the offline setting. Rather, our goal is to propose a general and unified framework applicable across a wide range of RL settings, including but not limited to the offline scenario.

Lemma 1 states that the optimal value functions Q* and V* can be solved by Equations 3 and 4, respectively. In Equation 5, however, the authors replace the maximisation with an expectation over actions sampled from \hat{\pi}. Does this modified operator still converge to V*?

Note that Lemma 1 shows that updating according to Equations (3) and (4) yields the optimal policy $\pi^{\ast}$ and value function $V^{\ast}$ that maximize the expected return under the constraint induced by $\mu$. In the online setting, we replace $\mu$ with a weighted mixture of $\pi^{\ast}$ and $\mu$, which ensures continual improvement of the reference policy. This allows $\pi^{\ast}$ to progressively improve toward the global optimum.

In Figure 5, learning curves are plotted after 250k offline pertaining. Yet this work claims that Uni-IVR (i) alleviates Q-value overestimation and (ii) avoids the performance drop typically seen at the start of offline-to-online finetuning. To substantiate these points, wouldn’t it be essential to plot the offline performance at the end of pre-training, alongside the subsequent finetuning curves?

The offline performance at the end of pre-training is reflected in the initial point of the online learning curve (which is non-zero). Our method improves upon this initial performance without dropping down during online fine-tuning, supporting claim (2). Additionally, the high sample efficiency observed during online learning further supports claim (1).

Please let us know if any further questions remain. We hope the reviewer can reassess our work with these clarifications.

Thank you for your detailed response. I appreciate the clarification regarding the goal of Uni-IVR. However, I still have several concerns.

While the authors claim that Uni-IVR outperforms strong baselines in both offline and offline-to-online settings, the baselines referenced (TD3+BC (2021), IQL (2021), Diffusion-QL (2023), and IDQL (2023)) have since been surpassed by several more recent SOTA algorithms. As such, I believe it is essential to compare Uni-IVR’s performance against a broader and more up-to-date set of baselines to validate its empirical contributions.

Regarding the authors’ comment that ''it is inaccurate to conclude that WSRL demonstrates better performance than Uni-IVR on the large dataset as a whol'': if the authors wished to make this argument, they should have included the WSRL scores for antmaze-medium-play and antmaze-medium-diverse in the first rebuttal. As a reviewer, I made a reasonable inference from the available results of the large dataset that WSRL likely outperforms Uni-IVR on the medium datasets. In the first rebuttal, the authors said that ''Uni-IVR continues to demonstrate strong performance compared to these baselines.'' As the authors themselves argue, it is inaccurate to conclude that Uni-IVR demonstrates better overall performance than other baselines without including the performance of WSRL on the medium dataset. Without those values, it is unclear why they were omitted, and the concern remains unaddressed.

Thank you for your feedback.

Regarding your first concern:

We agree with the reviewer and do had included additional comparison results with more advanced methods in the rebuttal. As shown in the updated results, Uni-IVR maintains competitive performance. Note that Uni-IVR also consistently outperforms strong baselines in the online setting, which we believe represents a significant empirical contribution that should not be overlooked.

Regarding your second concern:

There may be some misunderstanding, our point is that one cannot conclude WSRL is superior to Uni-IVR when WSRL is better than Uni-IVR on one dataset while underperforming on the other one. We clarify that we only report results that are available in the original publications of the respective methods. For example, WSRL does not include results on the medium datasets, so their absence is not due to selective reporting. We are not cherry-picking results that favor Uni-IVR. Taken together, both the original empirical results and the newly added comparisons with several stronger baselines are fairly enough to demonstrate the effectiveness of Uni-IVR.

We thank the reviewer for their thoughful review of our work. We address the reviewers questions and concerns below:

Given that IVR utilizes both KL divergence and chi-divergence in separate algorithmic formulations, it would be helpful to clarify which divergence is used in Uni-IVR.

Thanks for pointing out that, we use $\chi^2$-divergence throught the paper because SQL, which uses $\chi^2$-divergence, produces better and more stable results than EQL, which uses KL divergence. We will add this clarification in the revised paper.

Could you provide ablation results for some of the key hyperparameters in the algorithm?

Due to space limit, please see the response to Reviewer HJMu for a comprehensive ablation study including different $\lambda$, using InPG vs weighted BC vs MaxQ, using Uni-IVR vs IVR.

What specific method of policy mixing is used in the iterative refinement process, parameter mixing or probability mixing? If it is parameter mixing, could you clarify in what way this contributes to the novelty of the method?

We are using parameter mixing but do not claim the specific method of policy mixing as a contribution. Instead, the key contribution of this paper is identifying the ineffectiveness of online IVR and proposing a lightweight solution that that not only addresses this issue but also generalize to other settings. This solution further serves as a general framework applicable across various RL settings, which is a non-trivial extension from the original offline IVR that no previous work has achieved.

The use of 500K steps is an interesting design choice. However, it raises the question of whether training continues to improve beyond that point. Could you provide additional results or learning curves to illustrate this?

Note that the key metric for evaluating online RL algorithms is the sample efficiency, i.e., achieve higher performance given limited interaction steps. We addiontially provide the results of training 1000K steps below to further show the performance. It can be seen that the improvement is limited with additional 500k steps, indicating that the learning curve is almost converged (we will add the learning curve to the revised paper).

Would it be possible to include results using exactly the same hyperparameter settings as in the original IVR paper for the locomotion tasks in offline RL? This could help isolate the contribution of the proposed modifications.

Note that we didn't change the hyperparameter settings of IVT when training Uni-IVR. The full comparision of Uni-IVR and IVR is shown below to justify the benefit of using InPG policy extraction.

Please let us know if any further questions remain. We hope the reviewer can reassess our work with these clarifications.

We thank the reviewer for their time, effort, and constructive comments on our paper. We believe there may be some misunderstandings, which we address in detail below:

The core value learning objective is a direct and incremental extension of the IVR framework. Moreover, the paper only compare the improvement in Value learning objective in one task with no name (Figure 2).

Note that although the value learning objective appears similar to the IVR objective, extending IVR to the online setting is non-trivial. The key contribution of this paper lies in identifying the ineffectiveness of online IVR and proposing a lightweight solution that that not only addresses this issue but also generalize to other settings. This fix naturally leads to a unified framework applicable across various RL settings.

Figure 2 presents results on Hopper-v3. We apologize for omitting this information in the paper and will add it in the revised version. The observed behavior in Hopper-v3 is representative of all online environments; thus, we selected it as an illustrative example. For completeness, we provide the full performance comparison between Uni-IVR and IVR below. Note that IVR here uses the same policy extraction method (InPG) as Uni-IVR to ensure a fair comparison.

The novel policy extraction method InPG lacks a comprehensive evaluation, only demonstrated primarily on a single environment (Figure 3), which raises concerns about the generalizability of the results. Moreover, the absence of supplementary source code makes it difficult to check the effective of this novel method.

We respectfully disagree with the reviewer. As mentioned, Figure 3 is intended solely as an illustrative example to convey the motivation and intuition behind using InPG. The actual impact of InPG can be inferred from Table 1 in the paper by comparing Uni-IVR and IVR in the offline setting, where the value learning components are identical. For clarity, we include the relevant results here for easier reference.

In the online and offline-to-online setting, to validate the contribution of InPG, we provide the comparision of using InPG vs MaxQ and weighted BC as follows.

Lack of description about how to build and train w in the implementation. Moreover, computational cost analysis for the InPG module are missing.

Thank you for pointing that out. The w network is initialized as a value function and trained using the loss defined in Equation (7). The learning rate and network architecture are provided in the appendix. For clarity, we include a code snippet below demonstrating the training of the w network, which only requires less than 20 lines beyond the original IVR implementation.

# grad of q to pi
pi, _ = self.actor.sample(observations, deterministic=False)
q_pi = self.q(observations, pi)
q_grad_pi = torch.autograd.grad(
    q_pi.mean(),
    self.actor.parameters(),
    retain_graph=True
)
q_grad_pi_flat = torch.cat([grad.view(-1) for grad in q_grad_pi], dim=0)
w = self.w(observations, actions)
bc_losses = self.actor.evaluate(observations, actions)
policy_loss = torch.mean(w * bc_losses)
# grad of wbc to pi
wbc_grad_pi = torch.autograd.grad(
    policy_loss,
    self.actor.parameters(),
    retain_graph=True,
    create_graph=True,
)
wbc_grad_pi_flat = torch.cat([grad.view(-1) for grad in wbc_grad_pi], dim=0)
# w loss
w_loss = F.mse_loss(q_grad_pi_flat, wbc_grad_pi_flat)

The add of the InPG module doesn't add too much computational cost, as shown below.

The paper proposes two alternative objectives that enable the use of Flow-Matching and Diffusion models instead of traditional Gaussian policies. However, if I understand correctly, these objectives treat all actions from the dataset uniformly and do not incorporate information from the Q and V functions.

There seems to be a misunderstanding here. Note that our proposed two alternative objectives are intended to replace the log-likelihood maximization objective used for training Gaussian policies, as mentioned in lines 240–241. In other words, we are replacing the standard objective $\max E_D(\log \pi(a|s))$ with our alternative formulations. However, the learning of the w function remains essential for performing weighted behavior cloning. We will clarify this more explicitly in the revised version of the paper.

in Figure 2, the authors provide Q-curves to illustrate higher performance; however, Q-learning methods (IQ-learn [1] for example) can produce high Q-values without corresponding performance gains. There are no explanation for these Q value results.

Note that we provide the corresponding evaluation scores in the first row of Figure 2, and the Q-value is included only as an additional metric for reference. As shown in Figure 2, the actual performance aligns well with the Q-values. In fact, Q-value overestimation is a much more serious issue in the offline setting. In the online setting, the Q-value can still serve as a useful performance indicator—though it may exhibit slight overestimation due to TD learning, it is not nearly as severe as in the offline case.

In line 298-299, author stated that “DICE-based methods only work for the offline setting”. Can you justify the reason why they can not be applied by the reference policy?

Thanks for pointing out that, the reason is that if we apply DICE-based methods to the online setting, sampling from the stationary distribution $d^{\mu}(s, a)$ of the reference policy is needed.

More specifically, this is the dice learning objective:

$$\pi^{*} = \arg \max_{\pi} E_{(s, a) \sim d^{\pi}}[r(s, a)] - \alpha D_f(d^{\pi}||d^{\mu}).$$

Solve by duality we get

$$\min_{V} (1 - \gamma)E_{s \sim d_{0}}\left[ V(s) \right] + \alpha E_{(s, a) \sim d^{\mu}} \big[ f_{dual}\big(\left[T_r V(s, a) - V(s)\right] / \alpha\big) \big],$$

Samples from $d^{\mu}(s, a)$ are intractable in the online setting where $\mu$ is the previous policy.

This raises questions about the sensitivity of the method to this hyperparameter. Could you provide an ablation study that shows how performance varies with λ in each setting?

Due to space limit, please see the response to Reviewer kppM for ablation study of different $\lambda$.

To support the claim of scalability, could you please add the results for a Flow-matching policy to Table 1, as was done for the Diffusion policy?

We believe our clarifications to your responses naturally answered your questions listed. Please let us know if any further questions remain. We hope the reviewer can reassess our work with these clarifications.

We thank the reviewer for the effort engaged in the review phase and the constructive comments. Regarding the concerns, we provide the detailed responses separately as follows.

The algorithms compared in the online RL experiments are outdated (the latest is from 2018); it should consider comparing with more advanced algorithms.

Thank you for your suggestions. The main contribution of our paper is the proposal of a unified framework for off-policy RL. To highlight Uni-IVR’s scalability across different settings, we select representative baselines from online, offline, and offline-to-online RL. To further demonstrate Uni-IVR’s competitiveness in the online setting, we compare it with BAC [1], which, to the best of our knowledge, is the strongest model-free, off-policy online RL algorithm in recent years. Using the official implementation of BAC, our results show that Uni-IVR consistently outperforms it across all tasks.

The "iteratively refined behavior policy" used to address the over-conservatism issue in online IVR is a soft update (Equation 5). While more lightweight than dataset filtering, the choice of hyperparameter λ might affect performance.

Thank you for raising this point. We provide full ablation results for different values of $\lambda$ across all environments (with * marking the $\lambda$ values we selected and their corresponding results reported in the paper). The optimal range of $\lambda$ varies across settings, as the desired contribution of the previous policy differs. In the online setting, a smaller $\lambda$ is preferred because earlier policies are generally suboptimal. In contrast, in the offline-to-online setting, a larger $\lambda$ is beneficial due to the availability of a strong initial policy.

Nonetheless, within the same setting, the performance is relatively robust to the choice of $\lambda$, as shown in the table below. We note that we did not explore a broader range of values, which might yield even better results.

The paper does not provide detailed information on the computational resources used for the experiments.

Thanks for pointing out that, we use NVIDIA A40 for all experiments.

Although the paper considers various learning scenarios such as online, offline, and offline-to-online, it is limited to model-free RL. Verification of the proposed method in other settings, such as model-based RL, could further demonstrate its generality.

Thanks for pointing out that, the value and policy learning in Uni-IVR can be naturally extended to dyna-style model-based RL algorithms like MBPO [2] where the standard Q-value update rule is replaced by ours. We leave it as future work.

[1] Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic, Ji et al, ICML 2024.

[2] When to Trust Your Model: Model-Based Policy Optimization, Janner et al, NeurIPS 2019.