The paper deals with the combination of model-free and model-based approaches in online reinforcement learning. A combination of both approaches is presented and tested on the basis of several benchmarks.

This paper proposes a novel reinforcement learning (RL) algorithm, named Unified RL (URL), which introduces a way to switch between policies learned via model-free and model-based methods. To achieve this goal, the authors propose equivalent policy sets (EPS), which is the set of all policies for which there does not exist a provably better policy in terms of Bayesian return (expected return over the learned dynamics model parameter). In practical terms, URL works by learning a policy with a model-free algorithm, and a second policy using model-generated transitions. Then, the returns of both policies are evaluated via Monte Carlo estimation using the learned dynamics model with different dropout masks. The model-based policy is selected only if its return is greater than the return of the model-free policy when evaluated in all models/dropout masks. URL is evaluated in robotics tasks and compared with state-of-the-art model-free and model-based RL algorithms.

Summary

This work first introduces the concept of EPS (equivalent policy sets), which is used to compare whether a given candidate policy is provably suboptimal compared to a reference policy. Then, an algorithm is proposed that approximates this condition for two policies as input: (i) a candidate policy learned in a model-free soft-actor-critic (SAC) algorithm, and (ii) a reference policy learned in a model-based RL algorithm (MBRL baseline). If the condition holds, then the candidate policy is provably sub-optimal to the reference policy and the algorithm decides to use the reference policy for interacting with the environment. Otherwise, the candidate policy is utilized for this purpose.

Empirical analysis is performed to compare the proposed strategy to prior model-based and model-free methods as well as ablations of the proposed method in the form of an MBRL and SAC baseline which constitute the model-based and model-free algorithms used in the proposed method respectively. The analysis is focused on highlighting cases where the algorithm is able to keep the best performance of its individual components (MBRL and SAC), especially in cases where either component alone is expected to fail.

The paper studies the combination of model-based RL and model-free RL. The authors propose an approach that makes use of the concept of equivalent policy set (EPS) that, based on a Bayesian formulation, represents the set of policies that are not provably Bayes-suboptimal according to the current data and prior. The algorithmic contribution, Unified RL, switches between model-free and model-based RL according to the value of a variational lower bound which is evaluated from the prior and the model-based and model-free policies. The paper provides experimental validation on the set of Mujoco environments and a validation for testing the robustness to misalignment.

We have added a revised manuscript that incorporates reviewers' feedback. We additionally included a diff file as supplementary material that highlights the differences between the original and revised manuscript. We would once again like to thank the reviewers for their constructive feedback.

We thank the reviewer for their insightful feedback. Indeed a discussion of limitations is important. We have added a section on limitations to the manuscript where we discuss the following:

• Currently, our approach does not address exploration. In both the theory and implementation, we essentially assume that the policy that achieves the highest return is also the policy that is best for collecting more data. This is a common assumption in reinforcement learning; however, in certain cases it may be beneficial to execute a policy that achieves lower reward but provides more information about how to best to increase reward in the future. To address this issue, we could introduce an information-gain bonus into eq. 3 that preferences the agent towards policies for which our model has a high degree of epistemic uncertainty.
• Unified RL has the drawback that it maintains two separate policies (one model-free and one model-based), meaning that at least one of these policies must perform some amount of off-policy learning. This limits the set of model-free RL algorithms we can use to off-policy algorithms, such as SAC. Furthermore, empirically SAC has been shown to perform poorly off-policy, necessitating modifications [2]. This could be addressed in our approach by instead maintaining only one policy, which we update by performing a constrained optimization of a model-free RL objective. Here, the constraint would force the resulting policy to lie within the EPS.

To answer your questions:

1. We did not consider any stochastic MDPs. Are there any that you suggest?
2. Experiments are each repeated 5 times with distinct random seeds.
3. Uncertainties in table 1 are 95% confidence intervals (1.96*standard_error)

We have also addressed your comments in the revised manuscript. Thank you for pointing out this highly relevant related work [1]. We have added this reference to our manuscript. Indeed the approach in [1] is very similar to the MBRL component of our algorithm, with the primary differences being that we use a slightly different Bayesian neural network formulation and use KL divergence as opposed to alpha divergence to fit our approximate posterior.

[2] Ball, Philip J., et al. "Efficient online reinforcement learning with offline data." arXiv preprint arXiv:2302.02948 (2023).

To address your questions:

• Thank you for pointing out this inaccurate wording when we referred to the EPS as the smallest possible set of policies that are not provably suboptimal. It would be more accurate to say either that the EPS is the set of all policies that are not provably Bayes-suboptimal using a particular model, or that the EPS is the smallest set of policies pi for which a new policy pi' is not guaranteed to improve over pi. In this latter description, the word smallest appears because we perform an optimization over pi’ to try to rule out as many policies pi as possible.
• When we said that “in environments where either model-based or model-free RL strictly dominates, Unified RL matches or outperforms the better algorithm,” we meant the better performing algorithm between the model-based and model-free components of Unified RL. We have clarified this in the manuscript.
• Results for ALM were achieved by running the code released by the authors with the recommended hyperparameters for each environment. Differences between results reported by the authors and our results could be due to the fact that we use shorter episode lengths (100-200 timesteps vs. 1000 timesteps), which might hinder the Q learning step of ALM. We have clarified this in the manuscript.
• We do not know with certainty why Unified RL slightly outperforms SAC in the environment with distractors. One possible explanation is that Unified RL is able to gain some small benefit from using the model, perhaps because it has been trained on data collected from a higher performing policy and may therefore be more task-aligned, while the model used by MBRL alone never trains on data from a high-performing policy.
• What we meant by “rather than using the model to approximate a single optimal policy, we maintain a set of policies that may be optimal, which is then refined by model-free RL, thereby avoiding over-reliance on potentially inaccurate models”, is that we do not simply use the policy that is optimal according to our model. Instead, we implicitly maintain the set of policies that are not provably suboptimal, and use the model-free policy whenever it lies within this set. Otherwise, we use a policy that is guaranteed to be within the equivalent policy set (the model-based policy). Therefore, unified RL can be viewed as a model-free RL algorithm where the policy is constrained to lie within the equivalent policy set. We have clarified this in the manuscript.

We thank the reviewer for their in-depth and constructive feedback. In response to the weaknesses raised:

1. We would like to push back on the statement that Unified RL only achieves better performance in Hopper and Walker environments. We performed a t-test to determine whether Unified RL significantly improved over both SAC and MBRL in terms of cumulative reward throughout the training period, and found that the improvement over SAC was significant in 4 out of the 6 environments (all but Cartpole and DCLawTurnFixed), and that the improvement over MBRL was significant in 5 out of the 6 environments (all but DCLawTurnFixed) . Additionally, Unified RL was the most consistently high-performing algorithm out of all the algorithms we tested. To clarify this point, we have added the average rank that each algorithm achieved, which we also include below:

Average cumulative reward rank (lower is better) across all environments:

Unified RL (ours): 2.33

SAC: 3.5

MBRL: 4.5

ALM: 7.33

DDPG: 6.83

SVG: 4.83

TD3: 5.83

HL: 4.16

PPO: 5.66

Unified RL achieves the highest average rank across all environments, indicating it is the most consistently high-performing algorithm that we tested.

2. It is true that we do not explicitly represent the set of policies comprising the EPS, as you are correct that this would be intractable. However, it is not necessary to explicitly represent the EPS given that our goal is to constrain the policy of Unified RL to be within the EPS. For this, we require only a condition that allows us to check whether a policy is within the EPS, which we accomplish using the bound in eq. 6. We would therefore argue that we do implicitly consider the full set of policies that are not provably suboptimal. We have modified the manuscript to reflect this point.
3. Thank you for pointing out this inaccurate wording. It would indeed be more accurate to say that the EPS is the set of all policies that are not provably Bayes-suboptimal. Our intent was to convey the fact that the EPS is the smallest set of policies pi for with a new policy, pi’ provably achieves higher Bayesian return (hence the maximization over pi’ in eq. (4)). This is because we wish to exclude as many suboptimal policies from the EPS as possible. However, we acknowledge that our wording in the paper is unclear and we have corrected this issue.
4. The results we report for ALM were obtained by running the authors’ released code with the recommended hyperparameters. It is possible that the deterioration in performance of ALM resulted from the fact that our experiments used shorter episode lengths (100-200 timesteps vs. 1000 timesteps). We have clarified this in the manuscript.

Responses to questions:

1. You are correct, this was a typo. We have corrected this in the manuscript.
2. Yes, we have changed the x axis on the plot from episodes to timesteps.
3. Version 2
4. Yes, for the most apples-to-apples comparison we ran SAC with the same two modifications that were used in Unified RL (i.e., layer normalization and removal of entropy backups), though we found these modifications to not have a major impact on the algorithm’s on-policy performance.
5. Thank you, we appreciate that feedback!

We thank the reviewer for their thoughtful and constructive feedback. In response to the weaknesses raised:

[Choice of function f] An arbitrary concave f can be chosen because our bound is derived using Jensen’s inequality[1], which applies to any concave f. It is true that this choice of f may have a significant impact on the performance of the algorithm. However, this is also a feature of all variational inference-based approaches, with several papers proposing new variants on VI that use different f’s when deriving their evidence lower bounds [2, 3, 4, 5]. Because choice of f is a rich area of research in its own right, we left evaluating the effects of f to future work. We chose to use log for f, as mentioned in the Lower Bound Estimation subsection of Sec. 3.2, because log is the most well-studied choice of f for VI [6,7]. We have clarified this choice of f in the manuscript.

[Choice of p and q] The distribution family from which p and q are chosen is implicitly decided by our choice of Bayesian neural network formulation. As mentioned in Sec. 3.2, we chose to use binary dropout Bayesian neural networks, proposed by [8], due to their simplicity, computational efficiency, and the fact that they have been shown to perform well in an MBRL context [9, 10]. We also chose p and q to be as generic as possible, so that we encode minimal prior knowledge into our models. For the type of BNN that we used, p approximately corresponds to a normal distribution, while q is a Bernoulli distribution [11]. In terms of how a particular p and q were chosen for a given problem, p has one free parameter (the prior length-scale) which was treated as a hyperparameter. To minimize fine-tuning for particular environments, all environments use one of only two different length scales (see eta column, Table 2 in the appendix). q was trained in a standard Bayesian supervised learning fashion on the available data using variational inference (see Model-Based RL subsection in Sec. 3.2). We have clarified these points in the manuscript, and have included choice of prior in our formalization of the problem in the Bayesian context.

In response to the minor issues raised:

• Thank you for pointing out this oversight on our part. All results are computed by averaging over 5 distinct random seeds. Shaded areas on plots are 95% confidence intervals. We have clarified this in the text.
• We have corrected the order of citations in the manuscript.

[1] Weisstein, Eric W. "Jensen's Inequality." From MathWorld--A Wolfram Web Resource. https://mathworld.wolfram.com/JensensInequality.html

[2] Li, Yingzhen, and Richard E. Turner. "Rényi divergence variational inference." Advances in neural information processing systems 29 (2016).

[3] Wan, Neng, Dapeng Li, and Naira Hovakimyan. "F-divergence variational inference." Advances in neural information processing systems 33 (2020): 17370-17379.

[4] Adji Bousso Dieng, Dustin Tran, Rajesh Ranganath, John Paisley, and David Blei. Variational inference via χ upper bound minimization. Advances in Neural Information Processing Systems, 30, 2017.

[5] Liqun Chen, Chenyang Tao, Ruiyi Zhang, Ricardo Henao, and Lawrence Carin Duke. Variational inference and model selection with generalized evidence bounds. In International conference on machine learning, pp. 893–902. PMLR, 2018.

[6] Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." arXiv preprint arXiv:1312.6114 (2013).

[7] Blei, David M., Alp Kucukelbir, and Jon D. McAuliffe. "Variational inference: A review for statisticians." Journal of the American statistical Association 112.518 (2017): 859-877.

[8] Li, Yingzhen, and Yarin Gal. "Dropout inference in bayesian neural networks with alpha-divergences." International conference on machine learning. PMLR, 2017.

[9] Gal, Yarin, Rowan McAllister, and Carl Edward Rasmussen. "Improving PILCO with Bayesian neural network dynamics models." Data-efficient machine learning workshop, ICML. Vol. 4. No. 34. 2016.

[10] Depeweg, Stefan, et al. "Learning and policy search in stochastic dynamical systems with bayesian neural networks." arXiv preprint arXiv:1605.07127 (2016).

[11] Gal, Yarin. "Uncertainty in deep learning." (2016): 3.

We thank the reviewer for their thoughtful and constructive feedback. In response to the weaknesses raised:

• One of the primary contributions of our approach is a principled method for dealing with function approximation errors in the model. We fully agree that modeling error may cause the model to overestimate the return of the model-based policy. However, because we use a lower bound on the improvement of the MB policy over the MF policy, Unified RL underestimates this improvement during the policy selection step. The lower bound in eq. 3 is an exact lower bound on improvement in Bayesian return of pi’ over pi. Because function approximation errors cause this lower bound to be loose (and therefore lower), these errors bias the agent away from the MB policy. The way this practically manifests in the algorithm is as follows: when function approximation error exists (e.g., when our model class is misspecified), the approximate posterior assigns probability mass to the set of candidate dynamics models that best explain the data while not being too improbable a priori. We should expect the approximate posterior to assign some mass to the dynamics models that underestimate as well as overestimate the return of the model-based policy. When it comes time for our agent to select a policy using a Monte-Carlo estimate of the lower bound on policy improvement, due to the specific form of the bound we use, the agent will only select the MB policy if it yields higher predicted reward according to all dynamics models sampled from the posterior. Therefore, even if the approximate posterior overestimates the improvement of the MB policy over the MF policy on average, the agent will select the MF policy if at least one of the sampled dynamics models predicts that the MF policy will outperform the MB policy.
• It is true that we do not explicitly represent the set of policies comprising the EPS, as this would be intractable. However, it is not necessary to explicitly represent the EPS given that our goal is to constrain the policy of unified RL to lie within the EPS. For this, all we need is a condition that allows us to check whether a policy is within the EPS, which we accomplish using the bound in eq. 6. We do therefore consider the entire set of policies that are not provably suboptimal, in the sense that model-free RL can choose any policy within this set. It is true that our method of constraining the policy to lie within the EPS is simplistic; however, we thought it best to favor simplicity in our approach given that this is the first work on the topic.
• It is true that our method introduces additional computational overhead. However, we argue that for most real-world systems, the data efficiency gains more than compensate for this increased computational overhead. For example, in the Ant environment, to achieve 80% of its maximal reward, SAC requires 2480 episodes, while Unified RL requires only 1404 episodes. SAC requires 0.008s of wall clock time per episode to update the policy, while time taken to collect one episode of data in the real environment is on the order of 1s (assuming our RL controller operates at 100Hz, which is the frequency used by the Mujoco simulator). Under these assumptions, we can tolerate a 97-fold increase in computational burden and still require less wall clock time to reach 80% of maximal performance, because the time required to perform learning updates is practically negligible compared to the time required to collect data.

This paper proposes the idea of equivalent policy sets to combine model-based RL and model-free RL. All the reviewers agree that the idea of EPS is novel.

While the authors clarified several concerns during the rebuttal, two of the reviewers questioned the significance of empirical claims in the paper. While the proposed method is compute-intensive two train two agents at a time, the performance gain is not very high when compared to max(MF, MB). The authors provided the ranking of the models. But I don't think ranking gives the complete picture. This is a weakness the authors must address to convince others of the usefulness of the proposed method. one suggestion is to find environments where both model-based and model-free methods struggle and show that your method can actually solve the task.

I recommend authors to improve their experiments and resubmit.

Reject