Vlearn: Off-Policy Learning with Efficient State-Value Function Estimation

Dear Reviewer WLtt,

Thank you for your thoughtful review of our paper, we appreciate the time and effort you dedicated to providing detailed feedback. We are pleased to note your recognition of the strong empirical performance demonstrated in our experiments, especially when compared to V-trace. Below, we address your open questions and concerns.

• Derivation Details and Theoretical Justification: We acknowledge your concern about the ambiguity in the derivation of the upper bound for the value function. In the revised manuscript, we have improved the notation in the main text and provided a comprehensive derivation in the appendix. In addition, we have given more insight into the method, explaining its relationship to and differences from V-trace. We have also now included a full proof establishing the consistency of the upper bound. We hope that these revisions not only answer your questions about the derivation, but also provide a more intuitive understanding of the method. Should you require further clarification, we are available to provide additional information.

  To answer some of the questions already at this point. The $\sum_j^K$ over actions in state $s_t$ is a Monte Carlo estimate of the expectation over actions from our policy. In practice computing this is hardly possible as we would have to be able to execute multiple actions from the same state. Our method now moves this expectation outside of the square and allows us to use the joint expectation over states and actions. But we agree this should be also reflected in the notation, we updated it accordingly.

• "Upper Bound" and "Lower Bound" Clarification Eq (3): The upper bound here simply refers to the truncated importance weights that clip the importance weights at typically 1. While we do not leverage a specific lower bound, the importance ratio has a natural minimum at 0.

• Contribution and Connection to Mahmood et al. 2014: Our work is indeed similar to this previous work and could be seen as an extension to the general case of function approximation. However, direct application of this solution is still challenging as seen in our added ablation study, hence we also propose a proper way on how to leverage these results in an efficient way as part of an off-policy trust region method. Although a direct application is not possible, we believe that this related work further supports our argument that using importance sampling for the full loss function is preferable. We have made this connection clearer in the revised version.

• Minor Issues: Thank you for pointing out the typographical errors. In the revised manuscript, we corrected these inaccuracies.

We believe that the newly added changes strengthen the overall clarity and robustness of our paper. Your feedback has been very helpful, and we remain at your disposal for any further inquiries.

Dear Reviewer aLJf,

Thank you for your thorough review of our paper, we appreciate your time and the constructive feedback you provided. We are grateful for your positive feedback on the clarity of the research question, the well-written nature of the paper, and the comprehensive summary of related work. Additionally, we appreciate your acknowledgment of the convincing use of a trust-region method to stabilize training. We have carefully considered your questions and concerns and aim to address them in the revised version of our manuscript as well as in our answer.

• Derivation Details and Theoretical Justification: We acknowledge your concern about the ambiguity in the derivation of the upper bound for the value function. In the revised manuscript, we have improved the notation in the main text and provided a comprehensive derivation in the appendix. In addition, we have given more insight into the method, explaining its relationship to and differences from V-trace. We have also now included a full proof establishing the consistency of the upper bound. We hope that these revisions not only answer your questions about the derivation, but also provide a more intuitive understanding of the method. Should you require further clarification, we are available to provide additional information.

  To already address the question regarding the summation at this point, we indeed use a one sample estimate (as does V-trace), the formulation was probably misleading there. By moving the importance sampling out of the square in Vlearn, it allows us to work with the joint expectation over states and actions (this is what we referred to as subsummation). This way having only one action sample per state is less of an issue and leads to lower variance than having a one sample Monte Carlo estimate just for the expectation over actions, such as in V-trace. This is similar to what SAC is doing by only generating one action sample for its expectation over the Q-function. V-trace effectively splits the expectations of states and actions and still uses a one sample estimate for the expectation over actions in Eq. (2), leading to potentially higher variance.

• Utility of State-Value Estimation: We agree with the reviewer that the effects of the individual components are currently difficult to filter out. We have extended the ablation section of the paper by performing additional variations of our proposed method. However, it should already be noted here that in our experiments we already compare to V-trace in an equivalent setting, using the same techniques and only changing the way the value function is trained. Nevertheless, V-trace is not able to achieve the same performance. For this reason, we also think that a comparison with a Monte Carlo estimator is difficult, since V-trace is not able to learn a reasonable policy, which would also be reflected in a relatively poor estimate of V-trace in this comparison.

• Language & Clarity: We acknowledge there are discrepancies between Algorithm 1 and the method as stated in the main text. In the revised manuscript, we improved the algorithm description to make it more self-explanatory and aligned. We also fixed the mentioned language issues.

We sincerely appreciate your thoughtful review and are committed to enhancing the clarity, soundness, and presentation of our work based on your suggestions. We look forward to your feedback on the revised version that aims to address the mentioned points.

Dear Reviewer FMyp,

Thank you for your detailed review of our paper, we appreciate your time and thoughtful evaluation of our work. We are particularly encouraged by your positive feedback on the strong empirical results and the significance of our work. Below we aim to address the remaining questions and concerns.

• Comparison to V-trace: By moving the importance sampling out of the square in Vlearn, it allows us to compute the joint expectation over states and actions. This way having only one action sample per state should generally lead to less variance. V-trace effectively keeps the split expectations of states and actions and still uses a single sample to estimate the expectation over actions in Eq. (2), leading to potentially higher variance. Furthermore, V-trace is interpolating between the current (target) value function and the 1-step Bellman target

The smaller the importance ratio, the more the value function optimization focuses on the current (target) value function. However, for these samples, the overall loss scale and consequently the gradient remain unchanged. In the case of Vlearn, it is different – samples with small importance ratios contribute less to the total loss but still work towards the Bellman target. V-trace was initially designed for correcting off-policy asynchronous processing, where policy differences (i.e. importance ratios) are typically limited, so this is not much of a constraint. However, in a complete off-policy scenario, the samples in the replay buffer and the current policy deviate more rapidly and significantly from each other. We tried to emphasize this insight more in the revised version to make the distinction clearer.

• Estimation of Importance Sampling Ratio: First of all, we want to mention that we can compute the importance ratio directly from our predicted Gaussian distributions in closed form, hence, unlike the Q function, it does not need to be estimated. While in higher dimensional action spaces the two Gaussians will naturally overlap less and approach 0 faster, we found that the state-wise trust regions from TRPL help stabilize this as the policy cannot move arbitrarily fast from any previous policy distribution. This ensures there are always sufficiently many samples in the replay buffer for which the ratio is $>0$. However, we are not 100% sure, we understand the reviewer's question regarding this correctly. If our provided answer is not sufficient, please feel free to follow up on this.

• Low-Dimensional Tasks: We agree that our performance does not provide significant improvements for the low-dimensional task, but we did not expect it to, since the added complexity of learning a Q-function is likely negligible in these cases. Therefore, similar performance is a desirable result for our method. SAC on the HalfCheetah-v4, we see as an outlier, as other off-policy methods, such as MPO, also do not perform as well on this task, and it has been shown in previous work [1] that this particular task exhibits extreme behavior.

Thank you once again for your review, and we look forward to your feedback of the revised version that incorporates your suggestions.

References

[1] Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, Andr ́e Biedenkapp, Kurtland Chua, Frank Hutter, and Roberto Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 4015–4023. PMLR, 2021

Dear Reviewer syZi,

Thank you for taking the time to review our paper. We are pleased to note your positive remarks on the clarity of our motivation, the comprehensive background, and the strength demonstrated in our experiments. Further, we appreciate your valuable feedback and constructive comments. Below, we address each point to ensure clarity and address any outstanding issues.

• Derivation Details and Theoretical Justification: We acknowledge your concern about the ambiguity in the derivation of the upper bound for the value function. In the revised manuscript, we have improved the notation in the main text and provided a comprehensive derivation in the appendix. In addition, we have given more insight into the method, explaining its relationship to and differences from V-trace. We have also now included a full proof establishing the consistency of the upper bound. We hope that these revisions not only answer your questions about the derivation, but also provide a more intuitive understanding of the method. Should you require further clarification, we are available to provide additional information.

• Pseudocode: Here, the "for" is just an indicator that we compute this loss for both critic networks and equivalently for the target critics in line 13, i.e. a for loop over critics. We decided on this notation since it is also commonly used in other works, e.g. SAC [1]. The $\theta$ in line 11 is indeed inconsistent, and should be the minimum of the two critic estimates. Overall, for the updated version, we have aligned the pseudocode with the main text.

• Overestimation Bias: We agree the overestimation bias is not a main concern in our setting as compared to the Q-function based setting. We found, however, in practice that using two networks is also beneficial in our case (see new ablation study). We assume the second network can be seen as a small ensemble that acts as a form of regularization and by selecting the minimum value, we avoid introducing any overestimation bias by training two networks. As the twin network approach is common practice for the baseline methods, we kept it also in our case.

• Effect of Individual Components: We agree with the reviewer that an ablation study on the effects of the individual components can be of good value. We extended this section of the paper by running additional experiments. However it should already be noted here that in our experiments we compare to V-trace in an equivalent setting using the same techniques and only changing the way the value function is trained. Yet, V-trace is not able to achieve the same performance.

We sincerely appreciate your time and thoughtful consideration of our work. Should you have any additional questions or require further clarification, we are readily available to assist. Thank you once again for your valuable feedback and engagement.

References

[1] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861–1870. PMLR, 2018.

Thank you for the update. The current version is certainly more readable than the previous version. Though not as comprehensive as I expected, the ablation study revealed some contribution of each component to the algorithm's performance.

I have some follow-up questions and comments on this version of the work.

• Regarding equation (2), the authors claimed ``Directly optimizing this can be difficult as approximating the inner expectation with Monte Carlo samples requires multiple action executions per state, an unrealistic assumption for most RL problems." I wonder why multiple action execution is needed to estimate the inner expectation. The state $s_t$ is sampled from $\mathcal{D}$. The action is also sampled from $\mathcal{D}$. Do they come from the same transition $(s_t, a_t, r_t, s_{t+1})$? In addition, the next state $s_{t+1}$ is sampled from $\mathcal{T}$. Does that mean we should sample $s_{t+1}$ by running the sampled $s_t, a_t$ in the environment? An analogous question goes to equation (4).
• What is the difference between the loss in equation (4) and the loss for the off-policy semi-gradient TD(0) (Sutton and Barto, 2018, pp. 258), except for the use of a target network?
• The derivation of the upper bound objective now includes more details but again skips Jensen's inequality part. It is not trivial to show how to move the importance sampling ratio out of the square. Let $\delta (s, a)$ denote the TD error, $\rho(s, a) = \pi(a|s)/b(a|s)$ denote the importance sampling ratio. The inner expectation of $L^*(\theta)$ is $E_b[(\rho(s,a)\delta(s,a))]^2$. If we directly apply Jensen's inequality, we get $E_b[(\rho(s,a)\delta(s,a))]^2 \le E_b[(\rho(s,a)\delta(s,a))^2]$. $\rho(s,a)$ is trapped in the square. I assume the other reviewers are concerned with this step as well. Instead, I believe it's better to apply Jensen's inequality early on and add the importance sampling ratio afterwards: $E_\pi [\delta(s,a)]^2 \le E_\pi [\delta^2(s,a)] = E_b[\rho(s,a) \delta^2(s,a)]$.

Given the current state of the work, I am going to raise my rating slightly to reflect it.

Reference

Sutton, R. S., Bach, F., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT Press Ltd.