Improving Value Estimation Critically Enhances Vanilla Policy Gradient

We would like to thank the reviewer for their positive and supportive review!

However, the assertion that trust regions (“trust regions are neither sufficient nor necessary for guaranteed performance improvement”) requires further theoretical backing.

We will revise the statements in the paper to more accurately reflect both the current theoretical analyses and the empirical results.

Including experiments with PPO and TRPO would provide a more comprehensive evaluation.

Please see the answer to Q2 below.

Broader evaluations across more algorithms (TRPO) and environments would strengthen the conclusions.

That's a good point. We had considered running additional experiments in the DM Control Suite but finally decided against it after realizing that almost all on-policy algorithms perform poorly in that setting. For example, as shown in Appendix C.2 of https://arxiv.org/abs/2501.16142, PPO struggles in these environments. In contrast, only off-policy algorithms like SAC and TD-MPC consistently achieve strong performance.

Regarding the weaknesses:

1. We agree that further investigation into enhancing value estimation in PPO is a promising direction for future work. Please refer to the answer to Q1 below for a more detailed response.

2. We will revise the statements in the paper to more accurately reflect both the current theoretical analyses and the empirical results.

3. Please see the answer to Q2 below.

And for the questions:

Q1: While this paper focuses on the pivotal role of value estimation in policy gradient methods, as highlighted in the title, this question is one of the directions we plan to explore in future work. It will be interesting to investigate how much PPO can benefit from enhanced value estimation.

In the default PPO setting considered in this paper, the value network is updated with 320 gradient steps per iteration, which is generally sufficient for good value estimation. However, since PPO optimizes both the policy and value networks within a single loop, the same number of steps is applied to the policy network. This results in the policy network being over-updated, which contributes to the poor performance observed in the Ant and Humanoid tasks.

Therefore, one possible direction is to reduce the number of optimization steps in PPO, as we have already shown that 50 value steps are sufficient for good value estimates. Additionally, the empirical results in this work indicate that even a single policy step per iteration can lead to good performance, suggesting that we can also reduce the number of policy steps in PPO. In Figures 9(c) and (d) in the Appendix, we demonstrate that a PPO implementation with 8 gradient steps per iteration significantly outperforms the default PPO implementation on the Ant and Humanoid tasks. These results provide concrete evidence that this approach could be a promising direction.

We believe that by reducing the number of optimization steps, PPO can become more wall-time efficient and more robust to hyperparameters.

Q2: That is an insightful question! In Figure 4, we run the VPG algorithm with varying numbers of value steps, which demonstrates that having the maximum number of value steps per iteration does not always result in the best performance. Although no significant performance decline is observed as the number of value steps increases, we found that 50 value steps per iteration yields the optimal final return across five different environments. Additionally, we did not observe any instability even with an increased number of value steps.

The key insight we aim to highlight through this work is that enhanced value estimation can significantly improve the simplest VPG algorithm. The empirical results presented in Figure 4 suggest that even for those with limited experience in hyperparameter tuning, using a sufficiently large number of value steps can yield decent results. However, we agree that incorporating additional techniques to prevent overfitting could further enhance the robustness of VPG. In particular, since optimizing the value network is essentially a supervised learning problem, methods like early stopping and regularization, commonly used in deep learning, could be beneficial.

Q3: Of course! The code for the experiments will be released once the paper is accepted.

Thank you again! We will revise the paper accordingly based on your suggestions.

Thank you for the important review. Hope that our response adequately addresses your concerns.

For claims:

1. This argument directly corresponds to the Theorem 1 from the TRPO paper, which states that $J(\theta) \geq L^{TRPO}(\theta) - \frac{4 \beta \gamma}{(1 - \gamma)^2} D_{KL}^{max}(\pi_\theta \ \| \ \pi_{\theta_{old}})$ where $J(\theta)$ is the original policy objective, $L^{TRPO}(\theta_{old})$ is the surrogate objective and $D_{KL}^{max}$ is the KL distance between the old and new policies. Since it always has $J(\theta_{old}) = L^{TRPO}(\theta_{old})$. Therefore, the original policy objective is guaranteed to improve only if $J(\theta) - J(\theta_{old}) \geq L^{TRPO}(\theta) - L(\theta_{old}) - \frac{4 \beta \gamma}{(1 - \gamma)^2} D_{KL}^{max}(\pi_\theta \ \| \ \pi_{\theta_{old}}) \geq 0$ which requires that the surrogate objective is improved by a margin greater than $\frac{4 \beta \gamma}{(1 - \gamma)^2} D_{KL}^{max}(\pi_\theta \ \| \ \pi_{\theta_{old}})$. While the theory proves the above inequality when two policies are close enough, there is no way to guarantee that the second part of the inequality above always holds in practice.

We will add the citation there and rephrase the statement as follows for clarification:

According to the theory of trust regions (cite TRPO paper here), there is a gap between theory and practice because the theory guarantees improvement of the original policy objective only when the surrogate objective improves by a margin greater than the distance between the old and new policies.

2. The landscape is fractal due to the chaotic dynamics of the MDP, which are independent of the policy parameterization. The loss landscapes in [1, 2] correspond to the actor loss over a given batch of data, rather than the return landscape shown in Figure 1. For further reference on fractal landscapes, please refer to [1].

3. That is a good point. This paper focuses on continuous control problems where the MDP is deterministic. The only source of randomness comes from the Gaussian policy we use, which is why we can make the Lipschitz assumption on the dynamics and reward. The proof of Theorem 5.2 can be found in Appendix C. The intuition behind having $\beta_2$ in the denominator is that smaller learning rates require more value steps to compensate.

4. We agree that PPO is more sample efficient than VPG, as confirmed in the second paragraph of the Conclusion. However, this comes at the cost of reduced stability, as demonstrated in the Ant and Humanoid results in Figure 4, where VPG-repeat-k significantly outperforms PPO when k is greater than 50. When referring to performance, we mean the final performance of each algorithm, and we will clarify this in the draft. We believe that the current empirical results are sufficiently convincing to show that VPG with enhanced value estimation can achieve comparable performance to PPO, as initially claimed.

We will also smooth the reward curves.

For experiments:

1. We ran that for a single seed as it was only an illustrative example, and we have re-run the Hopper task with multiple seeds for further validation. Please see our response to Reviewer duhH for the detail.

2. We hypothesize that larger gradient magnitudes used in TRPO may positively contribute to its performance improvement over VPG. However, if this were the case, it would contradict the core idea in the TRPO paper, which favors small policy updates. Additionally, it has been shown that larger learning rates often lead to poor performance [2]. Furthermore, in Section 6, we examine how learning rates influence performance, demonstrating that excessively large gradient steps can negatively impact the final return.

3. $r_t = \frac{\pi_\theta}{\pi}$ is exactly the probability ratio in PPO, so it depends on $\theta$, which is estimated in the same way as in PPO. $r_t^*$ is the target probability ratio, as defined in lines 170-171.

4. Yes, in lines 266-268, we mentioned that PPO uses a single optimization loop for both networks, which leads to ratio clipping that prevents the policy from over-updating.

For the suggested references, we will add the following discussion accordingly to the main text:

"It has been demonstrated that proper reward scaling can lead to overall performance improvements in deep RL [3]."

"Like VPG, PPO uses a single optimization loop for both networks, which can be found in commonly used implementations such as CleanRL [4]."

We will also revise the paper to address the minor issues you suggested.

[1] Rahn et al., Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control, NeurIPS '23.

[2] Andrychowicz et al., What matters in on-policy reinforcement learning? a large-scale empirical study, ICLR '21.

[3] Schual et al., Return-based Scaling: Yet Another Normalisation Trick for Deep RL, 2021.

[4] Huang et al., CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms, JMLR '22.

We appreciate the reviewer's thorough and insightful feedback.

For claims:

1. In Figure 2(c)(d), we observe that the new loss with 50 epochs effectively enforces the trust region compared to runs with fewer epochs, despite its poorer performance. This coincides with the suggested alternative approach. Alternatively, we may adopt a weaker statement: trust regions do not necessarily need to be enforced for policy improvement.

2. In Figure 5 and 11, we observe a significant value estimation error in the original VPG implementation with only one value step per iteration. Therefore, reducing the number of value steps in TRPO and PPO would likely lead to similar results, as the value estimation module is identical across all on-policy algorithms.

3. See later responses.

4. This point is very interesting, and we have considered. Actually, we found that it is hard to say if an improvement in trust region enforcement is equivalent to a given improvement in value estimation. One possible metric is the number of optimization steps made to the policy and value networks in PPO. However, it is not appropriate, as Theorem 5.2 indicates that the policy and value networks evolve at different rates, meaning that applying the same number of gradient steps does not necessarily lead to equal improvements. Additionally, ratio clipping in the policy further complicates the comparison. While Theorem 5.2 provides a lower bound, determining the exact ratio of gradient steps that results in equivalent improvements in value estimation and trust regions is impossible. We will rephrase the draft accordingly based on your suggestions.

For experiments:

1. In the default PPO setting used in our paper, the value network takes 320 gradient steps per iteration, which is already sufficient to provide strong value estimation, as shown in Figure 5.

2. We adopted the maximum ratio metric from Engstrom et al.. In Figure 2, only a single random seed was used, as it serves primarily as an illustrative example. However, we have re-run the Hopper task with five seeds for further validation.

Regarding the empirical results, we observe that the mean fraction of ratios violating the trust region bounds increases from 0.19 to 0.32 through training, with a standard deviation of 0.01 across five random seeds using the original PPO implementation from Tianshou. A similar trend is also reported in Figure 3 of [1]. These observations support our claims, as a non-negligible fraction of ratios consistently violate the trust region.

Gradient clipping is applied only to the experiments in Sections 5 and 6 and is no longer used in the illustrative examples in Section 4.

3. Only the observation is normalized.

For weaknesses:

In the default PPO setting, it takes 320 optimization steps per iteration, compared to 501 steps in VPG-repeat-500. However, Figure 4 shows that VPG-repeat-50 already outperforms PPO with just 51 steps. This suggests that the approach is scalable.

For comments:

1. When referring to performance, we talk about the final return and will clarify this.

2. Figure 1 shows that the return landscape of TRPO is fractal and violates the assumptions made in the theory. However, despite this, TRPO still performs well in practice. This suggests that its empirical success may not be entirely explained by the theory of trust regions, which then motivates the findings presented in our paper.

For questions:

1. For the experiments in Section 6, we used the PPO implementation from Tianshou with learning rate decay disabled (disabled in VPG, too), which could explain the performance decline observed in Walker and Ant. We will update the hyperparameters in Appendix A to reflect these. We believe that anyone using this setting can reproduce the same results and will release the code afterwards.

As analyzed in Section 4, ratio clipping in PPO is not sufficient to effectively enforce trust regions or completely prevent large policy updates. This limitation helps explain why the default PPO struggles to solve the Humanoid task. Additionally, random seeds may also contribute to the observed performance differences. Similar declines in performance on Ant are reported in Figure 17 of the ICLR '21 paper by Liu et al..

Regarding the learning rate of VPG, we adopted 7e-4 from Stable-Baselines3. The 3e-4 was specifically chosen for Hopper is to allow for a better comparison with PPO on that task. As shown in Figure 7(a), using 7e-4 in Hopper actually results in better performance.

2. See previous responses.

3. Our results show that VPG can perform similarly to PPO with enhanced value estimation, which is the key contribution of this work. However, with PPO, other factors also play a role. As previously mentioned, PPO's performance may decline due to its aggressively updated policy, which explains why VPG-repeat-500 outperforms PPO despite both having similar value estimations.

[1] Wang et al., Truly Proximal Policy Optimization, UAI, 2020.

We thank the reviewer for their positive feedback and hope that our response can address your concerns.

Regarding the weaknesses:

We used the PPO implementation from Tianshou with learning rate decay and reward normalization disabled to better compare algorithmic performance without excessive code-level optimizations. We also acknowledge that mini-batch size is a crucial factor in PPO's performance and have conducted an ablation study on this in Figure 9(b) in the Appendix.

The value estimation error of PPO and VPG is compared in Figure 5, which shows that the default PPO implementation provides good value estimates, as it performs 320 value steps per iteration, which is sufficient for accurate estimation. We agree that policy initialization is crucial for on-policy algorithms, so we fixed the random seed to ensure that both PPO and VPG start from the same initial policy and value network.

For the question:

The question raised here is insightful. Our algorithm focuses on continuous control settings, where PPO is widely believed to consistently outperform VPG. However, in other domains, such as language modeling (LLM), even the simplest REINFORCE algorithm has been reported to be able to outperform PPO ([1]).

We agree that despite being the standard benchmark in deep RL, the success in MuJoCo tasks does not guarantee its effectiveness in other domains. In this paper, both our theoretical analysis and empirical results are focused on continuous control and robot learning, as emphasized in the Conclusion. We will revise the paper to ensure that our claims are clearer and more specific to this context.

[1] A. Ahmadian et al., Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs, arXiv:2402.14740.