Revisiting On-Policy Deep Reinforcement Learning

1. Limited Comparisons: The paper's claims of state-of-the-art performance are only made relative to PPO. Including comparisons with other on-policy methods like VMPO (https://arxiv.org/abs/1909.12238) or SPO (https://arxiv.org/abs/2402.07963) would provide a more comprehensive context and strengthen the results as I do not believe you can make this claim given only PPO as a comparison.
2. Backbone Usage: PPO+ uses a separate backbone for the critic and actor networks, while PPO does not. Previous work by Andrychowicz et al. (2021) suggests that separate networks generally perform better. The lack of an ablation study comparing the shared and separate backbones in PPO+ raises serious concerns about the true source of performance gains.
3. Hyperparameter Sensitivity Claims: The authors don’t explicitly claim that PPO+ is less hyperparameter-sensitive than PPO however they state it as a motivating factor for the creation of it and the paper does not provide any empirical evidence to support this. Without testing the robustness across different environments or hyperparameter settings, it seems that PPO+ doesn’t necessarily address a core motivation.
4. Limited Evaluation: The evaluation is restricted to Mujoco environments. While this is a common benchmark, it is not sufficient to demonstrate that PPO+ consistently outperforms PPO or is more generalised. Testing in a broader set of environments, like grid-based or discrete action spaces, would provide more robust support for the authors' claims.
5. Overfitting to MuJoCo: Given the limited environment diversity and potential over-tuning for Mujoco tasks, it is unclear if PPO+ is truly an improvement over PPO or merely a set of optimisations tailored to a specific domain.

--This paper presents some interesting ideas, but I think it could be strengthened by highlighting its contributions more clearly. For example, incorporating off-policy data with a replay buffer and learning a Q-function instead of a state-value function shifts the algorithm towards the off-policy RL domain. To really showcase the algorithm's effectiveness, it would be beneficial to see comparisons with established off-policy algorithms like SAC and TD3. This would provide a clearer picture of its performance within the broader context of off-policy RL.

--It's also worth noting that one advantage of on-policy algorithms is their ability to learn by fitting only a value function, which can be simpler than fitting a Q-function. Introducing Q-learning in this context might add complexity, which seems to contrast with the authors' claim of increased simplicity. It would be helpful to see further discussion on this design choice and its potential implications in the context of on-policy RL.

--Adding an entropy bonus is a well-established technique, having been introduced in the original PPO paper. The entropy weight is already a standard hyperparameter in most PPO implementations. More discussion on how the use of the entropy bonus here differs from standard PPO would be helpful.

--Authors noted, reintroducing discounting to the state distribution doesn't yield significant performance improvements. A discussion on in which scenarios using a discounted state distribution would be beneficial would also be helpful.

--Finally, the experimental results presented aren't entirely conclusive. In some domains, PPO performs better than PPO+. It's more fair to compare PPO+ with off-policy algorithms. However, as the authors mentioned, their method doesn't currently outperform SAC or TD3, despite incorporating many of the components from those algorithms. This raises questions about the specific benefits and potential advantages of the proposed modifications.

• The experiments are currently limited to MuJoCo control tasks. Evaluating PPO+ on a wider range of environments would provide more comprehensive proof of its capabilities.
• A discussion of the performance gap between PPO+ and off-policy counterparts would strengthen the paper.
• While the focus on PPO is understandable given its popularity, the paper would benefit from comparing PPO+ to other on-policy algorithms beyond PPO.
• The authors acknowledge that optimizing for the discounted objective increases sensitivity to the choice of discount factor. While they present results for two different discount factors, further investigation into this sensitivity and strategies for mitigating it would enhance the practicality of PPO+.

None of the proposed modifications is novel, they have all been well studied in the literature. The paper dedicates a significant amount of space to reviewing these fairly well known ideas. I don't think that merely putting them together in a new combination is in itself a significant contribution.

The experiments are not conclusive since important comparisons to SOTA off-policy algorithms are missing. Since the paper introduces effectively an off-policy component into the algorithm (with the need to implement a replay buffer etc.), I would have really liked to see this comparison. Indeed the authors state (in the limitations) that the proposed combination of algorithmic components underperforms such existing algorithms which begs the question why one should use the combination proposed in the present paper. (NB, some off policy algorithms such as MPO also use a trust region and in that respect bear similarities to PPO.)

For this to be a strong paper I would have expected an insightful discussion why the specific algorithmic combination should be particularly useful / interesting, a demonstration that it clearly outperforms existing algorithms on relevant problems, and a detailed analysis why this is the case.

1. The experiments are unfortunately fairly limited in scope. Only 6 mujoco control domains are used and only two of them (and and humanoid) would be considered high dimensional in 2024. This limits the evidence that the paper can present for its suggested modifications seveerly.
2. The presentation of the experiments is lacking: 2a: A comparison to baseline PPO is presented on two domains in Figure 1 and 2. With PPO failing on the high dimensional domains. This doesn't inspire huge confidence in the results. What is causing this? Is the asymptotic performance fine and the main difference is just the speed-up from the Q-function and standard PPO would just need to run much longer? 2b: Further Figure 3 ablates some choices of the algorithm but again seems lacking. We get no insight into which of the proposed modifications exactly makes things work. For example: how would standard PPO but with a Q-function do? It also seems like PPO without discounting could be fine on-policy (but we are missing those results here, i.e. the combination of on-policy and no discounting). 2c: A the practical implementations of PPO for any domain with higher dimensional observations (or larger models) might consider computing the loss only on a trajectory snippet extracted from a full episode. It is unclear how that would affect e.g. the discounting.
3. Out of the three proposed modifications two are already routinely considered in the literature/implementations: entropy regularization is a standard feature in many PPO implementations; using discounting for the 'policy gradient' loss has been considered multiple times in the literature (also partially noted by the authors) and not been consistently proven to make a big difference, so most implementations omit it. This leaves the reviewer thinking that the main contribution is to consider learning an action-value critic off-policy, but unfortunately the experiments do not properly ablate and compare this modification (see above).
4. In many applications it is generally hard to learn an action-value critic (since conditioning on high-dimensional actions comes with it's own problems) especially when dealing with large models and or large action spaces so the algorithm here may not be generally an improvement in all cases (e.g. the situation might look very different for RLHF of large models or for experiments requiring vision inputs).