Dear Reviewer Dgb4,

We are truly grateful for your thoughtful review of our paper and for the encouraging support you've shown towards our work! In the following, we will provide a comprehensive response to your comments.

To address your concerns, we have submitted all the code for PPO and SPO used in this paper as supplementary material. Additionally, a new version of the paper has been uploaded, with the main revisions highlighted in red for your convenience in further review.

I. Results Lack of Statistical Testing

Lack of Confidence Intervals and Statistical Tests

Could you provide confidence intervals and other statistical measures for the experimental results to help clarify performance variance and statistical significance?

We sincerely appreciate your thoughtful consideration. Based on your suggestion, we have referred to [1] and plotted the aggregate metrics on Mujoco-v4 with 95% confidence intervals based on data from 6 environments (Figure 5) using PPO-normalized score. We found that SPO achieved competitive performance across various statistical measures, which enhances the paper's persuasiveness.

II. Potential Baseline Tuning Issues

PPO and similar RL algorithms are known to be highly sensitive to hyperparameters like batch size, mini-batch size, and the number of rollout steps. The paper does not discuss any dedicated hyperparameter tuning for the baselines, raising concerns about potential underperformance.

Your feedback is extremely beneficial to us. To address your concerns, we referred to the open-source standard reinforcement learning library, CleanRL [2], and attempted to align all hyperparameters. We found that the only difference was in the mini-batch size setting, where our paper defaults to 512. Therefore, we focused on comparing the performance differences between SPO and PPO under different network depths and mini-batch size settings, and we supplemented the results with statistical tests, as shown in Figure 7. We found, as you mentioned, that PPO is quite sensitive to hyperparameters and network depth. In contrast, SPO showed significant improvements across various statistical measures and demonstrated relative robustness to mini-batch size. We hope this addresses your concerns!

III. Code-Level Optimizations

As noted in the paper, code level optimisations can significantly impact the performance of certain algorithms, but the paper does not specify what implementation tricks were used in the baselines and SPO itself.

Were any efforts made to tune the baseline hyperparameters specifically? If so, could you detail the tuning process? Additionally, how much effort was made tuning SPO?

Thank you for your question! We want to emphasize that for all the comparative experiments involving SPO and PPO in the paper, the only difference is the policy loss. You can also refer to the code implementation for Mujoco and Atari in the supplementary material, where the only difference is in the compute_policy_loss function within the trainer.py file. No further code-level tuning is applied to SPO, highlighting its simplicity and efficiency. For code-level details, our implementation is primarily based on [2] and [3], where these details are thoroughly explained. The code details for SPO are completely consistent with those for PPO.

IV. Additional Supplementary Experiments

Some additional interesting experiments have been added :), as shown in Figures 8 and 9. We found that naively reducing PPO's $\epsilon$ does not prevent excessive ratio deviation, further indicating that SPO's effective constraint on the probability ratio is non-trivial.

References

[1] R Agarwal et al. Deep reinforcement learning at the edge of the statistical precipice. https://arxiv.org/abs/2108.13264

[2] S Huang et al. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. https://github.com/vwxyzjn/cleanrl

[3] S Huang et al. The 37 implementation details of proximal policy optimization. https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/

III. Theoretical Derivation

To me, the transition from Equation (10) to Equation (14) feels somewhat abrupt. Could you elaborate further on this derivation?

Sure! Let us first write down the lower bound of performance improvement $\eta(\tilde{\pi})-\eta(\pi)\geq E_{s\sim{\rho_{\pi}(\cdot)},a\sim{\pi(\cdot|s)}}\left[\frac{\tilde{\pi}(a|s)}{\pi(a|s)}\cdot A_{\pi}(s,a)\right]-\frac{C}{4}\max_{s}\left[{E}_{a\sim\pi(\cdot|s)}\left|\frac{\tilde{\pi}(a|s)}{\pi(a|s)}-1\right|\right]^2=\mathcal{L}$ where $C=\frac{4\epsilon\gamma}{(1-\gamma)^2}$ is unknown for us. Therefore, it is impossible for us to directly optimize the lower bound $\mathcal{L}$. The commonly used approach is to attempt to constrain the latter term while maximizing the former term (the surrogate objective).

Considering that the maximization operation of the latter term requires traversing all states $s\in\mathcal{S}$, this is also not feasible. Therefore, we can only empirically impose the ratio constraint on each current data point $(s_t,a_t)\sim\pi_{\theta_{\mathrm{old}}}$, which is $\left|\frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t|s_t)}-1\right|\leq\epsilon,\enspace\forall (s_t,a_t)\sim\pi_{\theta_{\mathrm{old}}}$ and this leads us to derive the optimization problem (15) in our current paper, which is $\max_{\theta}\enspace E_{(s_t,a_t)\sim\pi_{\theta_{\mathrm{old}}}}\left[\frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t|s_t)}\cdot A(s_t,a_t)\right]$ $\mathrm{s.t.}\enspace\left|\frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t|s_t)}-1\right|\leq\epsilon,\enspace\forall (s_t,a_t)\sim\pi_{\theta_{\mathrm{old}}}$

Simplifying the notation, let the probability ratio be denoted as $r$, and the advantage function as $A$. Then, we aim to obtain an objective function $f(r,A,\epsilon)$ (where $r$ is the optimization variable) that maximizes $rA$ while constraining $|r-1|\leq\epsilon$. Additionally, we want this function to be convex, so why not consider a quadratic polynomial in $r$? For simplicity, let's assume $f(r,A,\epsilon)=rA-k\cdot (r-1)^2$ This is a quadratic function of $r$, with an unknown coefficient $k$. We also want the optimal solution to be at the boundary of our constraint $r=1+\mathrm{sign}(A)\cdot\epsilon$. By solving this simple differential equation, we obtain our objective $f(r,A,\epsilon)=rA-\frac{|A|}{2\epsilon}\cdot (r-1)^2$ we presented this very intuitive approach to the derivation.

IV. SPO in Large-Scale Scenarios

Is SPO also more applicable to large-scale scenarios and does it still show superiority in terms of ratio deviation? I’m curious about its performance difference from PPO in cases with extensive parallelism.

Thank you for your suggestion. We have included an experimental comparison of PPO and SPO with a large mini-batch size of 32,768 (see Figure 11 on page 15). We were surprised to find that with a large amount of data, the ratio deviation of PPO is well constrained (indicating that large-scale training data seems to mitigate excessive ratio deviation in PPO). In contrast, SPO generally has smaller variance and robust performance, and often matches or even surpasses PPO's performance in the later stages of training.

Additionally, for training with very large batch sizes, we recommend using a larger $\epsilon$ for SPO to match the early training advantages of PPO. We are also eager to further explore the use of adaptive $\epsilon$ in SPO during training to combine PPO's sample efficiency in the early stages with SPO's robust performance in the later stages.

Thank you again for your valuable suggestions and for your support of our work

References

[1] S Huang et al. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. https://github.com/vwxyzjn/cleanrl

[2] N Rudin et al. Learning to walk in minutes using massively parallel deep reinforcement learning. https://proceedings.mlr.press/v164/rudin22a.html

[3] Y Wang et al. Truly proximal policy optimization. https://proceedings.mlr.press/v115/wang20b.html

Dear Reviewer ESWe,

We are truly grateful for your thoughtful review of our paper and for the encouraging support you've shown towards our work! In the following, we will provide a comprehensive response to your comments.

Additionally, we would like to gently remind you that all code implementations of SPO and PPO used in our work have been uploaded to the supplementary materials. The revised version of the paper has also been uploaded, with the main revisions highlighted in red for your further review.

I. The Constraint on the Ratio Deviation

SPO tends to make the ratio deviation overly small in cases with fewer layers (see Table 2). Could the authors address whether the tighter ratio deviation in SPO may be limiting performance in some simpler tasks, and if so, how this trade-off might be managed?

SPO does not outperform PPO with a 3-layer network. Could you clarify this? In tasks where SPO underperforms (e.g., Ant, HalfCheetah, Walker2d), the ratio deviation appears to be constrained overly tightly.

When using a 3-layer network, what is the performance difference between SPO-0.3 and SPO-0.2? If SPO-0.2 constrains the ratio deviation overly tightly, causing it to underperform compared to PPO, SPO-0.3 might perform better due to the slightly looser upper bound on ratio deviation. If this is the case, it would provide stronger theoretical support for the approach.

• These issues directly hit the point! We found that the policy network with fewer parameters exhibits relatively conservative ratio deviation during training (see Figure 13). This indeeds limit the performance of SPO on some simpler tasks (as shown in Figure 6). The 3-layer MLP used in the experiments, with hidden layers of [64, 64] (based on the default settings from CleanRL [1] and most papers), has very limited expressive power. Our idea is that simply increasing the network parameters can effectively address this issue (Figure 8 somewhat proves this point).

• Then, based on your suggestion, we conducted experiments with a larger $\epsilon$ on Ant-v4, HalfCheetah-v4, and Walker2d-v4 (see Figure 14 on page 18), and we found that the results align well with your expectations. This suggests that the setting of $\epsilon$ indeed involves a trade-off: smaller epsilon leads to more stable performance improvement (based on the lower bound of policy improvement), but it may also make the policy updates more conservative. Our current response to this is to recommend using a relatively small $\epsilon$ to ensure robust policy improvement, while selecting neural networks with as many parameters as possible to provide sufficient expressive power.

II. Comparison to Existing Methods

Could the authors include a comparison of SPO with TR-PPO and other recent methods addressing PPO's clipping issues, particularly focusing on performance and ratio deviation metrics across different network depths?

Certainly! Following your suggestion, we added an additional section and compared PPO with a smaller $\epsilon$ under the same settings, as well as adaptive learning rates [2] and the TR-PPO [3] you mentioned. These methods also help to some extent in limiting aggressive policy updates, and the results can be found in Figure 9 on page 10. We found that naively using a smaller $\epsilon$ does not alleviate the issue of excessive ratio deviation in PPO. Furthermore, while adaptive learning rates do limit the ratio deviation, we observed that they lead to overly conservative policy updates (we noticed that throughout the experiment, the learning rate almost remained at 1e-5).

Dear Reviewer HKZk,

We sincerely thank you for your thorough review and valuable feedback on our paper, which will greatly help us improve the quality of our work. Below, we will address your concerns.

Please note that, in order to address your concerns, we have submitted all the code for PPO and SPO used in this paper as supplementary material. Additionally, we have uploaded a new version of the paper, with the major revisions highlighted in red for your convenience in further review.

I. Limited Novelty and Lack of Related Work

Limited novelty compared to prior works, with several missing citations to relevant prior works

the paper does not adequately frame its contribution with respect to prior work (there is no Related Work section in the paper).

(i) Lack of related work. We appreciate the reviewer for carefully reading and pointing out these two issues. We realize that this work indeed lacks proper citations to related work [1-7]. As you suggested, we have included the references you listed and added a section on related work (Appendix A) to introduce prior research. Additionally, we have briefly outlined the implementation details (Appendix B), which addresses concerns from most reviewers regarding the potential difficulty of reproducing the results.

(ii) Limited novelty, as you mentioned, previous work has established the relationship between TV divergence and the probability ratio [1], and we fully acknowledge this. However, despite this, we also have our own unique contributions:

• Firstly, previous work focused on probability ratio constraints based on clipping function, whereas we have thoroughly pointed out the potential drawbacks of clipping function (Figures 2 and 4) in our paper. For deeper networks, these drawbacks are highly detrimental to policy improvement.

• Secondly, we formally prove why using a trust region based on TV divergence constrain is a better choice (Theorem 3.3). This is due to the fact that the KL divergence constraint is looser, which leads to a smaller solution space, and the conclusion is non-trivial.

• More importantly, our novel objective provides theoretical guarantees for constraining probability ratio. The objective of SPO is convex with respect to the probability ratio, and the optimal value is exactly attained at the constraint boundary. Therefore, SPO offers strong theoretical guarantees for the probability ratio constraint and only requires slight modifications to the PPO objective to achieve significant performance improvements. In contrast, while previous work has also made efforts to constrain the differences between continuous policies, we believe that the SPO objective is quiet simple and effective. Furthermore, SPO achieves competitive performance improvements (Figures 5, 7, and 8), especially for deeper networks.

We sincerely appreciate Reviewer HKZk's valueble feedback, which has encouraged us to further clarify the novelty of our work. We would also like to note that other reviewers have recognized the unique contributions of our approach:

Reviewer ESWe remarked: "Although several papers have already pointed out issues with the heuristic clipping technique in PPO and proposed alternatives, I find the authors' approach offers a novel perspective, leading me to lean towards acceptance."

Reviewer Dgb4 highlighted: "SPO is a straightforward modification of PPO by changing the surrogate objective slightly, achieving meaningful performance improvements, particularly in terms of ratio constraint handling. The simplicity of SPO, combined with its impact, makes it valuable for the community."

V. Experimental Suggestions

How does SPO compare to well-tuned versions of PPO across each setting?

How does SPO compare to PPO using the adaptive learning rate in [3], which is a simple implementation detail used in the literature to prevent probability ratios from becoming very large?

How does SPO compare to non-parametric policy update methods such as [6, 7]? Similar to SPO, these methods apply updates based on target probability ratios.

• Thank you for your question. To address your concerns, we conducted additional experiments. These include a comparison of SPO and PPO with different network depths and mini-batch sizes (Figure 7). For adaptive learning rates, we show several methods you mentioned (Figure 9) and created a new section for this [3, 4]. We found that while adaptive learning rates improved performance, they often led to overly conservative policy updates, as we observed that the learning rate magnitude remained around 1e-5 throughout the learning process.

• Finally, regarding the suggestion to compare with non-parametric learning methods, since our work primarily focuses on parameterized policy improvement methods, we did not conduct additional comparative experiments. Nevertheless, we introduced these non-parametric works in the newly added related work section (Appendix A).

Thank you once again for your detailed review, which has greatly improved the quality of our work.

Reference

[1] J Queeney et al. Generalized Proximal Policy Optimization with Sample Reuse. https://proceedings.neurips.cc/paper_files/paper/2021/hash/63c4b1baf3b4460fa9936b1a20919bec-Abstract.html

[2] J Achiam et al. Constrained policy optimization. https://proceedings.mlr.press/v70/achiam17a

[3] N Rudin et al. Learning to walk in minutes using massively parallel deep reinforcement learning. https://proceedings.mlr.press/v164/rudin22a.html

[4] Y Wang et al. Truly proximal policy optimization. https://proceedings.mlr.press/v115/wang20b.html

[5] Y Cheng et al. Authentic boundary proximal policy optimization. https://ieeexplore.ieee.org/abstract/document/9376693?casa_token=_a37KD3Uk6YAAAAA:lpddziVA6z4-wSKRrnBpptwivJ48BQ3Pmxxd5cxJk_nullA3wQoeGF8BkJ7Dx1ek1bI4DvgK_9KpeYw

[6] Q Vuong et al. Supervised policy update for deep reinforcement learning. https://arxiv.org/abs/1805.11706

[7] HF Song et al. V-mpo: On-policy maximum a posteriori policy optimization for discrete and continuous control. https://arxiv.org/abs/1909.12238

[8] S Huang et al. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. https://github.com/vwxyzjn/cleanrl

II. Contributions are Overstated and Suboptimal PPO Implementation

At several points throughout the paper, the authors claim that SPO achieves state-of-the-art performance.

it would be much more convincing to show that SPO can perform similar to a well-tuned version of PPO across different tasks / architectures.

• Contributions are Overstated. We sincerely appreciate your thoughtful comments, and we recognize the issue of overstating our contributions. As a result, we have replaced all instances of "state-of-the-art performance" in the original paper with "SPO outperforms PPO" and "achieves competitive performance", on line 22, line 105, and line 532 in our current paper.

• PPO Implementation. To address your concerns, we aligned the hyperparameters with those from high-quality open-source reinforcement learning libraries, CleanRL [8]. We found that all hyperparameters for Atari are consistent, enabling PPO to achieve ~400 points in Breakout-v4. Additionally, we compared the hyperparameters for MuJoCo and found that the only difference was in the choice of mini-batch size. Based on your suggestion, we included a performance comparison for different network depths and mini-batch sizes (Figure 7). The results show that SPO outperforms PPO across various statistical metrics and demonstrates relative robustness across different network depths and hyperparameter settings. Additionally, all our code has been uploaded to the supplementary materials for your review. We hope this addresses your concerns.

III. Minor Technical Issues

I believe that (8) is not correct due to the min that appears in the PPO objective in (6). The cases must also consider the sign of the advantage function.

The connection between TV distance and the probability ratio in (9) / Appendix B requires the assumption that the support of $\tilde{\pi}$ is contained in the support of $\pi$.

• We appreciate your incredibly detailed review! As you mentioned, the gradient of PPO needs to account for the sign of advantage $\hat{A}_{\pi}(s_t,a_t)$. We have made the necessary modifications to (8), though this does not affect the subsequent derivations.

• Considering that typical policy networks usually output softmax-normalized probabilities (for discrete action spaces) or a multivariate Gaussian distribution (for continuous action spaces), we have ignored the assumption you mentioned in these two cases. For theoretical rigor, we have added this assumption in Appendix G of the current paper. Thank you for your suggestion.

IV. Some Problems

Can you please explain why each assumption in Definition 4.1 is important? When comparing the objective in (14) to PPO, it seems like the existence of a unique optimal probability ratio in SPO is actually the important feature compared to multiple optimal solutions in PPO.

Could you please explain the purpose of Section 3 and the optimization problem in (15)? It does not look like (15) is actually being used by SPO. Instead, the main algorithmic contribution is the objective in (14), which is motivated by considering the optimization problem in (17) at every state-action pair. Applying constraints at every state-action pair is more restrictive than the constraint in (15), and also does not account for the dependence across state-action pairs that occurs because $\pi$ must be a probability distribution.

Thank you for your questions; they have inspired us to some extent. We will address your concerns below.

• Firstly, each assumption in Definition 4.1 is important because the inclusion of the $rA$ term ensures that the gradient is indeed directed towards optimizing the surrogate objective, rather than just moving towards the constraint boundary. Additionally, $A\neq 0$ is necessary to express the optimal solution, as $A=0$ leads to the trivial case where both PPO and SPO objectives are always zero. The $\epsilon$ term is a constraint on the ratio deviation and thus must be positive. Finally, $f$ and $g$ being convex with respect to $r$ ensures that the gradient optimization algorithm will constrain the probability ratio within the boundary $r=1+\mathrm{sign}(A)\cdot\epsilon$.

• Secondly, this is a really good question. The optimization objective in (15) should impose ratio deviation constraints for all sampled $(s_t,a_t)\sim\pi_{\theta_{\mathrm{old}}}$ (we have corrected this and highlighted it in red). Constraining the ratio deviation expectation for all current data is similar to the heuristic approximation in TRPO, relaxing the maximization operation in the lower bound (10) to an average. Therefore, the current objective (15) aligns well with the SPO objective (14), as imposing constraints for each current data point $(s_t,a_t)\sim\pi_{\theta_{\mathrm{old}}}$ more closely aligns with the maximization operation in the lower bound (10).

Dear Reviewer HKZk,

We would like to kindly remind you that, perhaps due to your busy schedule, further feedback on our paper might have been overlooked. Based on your recent comments, we have made the following updates to our paper:

• We have added relevant works [1, 2] in line 222 and line 267.
• We have corrected the order of presentation in Section 3, first introducing problem (14) and then deriving objective (15).
• We have clarified the necessity of Proposition 3.2 and Theorem 3.3, noting that the performance improvement lower bound $\mathcal{L}=E_{s\sim{\rho_{\pi}(\cdot)},a\sim{\pi(\cdot|s)}}\left[\frac{\tilde{\pi}(a|s)}{\pi(a|s)}\cdot A_{\pi}(s,a)\right]-C\cdot D_{\mathrm{TV}}^{\max}(\pi,\tilde{\pi})^2$ cannot be directly optimized.
• We have included comparative experiments with the non-parametric method SPU [2] in Figure 4.
• We have also addressed the unreasonableness of the two objectives you mentioned: $rA-|A|\cdot|r-r^*|$ and $|r-r^*|$.
• Due to space limitations, we have included the necessary related work in Appendix A.

Overall, we believe that your concerns have been addressed. If this is the case, we sincerely hope that you can provide stronger support for our work (we noticed that you gave our work the lowest rating with the highest confidence).

We deeply appreciate the time and effort you have invested in reviewing our work. We have made revisions based on your feedback. If such efforts are not acknowledged, it may hinder constructive dialogue in the future review process.

Thank you for your understanding and consideration.

Sincerely,

Authors of Paper #3422

References

[1] J Queeney et al. Generalized Proximal Policy Optimization with Sample Reuse. https://proceedings.neurips.cc/paper_files/paper/2021/hash/63c4b1baf3b4460fa9936b1a20919bec-Abstract.html

[2] Q Vuong et al. Supervised policy update for deep reinforcement learning. https://arxiv.org/abs/1805.11706

Thank you for your response. Here are the issues you mentioned:

Proposition 3.2 / Theorem 3.3 do not provide insights about the optimization problem that is actually considered in this work. The optimization problem in (14) considers a strict subset of the TV divergence trust region $\Omega_{\mathrm{TV}}$, and the KL divergence trust region $\Omega_{\mathrm{KL}}$ is not a subset of the trust region in (14).

Thank you for your suggestion. We would like to reference TRPO [1]:

"While it is motivated by the theory, this problem is impractical to solve due to the large number of constraints. Instead, we can use a heuristic approximation which considers the average KL divergence."

TRPO is one of the most important papers in the field of model-free reinforcement learning, widely recognized by the academic community. However, even so, there remains a gap between theory and practical algorithms. In TRPO, the maximum KL divergence is relaxed to the average KL divergence, but this did not prevent the reviewers from accepting the paper. Therefore, we believe this suggests that there is always a gap between theoretical results and practical algorithms. It seems that the reviewer expects the gap between our theory and the practical algorithm to be infinitesimally small, which is clearly not achievable.

The function $f=rA-|A|\cdot|r-r^*|$ certainly seems to satisfy the imprecise statement "$f$ involves $rA$" term (which was the point of the example).

Thank you for your suggestion. However, we believe this is not correct. If that were the case, we could also consider $f=rA-rA=0$ as satisfying this condition, which would lead to the absurd conclusion that "$f$ involves everything". We believe this is a complete misinterpretation of Definition 4.1. Furthermore, we also tried the objective you proposed, $f=rA-|A|\cdot|r-r^*|$, and found its performance to be very limited, and it fails to constrain the probability ratio, you can verify this result by running the code we provided. We hope this addresses your concerns.

I also believe that the original submission should have been prepared much more carefully, rather than requiring significant revisions during the discussion period.

Thank you for your suggestions. However, changes to the paper during the review process are inevitable and can help improve the quality of the paper. Meanwhile, the additional experiments in the main text are unavoidable and are primarily meant to address your concerns. Specifically,

• SPU [2] (Figure 4)
• Figure 6
• the entire Section 5.3
• the Related Work in Appendix A.

We believe that these additional experimental results may have addressed most of the reviewers' concerns.

So far, we have been working hard to address all the issues you raised. However, we have noticed that you have now raised new questions, which we believe may indicate that most of your previous concerns have been resolved. Moreover, regarding your latest questions, we believe that some of them may be based on misunderstandings (see the text above). Additionally, we think that some of these questions pertain to common challenges in the field of model-free reinforcement learning, where inevitable gaps often exist between theoretical concepts and practically feasible algorithms.

Thank you again for your time and effort.

References

[1] J Schulman et al. Trust Region Policy Optimization. https://people.engr.tamu.edu/guni/csce642/files/trpo.pdf

[2] Q Vuong et al. Supervised policy update for deep reinforcement learning. https://arxiv.org/abs/1805.11706

We sincerely appreciate the effort you have put into reviewing our work. It seems inevitable to make revisions during the rebuttal phase, which helps authors improve the quality of their papers. We believe that it might be a better choice for other reviewers and the area chair to make the decision.

Dear Reviewer Xchv,

Thank you for your valuable feedback, which will greatly help improve our work! Below, we will address your concerns and look forward to further interaction with you.

To address your concerns, we have submitted all the code for PPO and SPO used in this paper as supplementary material. This code is primarily based on the high-quality reinforcement learning library CleanRL [1]. Additionally, a new version of the paper has been uploaded, with the main revisions highlighted in red for your convenience in further review.

I. The Novelty of Using TV Divergence Constraints

the idea of using total variation divergence is not new

What is the significance of proposition 3.2 and theorem 3.3? Could not we use the theorem (3) to conclude that Total Variation divergence is the better choice compare to KL divergence?

We acknowledge that the concept of Total Variation (TV) divergence is indeed not new and has been previously explored in various contexts. However, our application of TV divergence in this work presents several unique contributions:

• Firstly, we formally prove that optimizing the lower bound with a TV divergence constraint is better than using the KL divergence-constrained lower bound. Next, we will explain why this conclusion cannot be directly derived from Theorem (3). According to (3), the performance improvement lower bound is $\eta(\tilde{\pi})-\eta(\pi)\geq L_{\pi}(\tilde{\pi})-L_{\pi}(\pi)-C\cdot D_{\mathrm{TV}}^{\max}(\pi,\tilde{\pi})^2$ where $C=\frac{4\epsilon\gamma}{(1-\gamma)^2},\enspace\epsilon=\max_{s,a}|A_{\pi}(s,a)|,\enspace L_{\pi}(\tilde{\pi})=\eta(\pi)+E_{s\sim{\rho_{\pi}(\cdot)},a\sim{\tilde{\pi}(\cdot|s)}}\left[A_{\pi}(s,a)\right]$ Denote this lower bound as $\mathcal{L}=L_{\pi}(\tilde{\pi})-L_{\pi}(\pi)-C\cdot D_{\mathrm{TV}}^{\max}(\pi,\tilde{\pi})^2$ which we aims to optimize. However, note that this lower bound $\mathcal{L}$ can not be directly optimized due to the unknown coefficient $C$ and the maximum operator. Otherwise we could directly obtain the conclusion that TV divergence is the better choice compare to KL divergence, as TV divergence lower bound is tighter. However, one fact is that we optimize the lower bound $\mathcal{L}$ indirectly by maximizing $L_{\pi}(\tilde{\pi})-L_{\pi}(\pi)=E_{s\sim{\rho_{\pi}(\cdot)},a\sim{\pi(\cdot|s)}}\left[\frac{\tilde{\pi}(a|s)}{\pi(a|s)}\cdot A_{\pi}(s,a)\right]$ while trying to constrain the TV divergence $D_{\mathrm{TV}}^{\max}(\pi,\tilde{\pi})^2$. Therefore, we need to use Proposition 3.2 and Theorem 3.3 to demonstrate that, in this indirect optimization case, the TV divergence is still the better choice. In other words, there are cases where the TV divergence is large, but the $L_{\pi}(\tilde{\pi})-L_{\pi}(\pi)$ is much larger. Therefore, even if the TV divergence is not bounded, this is still a good lower bound. However, this situation is not within the scope of discussion for commonly used policy improvement algorithms due to the unknown coefficient $C$ mentioned earlier. We hope this can address your concern.

• Secondly, although previous work has made efforts to constrain the TV divergence, our work provides strong theoretical guarantees for constraining the probability ratio. This is because our new objective $f_{\mathrm{spo}}=rA-\frac{|A|}{2\epsilon}\cdot(r-1)^2,\enspace r=\frac{\tilde{\pi}(a|s)}{\pi(a|s)}$ has been proven to be convex with respect to the probability ratio $r$, and the maximum is achieved exactly at the constraint boundary $r=1+\mathrm{sign}(A)\epsilon$. Meanwhile, constraining the probability ratio indirectly constrains the average Total Variation (TV) divergence under the training dataset, which is $D_{\mathrm{TV}}^{\max}(\pi,\tilde{\pi})^2=\frac{1}{4}\max_{s}\left[{E}_{a\sim\pi(\cdot|s)}\left|\frac{\tilde{\pi}(a|s)}{\pi(a|s)}-1\right|\right]^2$ Since the max operator needs to consider all possible states, it is generally relaxed by using heuristic approximations to replace the max operation with an average operation (the expectation). Nevertheless, our objective involves imposing probability ratio constraints on each data point $(s_t,a_t)$, which results in stricter constraints compared to heuristic approximations and aligns more closely with the max operator in the lower bound $\mathcal{L}$.

• Lastly, we would like to quote reviewer ESWe and reviewer Dgb4's comments related to the use of TV divergence constraints in our paper: (ESWe) Although several papers have already pointed out issues with the heuristic clipping technique in PPO and proposed alternatives, I find the authors' approach offers a novel perspective, leading me to lean towards acceptance. (Dgb4) The paper is clearly written and provides strong theoretical grounding. New theorems and proofs substantiate the methodology, ensuring that the approach is technically sound.

II. Why does PPO Perform Worse with Deeper Policy Networks?

The authors' decision to use hyperparameters from the original PPO paper seems unfair, considering that changes in the setup necessitate retuning.

Could the authors provide a more compelling reason for why PPO in their experiments does not benefit from better representation learning, beyond the claim that PPO's performance deteriorates with an increasing number of parameters? Could tuning hyperparameters help PPO in this case?

In figure 7, is it possible to match the curve for SPO(epsilon=0.2) if we choose different epsilon for PPO?

Thank you for your thoughtful review and the insightful questions! We would like to provide a more detailed explanation regarding why PPO in our experiments does not seem to benefit from better representation learning, and whether tuning hyperparameters could help in this case.

• Performance Decline with Increased Parameters. Firstly, we provide an explanation for this phenomenon in the main text, suggesting that the clipping operation in PPO can result in the gradients of some data points being zero. A potential issue is that these data points can still affect the performace improvement lower bound $\mathcal{L}=L_{\pi}(\tilde{\pi})-L_{\pi}(\pi)-C\cdot D_{\mathrm{TV}}^{\max}(\pi,\tilde{\pi})^2=E_{s\sim{\rho_{\pi}(\cdot)},a\sim{\pi(\cdot|s)}}\left[\frac{\tilde{\pi}(a|s)}{\pi(a|s)}\cdot A_{\pi}(s,a)\right]-\frac{C}{4}\max_{s}\left[{E}_{a\sim\pi(\cdot|s)}\left|\frac{\tilde{\pi}(a|s)}{\pi(a|s)}-1\right|\right]^2$ even if they produce zero gradients, because all the current data come from the sampling of the old policy, and none of them can be ignored. Therefore, the clipping operation used by PPO is likely to cause the latter term in the lower bound to become very large. This is because these zero-gradient data points can be influenced by the gradients produced by non-zero gradient data points during further training, as the parameters of the policy are constantly changing. Optimizing the first term in the lower bound will inevitably push the current policy further away from the old policy. This situation is common during PPO training. While deeper networks have better representation capabilities, they are also more sensitive to parameter changes (with a larger Lipschitz constant). Therefore, even though deeper networks can allow for better representation learning, if the lower bound of policy improvement is poor or even negative, it can lead to worse performance.

• Hyperparameter Tuning. We have added more experiments to support our conclusions, including selecting different hyperparameters for PPO (Figure 6) and different $\epsilon$ (Figure 7), as well as further comparisons with other methods that restrict the magnitude of policy updates (Figure 8). These results indicate that naively reducing epsilon in the clipping objective does not overcome the issue of ratio deviation. Additionally, the comparative experiments between SPO and PPO with different hyperparameters and network depths indeed confirm the superiority and robustness of SPO's performance.

[1] S Huang et al. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. https://github.com/vwxyzjn/cleanrl

Dear Reviewer Xchv,

We have addressed your concerns and provided additional experimental results. What do you think of our response as well as new results? We also kindly ask you to consider stronger support for the paper if your concerns have been addressed.

Thank you very much.

Sincerely, The Authors of Paper #3422