Simple Policy Optimization

Dear Reviewer uY8N,

Thank you for your positive feedback. Below, we will address your concerns.

Figure 10: The results on Atari are averaged over only 3 random seeds, which I think is not sufficient.

Thank you for your suggestion. Given the high computational costs associated with Atari environments, the use of 3 random seeds represents a well-established practice in the field [2]. Furthermore, due to time constraints during the rebuttal period, we regret that we are unable to conduct additional experiments with more random seeds.

The results in Figure 8 look stronger than the MuJoCo results and would be better if put in the main paper instead of the appendix.

Thank you for your suggestion! Due to space constraints, we have included the key content in the main text (Figure 4 compares a broader range of baselines). If the paper is accepted, we will also incorporate Figure 8 into the main body.

Could you specify which implementation of PPO you are using (which implementation SPO is based on)?...

Our implementation of PPO is based on the standard CleanRL library: https://github.com/vwxyzjn/cleanrl, with the only modification being the computation of the policy loss in SPO. The complete code for this study has been submitted as part of the supplementary materials.

Have you tried scaling the width of the network besides the depth? I'm curious because a recent paper has shown that scaling the width can yield significant performance improvements on its own [1].

Thank you for the additional paper. Our three-layer network follows the default settings: [$\mathrm{dim}(\mathcal{S})$, 64, 64, $\mathrm{dim}(\mathcal{A})$], while the seven-layer network is both deeper and wider in architecture: [$\mathrm{dim}(\mathcal{S})$, 256, 256, 128, 128, 64, 64, $\mathrm{dim}(\mathcal{A})$].

Do you have any idea why PPO tends to perform worse and lose control of the probability ratio as the network scales?

Certainly! We'd be happy to provide a simplified explanation. Generally speaking, deeper networks tend to have larger Lipschitz constants, meaning that even minor changes in the network's input can lead to significant variations in its output. Consequently, when using deeper architectures, PPO's limitations become more pronounced, as slight parameter adjustments may cause the probability ratio to exceed its boundary.

Could you show the scaling performance in Atari? It would be more convincing if there are multiple benchmarks where the results support the claim.

Thank you for your suggestion. We primarily demonstrate the scaling performance in Atari environments through Figures 1 and 6. Specifically, Figure 1 shows that SPO can effectively train ResNets with over 100 layers, while Figure 6 highlights SPO's significant performance improvements using ResNet-18 across four Atari environments.

Additionally, due to the limited rebuttal period, we may require additional time to obtain the Atari results, potentially during the discussion phase.

Figure 2 (right): Could you specify the task on which the results are produced?

Sure! We generated Figure 2 using the Hopper-v4 environment. In fact, similar results can be reproduced in any environment (e.g., Humanoid-v4).

Best,

Authors

---

Reference:

[1] J Obando-Ceron et al. In value-based deep reinforcement learning, a pruned network is a good network.

[2] Y Gan et al. Reflective policy optimization.

Dear Reviewer eeAe,

Thank you for your constructive feedback. Below, we will address your concerns.

Partially. The claim regarding improved theoretical properties is plausible, but empirical support for SPO’s superiority over PPO and TRPO is limited.

The paper lacks a clear demonstration of significant performance improvements over PPO, with most experimental results showing only marginal gains.

Include a comparison with TRPO, as the paper suggests that SPO combines the advantages of both TRPO and PPO. A head-to-head comparison would strengthen this claim.

Thank you for your suggestion. However, as demonstrated in Figures 1, 4, and 5 of our paper, our experiments show that SPO indeed exhibits significant advantages over PPO. Notably, Figure 4 includes comparisons with a wide range of baselines. Unfortunately, due to TRPO's computationally expensive second-order optimization, it is typically not used as a baseline in existing works [1, 2, 3].

Claims regarding the theoretical advantages of SPO over PPO and TRPO are not fully substantiated with rigorous empirical validation.

Thank you for your feedback. As shown in Table 1 of our paper, when PPO trains deeper networks, the probability ratio becomes uncontrolled and leads to performance collapse in PPO. Please note that in this experiment, the only variable modified was the network depth - no additional code-level tuning was performed for SPO. We believe this strongly demonstrates SPO's theoretical advantage, as its probability ratio remains effectively constrained without performance degradation even with increasing network depth.

Key implementation details and hyperparameters (e.g., the reasoning behind comparing 3-layer vs. 7-layer architectures) are insufficiently explained...

In the experiments, provide more details on architectural choices (e.g., why 3-layer and 7-layer networks were compared)...

Thank you for your suggestion. The reviewer's concern primarily relates to the fairness of the experimental setup and the potential possibility that SPO might involve additional hyperparameters or code-level tuning. Regarding this point, please refer to our already uploaded SPO and PPO implementations, where you will find that the only difference between SPO and PPO lies in the policy loss computation (the sole modification in the trainer.py file, with all other settings remaining identical).

As for the network depth selection, we employed both the default three-layer mlp [$\mathrm{dim}(\mathcal{S})$, 64, 64, $\mathrm{dim}(\mathcal{A})$] and a randomly selected seven-layer mlp [$\mathrm{dim}(\mathcal{S})$, 256, 256, 128, 128, 64, 64, $\mathrm{dim}(\mathcal{A})$] to ensure comprehensive evaluation.

Several experimental setups (e.g., Figure 6) lack clarity in terms of legend explanations and parameter settings. Additionally, the small number of random seeds (only three) undermines the statistical robustness of the results.

Thank you for your careful review. In Figure 6, the red dashed line represents the value of 0.2, while all other elements are clearly indicated in the legend.

Regarding the random seeds, due to the computational costs in Atari environments, using 3 seeds is a common choice [4]. Furthermore, for the MuJoCo environments, the results presented in Figure 4 were indeed obtained using 10 random seeds to ensure statistical reliability.

Q2. Is there a tunable parameter in SPO that allows interpolation between PPO-like and TRPO-like behavior?...

Please note that TRPO and PPO exhibit fundamentally different optimization behaviors. SPO combines TRPO's theoretical guarantee (monotonic improvement) with PPO's computational efficiency (eliminating the need for second-order optimization).

Q3. What is the meaning of the values shown in the legends of Figures 5 and 6 (e.g., 0.1, 0.2)?...

The red dashed lines in these figures represent the hyperparameter epsilon (typically set to 0.2), and we will add the corresponding legend in subsequent revisions. The remaining horizontal lines, which are already labeled in the legend, indicate the performance of the original CNN at convergence (serving as baseline references).

Q4. Can you clarify the meaning of Theorem 5.1 (that SPO is $\epsilon$-aligned)? How does this translate into practical advantages in policy optimization?

Please refer to the performance improvement lower bound (10). Theorem 5.1 demonstrates that SPO can indirectly optimize this lower bound (10) by constraining the probability ratio deviation $\|\frac{\tilde{\pi}(a|s)}{\pi(a|s)}-1\|\leq\epsilon$.

Best,

Authors

---

Reference:

[1] K Cobbe et al. Leveraging procedural generation to benchmark reinforcement learning.

[2] Y Wang et al. Truly proximal policy optimization.

[3] Y Wang et al. Trust region-guided proximal policy optimization.

[4] Y Gan et al. Reflective policy optimization.

Dear Reviewer 8JyT,

Thank you for your comment. Below, we will address your concerns.

The authors claim that "TV divergence offers a larger solution space compared to methods incorporating a looser KL divergence constraint." However, the fundamental issue is that when optimizing the surrogate objective loss, the paper does not actually enforce a KL divergence constraint. Specifically, in equation (13), restricting the solution space to $\tilde{\pi}\in\Omega_ {\mathrm{TV}}$ is incorrect. The authors seem to have confused constraint-based optimization objectives with surrogate loss-based objectives.

Thank you for your comment. However, we respectfully disagree with this perspective. In our approach, we introduce the SPO objective in Equation (16) to constrain the probability ratio, thereby ensuring the $\epsilon$-aligned property defined in Definition 5.1. Furthermore, as established in Equation (9), constraining the probability ratio across batch data inherently bounds the total variation (TV) divergence, which naturally leads to a TV divergence-based trust region. Importantly, Theorem 4.3 demonstrates that the performance improvement lower bound under TV divergence constraints is indeed superior to that under KL divergence constraints.

The derivation from Eq. (17) to Eq. (18) does not appear rigorous. Specifically, replacing the absolute value $|r-1|$ with the squared term $(r-1)^2$ significantly changes the nature of the function. The paper lacks an analysis of the divergence introduced by this transition from Eq. (17) to Eq. (18). Additionally, it is unclear whether this modification will lead to actual policy performance improvements.

Definition 5.1 does not appear clearly motivated or sufficiently rigorous. The notion of an "$\epsilon$-aligned" surrogate is not strong enough to guarantee policy improvement. In fact, many surrogate functions can satisfy the condition of being "$\epsilon$-aligned" yet fail to yield any meaningful performance improvement. Therefore, the practical usefulness and theoretical significance of this definition remain unclear.

Can you rigorously justify why the absolute deviation (eq. 18) is preferred over squared deviation, given significant conceptual differences?

This might be a misunderstanding of our method. We employ the squared penalty because it is convex with respect to the probability ratio $r$, and its optimal solution naturally lies at the probability ratio boundary $r^*=1+\mathrm{sign}(A)\cdot\epsilon$. This demonstrates our method's effectiveness in constraining the probability ratio, which consequently leads to indirect optimization of the policy's performance improvement lower bound (10).

How do you explain the contradictory performances of SPO in Figures 1 and 8?

Please note that Figure 1 presents results using 50 to 101-layer ResNet architectures, while Figure 8 employs the default shallow CNN structure for comparison.

What are the detailed reasons for selecting only 35 out of the 57 available Atari environments, and how were these environments chosen?

We deliberately selected a subset of Atari environments where PPO can operate successfully. Among the 57 Atari games, some involve sparse reward settings where neither PPO nor SPO can effectively train reinforcement learning policies. Furthermore, given the substantial computational overhead of Atari experiments, running all environments would be prohibitively time-consuming. We maintain that the environments included in our experiments provide sufficient basis for meaningful algorithmic comparison.

Could you explicitly compare the performance of SPO with PPO-penalty, clearly identifying whether TV divergence or other modifications provide the primary benefit?

Thank you for your suggestion. Figure 4 in the paper includes comparative experiments between PPO-Penalty and other extensive baselines, with the results clearly demonstrating SPO's performance advantages over both PPO-Clip and PPO-Penalty in MuJoCo environments (10 seeds).

Best,

Authors

Dear Reviewer SqU8,

Thank you for your constructive feedback. Below, we will address your concerns.

PPO is widely used in large-scale RL problems, and KL regularization will be incorporated when the policy NN is complex, such as LLM...

We appreciate your suggestion. However, the purpose of KL regularization in PPO for LLMs is to prevent reward over-optimization and catastrophic forgetting caused by policy collapse. This does not prevent PPO's ratio deviation issue, as ratio deviation is defined with respect to the initial policy $\pi_ {\theta_ {\mathrm{old}}}$ at each update. Additionally, due to the limited rebuttal period, we may not be able to provide comparative results of SPO-fine-tuned LLMs. We sincerely apologize for this limitation.

Another interesting ablation will be to decrease PPO's $\epsilon$ or increase SPO's $\epsilon$ so that their average ratio deviation is the same and see if the ratio is the main reason for SPO's better performance.

An ablation on SPO's $\epsilon$ will help the readers better understand its robustness.

Thank you for your comment. We have supplemented the following experiments:

The results demonstrate that over-large $\epsilon$ (0.5) consistently leads to performance degradation, which aligns with the theoretical lower bound.

The paper [1] also studies replacing the divergence term to TV divergence but is not discussed.

Thank you for pointing out. We will include this in the related work section in our subsequent revisions.

The related works that use TV divergence or more general divergence are not thoroughly discussed.

We appreciate the suggestion. However, to our knowledge, [2] is the only existing work that discusses the relationship between total variation (TV) divergence and the policy ratio, we have discussed this paper in our related work section. KL divergence is first proposed in original TRPO paper [3]. To the best of our knowledge, beyond TV divergence and KL divergence, there appear to be no published works employing more general divergence measures in this context.

I didn't observe obvious typos. Authors may consider replace "can not" to "cannot".

Thank you for your thorough review. We will carefully incorporate your suggestions.

Can the authors comment on the potential drawbacks of the proposed method or, more generally, using TV divergence, compared to PPO and TRPO, such as instabilities? And will add KL regularization to PPO make it better?

In fact, we compared a PPO variant with KL regularization (PPO-Penalty), as shown in Figure 4 of our paper. The results demonstrate that SPO achieves superior performance.

Best,

Authors

---

Reference:

[1] X Chu et al. A strong on-policy competitor to PPO.

[2] J Queeney et al. Generalized proximal policy optimization with sample reuse.

[3] J Schulman et al. Trust region policy optimization.