Reflective Policy Optimization

We are profoundly grateful for the time you have taken from your busy schedule to provide a detailed evaluation of our work. We sincerely appreciate your professional feedback. We are committed to addressing the issues raised and are confident that our revisions will significantly improve the contribution of our work to the RL community.

Question 1: In the ablation experiment for k, when k increases the variance of the objective function will increase significantly, which puts forward a high requirement for fine adjustment of $\epsilon$ and $\beta$. So I want to know what is the setting of these hyperparameters in the ablation experiment. How to set such hyperparameters to get a reliable result, choosing the right k?

Answer 1: Your point is valid. In the ablation study, as shown in Fig. 5, when k=3, the objective function contains five hyperparameters, which take the values of ($\epsilon=0.2, \epsilon_1=0.1, \beta_1=0.3, \epsilon_2=0.1,\beta_2=0.1$). We fixed $\epsilon, \epsilon_1, \beta_1$ and $\epsilon_2$, and did not extensively fine-tune the hyperparameters, only $\beta_2\in${0.1, 0.3}. Considering the variance factor, we adjust the hyperparameters according to the following rules: $\epsilon\geq\epsilon_1\geq\epsilon_2\geq \cdots$ , and $\beta_1\geq\beta_2\geq\cdots$ . In this way, a more reliable result can be obtained. To verify the effectiveness of the RPO and how to choose k, we conducted experiments on the CliffWalking (see Fig. 2). It was found experimentally that when k = 3, the results obtained were a little better than when k = 2, but the improvement was not significant. Again considering the factor of the number of hyperparameters, we suggest a choice of k of 2.

Question 2: (The first point in Weaknesses) It is better to add some experiments on the environments with discrete action space, e.g. MinAtar[1].

Answer 2: We sincerely accept your expert guidance. Firstly, we have thoroughly read your reference: Young, K. & Tian, T. (2019). MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments. arXiv preprint arXiv:1903.03176. We have conducted additional experiments in the discrete action environment. We are immensely grateful for the expert guidance you provided on the MinAtar environment, as well as the relevant literature you recommended. The content of the literature has offered us a substantial opportunity for in-depth learning. Following your suggestions, we have extended our experimental scope to the more challenging Atari environment beyond MinAtar, aiming to demonstrate the comprehensive performance of our RPO algorithm. We look forward to your evaluation of these additional experimental results. We randomly selected twelve Atari environments for these experiments. The results of the MinAtar and Atari experiments have been incorporated into the new version of our paper for your review.

Question 3: (The second point in Weaknesses) The gap between the original reflective policy optimization and the clipped version cannot be ignored.

Answer 3: We also recognize the phenomenon you pointed out, which is indeed critical and important. However, our experiments show that our clipping operation did not significantly alter the optimal policy but rather made the training more stable during policy updates in most environments.

Question 4: (The third point in Weaknesses) The method is sensitive to some important hyperparameters, as shown in Figure 4, in Swimmer and Walker2d.

Answer 4: We are deeply appreciative of your thorough and professional insights. Indeed, as you have pointed out, we have observed sensitivity to parameter variations in specific scenarios. Through ablation studies, it gives some insight into fine-tuning the parameters of the algorithm. For some environments, we should reduce the parameter $\beta$ value, reducing the sensitivity of the parameters.

Once again, thank you for your thorough review and valuable comments on our work. We look forward to your further guidance.

Question 6: (Original question 3) Could you tell me if there was any correlation between the GAE parameters and multi-step k in experiments?

Answer 6: In this paper, GAE is evaluated in the same way as PPO, i.e., $\hat{A}^{GAE}_t=\delta^t+(\gamma \lambda)\delta^{t+1}+\cdots+(\gamma \lambda)^{T-t+1} \delta^{T-1}$,

where $\delta^t=r_t+\gamma V(s_{t+1})-V(s_t)$.

Assume that the sampled trajectory is $\tau = (s_0,a_0,s_1,a_1, \cdots, s_t, a_t, \cdots)$, we can calculate the advantage functions ($\hat{A}^{GAE}_0, \hat{A}^{GAE}_1, \cdots, \hat{A}^{GAE}_t, \cdots$).

When k=2, if we sample ($s_t, a_t, s_{t+1}, a_{t+1}$), the current policy is optimized by ($\hat{A}^{GAE}_t$,

$\hat{A}^{GAE}_{t+1}$). It can be seen that the relationship between them is that when k is fixed, the number of advantage functions is also fixed.

Question 7: (Original question 4) The performance is only shown in three environments about with the value of k.

Answer 7: Previously, due to limited resources and time, we did not perform parameter tuning for k=3. We only ran some experiments in three environments and included these results in our paper. Now, we have supplemented this experiment in the Appendix.

Thank you very much for your insightful feedback. Receiving such professional guidance from you is indeed a privilege for us. We are committed to addressing the issues raised and are confident that our revisions will significantly improve the contribution of our work to the RL community. We sincerely request that you reconsider your evaluation of our manuscript and consider revising your score accordingly. Your continued support and feedback are invaluable to us.

Question 1: (Weaknesses 1) The performance shown in the experiments does not seem to be much different from the performance of the baselines.

Answer 1: We are grateful for your query about the performance improvement and apologize for not articulating this clearly in our paper. In our experiments, we found that our method has a significant advantage over the original PPO. However, to showcase the robustness and broad applicability of our approach, we did not perform specific parameter tuning. The results post-tuning will be detailed in the paper's appendix to transparently exhibit RPO’s performance. In Fig. 3, our method is compared not only with the PPO algorithm but also with some algorithms that have modified PPO from an on-policy to an off-policy approach (GePPO and OTRPO) and it is found that our method is also advantageous. Figure 1 shows that the direct use of subsequent data plays a positive role and our approach can be seamlessly integrated into existing algorithm frameworks.

We once again apologize for any confusion caused by our lack of clarity in conveying that specific parameter tuning was not conducted to demonstrate the robustness and general adaptability of our method. Please forgive any inconvenience this may have caused during your review. Your guidance is invaluable, and we will take greater care in this regard in our subsequent paper submissions.

Question 2: (Weaknesses 2) The proposed environment is more appropriate to be considered in a constrained RL or safe RL.

Answer 2: Thank you for your approval of our proposed algorithm. Our proposed method serves as a supplement to existing reinforcement learning approaches. It is not only applicable to classic reinforcement learning scenarios but may also play a more significant role in constrained reinforcement learning or safe reinforcement learning. It also requires further research on this algorithm in order to better apply it in different environments. It will be a point that can be studied in the future.

Question 3: (Weaknesses 3) About the motivation of the proposed method.

Answer 3: I am very sorry that we did not make this example more straightforward. We will modify it in the introduction. With this example, we want to consider the impact of the subsequent data directly. The value function potentially contains information about the subsequent data. We need to consider the question: Is it the best way to optimize a policy using only value functions? The answer is definitely not. There is another way to optimize the current policy by directly using the subsequent data, unlike optimizing the policy using only the value function. Intuitively, the direct use of the subsequent data may speed up the algorithm's convergence and improve sample efficiency. In the subsequent sections of this paper, we verify this intuition theoretically and experimentally, respectively.

Question 4: (Weaknesses 4) Sensitivity of hyperparameters.

Answer 4: Fine-tuning the hyperparameters of our algorithm indeed leads to improved performance. In the main experiment (as shown in Figure 3), we fixed the hyperparameters without fine-tuning for a fair comparison. From Fig. 3, even with fixed hyperparameters, our approach achieves good results compared with PPO and the off-policy version of PPO.

Question 5: (Original question 2) About theorem 3.1 and 4.1, can you explain what the difference is, or can you compare the difference experimentally?

Answer 5: Consider $L_1(\pi, \hat{\pi})$ of the theorem 3.1 as an example. If the environment is unknown, it can only be optimized by sampling. If we sample a trajectory $\tau$, and $(s_0, a_0, s_1, a_1)\in\tau$, if $A^{\hat{\pi}}(s_1, a_1)<0$ and $r_0-1 <0$, we know that $(r_0-1)r_1 A^{\hat{\pi}}(s_1, a_1)>0$. Optimizing the $L_1(\pi, \hat{\pi})$ (considering the extreme case, using one sample), the probability of $a_1$ is increased. However, $A^{\hat{\pi}}(s_1, a_1)<0$, we should decrease the probability of $a_1$. It's a contradiction. Thus this term "1" of $r_0-1$ may adversely affect policy optimization though the theory is sound. This situation exists when the environment is unknown. But for $\hat{L}_1(\pi, \hat{\pi})$ in the theorem 4.1, there is no such problem. We also experimentally compared these two methods, i.e., RPO vs.* TayPO (with the term ”1"). The possibility of what we said earlier is also verified from the experiment.

Dear WKkT,

The author reviewer discussion period is ending soon this Wed. Does the author response clarify your concerns w.r.t., e.g., the empirical aspects of this work, or there are still outstanding items that you would like to see more discussion?

Thanks again for your service to the community.

Best,
AC

Thanks again for your time and effort in reviewing our paper! To further allay your concerns, we add some Atari experiments. We remain open to any further feedback and are committed to making additional improvements if needed.

Table 1 Comparison of RPO and PPO performance at 60% timesteps in MuJoCo.

Table 2 Comparison of RPO and PPO performance at 60% timesteps in Atari.

In order to better present the advantages of this method RPO over PPO, in addition to our experiments on MuJoCo, we randomly selected 12 Atari environments for our experiments (runs 50 million, 3 seeds). As shown in Tables 1 and 2, our proposed algorithm RPO achieves better results compared to PPO at the 60% timesteps in MuJoCo and Atari. For more information, please see the appendix.

Thank you very much for your insightful feedback. We greatly appreciate your constructive suggestions, which have provided valuable guidance for enhancing the quality of our manuscript. We are committed to addressing the issues raised and are confident that our revisions will significantly improve the contribution of our work to the RL community. We sincerely request that you reconsider your evaluation of our manuscript and consider revising your score accordingly. Your continued support and feedback are invaluable to us.

Question 1: About the examples of “cliff” and “treasure” stated in the second paragraph of the Introduction Section. Would the value function suffice whether a state is favorable?

Answer 1: I am very sorry for not making this example clear. We will modify it in the introduction. Let's briefly restate the example. Consider an environment with a "cliff." what would an agent do if it performed an action under a state and fell into a "cliff"? This action is dangerous. Meanwhile, this state might also be dangerous due to the fact that the next time the agent reaches this state again, it is likely to perform the same action. Hence, the agent also needs to avoid getting to this state again and keep out of this state as much as possible. The previous action, when reaching this state, the previous action also needs to directly avoid being performed because it is possible to fall into that state again. The same result is found for the "treasure" environment. Hence, it is necessary to optimize the previous action directly with the subsequent state-action pairs information, not only through the value function, thereby improving sample efficiency.

Yes, this way is feasible if the policy is optimized using the value function of the current state due to the fact that the value function potentially contains information about the subsequent states. We need to consider the question: Is it the best way to optimize a policy using only value functions? The answer is definitely not. What we are trying to convey with this example is that there is another, better way to optimize the current policy by directly using the subsequent data, unlike optimizing the policy using only the value function. Intuitively, the direct use of the subsequent data may speed up the convergence of the algorithm and improve sample efficiency. In the subsequent sections of this paper, we verify this intuition theoretically and experimentally, respectively.

Question 2 : About the citep & citet.

Answer 2: Thank you very much for pointing this out, we revise it in the next version.

Question 3 : In the second line of $\eta(\pi)-\eta(\hat{\pi})$, the first term on RHS should be with ($s_0, a_0$).

Answer 3: Thank you very much, we revise it in the next version.

Question 4 : The $\|\pi-\hat{\pi}\|_1$ has not been defined in the paper.

Answer 4: Thank you very much, we revise it. $\|\pi-\hat{\pi}\|_1$ is defined as $\max_s\sum_a|\pi(a|s)-\hat{\pi}(a|s)|$.

Question 5 : About $(s_0, a_0)\sim \rho$, and $(s_i, a_i)\sim \rho(\cdot|s_{i-1},a_{i-1})$.

Answer 5: I'm very sorry, I didn't make that clear. Let us explain in detail below. From the definition of $\rho(s)$ in Preliminaries Section, we see that $\mathbb{P}(s_t=s|\rho_0, \pi)$ has Markov properties. So, $\rho^{\pi}(s)$ has the same properties. From the proof of Theorem A.1 (see appendix), the link between the performance difference of $\pi$ and $\hat{\pi}$ and state visit distribution and conditional state visit distribution has been established. We know that $\rho(s_i|s_{i-1}, a_{i-1})= \rho(\cdot|s_{i-1}, a_{i-1}, s_{i-2}, a_{i-2}, \cdots, s_0, a_0)$, which is the conditional distribution under $s_{i-1}$ and $a_{i-1}$, and $s_{i-1}\sim\rho(\cdot|s_{i-2}, a_{i-2})$ and $a_{i-1}\sim \pi(\cdot|s_{i-1})$. For example, given the distributions of $\rho(s_0)$ and $\pi(a_0|s_0)$, $\rho(s_1|s_{0}, a_{0})$ must exist according to Bayesian theory, but the exact form is difficult to give directly. Their relationship is $\rho(\cdot) = \int \rho(\cdot|s_0, a_0) \rho(s_0) \pi(a_0|s_0) ds_0da_0$. For the general case, we have $\rho(\cdot) = \int \rho(\cdot|s_{i-1}, a_{i-1})\rho(s_{i-1}|s_{i-2}, a_{i-2})\pi(a_{i-1}|s_{i-1})\cdots \rho(s_0) \pi(a_0|s_0) ds_{i-1}da_{i-1}\cdots ds_0da_0$.

Question 6 : (Weaknesses) about $(r_0-1)r_1 A^{\hat{\pi}}(s_1, a_1) = [(r_0-1) A^{\hat{\pi}}(s_1, a_1)]\cdot r_1>0$ ... present a contradiction. Optimizing this objective can result in a bit more complicated consequence than what’s been stated here.

Answer 6: Your point is valid. From the perspective of parameter optimization, optimizing this objective be considered a more complex situation. In this paper, we consider this function intuitively without focusing on the specific form of the parameters. When the model is unknown, we optimize this objective function by sampling and give intuitively the possible problems which there is no way to avoid.

Question 7 : (Weaknesses) about the monotonic improvement statement of Theorem 4.1.

Answer 7: Thank you for providing a proof, the approach of proof is correct. We have identified a flaw in one of the proof steps, i.e., it is not true that $E_{s_0, a_0\sim\rho^{\hat{\pi}}}[(r_0-1) A^{\hat{\pi}}(s_0, a_0)]\frac{\|\pi-\hat{\pi}\|_1^2R\max}{1-\gamma}$,

but rather that $E_{s_0, a_0 \sim \rho^{\hat{\pi}}}[(r_0-1) A^{\hat{\pi}}(s_0, a_0)] \leq \frac{\|\pi-\hat{\pi}\|_1 R\max}{1-\gamma}$.

We can see that $G_1(\pi, \hat{\pi}) = E_{s_0, a_0 \sim \rho^{\hat{\pi}}(\cdot)}(r_0-1) E_{s_{1}, a_{1} \sim \rho^{\pi}(\cdot|s_{0}, a_{0})}A^{\hat{\pi}}(s_1, a_1)$ is different from $E_{s_0, a_0 \sim \rho^{\hat{\pi}}}[(r_0-1) A^{\hat{\pi}}(s_0, a_0)]$.

Therefore, the results of $G_1$ cannot be used directly. We give the complete proof below. $E_{s_0, a_0 \sim \rho^{\hat{\pi}}}[(r_0-1) A^{\hat{\pi}}(s_0, a_0)] - \frac{ \gamma R\max}{(1-\gamma)^2}\|\pi-\hat{\pi}\|_1^2\leq \frac{ R\max}{1-\gamma}\|\pi-\hat{\pi}\|_1 -\frac{ \gamma R\max}{(1-\gamma)^2}\|\pi-\hat{\pi}\|_1^2$.

When $0<\|\pi-\hat{\pi}\|_1<\frac{1-\gamma}{\gamma}$, the upper bound is greater than zero. So the lower bound of performance has solutions.

Dear KkoJ,

The author reviewer discussion period is ending soon this Wed. Does the author response clarify your concerns about technical correctness and presentation, or there are still outstanding items that you would like to see more discussion?

Thanks again for your service to the community.

Best,
AC

Thank you for your feedback. We greatly appreciate your constructive suggestions, which have provided valuable guidance for enhancing the quality of our manuscript. We will answer your questions as best we can. Lastly, we thank you for taking the time to engage with our work.

Re-Question 6: Rigorous and flawless descriptions should be given as an example.

Re-Answer 6: I am very sorry for not making this example clear. I've added a more rigorous description in our manuscript, and we've explained it to you to make it easier for you to understand.

Consider $L_1(\pi, \hat{\pi})$ in Eqn. (2) as an example. We consider this function without focusing on the specific form of the parameters. When the environment is unknown, it can only be optimized by sampling. Considering the extreme case, the function $L_1(\pi, \hat{\pi})$ is optimized by using a sample $(s_0, a_0, s_1, a_1)$, i.e., $L_1(\pi, \hat{\pi})\approx(r_0-1)r_1 A^{\hat{\pi}}(s_1, a_1)$.

If $A^{\hat{\pi}}(s_1, a_1)<0$ and $r_0-1 <0$, we know that $(r_0-1)r_1 A^{\hat{\pi}}(s_1, a_1)=[(r_0-1)A^{\hat{\pi}}(s_1, a_1)]r_1>0$. The probability of $a_1$ is increased. However, when $A^{\hat{\pi}}(s_1, a_1)<0$, we should decrease the probability of $a_1$. It's a contradiction. Thus, this term "1" of $r_0-1$ may adversely affect policy optimization, though the theory is sound. This situation exists when the environment is unknown.

Re-Question 7 : (Weaknesses) About the monotonic improvement statement of Theorem 4.1.

Re-Answer 7: Firstly, we define $\rho(\cdot|s, a)$ from another perspective.

Inspired by discount state visitation distribution $\rho(\cdot)$, i.e., $\rho^{\pi}(s) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\mathbb{P}(s_t=s|\rho_0, \pi),$ we can similarly give a definition of $\rho^{\pi}(s,a,s')$. It is defined as $\rho^{\pi}(s,a,s') = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\mathbb{P}(s_t=s,a_t=a,s_{t+1}=s'|\rho_0, \pi),$ where $\mathbb{P}(s_t=s,a_t=a,s_{t+1}=s'|\rho_0, \pi)=\mathbb{P}(s_t=s|\rho_0, \pi)\pi(a_t=a|s_t=s)P(s_{t+1}=s'|s_t=s,a_t=a),$ Further, we can give the definition of $\rho^{\pi}(\cdot|s, a)$, i.e., $\rho^{\pi}(s'|s, a)=\frac{\rho^{\pi}(s, a,s')}{\rho^{\pi}(s, a)}$.

By this way, $\rho^{\pi}(s_2|s_1, a_1)=\frac{\rho^{\pi}(s_0,a_0,s_1,a_1,s_2)}{\rho^{\pi}(s_1,a_1|s_0,a_0)\rho^{\pi}(s_0,a_0)}$. Similarly, we can define $\rho^{\pi}(s_i|s_{i-1}, a_{i-1})$.

Secondly, thank you for re-providing proof. The approach of proof is correct. Moreover, we have identified a flaw in one of the proof steps. The penalty term in the lower bound is $\frac{ \gamma R\max }{(1-\gamma)^2}\|\pi-\hat{\pi}\|_1^2$ instead of $\frac{ \gamma R\max }{(1-\gamma)^3}\|\pi-\hat{\pi}\|_1^2$ (see theorem 4.1 of my paper or theorem 1 of the TRPO paper [1]). So, the quadratic form takes the maximal when $\|\pi-\hat{\pi}\|_1=\frac{1-\gamma}{2\gamma}$.

Lastly, we are well aware of your concern that $\|\pi-\hat{\pi}\|_1$ may be too small. By the analysis of the theorem, we are more interested in the existence of solutions. In practice, as the TRPO paper says, "the KL divergence is bounded at every point in the state space, ..., is impractical to solve due to the large number of constraint". This issue is familiar to our approach but rather a challenge universally encountered by all lower bound algorithms, e.g., TRPO [1], CPO [2], GePPO [3], and so on.

[1] Schulman J, Levine S, Moritz P,et al. Trust Region Policy Optimization. ICML, 2015.

[2] Achiam J , Held D, Tamar A,et al. Constrained Policy Optimization. ICML, 2017.

[3] Queeney J , Paschalidis I C,et al. Generalized Proximal Policy Optimization with Sample Reuse. NeurIPS, 2021.

Thank you for your feedback. We would be honored to respond to each of your queries in detail. Lastly, we thank you for taking the time to engage with our work.

From theorem 4.1, when $k=1$, we can obtain the following inequality

$\eta(\pi)-\eta(\hat{\pi})\geq \alpha_0 E_{s_0, a_0 \sim \rho^{\hat{\pi}}}[r_0 A^{\hat{\pi}}(s_0, a_0)]- \hat{C}_1(\pi, \hat{\pi}),$

where $\alpha_0=\frac{1}{1-\gamma}$ and $\hat{C}_1(\pi, \hat{\pi})=\frac{\gamma R\max}{(1-\gamma)^3}\|\pi-\hat{\pi}\|_1^2$.

If the right-hand side of the above inequality is optimized as a whole, the penalty term is $\frac{\gamma R\max}{(1-\gamma)^3}\|\pi-\hat{\pi}\|_1^2$.

If we omit $\alpha_0$, the objective function we optimize is given by

$E_{s_0, a_0 \sim \rho^{\hat{\pi}}}[r_0 A^{\hat{\pi}}(s_0, a_0)]- \frac{\gamma R\max}{(1-\gamma)^2}\|\pi-\hat{\pi}\|_1^2,$

so, the penalty term is $\frac{\gamma R\max}{(1-\gamma)^2}\|\pi-\hat{\pi}\|_1^2$.

Thank you for your feedback. We must clarify, however, that our paper does not address the issues of "0" and "true gradient" mentioned in your review. Moreover, we have not received specific questions regarding our manuscript. Should you have any particular concerns about our work, we earnestly request that you point out the key aspects of the Weaknesses and any questions. We would be honored to respond to each of your queries in detail. Lastly, we thank you for taking the time to engage with our work.

Thank you for your feedback. I'm sorry I don't particularly understand what you're saying. We will do our best to answer your questions. Lastly, we thank you for taking the time to engage with our work.

Re-Question 1: Did you do that mixing Monte-Carlo with TD would be using subsequent experience? Accuracy of the value function?

Re-Answer 1: No, my proposed method doesn't use it. Our approach is in line with TRPO [1] and CPO [2]. The key of those method is to not update the policy too much, thus making the algorithm more stable, when the dynamic model is unknown.

Below, we we illustrate the differences between our proposed approach RPO and TRPO [1].

The objective function of TRPO [1] (derived rather than adding regular terms):

$E_{s,a}[\frac{\pi(a|s)}{\hat{\pi}(a|s)} A^{\hat{\pi}}(s, a)]- \hat{C}_{1}(\pi, \hat{\pi}),$

where $\hat{C}_{1}(\pi, \hat{\pi})=\frac{\gamma R\max}{(1-\gamma)^2}\|\pi-\hat{\pi}\|_1^2$.

When $k=2$, the objective function of my method RPO (derived rather than adding regular terms):

$E_{s,a}[\frac{\pi(a|s)}{\hat{\pi}(a|s)} A^{\hat{\pi}}(s, a)]+\alpha E_{s,a, s',a'}[\frac{\pi(a|s)}{\hat{\pi}(a|s)}\frac{\pi(a'|s')}{\hat{\pi}(a'|s')} A^{\hat{\pi}}(s', a')] - \hat{C}_{2}(\pi, \hat{\pi}),$

where $\alpha$ is a constant.

Our approach adds an additional term compared to TRPO. It can be seen that our method is very different from previous methods [1,2,3]. The methodology of previous studies [3,4,5] of sample efficiency does not directly apply in my case.

Note that the objective function of TRPO can be obtained in terms of adding the regular term (Bregman divergences, e.g., KL), but the original objective functions of TRPO is derived from policy performance (see paper TRPO [1] and my paper).

Re-Question 2: About 'r' and example.

Re-Answer 2: Thank you very much for your suggestion. We will use a different symbol to represent it.

We approach it from an optimization perspective, considering the action probabilities of the policy, and the penalty term prevents the policy updates from being too large. In the TayPO paper [6], they use $L_0(\pi, \hat{\pi})+L_1(\pi, \hat{\pi})$ as an objective function and omit the penalty term. Optimizing this objective function may pose issues. We consider this function without focusing on the specific form of the parameters. When the environment is unknown, it can only be optimized by sampling. Considering the extreme case, the function $L_1(\pi, \hat{\pi})$ is optimized by using a sample $(s_0, a_0, s_1, a_1)$, i.e., $L_1(\pi, \hat{\pi})\approx(I_0-1)I_1 A^{\hat{\pi}}(s_1, a_1)$.

If $A^{\hat{\pi}}(s_1, a_1)<0$ and $I_0-1 <0$, we know that $(I_0-1)I_1 A^{\hat{\pi}}(s_1, a_1)=[(I_0-1)A^{\hat{\pi}}(s_1, a_1)]I_1>0$. The probability of $a_1$ is increased. However, when $A^{\hat{\pi}}(s_1, a_1)<0$, we should decrease the probability of $a_1$. It's a contradiction. Thus, this term "1" of $I_0-1$ may adversely affect policy optimization, though the theory is sound. This situation exists when the environment is unknown.

Re-Question 4: About "sample efficiency" and "contribution ".

Re-Answer 4: I'm guessing you're asking both questions.

Sample efficiency refers to how many samples are needed to achieve the same level of performance. In other words, how much algorithm performance can be achieved using the same samples.

From theorem 4.2, we shows that the solution space is contracting when $k=2$. Reducing the solution space is beneficial in accelerating the convergence of the algorithm in some sense. Better performance can be achieved in the same number of steps of the algorithm iteration. In other words, with the same number of samples, we achieve better policy performance.

We generalized the TRPO algorithm, and proposed a new lower bound, combining the current data and subsequent data to optimize the current policy. Theoretical analyses show that our proposed method, in addition to satisfying the monotonic improvement of policy performance, can effectively reduce the solution space of the optimized policy, resulting in speeding up the training procedure of the algorithm.

[1] Schulman J, Levine S, Moritz P,et al. Trust Region Policy Optimization. ICML, 2015.

[2] Achiam J , Held D, Tamar A,et al. Constrained Policy Optimization. ICML, 2017.

[3] Hepeng L and Nicholas C ,et al. An Analytical Update Rule for General Policy Optimization. ICML, 2022.

[4] Jalaj B and Daniel R. On the Linear Convergence of Policy Gradient Methods for Finite MDPs. AISTATS, 2021.

[5] Alekh A and Sham M,et al. Optimality and Approximation with Policy Gradient Methods in Markov Decision Processes. 2020.

[6] Yunhao T,and Michal V,et al. Taylor expansion policy optimization. ICML, 2020.