Value Improved Actor Critic Algorithms

Dear reviewer,

Thank you for your valuable time, thoughtful review and accurate comments.

2. Limited empirical study on stability-greediness tradeoff. This is a keystone of the paper’s narrative, but there is only limited empirical analysis of this.

We agree with the reviewer that the tradeoff can be investigated further. We note that aspects of it have been investigated in prior work (for example [1], beyond the works already cited in the paper), and that different faces of it are investigated in Figures 1, 2, 4, 6 and 7.

4. Introduces a new greedification hyperparameter to tune in RL.

One could say that the tradeoff was there before VIAC, and with it being there, there was a possibility to tune it. However, we are of course in agreement with the reviewer that the introduction of any additional hyperparameter is unfortunate. We will include in the appendix a figure like Figure 4, extended for all environments, that sheds more light on the effects of different $\tau$.

Q1: In explicit VI-TD3, does the extra greediness (e.g., more PG steps) lead to an increase in total compute/wall-clock time, even if environment step efficiency is improved?

TLDR: Yes, absolutely. However, it is possible to get signficant (stronger, in fact) performance improvement with much smaller compute cost than PG with other explicit greedification operators. Results for which are included below.

Yes indeed, increasing PG steps comes with an increase in compute cost, which is important to mention (included in line 299). However, it is also possible to see significant performance improvement with explicit greedification of the evaluated policy without insignificant compute increase:

A simple example of a value improvement operator that is inexpensive and shows strong performance benefits in these domains is best-of-n (BON): sample N actions, take the value of the maximizing action as the bootstrap for the value target. This is a popular operator in RL for language models [2]. We include a table that summarizes final performance for VI-TD3 with this operator across the same domains as in Figure 3. We include baseline TD3, VI-TD3 with the expectile operator ($\tau=0.75$) and VI-TD3 with BON ($N=64$).

BON shows similar performance gains to IQL across most domains.

Below we report final-performance results normalized per environment between minimum and maximum final return over all seeds (20 for TD3 and VI-TD3 (IQL), and 12 for VI-TD3 (BON)), averaged per agent over environments, with +- one SEM. The IQL and BON variations are within each others 2 SEM confidence bound.

Below are the unnormalized final-performance results per environment along with 1 SEM which show the similarities in performance gains across the majority of environments in more detail.

[1] Thrun, Sebastian, and Anton Schwartz. "Issues in using function approximation for reinforcement learning." Proceedings of the 1993 connectionist models summer school. Psychology Press, 2014.

[2] Huang, Audrey, et al. "Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment." arXiv preprint arXiv:2503.21878 (2025).

Dear reviewer,

Thank you for your valuable time and clear comments and questions.

1. The presentation has a lot of room to be improved. Too much space is spent for the justification of generalized policy iteration. The authors should clearly explain the motivation and connection to AC algorithms at first, otherwise it is hard to follow.

Thanks for pointing that out, we will make sure that motivation and connection to ACs is made earlier and clearer.

2. The authors claim that the slower policy improvement brings stability, which we believe is also a key difference between the method and other off-policy algorithms that improve UTD ratios. However, there is no ablations to show the effectiveness of this design choice.

The claim that the slower policy improvement can bring stability is motivated by prior work: [6] explicitly delay the policy updates for learning stability. [7] and [8] explicitly limit the maximum change allowed to the policy before each new data-gathering step. [9] explicitly refers to and addresses issues that rise from too-fast changes to the policy.

In regards specifically to VIAC, Figure 7 demonstrates that increased UTD destabilizes the learning process much earlier than the increased greedification of the evaluated policy, equated over the same budget of gradient steps.

3. There is no proofs of convergence or theoretical guarantees of improvement over standard actor-critic for both deterministic and stochastic policies. Compared to the dense proofs for Section 3, this looks weird.

In regards to convergence: Using DP to analyze convergence properties of AC algorithms is rather standard [1, 2]. The main advantage of this analysis, from our perspective, is in its generality: the results are not limited to specific improvement operators, gradient based updates, presence of local optima, etc. and thus can be used to underly the convergence of algorithms that go beyong AC, such as MuZero [10].

In regards to improvement over standard actor-critics (or even policy iteration): Under strict-greedification ($>$) assumptions, we would expect that it is possible to show that value-improved GPI can outperform GPI for specific combinations of operators. Since a small number of recent algorithms already fall within the framework of VIAC in practice, albeit without the perspective of the framework (see lines 323 to 336 in the related work), we viewed proving soundness of this framework as more important than proving the possibility of benefit (which from that perspective, has already been observed). However, we agree that this result is important as well and we hope it will be addressed in future work.

4. For experiments in dm-control suite, the highest rewards the method achieved are still far from sota performance. For example, for humanoid stand and run, sota algorithms can reach 1000~ and 600~ [1,2], which are almost doubled compared to the proposed method. I note this is a minor issue.

BRO [3] specifically focuses on scaling up the DNN architectures and pairing the agent with dedicated optimism-based exploration, while Dime [4] explicitly changes the policy class and DNN architectures. In contrast, the baseline TD3 and SAC over which we investigate the effect of VI do not use any of these advancements. However, the results in Figure 5 in the appendix (VI-TD7) show that the improvements of VIAC are not limited to the older architectures of TD3 and SAC, and indeed VI-TD7 results in better performance across several environments compared to VI-TD3 / VI-SAC on both humanoid-stand and humanoid-run, suggesting that the improvement from VIAC is orthogonal to improvement from DNN architectures.

Q1. Is there any other greedy operators that also works well and where is the experiment for Gumbel MuZero operator? Have the author explored this direction?

TLDR: yes, and they have also showed significant performance gains in this domain.

We have tried several explicit greedification operators, in addition to PG and the implicit IQL. Overall, all resulted in performance gains in this domain. We include best of N (BON) as an example: BON samples $N$ actions and takes the value of the maximizing action as the bootstrap for the value target, and is popular in RL for language models [5]. The main advantage of IQL in comparison to the explicit operators is not only in its minimal computational and implementation cost but also that the greedification can be tuned with $\tau$ without increasing cost. In contrast, in operators such as BON, increasing greedification translates to increasing $N$, where the compute cost of the operator also increases with $N$.

In the experiments reported below, BON shows similar performance gains to IQL across most domains.

We first report final-performance results normalized per environment between minimum and maximum final return over all seeds (20 for TD3 and VI-TD3 (IQL), and 12 for VI-TD3 (BON)), averaged per agent over environments, with +- one SEM. For BON, we use $N=64$, and for IQL we use the same $\tau=0.75$. The IQL and BON variations are within each others 2 SEM confidence bound.

Below are the unnormalized final-performance results per environment along with 1 SEM which show the similarities in performance gains across the majority of environments in more detail.

We will include these results in the appendix.

I didn't find any place the author discussed the limitation.

The main limitation we observe for our contribution is the limitations of the theoretical results (finite horizon MDPs, DP, etc) which we mention per theorem. We will make sure this is made more clear and explicit in the text.

Please let us know if there remain any unaddressed concerns, comments or questions.

[1] Tsitsiklis, John N. "On the convergence of optimistic policy iteration." Journal of Machine Learning Research 3.Jul (2002): 59-72.

[2] Smirnova, Elena, and Elvis Dohmatob. "On the convergence of approximate and regularized policy iteration schemes." arXiv preprint arXiv:1909.09621 (2019).

[3] Nauman, Michal, et al. "Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control." arXiv preprint arXiv:2405.16158 (2024).

[4] Celik, Onur, et al. "Dime: Diffusion-based maximum entropy reinforcement learning." arXiv preprint arXiv:2502.02316 (2025).

[5] Huang, Audrey, et al. "Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment." arXiv preprint arXiv:2503.21878 (2025).

[6] Fujimoto, Scott, Herke Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." International conference on machine learning. PMLR, 2018.

[7] Schulman, John, et al. "Trust region policy optimization." International conference on machine learning. PMLR, 2015.

[8] Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).

[9] Capone, Cristiano, and Paolo Muratore. "Learning fast while changing slow in spiking neural networks." Neuromorphic Computing and Engineering 4.3 (2024): 034002.

[10] Danihelka, Ivo, et al. "Policy improvement by planning with Gumbel." International Conference on Learning Representations. 2022.

Dear reviewer,

Thank you for your valuable time, detailed review and many thoughtful comments and questions.

1. It is ultimately unclear to me where the performance gain really comes from. Is the greedification of the acting policy important in any way, or is the implicit value improvement really doing all the work, in which case the algorithm is question is an online (offpolicy) version of IQL?

TLDR: experiments with other value-improvement operators show that signif. performance gains from VI are not unique to IQL. We conclude that VIAC is not simply online IQL.

Best-of-n (BON) is an explicit value improvement operator which samples N actions and takes the value of the maximizing action as the bootstrap for the value target. BON is a popular operator in RL for language models [1]. We include a table that summarizes the final performance for baseline TD3, and the IQL and BON variations of VI-TD3, where BON shows similar performance gains to IQL across most domains. For BON, we use $N=64$, and for IQL we use the same $\tau=0.75$. Normalized per environment between minimum and maximum final return over all seeds (20 for TD3 and VI-TD3 (IQL), and 12 for VI-TD3 (BON)), averaged per agent over environments, with +- one SEM. The IQL and BON variations are within each others 2 SEM confidence bound.

Below are the unnormalized final-performance results per environment along with 1 SEM which show the similarities in performance gains across the majority of environments in more detail.

We will include these results in the appendix.

2. Much of the paper is dedicated to introduction of the theory, that ultimately does not reveal much other than sort of a sanity check with very limited scope.

Combining value improvement with policy improvement is already done in practice (see lines 326 to 336), albeit generally without this perspective in mind. For that reason, we think that the sanity check is important - if this approach is not generally sound we should not or should be careful when we use it. The choice of very abstract, very general scope of GPI allows us to establish Theorem 2, policy improvement is not enough, and the nec. / suff. conditions of Definitions 3-5, under the most abstract conditions, that are not limited to function approximation or even to RL. Analyzing VIAC in this scope allows us to further establish that this novel algorithms approach to RL/DP is sound under the most general conditions (excepting for finite horizon). We believe that it is an important first step towards establishing the theoretical understanding that underpins this general perspective which diverges from the standard RL iteration of policy improvement / policy evaluation.

C1. I was left uncertain as to whether the generalized policy iteration is the right framework to study the VIAC algorithm in its implicit value improvement flavor.

TLDR: We conclude that: 1. implicit improvement is within VIAC (including specifically the way we use IQL), and 2. that the value and policy improvement compliment each other, and unless one of them is explicitly the argmax, they do not render each other obselete.

If we understand correctly (please correct us otherwise), the concern is that the improvement in the value is doing all the work, and the improvement in the policy becomes unnecessary / unimportant.

We first note that if the greedy operator is used the VI-GPI collapses to Value Iteration / Policy Iteration (which are the same algorithm, from the perspective of GPI). In which case, the value improvement and the policy improvemnt coincide. However, as long as both operators take $\pi$ into account (for example, operators like $I_{GMZ}$ or policy gradient), every improvement step depends strongly on $\pi$. If $I(\pi, Q)$ does not diverge far from $\pi$ the value-improved value function does evaluate policies that are very different from $\pi$.

In fact, in a very simple set of experiments with dynamic programming and VI-GPI, discrete actions, and the same operators for both value and policy improvement ($I_{GMZ}$), we found that the improvement to the policy is the dominating effect. If this experiment sounds interesting despite its simplicity, please let us know and we will include it.

Specifically in relation to the implicit improvement of the expectile - note that we use it with respect to freshly sampled actions from the acting policy $\pi_\theta$, not with respect to actions stored in the replay buffer. [2] prove that this evaluates an explicitly greedier policy with respect to $\pi_\theta$, and [3] show that this policy is real and can be extracted.

Please let us know if the comment was not understood correctly, and this does not address / answer the concern the reviewer has in mind.

C2. The presentation of Figure 1 with x axis being the environment steps is questionable, given that the compute would be the right reference to use for the comparison.

Thanks for raising this point. The objective of Figure 1 is to show that when stripping other effects (such as IQL as the additional operator), there remains an advantage in sample efficiency from the greedification of the evaluated policy. To answer this question, we cannot put compute cost on the x axis. We agree indeed that the agents in Figure 1 result in additional compute cost (which we also mention following the figure, line 299). We will make sure the objective of the experiment is made more clear.

Q1. Typo: Thanks, good catch! Corrected to "the sets are also not disjoint".

Q2. Is $I_{GMZ}$ a generalization of soft policy iteration?

Interestingly, we believe the answer is no: In soft (max. entropy) policy iteration (/SAC), we aim to get $\pi_n = softmax (q_n)$. That is in fact not an improvement operator outside of soft-RL (an easy proof is that $softmax (q) \neq argmax (q)$, and we know that any optimal policy is an argmax policy). In contrast, when we use $I_{GMZ}$ we aim to get $\pi_{n+1} = softmax (q_n + \log \pi_n)$. Interestingly, this is a policy improvement operator [4]. In fact, this is also a limit-sufficient policy improvement operator (because $\pi_\theta$ will converge to an argmax policy that is uniform with respect to all maximizing actions, see Lemma 5 in Appendix A.6 in our paper).

Q3. non-detriment is not clear:

Thank you for pointing it out. We mean any operator that satisfies at least (in)equality 4. Note that this is not the same as greedification, which requires satisfying also inequality 5 (Which is the difference between the two). We will make sure this is made clear in the text.

Q4. Lines 282-283 are not clear:

Thank you for pointing that out. The first question aims to establish that the effect is there: that the greedification of the evaluated policy can by itself increase sample efficiency.

The second question aims to establish that the effect justifies the cost: that it is also possible to see significant performance gains without significant increases in implementation complexity / compute.

We will make sure to make it clear in the text.

Q5. It is unclear how you map the greedification operators in Alg2 to those of Alg3. When you increase the $n$ of $pg=n$, do you increase it for both the evaluated policy and acting policy?

Thanks for pointing that out. The increases are only to the evaluated policy (line 291). We will make sure that this is made more clear in the text. We investigate the effects of increases to the acting policy in the Appendix in Figure 6 (which is detrimental for PI while beneficial for VI).

Please let us know if there remain any unaddressed concerns, comments or questions.

[1] Huang, Audrey, et al. "Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment." arXiv preprint arXiv:2503.21878 (2025).

[2] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit q-learning." arXiv preprint arXiv:2110.06169 (2021).

[3] Hansen-Estruch, Philippe, et al. "Idql: Implicit q-learning as an actor-critic method with diffusion policies." arXiv preprint arXiv:2304.10573 (2023).

[4] Danihelka, Ivo, et al. "Policy improvement by planning with Gumbel." International Conference on Learning Representations. 2022.

Dear reviewer,

Thank you for acknowledging our rebuttal and clarifying the remaining questions.

Ok, so perhaps my question would be (this also relates to the previous point again) about "online IQL", where there is no explicit acting policy, just one that acts greedily w.r.t. the Q function learned by IQL from replay buffer actions. What are your thoughts about this / did you try comparing with such an algorithm?

In order to act greedily with respect to a Q function (learned by IQL with respect to a replay buffer, for example), one would have to search for $argmax_{a \in A} Q(s,a)$ in the continuous action space $A$ in order to act with these actions. In discrete actions we can find the $argmax$ by comparing the values of all actions, but in continuous actions we need other tools.

The most popular approach is that of actor critic: parameterize a function $\pi_\theta$, and search for $\pi_\theta(s) \approx argmax_{a \in A} Q(s,a)$ with gradient descent on $\theta$. This is exactly the policy improvement of policy gradient, and such an agent is exactly VIAC.

In contrast, a more "traditional" online-IQL agent will not act greedily with respect to the $Q$ critic (learned by IQL), but attempt to extract the policy implicitly evaluated by IQL, and act with it. Unlike VIAC, this agent, among other possible differences, will not converge to the optimal policy unless: 1. $\tau \to 1$ and 2. The maximizing action for the optimal policy for each state appears in the replay buffer.

Does this reply address the setup the reviewer has in mind?

But did you try increasing the update-to-data ratio of the non-vi algorithm (td3) to the point where it gets the same compute budget as that of the vi variants? If not, how do you conclude it is more sample efficient?

Thanks for clarifying the question, we now better understand the reviewer's perspective on this Figure. Yes, in fact, we were wondering the same, and this question is investigated in Figure 7. TLDR Figure 7: when thinking of compute (gradient steps) as a resource, and asking whether to spend it on VI or on RR, Figure 7 suggests that for small increases RR results in better performance than VI. For larger increases the script flips, RR destabilizes faster, and VI is the better performer.

We stress again that the aim of Figure 1 is simply to answer - can sample efficiency be increased by greedifying the evaluated policy directly?

Please let us know if you find Figure 1 in its current form not suited to answering the above question. We will make sure the purpose of Figure 1 is made more clear in the text, and that it does not suggest that VI outperforms an increase in RR, when one considers using specifically PG for VI (which is not the method we recommend: IQL and BON show much better performance for negligable cost).

Perhaps my terminology was incorrect, I meant policy iteration where you use a softmax in place of the max. I.e., policy mirror descent with negative-entropy regularization.

There is a subtle but major difference between the entropy-regularized objective $\pi_{n+1} = softmax (q_n)$, and that of $I_{GMZ}: \pi_{n+1} = softmax (\log \pi_{n} + q_n)$.

The operator $\pi_{n+1} = softmax (\log \pi_{n} + q_n) = softmax (\log \pi_0 + \sum_{i=1}^n q_i)$ (proven in Appendix A.5).

Looking at what happens when $q_n = q$, i.e. the value is stationary, perhaps sheds more light on the effect of the difference between the operators. In that case $\pi_{n+1} = softmax (\log \pi_{n} + q) = softmax (\log \pi_0 + n q)$. If we ignore $\pi_0$ (or if it is uniform), this is very similar to the regular softmax operator: $\pi_n = softmax (\beta q)$, if the inverse temperature $\beta_n = n$ is increasing linearly over iterations. At the limit, this converges to the hard argmax (A.5), while the softmax remains a softmax regardless of $n$.

More generally when the $q_n$ is not stationary, the logits of $\pi_n$ "accumulate" the information from all the previous $q_{i < n}$, concentrating over time to the action(s) with the largest sum: the maximizing actions for the sequence. If the sequence $q_n$ converges for some $q$, $\pi_n$ will also converge, and to the $argmax_a (q)$.

Please let us know if the above answers do not fully address the reviewer's questions or concerns and if there remain any additional questions or concerns.

Dear reviewer,

Thank you for your time, thoughtful review and detailed comments and questions.

1a. Does IQL fit within VIAC?

TLDR: We conclude that yes, the expectile operator fits within the VIAC framework.

[1] prove that the expectile loss explicitly evaluates greedier policies with respect to some sampling policy $\pi_b$. In their setting of offline RL, $\pi_b$ is captured as the distribution over the actions per state in the replay buffer. However, since we are not in the offline RL setting, we use the expectile loss with respect to fresh actions from $\pi_\theta$ directly (see Appendix D.3), which results in an operator that explicitly greedifies with respect to $\pi_\theta$. Further, [2] show that the policy $I_{IQL}(\pi, Q)$ can be extracted explicitly. We conclude that when $\tau > 0.5: \sum_{a \in A} I_{IQL}(\pi, Q)(a|s) Q(s,a) > \sum_{a \in A} \pi(a|s) Q(s,a)$, i.e., the expectile loss results in explicit greedification, and by using it with respect to fresh actions from $\pi_\theta$, it explicitly greedifies with respect to $\pi_\theta$.

The expectile operator however will not converge to the greedy policy unless $\tau \to 1$, and for that reason, it is not a sufficient improvement operator unless $\tau \to 1$. This raises again the question whether IQL fits within the VIAC framework or if it conflict with the theory. Corollary 2 addresses this by proving that even when insufficient ("non-detriment") operators are used for value improvement, the algorithm still converges.

1b. It remains unclear whether the observed improvements are due to a shift toward a greedier policy or simply the result of over- or underestimation.

TLDR: We conclude that the results presented in Figures 1 & 4 & additional results included here, suggest that performance gains of VIAC go beyond simply the result of over or underestimation.

In Figure 1 (second row, right) we see performance improvement without change in estimation bias. This shows that performance gains with VIAC can go beyond simply the result of over or underestimation.

Overall, we would expect to see 3 behaviors, differing across environments: (i) Where some overestimation is beneficial, VIAC may benefit from increased greedification up to the point of over-estimation increase. As long as over estimation is beneficial, it can benefit from the over estimation induced by increased greedification. (ii) Where overestimation is mostly detrimental, VIAC may benefit from increased greedification up to the point of over-estimation increase. Additional greedification will tradeoff against any negative effects of overestimation. (iii) As long as the effects of overestimation are minor, VIAC may benefit from increased greedification irrespective of overestimation.

In Figure 4, we see that humanoid-run belongs essentially to (iii): the entire performance improvement is experienced when greedifying $\tau \to 0.6$, up to which point the estimation bias is not strongly influenced. From $\tau > 0.6$, the estimation bias monotonically increases while performance remains roughly the same, until a point where the overestimation is too high and performance decays.

We include a table which extends and summarizes Figure 4 to all environments. We report:

(i) The maximum final performance increase (final performance - baseline final performance) observed over $\tau$s, without (<1%) increase in overstimation. The $\tau$ associated with this final performance.

(ii) The maximum final performance increase observed over $\tau$s. The associated $\tau$. The % overestimation increase out of the maximum overestimation observed in this experiment. The % overestimation / performance for this $\tau$.

Averaged across 10 seeds per $\tau$, and along with SEM. Stat. signif. differences from baseline are marked in bold.

In 5 environments (hopper-hop, humanoid-stand/walk/run, acrobot-swingup) there is significant performance gain without an increase in over estimation. In 6 environments (hopper-stand, quadruped-walk/run, fish-swim, walker-walk/run) any performance gain is correlated with some increase in over estimation. In 1 environment (cheetah-run) there is no statistically significant difference between the baseline and VI-TD3.

We conclude that the benefits of VIAC go beyond increasing over estimation bias. On the other hand, VIAC can be used to increase estimation bias, and when it is beneficial, VIAC can benefit from it (and when it is not beneficial, it's one of the reasons it is important that the amount of greedification be tuned).

2. Practical value improvement with other, low cost, explicit greedification operators:

TLDR: In additional results we include here other - low cost - explicit greedification operators demonstrate significant and comparable increases to performance.

We include an example of an explicit improvement operator that is inexpensive and shows strong performance benefits in these domains: Best-of-n (BON). BON samples N actions, takes the value of the maximizing action as the bootstrap for the value target. This is a popular operator in RL for language models [3]. We include a table that summarizes a figure like Figure 3 for baseline TD3, and the IQL and BON variations of VI-TD3. $N=64$ for BON and for IQL we use the same $\tau=0.75$.

BON shows similar performance gains to IQL across domains.

Below we report final-performance results normalized per environment between minimum and maximum final return over all seeds (20 for TD3 and VI-TD3 (IQL), and 12 for VI-TD3 (BON)), averaged per agent over environments, with +- one SEM. The IQL and BON variations are within each others 2 SEM confidence bound.

For results per env., due to limited space, see the reply to reviewer HLEH.

3. Sparse rewards:

Due to the possibility of explicitly tuning over-estimation bias through the tuning of the greedification with VIAC, we would expect VIAC to be able to induce pseudo-optimism-in-the-face-of-uncertainty exploration which should perform even better than baselines that do not have it, in sparse reward domains.

Q2. Could the authors elaborate on how the mismatch between the actor and critic policies affects learning?

In our experiments, we have only tried operators that sample from the acting policy $\pi_\theta$ and thus do not majorly diverge from it (like BON or IQL). We would conjecture that the important factor is whether the critic predictions are reliable for the evaluted policy. if OOD actions are evaluated, we would expect off-policy-evaluation concerns.

Q3. The theoretical results are grounded in finite MDPs, but modern deep RL involves continuous state and action spaces, along with approximation error. How well do the guarantees hold up under function approximation and in the presence of non-stationary Q estimates?

Cont. actions: While this is not usually analyzed in the setup of DP (to our knowledge), our theory is not fundementally limited to discrete actions. The induction does not depend on the action space, only on whether the improvement operators can greedify the policy all the way to an argmax.

Approximation error: When taking specific operators (say policy gradient for PI and the argmax for VI), under specific function approximation assumptions (presence of local optima, bounds on the possible error, etc), we don't see a reason to assume VIAC will not converge: VIAC simply maintains a Q estimate that is larger at any step than that of AC, but is still anchored in real policies.

Note that our analysis is already in the presence of non-stationary and approximate Q estimates. Similarly, we conjecture that the results can generally extend to other standard methods to approximate Q, such as Monte-Carlo / TD-$\lambda$ targets.

Q4. The balance between greedy updates and stable improvement somewhat reminds me of TRPO, which deliberately avoids overly greedy policy updates. Do the authors see any conceptual connection here?

Yes absolutely. We touch on a similar connection in the introduction, in lines 43-49. If we think of the improved policy of TRPO $\pi'$ as the "optimally greedy" policy, the evaluated policy is still limited to $\pi_\theta$, which is only moved towards $\pi'$ with gradient steps. VIAC in contrast would suggest directly evaluating $\pi'$, which seems very natural for TRPO. Does that make sense?

All results presented in the rebuttal will be included in the appendix.

Please let us know if there remain any unaddressed concerns, comments or questions.

[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit q-learning." arXiv preprint arXiv:2110.06169 (2021).

[2] Hansen-Estruch, Philippe, et al. "Idql: Implicit q-learning as an actor-critic method with diffusion policies." arXiv preprint arXiv:2304.10573 (2023).

[3] Huang, Audrey, et al. "Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment." arXiv preprint arXiv:2503.21878 (2025).