BiVWAC: Improving deep reinforcement learning algorithms using Bias-Variance Weighted Actor-Critic

1. This paper addresses the issue of bias between the function approximation $\bar{Q^\pi}$ and its target $y$. My understanding is that this approach aims to mitigate the challenge that the true value of the function is unknown. However, as shown in [1], the TD error is an inadequate substitute for the true value error, which can significantly impact RL performance. Although controlling the bias between the function approximation and its target is possible, the value error may still remain substantial. Could the author provide some theoretical analysis on this value error aspect?
2. Although the $\alpha$ is a novel contribution, the result does not reflect on this. Based on Figure 1, for SAC, $\alpha = 0$ or $\alpha$ close to 0 result in best performance. However if the $\alpha = 0$, based on Equation 6, the loss function is exactly AVEC cost.

[1] Fujimoto, Scott, et al. "Why should i trust you, bellman? the bellman error is a poor replacement for value error." arXiv preprint arXiv:2201.12417 (2022).

• Inconsistency of the theory and the experiment.

  • Throughout the exposition of the theoretical arguments, the "true" target is un-regularized value $Q^{\pi}$ (or $V^{\pi}$). Theorem 3 also states that $g_\phi$ can be used to construct an unbiased estimate of conventional un-regularized policy gradient.
  • On the other hand, the base algorithms in the experiments are SAC and PPO, both of which do not align these un-regularized argument.
    • For SAC, the true target is the entropy regularized value, $Q_{\tau}^{\pi} = R(s,a) + \gamma \sum_{s',a'} P(s'|s,a)\pi(a'|s') (Q_{\tau}^{\pi}(s',a') - \tau \log \pi(a'|s'))$. In addition, the actor loss is a KL divergence between the parameterized policy and the energy based policy induced by the regularized value.
    • For PPO, the actor loss incorporates clipped log-ratio values to implicitly apply trust-region constraint.
  • Therefore, the theoretical results obtained in this paper does not explain the experimental behaviors of BiVWAC-SAC and BiVWAC-PPO.

• A rather minor concern is that, the quality of writing seems not satisfactory.

  • The expositions are not consistent in some parts.
    • Section 2.2, the author explored to which quantities Lemma 2.1 should be applied, and states that "As the policy gradient $\nabla_{\theta} J$ directly reflects our objective of maximizing $J$, it is the best candidate". However, Lemma 2.1 is not directly applied to $\nabla J$ but to the critic's residual error.
    • In L.118-119, it is stated that "In this work we limit our scope to policies which can be represented by Gaussian distributions". However, I found no gaussian requirements in the theoretical arguments.
  • Followings are the flaws in the writing that I noticed.
    • Italic and normal characters are mixed up for MSE, Bias and Var. it is recommended to use either of them consistently.
    • L.239/240: to be to be
    • the sentence after Eq. (8) lacks a period.
    • L.318: $\delta_{\rm BiVWAC}$ is used without clear definition.
    • L.414: Figure 2 -> Figure 1?
    • L.685; last one must be MSE_aloha(z,z-)

• To me, the main weakness of this paper in its current form is its presentation. Despite being a fairly straightforward reformulation of the mean squared error, I found the motivation and derivations hard to follow. For example, the equations in lines 64, 144, 180, 190, 229, and 246 all serve to arrive at the loss given in eq. 6 but in a rather cumbersome way, introducing several variables and notation that are not defined explicitly. I suggest that the derivation of Eq. 6 could be significantly more concise by showing that $\mathcal{L}_{avec}$ is in fact the variance term of a bias-variance decomposition of the MSE and that one can recover the MSE by adding to this the bias term with equal weighting.

• Conceptually, I am unsure whether this approach should be thought of as an "interpolation" between AVEC and MSE in the sense that AVEC is a special case of $\alpha=0$. There is a significant difference, in my opinion, between weighing the components in line 260 and 262 differently or having $\alpha=0$ (AVEC) and $\alpha=1$. This is because AVEC changes the minimizer of the objective function, whereas the shown approach mainly changes the weighting of gradients. For example, AVEC is not a sensible algorithm without the correction term, wheres one can argue that any value of $\alpha>0$ and $\alpha<1$ still shares the same minimizer as the MSE (assuming sufficient expressivity and continuous state-spaces).

• The experimental results seem to speak to the above points, in that $\alpha=0$ is a significantly different algorithm. There are moreover a few experimental results that I find concerning:

  • In Fig. 3, several versions with $\alpha=0$ (AVEC) perform worse with the gradient correction. In my mind this is a highly counterintuitive result. In my understanding, the objective function of AVEC has no reason to provide accurate gradients without the correction term, so it is highly surprising to see higher performance without it.
  • I find Fig. 4 a very difficult to interpret plot. For example, I don't follow the authors suggestion that the trend of the curve is meaningful. The sign of the shown curves seems like a more relevant metric: For example, all curves on Ant-v4 l.h.s. seem to indicate that bias, variance, and MSE are reduced in the alpha version. This, however, does not really corroborate the trade-off nature suggested by the authors. But what I find more concerning is that I expected all curves to cross 0 at alpha=0.5, as the authors suggest that the loss equals the MSE for this value. Most curves, however, stray significantly far away from 0. The above points, in my opinion, cast serious doubt as to whether the bias-variance trade-off is the driving factor in the observed experimental results.

• The related work section seems rather short. Several classical works (e.g. many works by Sutton, Singh, Bertsekas) discuss the bias-variance tradeoff at length in temporal difference methods, policy gradients, etc.