Tackling Underestimation Bias in Successor Features by Distributional Reinforcement Learning

5.``No justification is given for the additive noise model (Equation 6).’’

Answer. Actually, we have claimed the justification of the additive noise model before Eq. (6). Specifically, in practical applications, SFs-estimation usually incorporates random errors, which presumably stem from function approximation and are induced by source tasks.

9.``Theorem 1, why is $s′, a′$ defined as input to $\Delta$ but then $s′$ appears in the expectation? Also, why do we have an expectation over $s′$ again?’’

Answer. Thank you for your valuable suggestion. We approve that $\Delta$ is not a function of $s’,a’$, and we will correct it in our further revision.

10.``Theorem 3 compares the expected return of the optimal policy with a risk measure of the estimated policy. Why? If you're being risk-sensitive, the goal is not to learn the mean-optimal policy. ’’

Answer. Actually, the estimated policy is trained by risk-sensitive RL algorithms, while the optimal policy is determined by the MDP independent with aleatoric risk measures. Thus, the estimated policy and the optimal policy should be considered in different scenarios, where the former (inherent) is objective and the latter (trained) is risk-sensitive.

11.``Algorithm 1, where is $\tau_{e}$ being used? Shouldn't it be used in Line 6 when computing the greedy action?’’

12.``Algorithm 1, quantiles are treated implicitly in some cases; this makes it hard to decipher what's going on. ’’

Answer. Thank you for your careful review. We will elaborate on how to use various kinds of quantiles in our further revision. Specifically, indeed, $\tau_{e}$ is used in Line 6 when computing the greedy action. $\tau_{j}$ and $\tau_{k}$ are used to compute the loss $J_{Z}(\boldsymbol{\theta})$. All these issues will be clarified.

1.``I believe they are learning the wrong object with their proposed distributional SF algorithm. They should be learning the features' joint distribution, but they treat each feature as independent. This is implicitly done in how the author's sample $\tau$; to learn the proper distribution, you'd need the quantile of a multivariate random variable. ’’

2.``The paper is poorly positioned in the literature regarding multi-dimensional reward functions in distributional RL. My above point on learning the wrong object has been solved by [1] and [2], where they learn the correct joint distribution.’’

4.``Assumption 1 seems dubious; this should be impossible with stochastic transitions; a single application of the Bellman operator will construct a mixture distribution, so, at the very least, you’d expect the target to be a mixture of Gaussians.’’

Answer. Thank you for your valuable suggestion. As in [3], we will consider a more general case elliptical distribution.

3.``Furthermore, I expected a more in-depth discussion about [3] as they also learn “distributional SFs” (although they also aren’t learning the joint distribution).’’

Answer. Thank you very much for the insightful references. We will add [3] to the related work section. Specifically, Gimelfarb et al. proposed risk-aware successor features (RaSF) to realize policy transfer between tasks with shared dynamics by incorporating risk-awareness into the decision-making. Differently, our proposed concept of DSFs considers multiple different risk measures, which consider more scenarios of the return distribution and are more general.

[1] Zhang et al.. Distributional reinforcement learning for multi-dimensional reward functions. NeurIPS 2021.

[2] Freirich et al.. Distributional multivariate policy evaluation and exploration with the bellman GAN. ICML 2019.

[3] Gimelfarb et al.. Risk-aware transfer in reinforcement learning using successor features. NeurIPS 2021.

We appreciate the time and effort that you dedicated to providing feedback on our manuscript and are grateful for the insightful comments on our paper.

4.``It is very hard to read the paper when there are a large portion of different concepts and their corresponding theorems and equations. I would recommend moving most of these items into the appendix and use the main portion of the paper to explain what these different concepts are and how they are related to the research question that you are attempting to address. You can also move the pseudocode for Algorithm 1 into the appendix. This will also allow you to make more space for the section for conclusion and discussion. ’’

5.``The overall paper structure should be re-visited. The fact that a whole chunk of related work is in your appendix is a missed opportunity for the readers that they can follow along.’’

Answer. Thank you for your valuable suggestion. We will reorganize the whole paper for a more clarified presentation.

1.``First of all, the writing and the overall presentation is not very clear and does not flow well at times. Despite reading the paper a couple of times, it still seems confusing and overly complicated. Lastly, some of the sentences in the paper do not even make sense and makes me wonder if they were generated by LLMs. Here are some examples: a. “Explosively, we take an impressive TRL method - successor features (SFs) (Barreto et al., 2017; 2018; Carvalho et al., 2023) as an example, to study the underlying overestimation/underestimation bias.” b.“They enrich our concepts mutually.” c.“Extensive quantitative evaluations support our analysis.”’’

Answer. Thank you for your careful review. Actually, we do not use LLMs while writing the paper, and we are sorry for all these unclear expressions. We will update these sentences in our further revision. The specifics are as follows.

Explosively, we take an impressive TRL method - successor features (SFs) (Barreto et al., 2017; 2018; Carvalho et al., 2023) as an example, to study the underlying overestimation/underestimation bias. -> Typically, we study the underlying overestimation/underestimation bias in TRL by taking a representative method - successor features (SFs) (Barreto et al., 2017; 2018; Carvalho et al., 2023) as an example.

They enrich our concepts mutually -> We will delete this meaningless sentence in our further revision.

Extensive quantitative evaluations support our analysis -> Extensive quantitative evaluations demonstrate the Q-value underestimation phenomenon in SFs transfer framework and the effect of DSFs mitigating such underestimation.

2.*``It seems that the research question about addressing underestimation was motivated by RL/ But at the moment, I fail to understand the need of using risk-sensitive frameworks and multi-objective RL. What is the main motivation for considering these frameworks and theories? Furthermore, the lack of clarity from the section on bridging successor features and multi-objective RL does not help the cause.’’

7.``What is the purpose of analyzing using the risk-sensitive framework?’’

8.``What is the purpose of considering multi-objective RL which only further complicates the study?’’*

Answer.

The main motivation for risk-sensitive frameworks.

As mentioned in [1], ‘risk’ refers to uncertainty over possible outcomes, and risk-sensitive policies are those that depend upon more than the mean of the outcomes. Meanwhile, [2] claims that using the expected return as a measure of optimality could still lead to undesirable behavior such as excessive risk-taking, since low-probability catastrophic outcomes with negative reward and high variance could be underrepresented. For this reason, risk awareness is becoming an important aspect of practical RL. With the aid of the risk-sensitive framework, a series of RL works have been successfully conducted [3, 4]. In our paper, we leverage this framework to relieve the underestimation problem in the scenario of SFs.

The main motivation for MORL.

Incorporating MORL will mitigate the underestimation, and subsequently ensure possessing the optimal policy. Specifically, GPI/DGPI can only be used to construct a policy that performs no worse than the existing one but brings about the limitation of the policies' optimality. The prior knowledge of the set of SFs/DSFs is crucial for the precise and efficient learning of optimal policies using GPI/DGPI. To tackle this issue, MORL literature has extensively studied the problem of constructing an excellent prior, convex coverage set (CCS). For detailed literature, please refer to [5, 6].

Actually, we should also conduct experiments on separate SFs. However, due to program limitations, we only conduct experiments by combining SF with MORL. We aim to address this limitation in the future and include experiments on separate SFs to improve our paper.

The lack of clarity from the section on bridging successor features and multi-objective RL does not help the cause.

Thank you for your valuable suggestion. We will complement the purpose of bridging SFs and MORL in Section 2.2 of our revision.

[1] Dabney et al.. Implicit quantile networks for distributional reinforcement learning. ICML, 2018.

[2] Gimelfarb et al.. Risk-aware transfer in reinforcement learning using successor features. NeurIPS 2021.

[3] Ma et al.. DSAC: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint arXiv: 2004.14547.2020.

[4] Zhou et al.. Distributional generative adversarial imitation learning with reproducing kernel generalization. Neural Networks, 2023.

[5] Yang et al.. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. NeurIPS 2019.

[6] Hayes et al.. A practical guide to multi-objective reinforcement learning and planning. AAMAS, 2022.

We appreciate the time and effort that you dedicated to providing feedback on our manuscript and are grateful for the insightful comments on our paper.

1. ``The results indicate that both the two DSFs-based algorithms (RDSFOLS and DGPI-WCPI)". The latter isn't even in Figure 2? I am assuming the latter refers to "WCDPI+DGPI", but this should be clarified. ’’

Answer. Thank you for pointing this out. The label "WCDPI+DGPI" should be "DGPI-WCPI". We are sorry for this typo, and we will correct it in our revised paper.

2.``Awkward wording.

"resulting in a “zero-shot” somewhat fantastical"

"For a new task $w_{n+1}$, it is practicable to evaluate all policies". Perhaps you mean practical?

"standpoint to expose the mystery of underestimation". The word 'mystery' seems akward here.

"Due to the disorder of exploration": I don't know what "disorder of explanation refers to"

"is lack of stability"

"DSFs exhibits" -> "exhibit"

"We remark that $\delta_{\phi}>0$ makes no focus". I didn't understand what was meant by "focus".

"if the set of DSFs enough close"’’

Answer. Thank you for your careful review and suggestions. We will make updates to these linguistic issues. The specifics are as follows.

For a new task $w_{n+1}$, it is practicable to evaluate all policies -> For a new task $w_{n+1}$, it is practical to evaluate all policies

standpoint to expose the mystery of underestimation -> standpoint to expose the phenomenon of underestimation

Due to the disorder of exploration -> Due to the randomness of exploration is lack of stability -> is not stable

DSFs exhibits -> exhibit, and we will correct this issue throughout the whole paper.

The agent will attain near-optimal performance, if the set of DSFs enough close to task $w_{n+1}$. -> The agent will attain near-optimal performance in task $w_{n+1}$, if it has solved a similar task before.

4.``Can you describe again the y-axis in Figure 2?’’

Answer. In Figure 2, the y-axis represents the expected return (value) obtained from each method when evaluated over a test set of 60 tasks uniformly sampled from $\boldsymbol{W}^{\boldsymbol{\phi}}$.

Reviewer dNHm-5.``Did you look at Q-value predictions of the agents to show/demonstrate lower underestimation?’’

Reviewer 3ktK-6.``Although the author did provide the theoretical proofs showing the existence of the underestimation bias in the SF & GPI framework, this point will make a stronger case with empirical evidence as well.’’

Answer. Actually, the dashed lines in Figure 2 denote the true Q-values in the new task, and the solid lines mean Q-value predictions of the agents. Obviously, Q-value predictions are lower than the true Q-values in the new task. This supports the theoretical analysis in Section 3.1.

Reviewer 3ktK-3.``Eq 4. Is y the target that you are regressing towards? It is confusing if that is not the case. ’’

Reviewer mAvf-7.``In Section 3.1, the prediction $\Psi(s′,a′,\theta_{i})$ should be $\Psi(s,a,\theta_{i})$? Only the TD target should have the next state-action term. This error is propagated from this point forward, e.g., the gradient term is wrong, and eq (5) contains the same error. ’’

Reviewer mAvf-8.``In Section 3.1, why is there an expectation over s’ in the loss? Aren’t we trying to write down the stochastic approximation algorithm for learning SFs via TD? Citing equation 1 makes it seem like that’s what we’re trying to do. ’’

Answer. We are sorry for this typo. Actually, term $y$-$\psi^{\pi_{n+1}}(s',a';\theta_{i})^{T}w_{n+1}$ should be $y$-$\psi^{\pi_{n+1}}(s,a;\theta_{i})^{T}w_{n+1}$, which denotes the TD error.

On the other hand, owing to the fact that the updated parameter $\theta_{n+1}$, rather than $\boldsymbol{\theta}_{i}$, is included in the greedy target value with respect to $s’,a’$, the gradient term is subsequently with respect to $s’,a’$ and contains the expectation over $s’$.

We will correct it in our further revision. Thanks a lot.

Reviewer dNHm-3.``The results in Figure 2 do not appear to be very compelling. It seems the proposed method does not significantly outperform the baselines.’’

Reviewer mAvf-6.``It’s hard to judge the effectiveness of the approach as the empirical results don’t differentiate much between distributional GPI and regular GPI. I would have had to see a better-executed empirical study to be convinced. ’’

Answer. Thank you for your insightful comment. Actually, we should conduct experiments on separate SFs and SFs with MORL. However, due to program limitations, we only conduct the latter. The reason why the experiment is not obvious is mainly because the x-axis is the number of iterations, which is followed by [1]. It is convinced that there will be an obvious good performance in both sample efficiency and stability for DSFs under the measure of training steps. We will strive to complement the experiments on separate SFs in the future.

[1] Alegre et al.. Optimistic linear support and successor features as a basis for optimal policy transfer. In ICML, pp. 394–413, 2022.