Finite-Time Analysis for Conflict-Avoidant Multi-Task Reinforcement Learning

Q1: The analysis for the TD learning part should be supported with appropriate references

Thanks for the comments. We agree that the TD analysis part is quite standard, however, our major contribution lies in the other analysis part (discussed below). We have modified the TD part to make this more clear.

Q2: The overall novelty of the analysis remains questionable

Thank you for the question. We face two main technical difficulties in our analysis. The first difficulty lies in addressing gradient estimation bias, as MOO analyses typically rely on unbiased gradient estimators. However, in our setting, the function approximation introduces a non-vanishing gradient estimation error.

To elaborate, in supervised learning, unbiased gradient estimators for each task can be easily obtained using a double sampling strategy as shown in (Chen et.al., 2023), (Xiao et.al., 2022). Although previous work (Fernandoetal et.al., 2022) introduced a tracking variable to manage this bias, it requires the bias to vanish over iterations. In contrast, in multi-task reinforcement learning, obtaining an unbiased gradient estimator for both task gradient and weight gradient is impossible due to function approximation error in the critic. These non-vanishing errors introduce additional bias, as the CA direction is updated by biased estimated policy gradients. Bounding this extra error remains an open question in RL and MOO problems.

The second difficulty arises in bounding the CA distance. Due to the non-vanishing error, directly bounding this CA distance (see line 446) leads to non-vanishing error terms. Instead, we introduce a surrogate direction and decompose the error into three error components. The CA distance can then be bounded by the gap between $\lambda_{t,N_{**CA**}}$ and $\widehat{\lambda}_t^\prime$, plus the gap between $\widehat{\lambda}_t^\prime$ and $\lambda_t^*$. We also refer the reviewer to Section 5 for a more detailed discussion.

Q1 & Weaknesses 1: Are there any unique challenges, when using their results in the RL setting?

Thanks for the comment. We are aware of the techniques to handle the MOO problem with unbiased gradient estimator in (Xiao et.al., 2023) and with biased gradient estimator where the bias vanishes with the number of iterations in (Fernandoetal et. al., 2022). Nevertheless, unlike the {\bf vanishingly biased} or {\bf unbiased} gradient estimator in the supervised case, the bias of policy gradient estimation is {\bf non-vanishing} in our problem. Thus, the analysis in (Xiao et.al., 2023) can not be applied to our work directly. With a non-vanishing biased gradient estimator, analyzing the CA distance (i.e., the distance between the real CA direction and the estimated CA direction) becomes significantly more challenging. To address this, we propose a {\bf surrogate CA direction} to effectively bound the CA distance. In addition, our analysis derives a sample complexity of $O(\epsilon^{-5})$, which is {\bf lower} than $O(\epsilon^{-6})$ in the newest version of (Xiao et. al., 2023) with small CA distance. Besides, we need to note that our analysis technique can be extended to (Xiao et. al., 2023) easily for settings settings involving non-vanishing gradient bias. This highlights that our contributions include both significant new developments and differences from the work in (Xiao et. al., 2023).

Q2 & Weaknesses 2: Why SDMGrad not shown in the experimental evaluation?

We would like to emphasize that our work focuses on addressing the challenge of non-vanishing bias in the gradient estimator within the multi-task reinforcement learning (MTRL) setting. Adding a regularization term like SDMGrad does not solve this issue, and hence its performance is expected to be similar to the baseline we used. The regularization term can be similarly incorporated in our algorithm, but it is orthogonal to our contribution. Besides, adding a regularization term may introduce confusion and detract from the clarity of our technical contributions.

Weaknesses 1:

Thank you for the questions. We would answer them accordingly.

(1). Eq. (1) represents a standard formulation commonly used in multi-objective optimization (MOO) or multi-task learning (MTL) frameworks, such as MoCo, MoDo, and SDMGrad. In our case, different objectives have shared policy $\pi$, yielding multiple reward functions. The goal of the MTRL problem is to simultaneously maximize these reward functions.

(2). A Pareto stationary policy (or point) is a widely accepted convergence measure in MTL and MOO. Since different objectives rarely share the same optimal point, the aim is to find Pareto points where no objective can be improved without sacrificing another. The point you mentioned is indeed on the Pareto front but does not represent a balanced solution. Fixed-preference MTRL methods, such as averaging objectives, tend to converge to such points if one objective is easy to optimize. To address this, we focus on dynamic-preference MTRL methods, which aim to mitigate gradient conflicts and avoid unbalanced results by searching for the conflict-avoidant (CA) direction.

(3). In our simulation, we report the average success rate but this does not mean that different objectives are updated equally. The objective is not to optimize the average rewards but to maximize multiple reward functions in a balanced way.

(4). Lastly, even though objective functions can be equally weighted, it is a special case of dynamic weights that weights are all the same along the training process. Furthermore, in most cases, we have no prior knowledge of the objective preferences. Updating objectives with fixed weights will lead to gradient conflict and undesirable direction.

Weaknesses 2:

We face two main technical difficulties in our analysis. The first difficulty lies in addressing gradient estimation bias, as MOO analyses typically rely on unbiased gradient estimators. However, in our setting, the function approximation introduces a non-vanishing gradient estimation error.

To elaborate, in supervised learning, unbiased gradient estimators for each task can be easily obtained using a double sampling strategy as shown in (Chen et.al., 2023), (Xiao et.al., 2022). Although previous work (Fernandoetal et.al., 2022) introduced a tracking variable to manage this bias, it requires the bias to vanish over iterations. In contrast, in multi-task reinforcement learning, obtaining an unbiased gradient estimator for both task gradient and weight gradient is impossible due to function approximation error in the critic. These non-vanishing errors introduce additional bias, as the CA direction is updated by biased estimated policy gradients. Bounding this extra error remains an open question in RL and MOO problems.

The second difficulty arises in bounding the CA distance. Due to the non-vanishing error, directly bounding this CA distance (see line 446) leads to non-vanishing error terms. Instead, we introduce a surrogate direction and decompose the error into three error components. The CA distance can then be bounded by the gap between $\lambda_{t,N_{**CA**}}$ and $\widehat{\lambda}_t^\prime$, plus the gap between $\widehat{\lambda}_t^\prime$ and $\lambda_t^*$. Full details can be found in Appendix A.

I thank the authors for the response. Regarding your response to weakness 1.1-1.3 -- I completely agree that you should and would like to show that the proposed algorithm performs well not just on the averaged objective or on solving any single task. That is exactly where the problem lies in. The theoretical and empirical results do not show what you would like to show. I understand that Pareto stationary point may be a common metric in the literature, but it really does not provide much information on the quality of a MTRL policy if you can essentially hack the metric by solving a single task.