Decentralized Decoupled Training for Federated Long-Tailed Learning

1. In Eq.6, how to calculate the gradient of $\min_{W_P}L(W_P;P,D^{bal})$? From my understanding, in Alg.1, line 3 is to optimize $L(P,W,\hat{W};D_k)$ while line 4-6 can be considered as jointly optimize $L(P,W,\hat{W};D_k)+\lambda L(W;P,D^{bal})$. But I don’t see how to optimize $-\min_{W_P}L(W_P;P,D^{bal})$ where $P$ should be the optimization variable.
2. Even though I know the whole training process as shown in Alg.1 and Fig.1, I am still confused about the effect of $\hat{W}$. Except that $\hat{W}$ will change the optimization goal as in Eq.5, I cannot get the point that what $\hat{W}$ can really learn in the training process. My intuition is that $\hat{W}$ will learn a residual of the output of a classifier trained with the long-tailed distribution and balanced distribution. But it makes no sense to me to introduce $\hat{W}$ here. We can solely train $W$ since the inference only involves $W$ but $\hat{W}$. Moreover, I notice that the author shows the results in Fig.4 to show the effectiveness of the additional classifier. Can you provide more intuition or explanation about why the additional classifier leads to better performance from a learning perspective?
3. Why $g_{W^{t-2},c}^{pro}$ is a constant? I think it depends on $W^{t-2}$ and $P^{t-2}$. So I don’t quite get the point why we need to normalize the scale of $g_{W_k^{i-1}}^{bal}$ in Eq.13.
4. In Eq.9-11, it requires at least one client to have sufficient samples for each class. I doubt the applicability of such a requirement in a realistic scenario. What if the data distribution is a heavy long-tailed distribution, e.g., only a few samples for some classes? In this case, it is very possible that no clients have sufficient samples, which makes their method fail or makes it very hard to tune the threshold $T$. Even though in Fig.3 it seems that the proper choice of $T$ is 2-8, I still think it is too small for sufficient samples and encourage the author to increase $T$ to see whether a larger $T$ can lead to a better performance. Also, when $T=\infty$, no clients have sufficient samples and all the local real data are removed, then how to get the global gradient prototype?
5. I don't quite understand why Eq.3 is a contradictory target. Because from the formulation of Eq.3, $P, W$ can be achieved to satisfy both $\min L(P,W;\cup_k D_k)$, and $\min L(W;P,D^{bal})$. It is not contradictory based on the math formulation.

1. There are certain issues with the formulas and symbols used throughout the paper, causing confusion. Specifically,

   • In Eqs (4), (5), and (6), what distinguishes $L(P,W,\widehat{W};\mathcal{D})$ from $L(W;P,\mathcal{D})$? It would be clearer to use distinct symbols for differentiation.

   • In Eqs (7), (8), (9), and (11), the subscripts for the differentiation symbols appear to be incorrect. The correct notation should be $x^{t}=x^{t-1}-\eta\nabla_xf(x^{t-1})$.

   • Eqs (4) and (5) raise some queries: How does $L(W;P,\mathcal{D}^{bal})$ constrain $L(P,W,\widehat{W};\mathcal{D})$? Does the expression $L(W;P,\mathcal{D}^{bal})=\min_{W_P}L(W_P;P,\mathcal{D}^{bal})$ imply that $W=\arg\min_{W_P}L(W_P;P,\mathcal{D}^{bal})$?

2. This paper seems to rely on an implicit assumption that the real data, which is exclusively stored on the clients, is abundant. Such an assumption seems strong, especially in cross-device contexts where locally stored data is typically limited.

3. The Algorithm focuses solely on the local training process. Incorporating a global process for clarity, which includes both local training and global aggregation, would be beneficial.

4. The description, specifically for each class $c$, in point (1) is some ambiguous. Are points (1) and (2) referring to different cases?

5. For the final logits given as $z=W^\top h+ \widehat{W}^\top h$, is there a need for an additional normalization coefficient to satisfy the properties of logits?

6. It would be beneficial if the authors could provide a more comprehensive discussion in the experimental section regarding the additional communication overhead of the proposed method. This encompasses the transmission of extra model parameters or gradient information.

I am not fully convinced by why the supplementary classifier can capture the global distribution precisely. Could the author provide more analysis on why and how precisely it can recover the global class distribution?