ConDa: Fast Federated Unlearning with Contribution Dampening

• The paper assumes the following:

  • A non-IID FL setup is highly challenging to unlearn.
  • However, non-IID FL is difficult in general federated optimization scenarios but not in federated unlearning.
  • Because of this claim, I suspect the method might not work in the IID setting since it creates another spectrum of problems associated with the similarity of local data distributions. Let's compare it to traditional machine unlearning, where you have class-level unlearning, which is similar to this setting of client-level forgetting in federated unlearning since you are assuming only a non-IID setting. Which, in turn, makes the problem solvable.

• Experimental settings:

  • You must consider more settings (data partitioning settings) by employing different kinds of non-IID-ness [1, 2]. There needs to be more than the settings provided in the paper (Tables 1, 2, 4, 5) to validate the method's validity.
  • Choice of the number of local epochs. Setting the local epochs to 2 in the non-IID setting favors overfitting each client's local data and makes them diverge more in the non-IID settings. This specific setting may help the proposed method obtain a better result since the gradients shared with the server from each client are diverse and easy to manage in the forgetting phase.

• The accuracy of CIFAR-10 (Table 3, Figures 4-7) is below 50%, which is unfavorable even though you use ResNet-18 architecture. It makes the validity of the results questionable.

• There are many hyperparameters ($\alpha, \lambda, U$), and no proper explanation or ablation study exists. The ablation study for the cut-off ratio ($\alpha$) is very sensitive, ranging from $0.01$ to $0.9$ [Line 511].

• [Line 204] Abuse of notation. Shouldn't the gradient update be multiplied by a learning rate scaler ($- \dfrac{1}{\eta_{\ell}}$)?

• [Line 369] IID FUL has been explored extensively (you need to add a citation to back this up).

• The tables offer minimal insight into the non-IID-ness nature of the data partitioning. It would be more informative to describe the method used for generating the data splits and specify the random seed rather than including the tables, particularly those for CIFAR-100 (Tables 4 and 5). A heatmap could be a more effective alternative to the table for visualizing this information (only for a small number of classes).

• Table 3: The CIFAR-100 (Backdoor Attack) case is missing. According to the table, the proposed solution outperforms other methods in only half of the cases.

• The organization of the tables and figures can be improved. They can be better positioned with their first references in the paper.

[1] Li, Qinbin, et al. "Federated learning on non-iid data silos: An experimental study." 2022 IEEE 38th international conference on data engineering (ICDE). IEEE, 2022.

[2] Kairouz, Peter, et al. "Advances and open problems in federated learning." Foundations and trends® in machine learning 14.1–2 (2021): 1-210.

1. The main problem is that the design of the unlearning algorithm in this paper simply follows the work by Foster et al. 2024 [42] (Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening) and applies it to federated learning. There is the lack of description of how the method proposed in this paper differs from the Selective Synaptic Dampening (SSD) [42].
2. In general this algorithm (compared to the original SSD) brings even more additional parameters and this becomes a non-trivial endeavor to find their optimal values. Algorithm 1 in [42] only has parameters $\alpha$ and $\lambda$, whereas this work adds the $U$ dampening upper bound.
3. This method assumes that the server is fully trusted. However, in this case, the server can have access to the individual client’s data through, e.g., [1].
4. The method assumes rather a big overhead on the server side to provide the unlearning. If the global model is large, then given millions of clients and hundreds of training rounds, the size of the collected data can be larger than the biggest LLMs.
5. The results for the Retrained Model in Table 3 vs ConDa clearly indicate that the proposed method simply unlearns less (higher accuracy on U-Set – the forgotten set) and forgets more on the retained set (R-Set). ConDa also forgets much less about U-Set than other methods for unlearning, such as PGA and FedEraser. Further concerns regarding the evaluation are in the questions.

References:

1. Boenisch, F., Dziedzic, A., Schuster, R., Shamsabadi, A. S., Shumailov, I., & Papernot, N. When the curious abandon honesty: Federated learning is not private. https://arxiv.org/abs/2112.02918

I listed a few weaknesses below:

1. It is not clear to me why the proposed algorithm is designed for Non-IID cases? What is the special component in the proposed algorithm that used to tackle Non-iid case? Why other existing unlearning algorithm cannot deal with non-iid case? Any intuition?
2. There is no analysis on the additional storage requirements. Does it scale linear with the number of clients, and model size, etc?
3. As the author pointed out in the limitation section, the proposed algorithm contains a few hyperparameters and they are all sensitive to clients’ datasets. In other words, hyperparameter tuning is needed every time to unlearn some client data from the global model, which makes the proposed algorithm not practical at all.
4. Storing all clients’ model updates across all global training rounds may leak clients’ private data information, especially in the non-iid setting. There is no discussion on this topic in the paper.
5. There is no theoretical analysis to provide any guarantee on the proposed algorithm that it does remove the effect of removed data samples, and to what extent it removes the effect of those points.

Minor comment: It is better to have the figure/table and the text discussing them at the same page or at most one page after. It is a bit hard to read through the text and scroll back to check the figures.