FedNovel: Federated Novel Class Learning

We really appreciate your constructive comments and suggestions. We provide the following response to answer your questions and address your concerns.

Weaknesses:

1. The method lacks sufficient elaboration and requires additional effort to comprehend the underlying technique.

Thanks for your suggestion. We have revised Figure 1 in our manuscript to include more information and details, enabling it to more clearly express the method we propose. As for the algorithm flowchart, we have provided the optimization pipeline for the novel class learning stage in the appendix (Section C). In our revision, we present the complete GAL algorithm process, encompassing both the known class learning and novel class learning stages.

2. Some of the assertions made in the paper need additional justifications.

Thank you for your comment. We have answered your questions in detail in the subsequent Question Section. Please refer to the responses provided below.

Questions:

1.(a) The known class learning training loss.

We apologize for the confusion caused. To develop model $m^L$, during the known class learning, we train the model using both the modified Prototype Contrastive Learning (PCL) and cross-entropy loss simultaneously.

1.(b) The question about the prototype communication.

As illustrated in Figure 1, local prototypes are uploaded to the server only once at the beginning of novel class learning. Upon receiving these local prototypes, the server uses Potential Prototype Merging (PPM) to estimate the number of novel classes and to construct global prototypes. These global prototypes are then distributed to all clients, who use them to initialize the novel class classifier. Then, the novel class classifier is updated through local training and central aggregation. Thus, the update of the novel class classifier essentially equals the evolution of the global prototypes. In this case, the entire novel class learning process requires only a single communication of local prototypes at the beginning. The reduction in the number of prototype communications further enhances privacy protection.

2. Whether the number of novel classes is fixed or not during novel class learning.

In FedNovel, we assume that novel class learning can be conducted periodically. During each novel class learning stage, we only invite clients who are active and available at the time to participate, and we pause the update of their data memories. Therefore, in this case, the number of novel classes in each novel class learning stage remains constant. For the continuously incoming novel data, we can initiate another stage of novel class learning when the novel data memory of the majority of clients is nearly full.

3. Any justification to support that using weights as prototypes is as effective as data/features?

Your understanding is correct. The weights of a standard linear fully connected layer indeed differ significantly from features. However, under specific conditions, if we ensure that the bias of the linear fully connected layer are zero, and train the neural network using the cross-entropy loss, the weights of the linear fully connected layer can be viewed as prototypes [1]. Moreover, the classifier consists solely of a single unbiased linear fully connected layer, which is put right after the feature extractor. Consequently, the dimensionality of this layer's weights is the same as that of the extracted representations. With the bias consistently set to zero, the computation of cross-entropy loss is based on the softmax probabilities derived from the inner products between the representations and the weights of that linear layer. Minimizing cross-entropy loss increases the inner product between the sample representation and the corresponding neuron weight (associated with the sample's ground truth label). The inner product between vectors is proportionally related to the cosine similarity between them. Thus, an increase in the inner product equals an increase in cosine similarity. Therefore, we can consider the neuron weights of the linear fully connected layer as prototypes based on cosine similarity.

4. The typo of "anchor sample" that appears in Section 3.3.1.

Thanks for your question and careful reading. The term 'anchor sample' was a typo. Our original intention was to refer to each regular training sample. We have removed the term 'anchor' in our revision.

Dear Reviewer hsUT,

The conclusion of the discussion period is closing, and we eagerly await your response. We greatly appreciate your time and effort in reviewing this paper and helping us improve it.

Thank you again for the detailed and constructive reviews. We hope our response is able to address your comments related to the training pipeline clarification, questions about the novel class number, justification about neural parameter-based prototypes, and clarification of novel data memory. We take this as a great opportunity to improve our work and shall be grateful for any additional feedback you could give us.

Best Regards,

Authors of Paper 5479

Dear Reviewer hsUT,

As the rebuttal discussion phase ends in less than 24 hours, we want to express our gratitude for your engagement thus far. We shall kindly remind you that after the 22nd, we are not allowed to respond to your further questions you may have. We really want to check with you whether our response addresses your concerns during the author-reviewer discussion phase. We have diligently addressed every concern and question you raised during the initial review, and our extensive efforts are aimed at enhancing the clarity and quality of our work.

Your feedback is really important to us. We eagerly await any potential updates to your ratings, as they play a critical role in the assessment of our paper. We genuinely hope our responses have resolved your concerns and provided satisfactory explanations. Your thoughtful evaluation greatly aids in our paper's refinement and strength. We sincerely appreciate your dedication and time again.

Best regards,

Authors of Paper 5479

Dear Reviewer hsUT,

Thank you again for the initial comments. As the Reviewer-Author Discussion phase is closing in less than 12 hours (Nov. 22nd AoE), we would greatly appreciate any feedback on our rebuttal. We fully understand that you may be busy at this time, but hope that you could kindly have a quick look at our responses and assess whether they have addressed your concerns and warrant an update to the rating. We would also welcome any additional feedback and questions.

Best Regards,

Authors of Paper 5479

We would like to thank you for your constructive comments. We will address your concerns, answer your questions, and clarify your misunderstandings as follows.

1. Lacks of crucial reference literature.

To prevent misunderstandings, we need to clarify that Federated OpenSet/OpenWorld Learning (FOL) is related to but distinctly different from, the FedNovel problem we are focusing on. Firstly, their application scenarios differ. FOL is typically applied in static federated training, where the simultaneous learning of known and novel classes is achieved within a single training phase. In contrast, FedNovel focuses on dynamic federated training, where the federated model, after having been employed for a period following the training of known classes, is then found to require the learning of novel classes. For dynamic federated training, the goal extends beyond learning novel classes to effectively mitigate the forgetting of known classes. FOL, which only considers single-phase training, is not applicable in continual dynamic federated learning.

Secondly, the training settings differ. For FOL, given its focus on static federated training, clients' local data is also static, meaning there is no continual influx of streaming data, and labeled data can be statically retained for training. Furthermore, static federated training does not allow the participation of new clients. In contrast, new clients in dynamic federated training often have no data at all. Although their streaming data continues to arrive, it also lacks any labels. FedNovel specifically addresses the challenges posed by the continual arrival of streaming data. It assumes that during the novel class learning stage, there will be no labeled data available, whether for known or novel classes.

Certainly, we understand your concern. We have already discussed the similarities and differences between FedNovel and FOL in the related work section and removed the claim of the first work in our revised manuscript. Furthermore, we have modified the title of our paper to "Federated Continual Novel Class Learning" to avoid any confusion and misunderstanding.

2. Insufficient comparison and discussion.

Commonly used semi-supervised learning (SSL) typically assumes that the training data is partially labeled and that both labeled and unlabeled data belong to the same label space. The most prevalent method in SSL is progressive learning. This approach initially trains the model on labeled data, then tests it on unlabeled data, selecting high-confidence data to add back into the training set. As training goes on, an increasing amount of unlabeled data is incorporated, gradually improving the model's performance. In contrast, the novel class learning problem that FedNovel focuses on has no labeled data available, and the novel classes do not overlap with the known classes. Under such settings, even if a model trained on known classes is tested on novel classes, the confidence becomes meaningless due to the completely different label spaces.

We understand your concerns. During the rebuttal phase, we conducted comparison experiments between our GAL approach and a federated semi-supervised learning work, SemiFL [F]. The experiment results are provided below, and we can observe that GAL can still substantially outperform SemiFL.

Dear Reviewer RSBY,

The conclusion of the discussion period is closing, and we eagerly await your response. We greatly appreciate your time and effort in reviewing this paper and helping us improve it.

Thank you again for the detailed and constructive reviews. We hope our response is able to address your comments related to the comparison and discussion with more literature, novelty clarification, theoretical explanation of PPM, and issues about experiments. We take this as a great opportunity to improve our work and shall be grateful for any additional feedback you could give us.

Best Regards,

Authors of Paper 5479

Dear Reviewer RSBY,

As the rebuttal discussion phase ends in less than 24 hours, we want to express our gratitude for your engagement thus far. We shall kindly remind you that after the 22nd, we are not allowed to respond to your further questions you may have. We really want to check with you whether our response addresses your concerns during the author-reviewer discussion phase. We have diligently addressed every concern and question you raised during the initial review, and our extensive efforts are aimed at enhancing the clarity and quality of our work.

Your feedback is really important to us. We eagerly await any potential updates to your ratings, as they play a critical role in the assessment of our paper. We genuinely hope our responses have resolved your concerns and provided satisfactory explanations. Your thoughtful evaluation greatly aids in our paper's refinement and strength. We sincerely appreciate your dedication and time again.

Best regards,

Authors of Paper 5479

Dear Reviewer RSBY,

Thank you again for the initial comments. As the Reviewer-Author Discussion phase is closing in less than 12 hours (Nov. 22nd AoE), we would greatly appreciate any feedback on our rebuttal. We fully understand that you may be busy at this time, but hope that you could kindly have a quick look at our responses and assess whether they have addressed your concerns and warrant an update to the rating. We would also welcome any additional feedback and questions.

Best Regards,

Authors of Paper 5479

In our response to Reviewer a8Cy, we elaborated on the differences in learning difficulty between label available novel class discovery and continual novel class learning. For label available NCDL, there are generally two approaches. The first one is based on semi-supervised learning (SSL), where the model is trained using labeled known class data while simultaneously selecting high-confidence unlabeled novel class data to participate in training. As training progresses, more novel class data are incorporated into training, enhancing the model's ability to extract representations for novel classes. The model's ability to extract representations for known classes is directly determined by the labeled known class data, which means that the learning difficulty for known classes is fixed as long as the supervised training is applied. Under such a scenario, if learning for known and novel classes is regarded as two separate tasks, the SSL-based NCDL works like multi-task learning, leading to the learning of more task-shared features. Such task-shared features greatly benefit novel class learning. The second approach to labeled available NCDL typically involves sufficient supervised pre-training using labeled known class data, followed by learning from unlabeled novel class data under the guidance of the labeled known class data. With this approach, the model, trained on labeled known class data, tends to learn task-private features that are less beneficial to novel class learning compared to the task-shared features. However, the second approach still allows for the use of labeled known class data during novel class learning, facilitating cross-task transfer learning. If there is any forgetting of the knowledge about known classes during novel class learning, it can be alleviated by retraining on the labeled known class data. Our proposed GAL is similar in training setup to the second method, but differs in that there is no labeled known class data available to maintain the learned knowledge during novel class learning. Although a significant portion of the knowledge the model has acquired about known classes is task-private, there still exists some task-shared general knowledge. Without a dedicated design to combat knowledge forgetting, both task-shared and task-private knowledge will gradually fade away. Once the task-shared knowledge is forgotten to a certain extent, novel class learning becomes an impossible task. Indeed, the difficulty of training for a task largely depends on the supervision information. For labeled known class data, whether trained simultaneously with unlabeled novel class data or in supervised pre-training before novel class learning, the learning difficulty remains the same. For unlabeled novel class data, which lacks labels for novel classes, the learning difficulty is significantly higher, especially without even any supervision information from known classes.

We have removed the claim of being the first work in our revised manuscript, but we still hope to inspire further exploration into the scenario of federated continual novel class learning through our work. It is by addressing these challenges that federated learning can be sustainably applied in the ever-changing real world.

[1] Grow and Merge: A Unified Framework for Continuous Categories Discovery. NIPS, 2022.

[2] Class-incremental Novel Class Discovery. ECCV, 2022

We would like to thank you for your positive and constructive comments on our work. We will address your concerns one by one as follows.

1. Discuss more related works.

Thank you for your suggestion, we have discussed open-world semi-supervised learning (OSSL) in the Related Work of the revised paper. Although the setup of OSSL does share certain similarities with ours, there are still key differences: (1) The data distribution differs. In OSSL, the training data is centralized to locate a machine during training. However, in FedNovel, the data is distributed to multiple clients following the non-IID setting, which means the local data distributions of clients are distinct from each other. (2) The data availability differs. The data in OSSL are static, which means all labeled and unlabeled data can be accessed at the same time, thus they can be learned within a single training phase. In contrast, the data in FedNovel are dynamic. FedNovel first requires the model to learn only on labeled known classes, then moves to learn on unlabeled novel classes without access to labeled known class data. Therefore, in the novel class learning stage, there is one more challenge in FedNovel than OSSL, which is to preserve good performance on known classes.

2. Experimental results of SemiFL.

Thanks for your suggestion. Similar to what we mentioned before, the key difference between SemiFL and FedNovel is also the data availability. The data in SemiFL are static while the data in FedNovel are dynamic. SemiFL can train on all labeled and unlabeled data in the same training phase, while FedNovel can only train on labeled classes in the first training phase and unlabeled classes in the following training phase. To further evaluate the effectiveness of our GAL, we also carry out the comparison experiments with SemiFL as below. Based on the results, GAL still achieves better performance than SemiFL.

3. Why $n_{size}$ is set as 2.

In our Potential Prototype Merging (PPM), $n_{size}$ is used to control the data scale of high-density regions. As the data in FedNovel are non-IID distributed among clients, a particular client may have only a few data samples for certain novel classes. Besides, for a particular novel class, it may be possible that only a few clients own this class. Setting a small $n_{size}$ can help our PPM not ignore the novel classes that correspond to a small number of local prototypes.

Dear Reviewer a8Cy,

Thank you for your initial positive feedback and insightful suggestions. We appreciate your recognition of our efforts to address your concerns. We would like to check with you whether our response addresses your further concern about the learning difficulty of different application scenarios. We really hope our response can address your concern, certainly, we will be glad to answer any further questions.

We value your input and sincerely hope you consider raising your rating based on the improvements we’re implementing. Your endorsement would greatly enhance the credibility of our work.

Thank you once again for your time and valuable feedback.

Best Regards,

Authors of Paper 5479

First of all, we really apprepiate your positive and constructive comments on our work. We will address your concerns one by one.

1. The number of novel classes

In FedNovel, the number of novel classes is not private information. This can be verified by works [1] in federated open-world semi-supervised learning, where the server is assumed to know the exact number of unseen classes before training. In contrast, the proposed GAL can estimate the number of novel classes by using our designed Potential Prototype Merging (PPM).

[1] Towards Unbiased Training in Federated Open-world Semi-supervised Learning. ICML, 2023.

2. Definition of private information

The private information that needs to be protected in FedNovel is the meaningful semantic information of raw data for every client. If existing data reconstruction attacks can't recover such semantic information from the prototypes sent by each client, we can say that our GAL does not leak private information. We have carried out experiments of launching data reconstruction attacks on the prototypes in GAL. The detailed results are shown in Appendix B.8, and the reconstructed images shown in Figure 3 look very similar to random noise in human eyes. To further verify that semantic information is not contained in the reconstructed images, we compute the Structural Similarity (SSIM) between reconstructed images and all images from each class. Then we check whether there is a set of classes that always share higher similarity with the reconstructed images and whether this class set is major included in the clusters corresponding to the attacked prototypes. Specifically, we choose the top 50% classes (10 out of 20) according to the average SSIM and set their corresponding dimensions as 1 in a zero vector. We also create another zero vector with the major component classes of a prototype as 1. Then we calculate the Manhattan Distance (MD) between these two vectors. For a better comparison, we also obtain the same vectors for random noise images and calculate the MD between vectors of reconstructed and random noise images. According to the following results, we can conclude that data reconstruction attacks cannot recover meaningful semantic information, thus our GAL is privacy-preserving.