Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domains

Thank you for your insightful review. Here is our response to the mentioned weaknesses:

1. The key point of our dual-level clustering is local prototypes clustering. Different from easy domains, hard domains often exhibit looser clustering, increasing the risk of misclassification, especially for samples near category boundaries. Our dual-level prototypes clustering captures more feature distribution variation information by introducing the local prototypes clustering operation, since clustering locally provides more prototypes compared to simply averaging especially considering the sparse distribution in hard domains as shown in Figure 3 of our paper.

   To demonstrate this, we refer to Figure 1 of the attached pdf file. The y-axis shows the average number of local prototypes among classes generated at each selected round. Note that the number of prototypes for different classes can vary, resulting in non-integer averages. The easy domain (MNIST) has fewer prototypes generated compared to the hard domain (SVHN), showing that in hard domains, more prototypes are utilized to better capture the variance information. Furthermore, an average performance gain of 3.08% in experiment of impact of local prototypes clustering in Table 2 of our paper also supports this observation. At the global level, our clustering aims to reduce communication overhead and privacy risks by limiting the prototype variety each client sends, enhancing both efficiency and privacy while FPL intends to balance the impact of local prototypes from each domain, involving only server-level clustering.

   Meanwhile, our proposed α-sparsity prototype loss is also innovated and designed to complement the dual-level clustered prototypes operation. This loss function origins from reducing the potential risk of feature representation overlapping caused by our dual-level clustered prototypes operation. This loss function reasonably utilizes the variance information to reduce feature similarity across different classes while increasing it within the same class, promoting more effective and stable learning.

2. For each clustering operation, the installed FINCH python project provides several clustering results and we select the one with the smallest number of clustering prototypes. During different rounds, the number of local clustered prototypes vary for different classes among different clients, generally decreasing as training progresses. From Figure 1 of the attached pdf file, we observe that the average number of clustered local prototypes consistently decreases over the training process. This indicates that our local prototypes clustering method aligns with the convergence process among different domains.

3. We further conduct the ablation study of \alpha on the DomainNet dataset as shown in Figure 2 of the attaced pdf file. Our extensive experiments indicate that maintaining $\alpha$ within the stable range of 0.125 to 0.25 consistently yields the best performance. Although the results vary with different values of $\alpha$, our method consistently outperforms all baseline methods, demonstrating its robustness.

4. In FPL, the client domain distribution is unbalanced, with data ownership distributed among 20 clients as follows: MNIST (3 clients), USPS (7 clients), SVHN (6 clients), and SYN (4 clients). However, the reported final average accuracy is computed by summing the results from all clients and dividing by 20, which gives more weight to domains with more clients, leading to an unfair representation.

   In Section G, we address this issue by first averaging the results within each domain based on the number of clients in that domain. We then average these domain-specific results based on the number of domains. This weighted averaging ensures that all domains are given equal weight, regardless of the number of clients they have.

For Question 1, please refer to our response to Weakness 1.

Thank you for your insightful review. Here is our response to the mentioned weaknesses:

1. We introduce \alpha-sparsity loss to mitigate the potential risk of feature representation overlapping caused by the dual-level clustered prototypes operation. This loss focuses on maximizing inter-class distance and putting more attention on expanding the overall feature distribution to a broader range, thereby solving the feature overlapping problem efficiently and simply. While Optimal Transport (OT) may be employed in our framework, it introduces additional significant computational complexity [1]. Given our experimental results, it is effective to use our proposed \alpha-sparsity loss due to its simplicity.

   [1]. Peyré G, Cuturi M. Computational optimal transport: With applications to data science[J]. Foundations and Trends® in Machine Learning, 2019, 11(5-6): 355-607.

2. We apologize for any confusion caused by our paper's structure. Could you please provide us with a range of 'meaningless' and 'complicated' words so we can revise them accordingly? Moreover, we can move the DomainNet and Officie-10 parts from the Appendix to the main text per your suggestion in our final version.

   Regarding the label skew effect, we conduct label non-IID experiments using the Dirichlet method with a distribution parameter of 0.5, as detailed in Section E. The label distribution becomes more non-IID as the distribution parameter decreases. To illustrate the impact of different distribution parameters (0.1, 0.5, 1, 5, and 10), we compare the average accuracy with the best baseline method FPL from our previous non-IID experiment on Digit-5. Figure 3 of the attached pdf file shows that our method consistently outperforms FPL across all distribution parameters. However, the performance gap narrows as the datasets become more non-IID. This trend is expected, as in non-IID datasets, the number of samples for some classes can be very small, leading to less representative feature representations. Consequently, our clustered local prototypes may become similar to the averaged single prototype used in FPL.

   We also conduct the experiments on a larger scale of 20 clients, the same number in FPL, and the datasets of per four clients come from the same domain. As shown in Table 1 of the attached pdf file, our method still outperforms FPL under such a setting. Note we have already explored the unbalanced clients distribution setting of 10 clients in Section G.

Question 1:

Firstly, while both our method and FPL employ prototype clustering, the objectives and implementations are markedly different. FPL performs single-level global clustering, intended to balance the impact of local prototypes from each domain, involving only server-level clustering. This single-level global clustering is specifically designed to address imbalances in client distribution across domains, aiming to neutralize the skewed influence of domains with more clients on global model training. Conversely, our method integrates dual-level clustering at both the client (local) and server (global) levels. At the global level, our clustering aims to reduce communication overhead and privacy risks by limiting the prototype variety each client sends, enhancing both efficiency and privacy. Our local prototype clustering is crucial because it captures essential variance information, not just the average data, which is particularly crucial in hard domains. hard domains often exhibit looser clustering, increasing the risk of misclassification, especially for samples near category boundaries. Therefore, only capturing average local prototype suffices for easy domains but falls short for hard domains. To demonstrate this, we refer to Figure 1 of the attached pdf file. The y-axis shows the average number of local prototypes among classes generated at each selected round. The easy domain (MNIST) has fewer prototypes generated compared to the hard domain (SVHN), showing that in hard domains, more prototypes are utilized to better capture the variance information. Furthermore, an average performance gain of 3.08% in experiment of impact of local prototypes clustering in Table 2 of our paper also supports this observation.

Secondly, we introduce an innovative $\alpha$-sparsity prototype loss featuring a corrective component, which mitigates the potential risk of feature representation overlapping caused by the dual-level clustered prototypes operation.. This loss function reasonably utilizes the variance information to reduce feature similarity across different classes while increasing it within the same class, promoting more effective and stable learning.

Thank you for the AC tips. I have provided more details in my response to the review.

As for the details of the review,

1. The naive solution is to utilize a Gaussian distribution to describe class information. For example, FedFA [1] is based on Gaussian modeling of feature statistic augmentation. I encourage the authors to compare the conceptual differences between local prototype clustering and class Gaussian construction. Additionally, the authors should clearly explain the advantages of utilizing a dual-level prototype approach.
2. With respect to paper organization, the authors should spend more space on concept comparison with existing methods. But the authors focus on preliminary discussion.

Thank you for authors feedback. I still have the following questions:

1. You refer to a paper from 2019, but many papers related to Optimal Transport (OT) have been published in recent years. For instance, FedOTP [2] introduces the OT loss in regularization.
2. The authors did not address question 3: “Figure 1 is confusing. Why would a monitor capture the digits? It does not seem like a logical problem illustration.”
3. You mentioned that “This loss function reasonably utilizes the variance information to reduce feature similarity across different classes while increasing it within the same class, promoting more effective and stable learning.” Could the authors provide the corresponding theoretical or convergence analysis? Additionally, regarding “which mitigates the potential risk of feature representation overlapping,” could the authors offer visualizations to support this claim rather than relying on terms like “potential”?

[1] FedFA: Federated Feature Augmentation [2] Global and Local Prompts Cooperation via Optimal Transport for Federated Learning. CVPR 2024

Thank you for your insightful review. Here is our response to the mentioned weaknesses:

1. Our design principle aims to ensure our method excels particularly in challenging domains, which often have lower baseline accuracy, such as SVHN. This aligns with our motivation to tackle diverse learning challenges across various domains. In more challenging domains, the dual-level clustering method generates multiple clustered local prototypes that capture variance information effectively, thus enhancing focus and fairness, as shown in Figure 1 of the attached pdf file. For easier datasets like MNIST, where feature representations naturally concentrate, our local clustering tends to produce a centralized prototype similar to the traditional average method, leading to a smaller performance gap.

   We focus on hard domains beacuse in real-world settings, difficult domains are common and pose significant challenges that need effective solutions. Previous works often struggle in these hard domains, and our method addresses this gap by ensuring robust performance improvements where they are most needed.

   Contrary to the concern about insignificant results, our method does show considerable improvements in challenging domains. For instance, on the SVHN, Synth and MNIST-M, our approach achieved a notable performance gain of 5.3%, 2.72% and 3.14% respectively compared to SOTA methods. This demonstrates the effectiveness and robustness of our method in handling difficult datasets, which are critical for practical applications.

2. For the computation cost of FedPLVM, the feature representations of data samples can be obtained directly from the model without any additional computation. Our method introduces the dual-level prototypes clustering compared to other baselines, and we test the average running time for local training (0.0296s), local prototypes clustering (0.552s) and global prototypes clustering (0.0214s). Compared to the local training, our additional dual-level clustering step is negligible (approximately 0.0296 / 0.552 ≈ 5% of the local training time). Meanwhile, the proposed \alpha-sparsity loss does not change the network structure.

   For the communication cost of FedPLVM, we need each client to upload local clustered prototypes with the local model to the server and the client will then download the global clustered prototypes with the global model for the local training of the next round. Our modified lightweight ResNet-10 has approximately 4.9 million parameters. In our network, one prototype is only a 512-dimension vector, so the maximum number of total uploaded parameters for the local clustered prototypes can be approximately 3.2 (from the highest point of the red line in the above picture ) * 10 (number of classes) * 512 = 0.016 million and the estimated value of the downloaded parameters for the global clustered prototypes can be 21.20 (from table 3 in the paper) * 512 = 0.011 million which are both ignorable compared to the size of the model. Note the previous work FedPCL requires each client download the prototypes from all other clients from the server, which makes our clustering prototypes operation more realizable.

Question 1: A straightforward approach to balance the two sub-terms is introducing another hyper-parameter to control the weight. However, to maintain the simplicity of our proposed \alpha loss, we can adjust the temperature $\tau$ in the contrastive term to balance the two sub-terms. In our experiments, using our default $\tau$, the average loss value of the first term in the 50th round out of 100 is 0.364, while the second term is 0.105, demonstrating comparable values.

Thank you for your insightful review. Here is our response to the mentioned weaknesses:

1. Firstly, we want to highlight that the main objective of this paper is to develop a global model that works across all domain datasets. This approach aligns with the baseline methods, such as FedProto and FPL, which also address variations in client data domains.

   Secondly, training a unified global model, as opposed to personalized models, offers distinct advantages. For instance, a potential use case involves applying the trained model to broader scenarios, such as when a client has data from multiple domains and the domain of each data sample is unknown. In such cases, a global model remains effective, whereas personalized models may not be suitable.

   Lastly, developing a strong global model enhances our ability to implement personalization effectively. While our method can integrate personalization techniques, it is not the current focus of our work. We plan to explore personalization in more depth in our future research.

2. For the computation cost of FedPLVM, the feature representations of data samples can be obtained directly from the model without any additional computation. Our method introduces the dual-level prototypes clustering compared to other baselines, and we test the average running time for local training (0.0296s), local prototypes clustering (0.552s) and global prototypes clustering (0.0214s). The additional local prototypes clustering in dual-level clustering step is negligible, accounting for only about 0.0296 / 0.552 ≈ 5% of the local training time. Meanwhile, the proposed \alpha-sparsity loss does not change the network structure.

   For the communication cost of FedPLVM, each client uploads local clustered prototypes along with the local model to the server. The client then downloads the global clustered prototypes and the global model for the next round of local training. Our employed ResNet-10 has approximately 4.9 million parameters. Each prototype is a 512-dimension vector, so the maximum number of uploaded parameters for the local clustered prototypes is approximately 3.2 (from the highest average number of local clustered prototypes of the red line in Figure 1 of the attached pdf file) * 10 (number of classes) * 512 = 0.016 million and the estimated value of the downloaded parameters for the global clustered prototypes can be 21.20 (from table 3 in the paper) * 512 = 0.011 million. Both are negligible compared to the model size. Notably, the previous work FedPCL requires each client to download the prototypes from all other clients via the server, making our clustering prototypes operation more efficient and feasible.

3. Local prototype clustering is beneficial because it captures essential variance information, not just the average data, which is particularly crucial in hard domains. Easy domains tend to show tight clustering within the same category and clear distinctions between different categories, facilitating accurate classification. In contrast, hard domains often exhibit looser clustering, increasing the risk of misclassification, especially for samples near category boundaries. Therefore, only capturing average data suffices for easy domains but falls short for hard domains.

   To demonstrate this, we refer to Figure 1 of the attached pdf file. The y-axis shows the average number of local prototypes among classes generated at each selected round. The easy domain (MNIST) has fewer prototypes generated compared to the hard domain (SVHN), showing that in hard domains, more prototypes are utilized to better capture the variance information. Furthermore, an average performance gain of 3.08% in experiment of impact of local prototypes clustering in Table 2 of our paper also supports this observation.

4. We respectfully disagree. There are a few key distinctions between our work and FedPL.

   Firstly, while both our method and FPL employ prototype clustering, the objectives and implementations are markedly different. FPL performs single-level global clustering, intended to balance the impact of local prototypes from each domain, involving only server-level clustering. This single-level global clustering is specifically designed to address imbalances in client distribution across domains, aiming to neutralize the skewed influence of domains with more clients on global model training. Conversely, our method integrates dual-level clustering at both the client (local) and server (global) levels. At the global level, our clustering aims to reduce communication overhead and privacy risks by limiting the prototype variety each client sends, enhancing both efficiency and privacy. Our local prototype clustering is crucial because it captures essential variance information, not just the average data, which is particularly crucial in hard domains. Hard domains often exhibit looser clustering, increasing the risk of misclassification, especially for samples near category boundaries. Therefore, only capturing average local prototype suffices for easy domains but falls short for hard domains.

   Secondly, we introduce an innovative α-sparsity prototype loss, highlighted as “interesting” in the strengths part of your review, featuring a corrective component, which mitigates the potential risk of feature representation overlapping caused by the dual-level clustered prototypes operation. This loss function reasonably utilizes the variance information to reduce feature similarity across different classes while increasing it within the same class, promoting more effective and stable learning.

For Question 1, 2 and 3, please refer to our response to Weakness 1, 3 and 2 respectively.

Thank you for your insightful review. Here is our response to the mentioned weaknesses:

1. Local prototype clustering is beneficial because it captures essential variance information, not just the average data, which is particularly crucial in hard domains. Easy domains tend to show tight clustering within the same category and clear distinctions between different categories, facilitating accurate classification. In contrast, hard domains often exhibit looser clustering, increasing the risk of misclassification, especially for samples near category boundaries. Therefore, only capturing average data suffices for easy domains but falls short for hard domains.

   Our proposed method captures more feature distribution variation information by introducing the local prototypes clustering, since clustering provides more prototypes compared to simply averaging especially considering the sparse distribution in hard domains as shown in Figure 3 of our paper. For easy domains, where the average is sufficient, our method generates fewer prototypes.

   To demonstrate this, we refer to Figure 1 of the attached pdf file. The y-axis shows the average number of local prototypes among classes generated at each selected round. The easy domain (MNIST) has fewer prototypes generated compared to the hard domain (SVHN), showing that in hard domains, more prototypes are utilized to better capture the variance information. Furthermore, an average performance gain of 3.08% in experiment of impact of local prototypes clustering in Table 2 of our paper also supports this observation.

2. $C_y$ is the number of clustered prototypes for the class $y$.

3. Firstly, our extensive experiments indicate that maintaining $\alpha$ within the stable range of 0.125 to 0.25 consistently yields the best performance. We further conduct the ablation study on the large-scale Domain dataset. As shown in Figure 2 of the attached pdf file, we can still observe a performance benefit from the stable range.

   Secondly, it's important to note that $\alpha$ is a crucial element in our proposed $\alpha$-sparsity loss. If $\alpha$ is set to 1, our $\alpha$-sparsity loss reduces to a standard contrastive loss, although it still includes a specially designed corrective term (and also the proposed dual-level prototypes clustering operation). Therefore, it's natural for $\alpha$ to impact the results.

   Lastly, in our ablation study (refer to Figure 4 in the main paper), although the results vary with $\alpha$, our method consistently outperforms all baseline methods, demonstrating its robustness.

4. To the best of our knowledge, there is still no newer work under the same setting. If you are aware of any such studies, please let us know, and we will compare our work with them. Furthermore, there is one recent fair federated learning work, FedHeal [1], which compares with other fair federated learning methods by evaluating whether they can work well with some baseline methods used in our work, such as FedAVG and FedProto and bring any benefits. Although it operates under different setting compared to ours and does not compare with any baseline works in our setting, we conducted experiments to demonstrate that our proposed framework performs better than the SOTA method FPL + FedHeal in [1]. From Table 1 of the attached pdf file, we can observe that while FPL gains a performance boost from FedHeal, our proposed method still outperforms it.

   [1]. Chen Y, Huang W, Ye M. Fair Federated Learning under Domain Skew with Local Consistency and Domain Diversity[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 12077-12086.