FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts

Comment:

We appreciate your valuable comments. We were wondering if our responses have addressed your concerns. Please let us know if you have additional questions. Thank you!

Q1: The main weakness lies in the choice of baselines. Since this paper uses the CM metric to evaluate the overall generalization performance, the authors should conduct further experiments comparing against prompt learning methods that specifically focus on generalization ability in federated learning, such as FedCoCoOp and FedMaple. Please refer to the baseline methods shown in Table 2 of FedMVP.

A1: Thank you for your suggestion. We have added the experimental results of FedCoCoOp and FedMaple as follows. FedCoCoOp and FedMaple indeed demonstrate excellent performance in pure generalization capabilities. However, both methods exhibit significant deficiencies in client personalization abilities. In the CM comprehensive metric that balances personalization and generalization, FedMGP significantly outperforms these baseline methods. Moreover, FedMGP's training parameters (12.8k) are substantially lower than FedCoCoOp(35.4K) and FedMaple(3.55M). This will result in high communication costs in federal learning.

Q2: The paper proposes a multi-group prompt training and aggregation strategy, but it is unclear why FedMGP can better generalize to unknown datasets compared to existing methods. A more detailed explanation would strengthen the contribution.

A2: Thank you for this suggestion. We fully recognize the importance of interpretability and have addressed it in our paper as follows:

• As shown in Figure 1 of the appendix, our diversity loss effectively enforces distinct representations. The remarkably low inter-group correlation among visual prompts provides direct evidence that each group learns to capture unique and complementary visual features.
• In appendix F, signal-noise decomposition theory mathematically proves that the multi-group mechanism enhances the model's signal-to-noise ratio by promoting representational diversity . This framework provides a rigorous foundation for why our design choice is critical for balancing personalization and generalization.

Q3: The experimental results for FedOTP and PromptFolio appear unusual. These methods achieve significantly high performance locally while showing substantial performance drops on base and new datasets. To my knowledge, FedOTP and PromptFolio are not specifically designed for generalization scenarios, which may not provide a fair comparison to validate the effectiveness of FedMGP.

A3: We appreciate the reviewer's concerns. However, in the seminal federal learning paper[1]. highlighted the core motivations of federated learning:

• Decoupling model training from raw data access, safeguarding user privacy while avoiding risks of centralized data collection;
• Achieving high-quality models with fast convergence and low communication overhead via iterative model averaging under non-IID, unbalanced, and massively distributed data.

In pathological IID and Dirichlet distribution scenarios, focusing solely on local performance is equivalent to conducting fully-supervised training on each client. This not only fails to achieve cross-client or cross-domain generalization, but also undermines federated learning’s original intent of obtaining global data generalization performance under conditions of reduced communication costs and privacy preservation. Therefore, generalization should be a necessary consideration for every federated learning method. As representative works in federated prompt learning, FedOTP and PromptFolio exhibit outstanding local performance but suffer significant performance drops on both global seen classes and unseen classes in same data distribution, which highlights the critical importance of evaluating generalization and provides a fair benchmark for validating FedMGP’s effectiveness in broader scenarios.

[1] McMahan, Brendan, et al. "Communication-efficient learning of deep networks from decentralized data." Artificial intelligence and statistics. PMLR, 2017.

Q4: Conducting experiments on all CLIP datasets would be beneficial. The experimental settings can reference those used in CoOp.

A4: Thank you for your valuable suggestion. We have supplemented experiments on StanfordCars, SUN397, UCF101, and FGVC-Aircraft, demonstrating the superiority of our method on these benchmarks.The experimental results are shown in the table below.

We are unable to test EuroSAT and ImageNet for the following reasons.:

• EuroSAT consists of only 10 classes. After splitting into seen and unseen sets, the seen subset contains just 5 classes, making it infeasible to simulate pathological non-IID settings (which require one class per client, i.e., 10 clients) under the standard federated prompt learning protocol [1–3].
• ImageNet is prohibitively large. Due to time and resource constraints, we were unable to conduct ImageNet experiments within the current timeline. Moreover, previous federated prompt learning studies [1–3] did not evaluate on ImageNet, placing it beyond the scope of directly comparable baselines.

We believe this selection maintains experimental rigor while fully aligning with established federated prompt learning standards.

[1]Guo, Tao, et al. "Promptfl: Let federated participants cooperatively learn prompts instead of models–federated learning in age of foundation model." IEEE Transactions on Mobile Computing 23.5 (2023): 5179-5194.

[2] Li, Hongxia, et al. "Global and local prompts cooperation via optimal transport for federated learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3]Pan, Bikang, Wei Huang, and Ye Shi. "Federated learning from vision-language foundation models: Theoretical analysis and method." Advances in Neural Information Processing Systems 37 (2024): 30590-30623.

Q5: Other Paper Formatting Concerns

A5: Thank you very much for your detailed suggestions on paper formatting. We will address these issues in the final version of the paper.

Comment:

Thank you for your valuable feedback! Below are address all raised concerns of the paper.

Q1：Lack discussion on related group prompt based federated learning, [1] is crucial as it not only proposes learning group-specific prompts to capture heterogeneous client features but also provides a theoretical understanding of the effectiveness of such group-based methods. A thorough discussion against such foundational work is essential.

A1: Thank you for the insightful suggestion! We will include a detailed comparison with [1] in the final paper. Both methods leverage the group-based mechanism to balance global generalization and local personalization. However, the key distinctions are:

• SGPT[1] maintains a global prompt group pool on the server for different data domains; FedMGP keeps multiple prompt groups locally on each client to enhance diversity .
• In Inference time，SGPT requires explicit selection of a prompt group based on input features; FedMGP employs dynamic aggregation, performing average of group confidences at inference without additional selection .
• SGPT incurs approximately 100k parameters for communication and training; FedMGP reduces this to 12.8k training parameters and 2.6k communication parameters

[1] Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning.

Q2：The proposed dynamic prompt aggregation strategy, which relies on similarity-guided probabilistic sampling, introduces a potential for imbalance in the global aggregation process. The paper does not adequately address how it resolves the critical issue where certain client-side prompt groups might be consistently less similar to the global prompts, leading to them being "hardly selected" for aggregation.

A2: Thank you for you question. Our Method have considered this scenario.

• Our dynamic aggregation strategy incorporates a temperature parameter (τ) , which critically controls the selection probability distribution. As detailed in our ablation study in Appendix E.1 , this parameter allows us to fine-tune the balance between exploitation (deterministically selecting high-similarity prompts with a low τ) and exploration (enabling the selection of diverse, lower-similarity prompts with a higher τ).
• This probabilistic sampling , governed by the temperature, ensures a diverse set of prompts is aggregated in each round. As demonstrated in the selection statistics table below , even prompt groups with lower initial similarity are consistently given a chance to be selected across different communication rounds, effectively preventing premature convergence to a homogenized global prompt and fostering the continuous discovery of diverse semantic features .
• Ultimately, this mechanism creates an intelligent trade-off inherent to personalized federated learning. Prompts frequently selected across clients tend to encapsulate generalizable knowledge , strengthening the core model, while less frequently selected prompts often hold unique, client-specific features crucial for personalization . Our method dynamically balances these two competing objectives, achieving a robust equilibrium between cross-client generalization and local adaptation.

Q3: Equation 9 appears to imply a simple averaging of prompts from different clients. This raises a concern regarding potential bias in the global model. If some clients possess substantially more training data than others, a straightforward averaging scheme could disproportionately weigh the contributions of larger clients.

A3: Thank you for your valuable comment. Consistent with FedAvg, we weight the aggregation by each client’s data volume to prevent bias arising from unequal dataset sizes. The revised aggregation formula for Equation 9 is: $\tilde{P}^T_i = \sum_{c \in C_T} \frac{n_c}{\sum_{c' \in C_T} n_{c'}} \,P^T_{i,c},$ where $n_c$ denotes the number of samples at client c.

We will incorporate this clarification in the final paper.

Q4: Since only select prompts most similar to global prompts for aggregation, will this lead to a fixed prompts being aggregated while others always keep local? How is the frequency that each group prompt being aggregated globally?

A4: Thank you for this important question. Our method does not result in the same groups being consistently aggregated while others remain perpetually local. We provide aggregation frequency statistics for each group during training on OxfordPets in the table below. As mentioned in Q2, we introduce a temperature parameter to mitigate such extreme scenarios . Furthermore, our dynamic aggregation mechanism inherently requires different groups to maintain varying aggregation frequencies: prompt groups that are selected more frequently during training tend to aggregate broader, global semantics, while those with lower selection frequencies preserve more localized, client-specific semantics . These complementary behaviors balance generalization and personalization. Although we cannot include visualizations in this submission, we will add this discussion with qualitative examples and visualizations in the final version.

Comment:

Thanks a lot for your time and feedback. We have to say that the reviewer asks valuable questions and provides thoughtful clues. We appreciate your inspiring reviews. And we are happy to address the concerns.

Q1: The use of a fixed number of prompt groups (e.g., G=5) across all clients fails to consider data heterogeneity. Clients with simpler data (e.g., only 1–2 classes) may require fewer groups, while more complex clients (e.g., multi-domain or long-tail data) may benefit from more.

A1: Thank you for this insightful question.

• The referred setting corresponds to the Dirichlet distribution experiments in Table 2 and Appendix D (α=0.5 / α=0.3), a widely adopted benchmark for measuring data heterogeneity[1][2]. Even under severe class imbalance, FedMGP delivers outstanding performance, demonstrating the robustness and generalization of our dynamic aggregation mechanism across diverse client distributions.
• Moreover, according to your valuable suggestion. We have supplemented an ablation study on the number of prompt groups (G) under this imbalanced setting. The results are as follows. These findings show that G=5 strikes the optimal balance between performance gains and parameter efficiency, further enabled by our dynamic selection strategy that adapts to each client’s data distribution. Overall, FedMGP exhibits high efficiency, strong robustness, and superior generalization in handling data imbalance and heterogeneity.

[1] Li, Hongxia, et al. "Global and local prompts cooperation via optimal transport for federated learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Cui, Tianyu, et al. "Harmonizing Generalization and Personalization in Federated Prompt Learning." International Conference on Machine Learning. PMLR, 2024.

Q2：The dynamic aggregation process relies on the cosine similarity between local prompts and the global prompts from the previous round. In the extreme case where a local prompt directly adopts the previous global prompt, the similarity reaches its maximum, resulting in the highest selection probability. However, this similarity-based sampling strategy may not be optimal.

A2: Thank you for you question. Our Method have considered this scenario.

• Our dynamic aggregation strategy incorporates a temperature parameter (τ) , which critically controls the selection probability distribution. As detailed in our ablation study in Appendix E.1 , this parameter allows us to fine-tune the balance between exploitation (deterministically selecting high-similarity prompts with a low τ) and exploration (enabling the selection of diverse, lower-similarity prompts with a higher τ).
• This probabilistic sampling , governed by the temperature, ensures a diverse set of prompts is aggregated in each round. As demonstrated in the selection statistics table below , even prompt groups with lower initial similarity are consistently given a chance to be selected across different communication rounds, effectively preventing premature convergence to a homogenized global prompt and fostering the continuous discovery of diverse semantic features .
• Ultimately, this mechanism creates an intelligent trade-off inherent to personalized federated learning. Prompts frequently selected across clients tend to encapsulate generalizable knowledge , strengthening the core model, while less frequently selected prompts often hold unique, client-specific features crucial for personalization . Our method dynamically balances these two competing objectives, achieving a robust equilibrium between cross-client generalization and local adaptation. | round | t_g1 | t_g2 | t_g3 | t_g4 | t_g5 | v_g1 | v_g2 | v_g3 | v_g4 | v_g5 | |---|---|---|---|---|---|---|---|---|---|---| | 1 | 3(15%) | 2(10%) | 6(30%) | 5(25%) | 4(20%) | 3(15%) | 2(10%) | 6(30%) | 5(25%) | 4(20%) | | 2 | 6(30%) | 2(10%) | 6(30%) | 3(15%) | 3(15%) | 6(30%) | 2(10%) | 6(30%) | 3(15%) | 3(15%) | | 3 | 7(35%) | 2(10%) | 5(25%) | 3(15%) | 3(15%) | 7(35%) | 2(10%) | 5(25%) | 3(15%) | 3(15%) | | 4 | 6(30%) | 3(15%) | 5(25%) | 3(15%) | 3(15%) | 7(35%) | 3(15%) | 4(20%) | 2(10%) | 4(20%) | | 5 | 6(30%) | 3(15%) | 3(15%) | 3(15%) | 5(25%) | 7(35%) | 3(15%) | 3(15%) | 2(10%) | 5(25%) | | 6 | 6(30%) | 4(20%) | 3(15%) | 2(10%) | 5(25%) | 7(35%) | 4(20%) | 3(15%) | 1(5%) | 5(25%) | | 7 | 7(35%) | 4(20%) | 4(20%) | 1(5%) | 4(20%) | 7(35%) | 4(20%) | 4(20%) | 1(5%) | 4(20%) | | 8 | 6(30%) | 4(20%) | 4(20%) | 2(10%) | 4(20%) | 7(35%) | 4(20%) | 5(25%) | 1(5%) | 3(15%) | | 9 | 4(20%) | 5(25%) | 6(30%) | 1(5%) | 4(20%) | 5(25%) | 5(25%) | 7(35%) | 0(0%) | 3(15%) | | 10 | 5(25%) | 5(25%) | 5(25%) | 2(10%) | 3(15%) | 4(20%) | 7(35%) | 6(30%) | 1(5%) | 2(10%) |

Q3: At inference time, the model averages predictions across all prompt groups equally, ignoring confidence differences between groups. This uniform treatment could reduce prediction reliability, especially when some groups are more semantically aligned with the input than others.

A3: We appreciate your valuable comments. We wish to clarify that our inference strategy directly incorporates the confidence of each group, as you suggested. For each prompt group, the final logits , computed from the similarity between its guided text and image features, directly represent that group's confidence distribution over the classes for a given input. Therefore, averaging these logits is a principled way to ensemble the predictions, where each group's contribution is naturally weighted by its confidence. As rigorously demonstrated in our ablation study in Table 6 , this method significantly outperforms alternatives like selecting the single most confident group (Max logits) or averaging the prompt-guided features. This result powerfully validates our integrated design: the multi-group architecture enables diverse specializations—some capturing broad, generalizable knowledge while others master client-specific, personalized nuances . Our inference strategy then optimally synthesizes these varied perspectives, allowing the model to draw upon shared knowledge for general tasks while accurately reflecting personal characteristics when needed, thus achieving a superior balance between generalization and personalization .

Dear Reviewer:

We would like to ask if our response has sufficiently addressed your concerns. If you have any remaining questions or concerns, please feel free to share with us directly. Thank you for your time.

Thank you for your feedback. We understand your concerns and would like to further clarify the rationality of the similarity design. The detailed explanation is as follows:

• We would like to clarify that in the federated learning process(Appendix A), there is no scenario where a local prompt directly adopts the previous global prompt when aggregatting . After aggregation, only the top-s groups use the global prompt for initialization, but after local training in each client, they incorporate a substantial amount of local knowledge, resulting in significant differences from the previous round's global prompt when aggregation in current round.
• We understand your concern that the initialized groups with previous round's global prompt might always maintain the highest similarity, leading to the exclusion of other groups. However, in practice:
  • Due to the diversity loss, the initialized prompts will pull away from each other during local training, learning diverse features and ensuring that each group has the opportunity to capture general knowledge (as confirmed in Appendix D) .
  • Due to local data heterogeneity, in federated learning, client data is highly heterogeneous (such as Dirichlet distribution), causing the prompts after local training to differ greatly from their initial values, preventing situations where similarity to the previous round's global prompt is extremely high.
  • The temperature parameter (τ) introduces randomness, allowing low-similarity prompts (containing more local features) to participate in aggregation at a low frequency, balancing exploration and exploitation, preventing overfitting, and minimizing interference (as shown in the ablation study in Appendix E.1) .
  • The following table shows the similarity between each group's local prompt and the previous round's global prompt before and after each round of training for the OxfordPets dataset, cilent1. It can be seen that after local training, each group will have significant differences from the initial group, especially the group which parameters were updated by the previous round's global prompt. The bolded entries in the table represent the group with the highest similarity in each round, and * represents the group selected in the current round. Due to the presence of the temperature parameter, the group with the highest similarity is not always selected (e.g., round4, round6). The group with the highest similarity each time is not necessarily the group which parameters were initialized with the previous round's global prompt (e.g., round2, round4).

Thank you for your response. For Q2, although more justification is provided for dynamic aggregation and probabilistic sampling, they cannot address the question about the design of the similarity evaluation. “In the extreme case where a local prompt directly adopts the previous global prompt, the similarity reaches its maximum, resulting in the highest selection probability.” A more reasonable probability design is needed.

Comment:

We appreciate your valuable comments. We were wondering if our responses have addressed your concerns. Please let us know if you have additional questions. Thank you!

Q1:The paper does not report error bars or statistical significance tests, which limits the strength of its claims regarding performance robustness under data variability.The paper does not report error bars or statistical significance tests, which limits the strength of its claims regarding performance robustness under data variability.

A1: To rigorously evaluate performance robustness, we have conducted main experiments across the five datasets using three distinct random seeds (0, 1, 2) . The updated table presents the results, confirming that FedMGP not only achieves the highest performance but also demonstrates superior stability with the lowest standard deviation. Crucially, this leading performance and robustness are achieved with the lowest communication overhead (2.6k parameters) among all methods, highlighting the exceptional efficiency and reliability of our approach.

Q2: The paper claims that different prompt groups specialize in distinct semantics via diversity loss, but provides less interpretability and explainability.

A2: Thank you for this suggestion. We fully recognize the importance of interpretability and have addressed it in our paper as follows:

• As shown in Figure 1 of the appendix, our diversity loss effectively enforces distinct representations. The remarkably low inter-group correlation among visual prompts provides direct evidence that each group learns to capture unique and complementary visual features.
• In appendix F, signal-noise decomposition theory mathematically proves that the multi-group mechanism enhances the model's signal-to-noise ratio by promoting representational diversity . This framework provides a rigorous foundation for why our design choice is critical for balancing personalization and generalization.

Q3: Although the method is motivated by privacy-sensitive federated settings, the paper does not assess whether the shared prompt vectors, especially visual prompts could leak private information or be vulnerable to reconstruction attacks.

A3: We appreciate the reviewer’s concern. Our framework adheres to the FedAvg paradigm[1], which has been extensively proven to offer strong privacy protection because:

• Our method exchanges only updated text and visual prompt parameters, without sharing gradients, prediction logits, or data embeddings, eliminating the key information required for model inversion, and preventing adversaries from obtaining labels or embeddings to build shadow models.
• With only approximately 2.6K parameters per round, PromptFL[2] has shown that low-parameter communication is more privacy-preserving than full model uploads; this also drastically reduces the attack surface and minimizes the risk of membership inference and attribute inference.
• Furthermore, our method leverages diversity loss and dynamic selection to prevent overfitting of single prompts and enhance protection of privacy-sensitive features.

[1] McMahan, Brendan, et al. "Communication-efficient learning of deep networks from decentralized data." Artificial intelligence and statistics. PMLR, 2017.

[2]Guo, Tao, et al. "Promptfl: Let federated participants cooperatively learn prompts instead of models–federated learning in age of foundation model." IEEE Transactions on Mobile Computing 23.5 (2023): 5179-5194.

Q4:In many group-based FL algorithms, group membership tends to drift across rounds. Could the authors clarify whether prompt group assignments remain stable across training rounds? Is there any tracking or visualization of how group semantics evolve over time?

A4: Thank you for your suggestion! Our prompt group assignments remain highly stable throughout training. We quantitatively demonstrate this by tracking each group’s selection frequency per round.

• As shown in the table below, which demonstrates the selection frequency of each group during training on the OxfordPets dataset. Our dynamic aggregation mechanism is designed to guide the specialization of prompts; some evolve to capture shared global knowledge by being selected frequently, while others focus on client-specific local features through less frequent selection.
• The temperature parameter (τ) in our selection process is crucial for maintaining this dynamic balance, preventing any prompt group from being permanently excluded and thereby ensuring the model retains both generalization and personalization capabilities. | round | t_g1 | t_g2 | t_g3 | t_g4 | t_g5 | v_g1 | v_g2 | v_g3 | v_g4 | v_g5 | |---|---|---|---|---|---|---|---|---|---|---| | 1 | 3(15%) | 2(10%) | 6(30%) | 5(25%) | 4(20%) | 3(15%) | 2(10%) | 6(30%) | 5(25%) | 4(20%) | | 2 | 6(30%) | 2(10%) | 6(30%) | 3(15%) | 3(15%) | 6(30%) | 2(10%) | 6(30%) | 3(15%) | 3(15%) | | 3 | 7(35%) | 2(10%) | 5(25%) | 3(15%) | 3(15%) | 7(35%) | 2(10%) | 5(25%) | 3(15%) | 3(15%) | | 4 | 6(30%) | 3(15%) | 5(25%) | 3(15%) | 3(15%) | 7(35%) | 3(15%) | 4(20%) | 2(10%) | 4(20%) | | 5 | 6(30%) | 3(15%) | 3(15%) | 3(15%) | 5(25%) | 7(35%) | 3(15%) | 3(15%) | 2(10%) | 5(25%) | | 6 | 6(30%) | 4(20%) | 3(15%) | 2(10%) | 5(25%) | 7(35%) | 4(20%) | 3(15%) | 1(5%) | 5(25%) | | 7 | 7(35%) | 4(20%) | 4(20%) | 1(5%) | 4(20%) | 7(35%) | 4(20%) | 4(20%) | 1(5%) | 4(20%) | | 8 | 6(30%) | 4(20%) | 4(20%) | 2(10%) | 4(20%) | 7(35%) | 4(20%) | 5(25%) | 1(5%) | 3(15%) | | 9 | 4(20%) | 5(25%) | 6(30%) | 1(5%) | 4(20%) | 5(25%) | 5(25%) | 7(35%) | 0(0%) | 3(15%) | | 10 | 5(25%) | 5(25%) | 5(25%) | 2(10%) | 3(15%) | 4(20%) | 7(35%) | 6(30%) | 1(5%) | 2(10%) |

Q5:Fairness and bias: Do the multi-group prompts risk encoding biased semantic clusters, especially if clients have class-imbalanced datasets?

A5: Thank you for your valuable Question. Our FedMGP method demonstrates superior generalization and carries no risk of encoding biased semantic clusters. In the main paper (Table 2, Dirichlet α=0.5), we show exceptional performance under severe class imbalance . In the cross-domain experiments in Appendix D, we apply an even stricter Dirichlet distribution (α=0.3) on each domain; even under simultaneous feature and label shifts, FedMGP maintains robust performance, directly validating the effectiveness of our dynamic aggregation mechanism.

Q6：The diversity loss encourages group-wise differentiation, but it remains unclear what kinds of semantics each group learns. Could the authors provide qualitative examples or visualizations?

A6: Thank you for the suggestion. As demonstrated in the Q2 experiments, prompt groups that are selected more frequently during training tend to aggregate broader, global semantics, while those with lower selection frequencies preserve more localized, client-specific semantics. These complementary behaviors balance generalization and personalization. Although we cannot include visualizations in this submission, we will add this discussion with qualitative examples and visualizations in the final version.