Dual Prompt Tuning for Domain-Aware Federated Learning

Q1: Fed-DPT is only applicable for vision-language models

R1: Thank you for your valuable comments. We agree that our method is specifically designed for vision-language models, but this does not mean that this study lacks contributions to the community. On the contrary, we believe that this work holds significant importance for research in federated learning.

With recent advances in self-supervised and weakly supervised learning, pretraining a large-scale foundation model such as CLIP followed by lightweight prompt tuning for downstream tasks is becoming a new paradigm of transfer learning. Under this condition, the traditional fine-tuning-based protocols may not be as effective, and how to fully exploit the prompt learning techniques in federated learning has not been well explored. Our proposed Fed-DPT algorithm, which introduces coupled prompt learning at both vision and language ends, can effectively transfer the knowledge representations learned during the pre-training of CLIP models to downstream federated learning tasks without the need for excessive trainable parameters and communication overhead. We believe this study can offer valuable insights for the follow-up research of both vision-language models and federated learning.

Q2: Comparison to traditional FL methods

R2: Thank you for your valuable suggestion. In response, we have now included a comprehensive evaluation comparing our proposed method with traditional federated learning approaches, such as FedBN, in the revised version of our manuscript. Specifically, we replicated the FedBN setup on the DomainNet dataset, focusing on a 10-class subset of the original 345-class DomainNet to ensure a relevant and fair comparison. The results are summarized in the table below, where it shows that our method constantly outperforms the baselines.

Q3: Effect of visual prompts

R3: Thanks for the comment. Regarding the concern you raised, we have provided a detailed explanation in Q2 of our overall rebuttal response.

Thank you for your question about the privacy. In detail, during federated training of our method, there are class names as well as text prompts need to be shared across clients. In the context of federated learning studied in this work, the class names are not considered as private information since the clients are dealing with the same classification task so they all have the same class names. Also, sharing text prompt does not involve direct privacy leakage and we demonstrate that our method has the same level of privacy protection as FedAvg in our overall response. It is a good point to think over the privacy and we have added a detailed discussion for it in the revised paper.

Q1: Comparison to pFedPG [ICCV'23]

R1: Thank you for your valuable suggestion. The difference between our Fed-DPT and the pFedPG [ICCV'23] method lies in the model architectures and prompt learning mechanisms. Specifically, pFedPG focuses on single-modality (visual) prompt tuning with only a ViT-Base image encoder employed. In contrast, our Fed-DPT builds on top of a CLIP model, leveraging both visual and textual prompts. As we demonstrated, this dual-modality prompt tuning method facilitates the capabilities of detecting inter-domain correlations so it is suitable for domain-aware federated learning scenarios. Here we conduct a comprehensive evaluation using the experimental setup of pFedPG, which concentrates on a ten-class subset of the 345-class DomainNet dataset. The results are summarized below, where our Fed-DPT exhibits significantly superior performance than pFedPG.

Q2: Clarifying the overall process.

R2: Thank you for your advice. We have combined Figures 1 and 2 in the revised manuscript to present a more clear framework. Further, to clarify the overall process, we include a pseudo-code of our Fed-DPT in the appendix. We hope these improvements in clarity can well address your concerns.

Q3: Chosen of L2 loss.

R3: Thank you for your constructive feedback. In this paper, we apply a simple L$_2$ loss between the normalized visual and textual features to optimize our model. Formally, we have

$\text{L}_2\text{Loss} = -\text{sim}(f_v, f_t) = -<\frac{f_v}{||f_v||}, \frac{f_t}{||f_t||}>,$

where $f_v$, and $f_t$ denote the representation vectors of the image and its corresponding text label, respectively. The general cross-entropy loss can be written as

$\text{CrossEntropy} = -\log\frac{\exp(\text{sim}(f_v, f_t)/\tau)}{\sum_{i=1}^n\exp(\text{sim}(f_v, f_t^i)/\tau)}.$

Since CLIP is a large-scale pretraining model that has established good vision-language alignment, the L2 distance between an image feature $f_v$ and a mismatched text feature $f_t^i$ tends to be large and often exhibits low variance to different $f_t^i$. Consequently, the normalization term $\sum_{i=1}^n\exp(\text{sim}(f_v, f_t^i)/\tau)$ of the cross entropy loss tends to be a constant positive value, especially when the number of classes $n$ is big (e.g., 345 for DomainNet). So under this condition, minimizing the cross entropy loss is approximately equivalent to minimizing the L$_2$ loss $-d(f_v,f_t)$, and we personally find that leveraging this L$_2$ loss leads to slightly higher accuracy and faster convergence than cross-entropy. We also added a figure (Fig. 2) in the appendix which summarizes the comparison of their convergence.

Q4: Performance improvements on DomainNet

R4: Thank you for your advice. Compared with OfficeHome and PACs datasets on which the Zero-Shot CLIP model already achieves high accuracy levels of 78.6% and 95.8%, DomainNet has a larger margin for improvements. As detailed in Table 1, our Fed-DPT reduces the performance variance among the six distinct domains in DomainNet, attaining the most significant improvements on the "information graph" and "quick draw" domains. This suggests that our domain-aware prompt learning method addresses CLIP's limitations in domain generalization, leading to significant robustness to image patterns. We have included more discussions on this point in the revised paper.

Q5: Significance of the domain-aware mechanism

R5: Thank you for your observation regarding the ablation study results. As indicated in Table 3, the Domain-Agnostic DPT, which employs both visual and textual prompts but does not incorporate our domain-aware mechanism, achieves a 9.9% improvement over the baseline Zero-Shot CLIP. Our proposed Fed-DPT further enhances this by 4.9%. This is a significant improvement in performance since the domain-aware mechanism does not require any additional model parameters compared with the Domain-Agnostic DPT approach. We will make it more clear in the revised version.

Q1: Cross-silo vs. cross-device scenarios.

R1: Thanks for your valuable comments. We agree that the methods based on large-scale pre-training models are more suitable for cross-silo scenarios instead of cross-device setups. However, it is important to clarify that the two scenarios are characterized by the number of participants, the capability of computing resources, and the nature of the data involved, while many federated learning methods have a specific focus on one of them.

Our method is basically designed to tackle domain shift challenges in federated learning, introducing a domain-aware prompt tuning mechanism and demonstrates effectiveness. In scenarios where each domain possesses substantial data with a distinct statistical distribution, our approach is particularly suited to a cross-silo federated learning context. We do not view this as a limitation; rather, it's a strategic focus. Solutions optimized for cross-device scenarios may not effectively address the unique challenges posed by cross-silo problems.

We appreciate your suggestions to clarify the applicable scenarios of our method, and we have included this explanation in the revised version of the paper.

Q2: Fine-tuning CLIP with FedAvg/FedProx.

R2: Thank you for your constructive comments. In our study, we intentionally avoid fine-tuning the CLIP model due to a couple of significant considerations.

Primarily, fine-tuning the CLIP model may easily break its well-established vision-language alignment, which could undermine CLIP's strong capabilities in domain generalization and open-vocabulary inference. So, in practice, the existing methods used to freeze CLIP's encoders and tune additional parameters such as prompt tokens and adapter layers from downstream tasks. An additional benefit of this protocol is its ability to produce favorable results without requiring a substantial volume of training data. In Table 8 (in the appendix), we obtain very competitive few-shot results by our prompt tuning technique.

Secondly, fine-tuning a large model such as CLIP entails a substantial increase in communication cost and a decrease in convergence rate. So, given the same number of training iterations, the fine-tuning protocol often falls short of prompt learning. To provide a clear comparison, we have included the following results in our revised paper, which illustrates the advantages of our method.