Enhancing Foundation Models with Federated Domain Knowledge Infusion

We thank the comments and questions.

>>> W1

Yes, the MLP layers are optimized during the training. We will clarify it in the final version.

>>> W2

We do not share any label information with the server. As introduced in Sec 3.1, clients only share the style information via the text prompt (style is clear) or the generated textual inversion token ( style is vague), for the data generation.

>>> W3

We clarify that the domain shift among clients also represents the data heterogeneity in FL [1,2]. In our experiments, the data from each client represents a unique domain and there is no crossover/mixup between clients. Under such a heterogeneous setting, our approach is still able to help enhance the capability of the foundation models with the help of local clients’ domain knowledge. Furthermore, we add more experiments of the data class-heterogeneity. With ImageCLEF-DA maintaining settings unchanged in Table 1 and Table 2, we partition the dataset with respect to the classes using the Dirichlet distribution in [3] to simulate the class heterogeneity. The performance of the results decreases slightly but is still higher than the classical methods.

[1] Heterogeneous federated learning: State-of-the-art and research challenges. 2023.

[2] A review of federated learning methods in heterogeneous scenarios. 2024.

[3] Harmonizing Generalization and Personalization in Federated Prompt Learning, ICML 2024

>>> W4

[CVPR’24] designs a prompt-tuning approach to solve the data heterogeneity challenge in FL. They learn shared prompts at the server side and match them with the local groups and assign group prompts to them. The main design covers how to learn shared prompts, group prompts, prompt selection, and an optimization method to iteratively learn different knowledge. In [ICML’24] finds the balance between generalization and personalization for the data heterogeneity. The approach is mainly a prompt-based approach, which is based on utilizing local personalized prompts with the help of a global prompt and an adaption term. Also, most of their designed modules are at the local side and they focus on the evaluation on the local clients.

Compared with them, we have 2 key differences: 1, motivation part: we focus more on how to enhance the capability of foundation models at the server side with the help from the local clients by knowledge infusion; 2, method part: our key method is not a prompt-based approach and most operations happen at the server side to release the computation burden at the client side.

>>> W5

The results in Table 1 and 2 are already cover the setting of 3--->3 in Table 4. Due to the limited time during the rebuttal period to run experiments, we report the 2--->4 setting as below. (We omit the decimal part due to limited space)

We see (1) given 2 in-domain combinations for training, the overall performance decreases compared with the setting where we have 3. This is because of the less training data, which causes the performance degradation; (2) Our approach still outperforms other baselines.

>>> W6

Our method can also work even without knowing the in-domain and out-of-domain of a given task and it can be scaled with more domains. Given the initialized number of adapters equal to the number of domains, our designed modules are capable of handling out-of-domain tasks. In particular, we report the results of different training and testing domains in Table 4, where we scale our approach to different numbers of domains. As the reviewer mentioned, given the assumption if we may not be able to access to the domain information that it is in-domain or out-of-domain and minimize the change of current approach, we could equally treat the data and use the label index with the maximum value in $\eta^{i}$ as the predicted label via eq 9. We provide the results in this case following the setting in Table 4 as below.

The performance has a slight decrease in a certain range but still outperforms the baselines. The results show the effectiveness of our method given the condition without knowing the domain information during the inference process.

We hope our reply sufficiently answers your question and addresses your concern.

We genuinely appreciate the reviewer’s valuable comments and questions. We would like to address them as follows for your review.

>>> W1 and >>> Q1

Thanks for your constructive comment and question. If we have a dynamic or numerous clients, we will group clients by the domains they belong to. In Sec 3.4.2 or Figure 3.c, we can conduct a basic model aggregation approach to get the aggregated $\bar{W}_n$ for each domain group n, e.g., average-based approach. Then we can replace the original $W_n^t$ with the domain group-level $\bar{W}_n$ The aggregation process will merge the knowledge from the same domain and can be generally plugged into the current proposed framework. Also, as it is a basic average operation of the model parameters, it may not raise too much extra computational burden. In our proposed approach, the number of the adapters is equal to the number of domains, and each domain adapter can only interact with one aggregated model parameter, as we discussed above. However, we totally agree with the reviewer’s comments about the setting with a large number of domains. Our current work is based on the cross-silo setting and we have stated that we will further explore the potentials of extending this approach to the cross-device setting in the conclusion part (line 434-439). We hope our reply can sufficiently answer your question and address your concern.

>>> W2 and >>> Q2

Thank you for the comment. We have actually considered the issues of synthetic data quality problems and discussed them in Sec. 3.3.2. We totally agree that the quality of generated data would have an effect on the performance. To solve this, in our proposed work, we design an estimation mechanism to estimate the quality of the generated data in Sec 3.3.2. In particular, we first obtain the prototype representation for each label category. After that, we calculate the cosine similarity of the representation of the generated data and the prototype to obtain the score \alpha. With this estimation, we add it as a weight to equations (6) and (7) (lines 245 to 248) to conduct mutual learning using quality assessments of the generated data. With a low quality of the synthetic data, the weight will be small and help control its effect on the overall performance.

Furthermore, to examine the effectiveness of this quality evaluation module, we also provide the ablation study in Sec 4.4, and the results are shown in Table 3. We remove the quality estimation module for the synthetic data and name the setting as FedAG_quality. We observe that the performance drops in the evaluation of the in-domain and out-of-domain data, which demonstrates the effectiveness of the quality evaluation module to help control the utilization of the synthetic data.

We do appreciate that the reviewer mentioned the possibility of adversarial samples in the synthetic data. We will explore this direction to further improve the security and bias of our proposed approach in future work. We hope our response fully clarifies your question and resolves your concern.

We genuinely appreciate the reviewer’s valuable comments and suggestions. We would like to address them as follows for your review.

>>> W1

In our main experiment, the amount of the synthetic data is equal to 10% of the real data for each domain. To further examine how the amount of synthetic data affects the results, we conduct a further study about the synthetic data volume in Appendix I. In particular, we randomly sample 75%, 50%, and 25% from the existing synthetic data in our main experiment and repeat the experiments while keeping all the settings unchanged. We observe that the performance will degrade with the reduction of the synthetic data. With very limited synthetic data (25%), our approach is still able to maintain an acceptable performance. The results demonstrate how the amount of the synthetic data affects the final results.

>>> W2

Due to the limited page number in the main paper, we put the pseudo algorithm in the last page of the Appendix. To further enhance the readability of our work, we will try to put a simplified version of the pseudo code in the main paper.

>>> Q

In our proposed framework, the clients share very limited and vague information with the server. We let the clients provide the style information via the text prompt or the generated textual inversion token. In particular, for easily distinguishable styles such as “Ghibili cartoon”, “Pablo Picasso”, direct text prompt can be used. For vague or ambiguous styles, one can use textual inversion, a technique that enables Stable Diffusion to learn a new embedding vector to represent this style from just a few sample images. This involves a brief training process, the system optimizes only this new embedding vector. It adjusts the vector so that when it's fed into the frozen Stable Diffusion model along with descriptive prompts (like "a photo in <my-new-style>"), the model generates images that look like the sample images. After the style tokens are gathered from clients, the server can simply apply the template such as “a clock in <style-token> style” as text prompt input to Stable Diffusion to generate synthetic data.

Besides that, to avoid the effects of the low-quality synthetic data, we further design a quality estimation module to measure the quality of the generated data and add that score to our optimization equation. We will add a more detailed description to clarify this part in the final version of our paper.

We sincerely appreciate the reviewer’s constructive suggestions and comments. We would like to reply respectively as follows.

>>> Response to Other Comments Or Suggestions

Thank you for the feedback. We will follow your suggestion to improve the figure and fix the typos in the final version of the paper.

>>> Q1

Thank you for the question. In our proposed approach, the number of adapters is equal to the number of domains. If we have more clients, we can group them into different groups based on the domains to which they belong. Then we aggregate the model parameters into one for each domain before we conduct the mutual learning process in our design. As for the computational cost, we freeze the encoders of the foundation models and only allow the parameters of adapters and the local model to be trainable and updated. Furthermore, we put most of our designed operations on the server side to release the burden of the local clients, as it is more common that the server side may have more flexible computation constraints than the local side. As for the communication cost, as we mentioned in Sec 3.1, we may need to transfer the synthetic data from the server to the clients in a one-time manner, which can be negligible. We appreciate the reviewer’s question and will investigate how to further reduce the computation and communication cost in future work. We hope our response can sufficiently address your concern.

>>> Q2

Thank you for the comment. It is possible that the real-world scenarios could be more subtle, but we believe that our method can work for these different scenarios as demonstrated with three datasets having different levels of domain shifts. In our approach, we proposed a cross-domain learning mechanism to capture the relationship among different domains and quantify it with a weight score $\beta$. Besides that, we add a regularizer to further adjust the attention based on the domains of the data. As the reviewer commented, given a subtle domain shift in the real-world scenario, PEFT could be a more effective way. However, we may think our method is still advantageous because we do not directly access the local data, while centralized PEFT does not have such advantages. We will explore this direction with some real world data and compare it with PEFT in our future work. We hope our reply can sufficiently answer your question.

>>> Q3

Thanks for the observation and the valuable question. We would like to provide our explanations as follows. First of all, as described in Appendix C, we use the ViT-B-32 for the image encoder on the server side. As it is pretrained, it has the basic zero-shot inference capability for image-related tasks. Secondly, for the baseline ViT_cen, we put the data from the domains together and tune all the parameters, which shows the degraded performance. One possible reason could be that we fine-tune all the parameters of this large ViT-based model with not enough training data. This may harm the previous well-pretrained model because of the under-training. This can also be verified by the results of CLIP_L and CLIP_A, where we conduct PEFT approaches with training LoRA and adapters only. With these two approaches, the performance will get boosted compared with ViT_cen.

We appreciate the reviewer's valuable comment. We will add the analysis above to the result analysis in Sec 4.2 and 4.3, respectively, in the final version of our paper.