Federated Adapter on Foundation Models: An Out-Of-Distribution Approach

Dear Reviewer DApR,

Thanks for your valuable reviews, however, there are some misunderstandings about our paper and detailed responses to each question are provided below.

W1: Theoretical analysis is not related to the foundation model properties.

Our Theorem 1 and Theorem 2 are based on foundation model properties, which mainly rely on the pretrained head of foundation models promoting only the feature encoder to handle out-of-distribution (OOD) by learning invariant features. Specifically, the pretrained head of foundation models is fixed and taken as the optimal head for all tasks to reduce prediction disagreement for implicitly invariant features learning, similar inspiration as previous work [1]. Additionally, with the pretrained head, the generalization bound of the personalized model in FedFM can be further constrained by the invariant feature distance as illustrated in Theorem 2. However, conventional FL does not have the pretrained knowledge to guarantee invariant learning. We add more explanation of Theorem 1 and 2 in Section 3.1 and Appendix D for better understanding.

W2: Do not compare with conventional FL baselines.

We do compare our method with conventional FL methods adapted to FedFM, as described in Section 5.1 and Appendix C.2. The conventional FL methods we compared—pFedMe[2], FedLoRA [3], PRADA [4], and FedSDR [5]—have been adapted to FedFM with adapters for a fair comparison. The results shown in Table 2 support our claim that conventional FL methods are suboptimal for addressing OOD generalization in FedFM.

W3: The inappropriate use of foundation models.

We are focusing on the research direction termed as federated foundation models (FedFM) [6], where most foundation models are predominantly based on transformer architectures with large pretrained data originating from NLP. Our theories and methods are all universally applicable towards FedFM and do not rely on any specific characteristics of a particular subdomain. Similar to previous conventional FL methods (e.g. SCAFFOLD [7]) only conducting experiments in a subdomain (image models) for FL, we also utilize model from a subdomain (LLM from NLP) as exemplary instances for FedFM to validate our proposed theories and methods, and LLaMA stands out as one of the most popular open-source foundation models, boasting significantly greater parameter volumes (hundreds to thousands of times larger) compared to models like ViT or CLIP.

In addition, our method can be easily adapted to other transformer-based foundation models by simply replacing the LLaMA with other models like ViT. We also conduct experiments with ViT on OfficeHome dataset [8] as shown in the following table. The results show our method could outperform other personalized models (pFedMe, FedLoRA and PRADA) and have comparable results with global model (FedIT), which demonstrates the effectiveness of our methods towards different foundation models. Specifically, our proposed invariant feature-based regularization is designed to enhance the personalized model's OOD generalization ability by leveraging the global model's generalization capabilities. Since we employ FedIT as the global model, we expect that our personalized models will exhibit OOD generalization results comparable to those of FedIT. We have added a discussion in Appendix F.2 to discuss this.

Methods	Art	CliPart	Product	Real World	Avg
FedIT	68.11	56.66	77.18	77.94	69.97
pFedMe	54.72	41.25	59.22	60.67	53.96
FedLoRA	60.49	51.31	72.93	73.15	64.47
PRADA	54.73	41.25	59.24	60.68	53.96
FedOA	67.49	56.51	75.96	77.45	69.35

Dear Reviewer DPmc,

Thanks for your valuable reviews and your recognition of the key contributions of our work. Detailed responses to each question are provided below.

W1: The motivation of the proposed invariance-based regularization.

We propose a new framework towards OOD generalization of FedFM, both the integration of PEFT methods and the proposed invariance-based regularization are important parts of our design of the proposed framework. Yes, the integration of PEFT methods primarily addresses issues related to parameter scale, while the proposed invariance-based regularization, based on PEFT methods, targets OOD generalization issues. Furthermore, as explained in “Why feature distance-based regularization? ” of Section 3.2, the proposed invariance-based regularization could further address large parameter scale issues by using features for calculation, which are significantly smaller in size than parameters for computations in federated foundation models. Thanks for your advice, we have revised the introduction to make our motivation more clear.

W2: Evaluated on large-scale FL.

Yes, evaluation on a large-scale FL could further demonstrate our method’s effectiveness. However, given the constraints of computational resources and the use of large foundation models with billions of parameters, our experiments are strategically designed to validate the efficacy of our method and the soundness of the proposed theorems with limited resources. Additionally, we have conducted supplementary experiments that expand the number of clients to 30, as detailed in the table below, and compared the results with other baseline methods on reading comprehension (RC) dataset. The findings from these experiments underscore the scalability of our method. We have added this discussion in Appendix F.1.

	FedIT	PRADA	FedLoRA	FedOA
RC	58.04	39.30	46.90	58.84

Dear Reviewer Um8N,

Thanks for your valuable reviews, detailed responses to each question are provided below.

W1: Limit novelty and contributions.

Our paper is to further explore federated foundation models by proposing a new framework to consider out-of-distribution (OOD) generalization with theoretical guarantees.

• Our paper mainly focuses on OOD generalization of federated foundation models, while FedIT focuses on in-distribution performance.
• Our paper considers OOD generalization scenarios for both conventional and personalized FL, while FedIT only addresses conventional FL.
• We contribute theoretical analysis covering both OOD generalization and convergence, whereas FedIT lacks such theoretical analysis.
• We introduce a novel bi-level optimization framework targeting both global and personalized adapters for OOD generalization, in contrast to FedIT's singular focus on an aggregated objective for global adapter optimization on in-distribution performance.

W2: Objective (2) cannot learn invariant features.

Objective (2) focuses solely on the empirical loss related to the feature encoder $\Phi$, rather than on the empirical loss associated with the entire model $f$. This distinction implies that the empirical loss labels for the feature encoder of Objective (2) are based on features $z$ that are consistent across different environments (invariant features), rather than on the overall model empirical loss labels $y$ (e.g., [0,1]).

Objective (2) provides a succinct summary of previous FL work that employed invariant learning approaches (not only Invariant risk minimization). For instance, previous study [1] achieves objective (2) by directly using contrastive learning to narrow the gap between features of the same category across different domains and to distance features of unrelated categories, thus facilitating the learning of invariant features. In addition, work [2] implicitly achieves objective (2) by aligning inter-client’s heads to reduce prediction disagreement for implicitly invariant features learning. We add more explanation of the objective (2) in Section 2.1 for better understanding.

W3: Theorem 1 directly from previous work and Theorem 1, 2 does not hold.

Theorem 1 seeks to extend the theoretical results of previous work to federated foundation models settings. In Theorem 1, we claim that through the aggregation of federated foundation models with a pretrained head, it is possible to directly fulfill Objective (2). That is, during tuning, the pretrained head of foundation models is fixed and taken as the optimal head for all tasks ($w \in argmin_{w} R_e(w,\Phi_g)$ for all $e \in \mathcal{E}_{all}$) to implicitly learn invariant features, directly achieving the objective (following equation) like invariant risk minimization. By omitting the pretrained head, the objective of the aggregation of federated foundation models can be simplified as objective (2). This approach draws similar inspiration from previous work [2].

$\min_{\Phi_g} \sum_{e\in \mathcal{E}_{train}}\alpha_e R_e(w,\Phi_g)$

s.t. $w \in argmin_{w} R_e(w,\Phi_g)$ for all $e \in \mathcal{E}_{train}$

Based on the theoretical results of previous work, the generalization bound of the global model in FedFM can be further bounded as $Supp_{\cup\Phi}(|R_e(\Phi)-R_{e'}(\Phi)|)$. For easy understanding, we add more explanation of Theorem 1 in Section 3.1 and Appendix D.

W4: Details of the adopted PEFT framework.

Our methods are designed with high flexibility, allowing for the integration of various adapter-based PEFT methods, and all these adapter-based PEFT methods are unified under a common scheme that partitions parameters into two parts: one small trainable part and one large frozen part. Thanks for your advice, we already add more details of the adopted PEFT framework in Section 3.2.

W5: Scalability with larger clients.

Given the constraints of computational resources associated with large foundation models containing billions of parameters, our experiments are mainly designed in an ideal setting to validate the efficacy of our method and the proposed theorems. In addition, to demonstrate our method’s scalability, we conduct additional experiments involving an expanded group of 30 clients as detailed in the following table, compared with other baselines on reading comprehension task (RC). The results from this table further demonstrate the effectiveness of our method. We have added this discussion in Appendix F.1.

	FedIT	PRADA	FedLoRA	FedOA
RC	58.04	39.30	46.90	58.84