FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models

1 We are willing to release our code and have made the project available at https://anonymous.4open.science/r/FOCoOp/. It will be included in the final version of the paper.

2 $\mathfrak{c}()$ in Eq.(7) is the cost function, which can be computed via L2-norm distance.

3 d+ and d- indicate the positive and negative distance set, respectively. Specifically, $d^+$ means we compute the set by satisfying $\eta$ percentile of positive cosine distance, while $d^-$ means the set satisfying $\eta$ percentile of negative distance.

4 We will carefully revise our presentation and fix the typos.

5. Clarification on the Role and Aggregation of Prompts. As described in lines 78–81, the three sets of prompts serve distinct roles:

Global prompts capture the shared ID global distribution across all clients.

Local prompts capture each client's ID personalized but heterogeneous distribution.

OOD prompts are trained to mismatch with ID data, preventing them from mistakenly aligning with samples that are unseen locally but present in other clients.

During training, each client sends both global prompts and OOD prompts to the server, while leaving local prompts at clients. The server aggregates global prompts to learn a consistent alignment with the global distribution, and selects the most dissimilar OOD prompts to represent a consistent misalignment across clients. Since OOD prompts are shared and distribution-agnostic, they generalize well to open-world OOD datasets, leading to superior performance in OOD detection tasks. We will elaborate on these design choices more clearly in the final version.

6 Explanation on FOCoOp’s greater impact on OOD detection. We appreciate the reviewer’s observation. FOCoOp is the first method that is designed to enhance OOD robustness while maintaining strong performance on standard FPL tasks. This is indeed reflected in the results: while improvements on ID and ID-C generalization are moderate (as existing FPL methods already achieve strong performance), FOCoOp demonstrates substantial gains in OOD detection. This discrepancy arises from our explicit design: FOCoOp introduces OOD prompts and uses a DRO objective to explore a broader prompt space. While this helps generalization to some extent, the major benefit lies in detection. Specifically, we distinguish between seemly OOD prompts (which match with ID samples that are unseen locally but seen globally) and strict OOD prompts (which are mismatched with all ID data), enabling more accurate identification of OOD data. We will clarify this distinction and its impact in future revisions.

7 Clarification on aggregating prompts. In lines 157.left-158.left, we present a general formulation of PFL and refer to prompts matched with local data on each client as local prompts. As described in lines 246–247, we also explicitly introduce the method of FOCoOp, which first aggregates global prompts across all clients, and calibrates global prompts later by Eq. (10). We will clarify this distinction more clearly in the next version.

1 We will carefully refine current version and correct typos.

2 The supplemental materials consist of five sections: (A) related work details, (B) algorithms, (C) theoretical analysis of optimization, (D) datasets and implementation, and (E) additional experimental results. In Section E, we report the results on CIFAR-100 and TinyImageNet under different Dirichlet distributions (Tables 6–7), which are consistent with Tables 1–3 and further verify OOD robustness. Table 4 has been split into Tables 8 and 9 for clarity. Tables 10–11 provide numerical results for Figure 4 (OOD detection on various OUT datasets), while Tables 12–13 are numerical results corresponding to Figure 5 (generalization on different ID-C datasets). Figures 8–9 are also enlarged for better readability. We will add them in a future version.

3 Clarification on data usage in FPL

We do not use OOD data when optimizing prompts. Please refer to Answer 4 in Review zfft for further details. As indicated in Eq. (1), the final evaluation targets minimizing classification errors on ID and ID-C test sets, and minimizing the failure to detect OUT data. We will clarify this more explicitly in the future. A summary of training and evaluation settings is provided below:

Training datasets:

CIFAR-100, TinyImageNet, Food101, DTD, Caltech101, Flowers, and OxfordPets, each split into train/test sets and simulated as heterogeneous clients using Pathological or Dirichlet partitioning.

DomainNet and Office-Caltech10 use an N-1 domain split strategy.

Evaluation settings:

ID ACC: Accuracy on the test sets of the above training datasets.

ID-C ACC: Accuracy on CIFAR-100-C, TinyImageNet-C, and the held-out domain in DomainNet and Office-Caltech10.

OOD FPR95 / AUROC: Evaluated on Places365, Texture, iSUN, LSUN-C, and LSUN-R.

4 Baseline Selection. We select baselines from the state-of-the-art (SOTA) personalized federated learning (PFL) methods and centralized prompt-based OOD detection methods. For centralized prompt-based OOD baselines, we choose GalLop and LAPT due to their strong performance in both generalization and OOD detection. We also include LoCoOp, as GalLop is proposed as an improvement over it.

5 Our computation cost is comparable to existing FPL methods such as PromptFolio and FedOTP, with the global and local prompt computation cost being $\mathcal{O}(KEBCd)$. The introduction of OOD prompts adds an extra but manageable overhead of $\mathcal{O}(KEB(C+U)d)$, where $U$ is the number of OOD prompts. Since $U$ is a controllable parameter, incorporating OOD prompts is a reasonable choice for significantly improving OOD detection performance. All notations are defined in our main paper, i.e., client numbers $K$, local epochs $E$, averaged batch size $B$, class number $C$, and prompt embedding $d$. Furthermore, we measured the average computation time for each communication round. An extra set of OOD prompts only increases computation cost by 4.76%, and 1.61% for CIFAR100 and TinyImageNet. The most computational burdens stem from encoding features rather than prompt learning. This indicates that employing three sets of prompts remains a practical and efficient design. FOCoOp containing BOS and GOC causes slightly larger but still controllable computation, which increases cost by 1.58s(9.06%) and 1.64s(6.18%) for CIFAR100 and TinyImageNet, respectively.

6 We will carefully refine our presentation and correct typos.

1 Inconsistency in maintaining local OOD robustness in FPL.

FOCoOp aims to improve the OOD robustness of FPL concerning the global distribution, which covers all clients' training data. The consistency we focus on lies in detecting semantic shifts beyond all client data and ensuring generalization under covariate shift and client heterogeneity within the global distribution.

However, this consistency cannot be directly achieved by prompt tuning for local robustness. While prompts can transfer the knowledge of foundation models, they are adapted to heterogeneous local data during training, leading to inconsistent matching patterns across clients. This makes prompts wrongly identify any data different from local distribution as OUT, degrading both generalization and detection. Results in Tab. 1-3 reflect this limitation. PromptFL and FedLoCoOp lack consistency mechanisms and underperform in all metrics. Similarly, FPL methods like FedOTP and PromptFolio aim to balance generalization and personalization, yet still fail in detection.

2 Reason for OOD prompts in client match unseen data from other clients.

This is a general case caused by heterogeneous data distributions across clients. In federated settings, clients often have imbalanced or missing classes, e.g., client 1 may only have data of dog and cat classes, while client 2 has dog, cat, and fish. During training, OOD prompts are optimized to avoid resembling locally seen data. However, lacking the reference of global distribution, OOD prompts may mistakenly match with samples locally unseen but present in other clients. For instance, in client 1, OOD prompts may falsely match "fish" images as OUT, but fish images are in client 2.

3 We realize the goal of OOD prompts, i.e., capturing misalignment between local ID data and the OOD prompt context, by explicitly optimizing Eq. (5). Specifically, we penalize higher similarities between OOD prompts and local ID data(Eq. 3), and ensure the total similarity of OOD prompts is lower than that of ID prompts(Eq. 4). This design notably improves OOD detection across all evaluated tasks in Sec 4.2. For example, it consistently improves the detection of different semantic shift datasets in Fig.4. However, since each client only observes a subset of the global distribution, the learned OOD prompts may mistakenly match ID samples unseen in this client but valid from other clients as OUT. Therefore, the GOC module is used to enhance consistent OOD robustness across clients.

4 Experiments on exploring OOD data. We clarify that only in-distribution (ID) data is used during training, which aligns with federated settings where OOD data is typically unavailable. Our method does not explicitly explore OOD data, instead, it uses a DRO-based objective to explore prompt space, allowing global (OOD) prompts to better match (mismatch) ID samples.

All OOD data (both ID-C and OUT) are used only for evaluation(main paper Table 5), not for training prompts.

Next, we evaluate OOD robustness on various OOD datasets, i.e., (1) Tables 1–2 show FOCoOp’s strong generalization to ID-C datasets (e.g., CIFAR100-C, TinyImageNet-C) and its improved detection of OUT data (e.g., Texture), (2) Extra evaluation of various OUT datasets also verified that FOCoOp can consistently enhance OOD robustness. These confirm the effects of exploring prompts in FOCoOp.

5 We can verify the clear separations for both class and distribution levels by similarity matrix. In link, we model FOCoOp on Cifar10 (10 ID prompts and OOD prompts), and sample 100 images per class to compute the average of similarities between images and prompts. The diagonal of the ID prompt matrix shows the highest similarities, suggesting intra-class alignment and clear class separation. Meanwhile, the similarities of OOD prompts are notably lower than those of ID prompts, further indicating clear distribution separation.

6 We extend our evaluation to ResNet50 on CIFAR100 and TinyImageNet. The results validate that FOCoOp maintains strong generalization and detection capabilities even with smaller models.

Pathological Non-overlap (10 clients, K=10)

7 Privacy analysis. Prompts encode only class-level statistics rather than instance-specific information, making it non-trivial to infer original individual image data stored on clients. The privacy of class-level statistics can be strengthened by applying differential privacy on prompts before sending to server.

8 We will carefully fix the typos mentioned.

1Reasons for using optimal transport (OT).

OT is a powerful tool for comparing distributions, and is used for different goals in BOS and GOC.

In BOS, OT is used to constrain uncertainty set in distributionally robust optimization (DRO). Unlike KL-divergence, capturing categorical distribution without considering geometry in feature space, OT divergence preserves the geometry of latent feature spaces, which is vital to text-image feature matching in VLMs. As noted in lines 211.left-188.right, global prompts enhance robustness in ID data in all clients and covariate-shift data near ID data, making OT as a suitable constraint. However, we adopt unbalanced OT (UOT) to constrain the uncertainty set of OOD prompts. Because it combines the strengths of OT and KL-divergence, capturing both geometric and non-geometric semantic shifts. In GOC, it uses semi-UOT to improve discrimination between global prompts and OOD prompts. By coupling global prompts and all OOD prompts, we can select seemly OOD prompts based on the close distance, and distribute mostly distant (strict) OOD prompts to all clients.

2 The reason for creating worst-case prompts and how to realize wider distribution exploration.

We obtain the worst-case prompts via DRO, as defined in Eq. (6). Specifically, we construct an uncertainty set $P$ for global prompts, which is centered around the original distribution of global prompts $P_0$​, and bounded by an OT divergence threshold $\eta_1​$. Within this set, we identify and optimize the perturbed prompts that yield the worst performance, as formulated in Eq. (6). The distribution of worst-case prompts is denoted as $\hat{P}$. The same procedure with UOT-divergence is applied to OOD prompts, where original distribution $Q_0$, uncertainty set $Q$, and worst-case prompts $\hat{\boldsymbol{t}}^o \sim \hat{Q}$ are corresponding notations.

Since we match images with textual prompts that yield the worst performance, the matching process is no longer point-to-point. Instead, it becomes a point-to-uncertainty-set matching, enabling more robust alignment in a wider semantic space.

3 In Eq.(7), $\mathfrak{c}(\hat{t}^g,t^g)$ is the cost function of optimal transport, we implement it by computing L2-norm distance.

4 Explanation on computing Eq.(8) to get seemly and strict OOD prompts. As in line 270.left, we compute all costs between aggregated global prompts and OOD prompts from all clients, and we regularize the assignments for OOD prompts with soft KL regularization while the assignments for global prompts are balanced regularization. Then we solve the mapping $\pi$ by theorem 3.2, where we use ${\pi^*}^\top 1_{\text{KU}}$ as distance. Based on it, we can rank OOD prompts, and the seemly OOD prompts are close to global prompts, but strict ones are mostly distant. The whole procedure can be found in lines 266.left-265.right.

5 We will add explanations for server block in Fig. 2. Specifically, in global-view OOD consistency, we use Semi-UOT to couple aggregated global prompts and all client OOD prompts. Then we take ${\pi^*}^\top 1_{\text{KU}}$ as probability/distance, to select seemly OOD prompts and strict OOD prompts for the next communication round. Based on mapping, we can re-assign global prompts with seemly OOD prompts by Eq.(10), and remain strict OOD prompts by Eq.(11). Then server sends the final global prompts and OOD prompts to all clients for consistency.

6 We sincerely appreciate the reviewer’s constructive suggestions, and will improve the clarity of our revised version.