Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation

Dear Reviewer UBUx:

Thank you for your positive response! We value the constructive feedback and have carefully considered the concerns raised. Below we provide detailed responses to address the reviewer's concerns:

Weakness & Question

W1 & Q1: The rationale for employing dual adapters should be clarified.

A1: We apologize for the confusion. In the vision branch, we tend to introduce the adapter architecture to reshape image features. We observe that both global and local adapters contribute to optimizing the visual feature space, thereby enhancing vision-language alignment. Building on this insight, we explore their combination (dual adapters) to enable mutual reinforcement between consensus and style knowledge. This design takes full account of the nature of federated learning, helping to preserve the generalization capability while improving personalized representation.

W2 & Q2: Why does the performance on DomainNet appear more promising compared to the other three datasets?

A2: Thank you for your detailed observation. Compared to the other three datasets, DomainNet is significantly larger and exhibits greater domain diversity. This characteristic poses a greater challenge of visual feature diversity, which is the primary focus of our study. With targeted visual components (dual adapters with gate), our approach is more compatible with handling such complex scenarios compared to other methods. This is clearly demonstrated by its superior performance on DomainNet.

W3 & Q3: The empirical record on the associated costs.

A3: Thank you for this valuable suggestion. We measure the communication cost by the size of uploaded data per round per client, and computation cost by the number of floating point operations (FLOPs) in the same batch. Below we present a comparison of both communication and computation costs.

Table 2. Comparison of communication cost (KB) and computation cost (G).

The introduction of the general adapter in our method adds more global parameters, inevitably leading to an increased communication burden. Meanwhile, the local updates in our method follow the standard CLIP loss and involve only one combined prompt that needs to be encoded. As a result, the overall computational cost of our approach remains comparable to that of PromptFL.

Dear Reviewer 287K:

We sincerely appreciate your constructive and insightful comments. We will explain your concerns point by point.

Weakness

W1: Strange decoupling.

A1: A key challenge in decoupling prompts lies in achieving efficient collaboration between prompts with different attributes. One approach is to decouple and collaborate at the feature level, like FedOTP [1] and FedPGP [2]. Instead, we aim to decouple at the semantic level, i.e., in the prompt construction phase, and facilitate collaboration through the pre-trained encoder. Our goal is to coordinate decoupled prompts by leveraging prior knowledge from the encoder to generate a feature that incorporates both consensus and style knowledge.

W2: Unclear rationale for specific technologies.

A2: With respect to the textual modality, replacing the textual inputs of the CLIP model with prompts is a common approach to learning task-specific knowledge. Our work continues this textual paradigm, and we introduce a visual limitation building on this paradigm: visual feature diversity. We aim to address this challenge by dynamically adapting the high-dimensional visual feature space. A straightforward strategy is to leverage adapters, which are flexible and efficient. We will explicitly state the reasoning behind selecting each PEFT method for different modalities in Section 2.1.

W3: Writing issues.

A3: Thank you for your tips. We have corrected the error you noted in Line 81 and other potential issues.

[1] Global and local prompts cooperation via optimal transport for federated learning. Li et al. CVPR 2024.

[2] Harmonizing generalization and personalization in federated prompt learning. Cui et al. ICML 2024.

Dear Reviewer oran:

Thank you very much for your recognition of our work and for raising thoughtful questions and concerns about our work. We have carefully considered each comment and provided responses.

Weakness

W1: Analysis of t-SNE visualization.

A1: Thank you for your insightful question. Unlike traditional unimodal architectures, CLIP requires alignment between visual and textual space. In our work, each participant's visual space is optimized to better align with the textual space, which is also being refined simultaneously. This bidirectional optimization leads to improved overall performance, even though the visual space itself undergoes relatively minor changes.

W2: The results in Tab.1 and Tab.2 report improvement against a weak baseline rather than relatively SOTA methods.

A2: We will revise Tab. 1 and Tab. 2 to provide a more detailed comparison in the final version, not simply an improvement over the weak baseline. Thanks for your advice.

W3: The rationale behind selecting each specific technique is not clearly explained.

A3: For the textual modality, prompt tuning has emerged as an efficient way to learn specific semantic knowledge on downstream tasks. Therefore, recent research focuses on how to better use prompt tuning, as we do. For the visual modality, our goal is to go for dynamic adaptation of the visual space to address visual feature diversity, thus better aligning with textual space. The available methods are adapters or visual prompts. Similar to the textual prompt, we explore the combination of global and local components, which is easy to implement in adapter tuning. However, managing the trade-off between global and local components in visual prompts remains challenging, particularly in determining their placement and properties. To handle this trade-off in adapter tuning, we introduce a gating mechanism, which dynamically learns the summation weights to coordinate multiple generated features. Thank you for your constructive comments.

Dear Reviewer Zexp:

Thanks for raising thoughtful questions and concerns. We sincerely hope this point-to-point response allows the reviewer to update the score. We will add a more detailed comparison or analysis of the existing decoupled works in the final version.

Weakness

W1: Analysis of motivation. (Why decoupling prompts into independent components is insufficient to achieve better collaboration? Why define the decoupled in a combined manner instead of an independent component?)

A1: We apologize for the confusion. We define the problem with current decoupling methods as textual property loss. Specifically, after extensive training, the text encoder is capable of recognizing textual relationships between tokens—what we refer to as textual property. For instance, it tends to perform better with fixed texts like "[class] with a style of [domain]", demonstrating its ability to capture concepts such as style. However, methods that decouple prompts into independent components overlook this capability, as they process prompts in isolation while the encoder can only handle tokens that come in together. Additionally, this paradigm depends on newly designed losses to guide the optimization, which is irrelevant to textual properties. Therefore, such methods fail to learn the knowledge of textual relationships between independent prompts. We argue that textual relationship is crucial for coordinating decoupled prompts effectively. Unfortunately, current methods weaken these relationships, making them insufficient for effective collaboration.

Thus, we propose a combined approach, where multiple prompts are integrated into a single prompt. Decoupled prompts are fed into the textual encoder together. It can maximize the use of the encoder's knowledge to achieve better textual collaboration. As we demonstrate below, the method that simply combines global and local prompts outperforms existing methods, validating its effectiveness.

Table: Comparison of simple combination with several methods on Office31 and PACS with $\beta=0.0$. Simple combination is denoted as $[P_g, class, P_l]$, where $P_g$ is the global prompt and $P_l$ is the local prompt.

W2: The differences between FedDDA and DiPrompT.

A2: Thank you for your valuable feedback. The primary difference between FedDDA and DiPrompT is whether the decoupled prompts are independent or not. This also marks the key difference between FedDDA and other previous decoupled methods. Previous decoupled methods, including DiPrompT, decouple prompts into independent components, each designed with distinct properties. These properties are typically derived from global or local federated setups and are further enhanced through external loss signals or multi-stage training. Instead, our method combines decoupled prompts into a single prompt, which maximizes the use of prior knowledge to achieve better textual collaboration.

W3: The reason why DiPrompT is not included in the performance comparison.

A3: We fully understand your concern about DiPrompT as a baseline. However, DiPrompT and our FedDDA address different problems with distinct experimental setups. DiPrompT is designed for domain generalization, while our work focuses on personalized adaptation to optimize performance within each client's own domain. Unfortunately, the lack of open-source code ultimately prevents us from including this approach in our evaluation. Once it becomes available, we will make every effort to conduct a thorough comparison. Thank you for your insightful comment.

W4: Lack of experiments on analyzing the performance impact of the number of total clients.

A4: We conduct the experiments on the performance impact of the number of total clients as shown in the following Table. Our method constantly achieves better results than other methods.

Table: The performance impact of the number of total clients on PACS with $\beta=0.0$.