Personalized Prompt Tuning for Unsupervised Federated Learning

1. In line 68, the paper states that "textual prompts tend to learn category-related information which may introduce conflicts due to label skews." Could the authors provide references to any existing literature that discusses this problem or present simple results to illustrate this statement?

2. In FedPP, clients are required to deploy CLIP models, specifically ViT-B32 in the experimental settings. This model may be too large for some clients to deploy. Could the authors provide results using a smaller model, such as ResNet50, for CLIP?

3. The connection between the Partial Prompt Aggregation (PPA) and Cooperative Pseudo-label Selection (CPS) methods appears to be weak. I did not observe any clear connections between these two methods. Could the authors clarify the motivations and connections between Sections 3.3 and 3.4 more clearly?

4. In line 242, how to determine $M$ in the server? Is this parameter manually decided, and does it vary for different datasets?

5. In line 270, the paper states that "the textual branch focuses on determining the classification boundaries by leveraging category information." However, from Figure 3, it seems to suggest that "textual prompts are more personalized than visual prompts in CLIP." Figure 3 does not appear to illustrate any information about different categories. Could the authors clarify this point?

6. In the experiments, do all baselines use CLIP as their client models?

7. The number of clients in the experiments is relatively small, with only 5 or 10 clients. Is it possible to extend the experiments with a larger number of clients (like 50 or 100 clients) using smaller client models (smaller ViT or ResNet)?

1. The main limitation of the paper is the lack of theoretical analysis. FedPP introduces significant differences in training compared to previous baselines, which could impact training convergence, communication complexity, and privacy concerns. However, none of these issues are discussed in detail. Specifically, the following points require further clarification: a) Communication Cost: In FedPP, clients incur an additional cost by uploading the estimated pseudo-label distribution $U_k^{est}$, while avoiding the need to upload textual prompts since only visual prompts are aggregated by the server. How do these changes impact the overall communication cost? From a complexity perspective, is FedPP more or less efficient than other methods? b) Training Convergence: If clients update both visual and textual prompts locally but only visual prompts are aggregated by the server, can FedPP guarantee convergence? How does the convergence speed of FedPP compare to that of other methods? c) Privacy Concerns: The uploaded pseudo-label distribution $U_k^{est}$ could potentially reveal statistics about local data, as it is derived from high-confidence samples. This raises concerns that FedPP may pose a higher risk to client privacy compared to traditional federated learning methods, which only share model weights. Is there theoretical evidence to demonstrate that FedPP does not compromise client privacy?
2. The paper highlights the two modules in FedPP as key contributions. However, the Partial Prompt Aggregation (PPA) module can be simply summarized as aggregating only the visual part of the prompts on the server, and the aggregation method used in FedPP is essentially the same as prior federated approaches that rely on local data capacity for aggregation. Limited novelty and innovation is shown with this module. Furthermore, although Figure 3 demonstrates that the experimental divergence on textual prompt is more significant than that on visual prompt, it remains under-explored why aggregating only visual prompts leads to performance improvement compared to previous methods that tend to aggregate both prompts. Here some in-depth exploration to justify this aggregation approach is important for understanding but unfortunately is not provided.
3. The paper also contains numerous mathematical notation and spelling errors, impacting the readability to some extent. For example, in Equations (3) and (4), the symbol $i$ is used without explanation. It seems to represent client indexes, but the paper mentions that this is denoted by $k$. Additionally, in Tables 1 and 2, "zero shot" is misspelled as "zore shot." Furthermore, in appendix, there are also several symbol errors in Algorithm 1, such as $k \in M_t$ on line 9 and the inconsistent use of $p_\alpha$ and H_\alpha$on line 16. Equation 2 shows$p_{1/2}$and$H_{1/2} instead.

1. In the some federated learning settings, such as on-device federated learning, it is unrealistic to assume clients are able to run CLIP model which requires much more computational resources.
2. There is no clear insight for the partial prompt aggregation protocol. Why only uploading visual prompts are benefitical to the performance. It should be explained more clearly.
3. The number of clients in the experiments are too small. It would be helpful for the paper to have experiments with more than 100 clients.

-- Overclaimed statements about CLIP and FedPP. See C1.

-- The selection strategy for reliable samples needs further justification. See C2.

-- Generating pseudo-labels based on 'reliable' samples may not fit the local data distributions. See C3.

-- The paper lacks a corresponding security design for transmitting information between clients and the server. See C4.