Personalized Collaborative Fine-Tuning for On-Device Large Language Models

Thanks for your positive feedback.

Regarding privacy concerns, while the paper didn't extensively address them, we ensured a degree of privacy by keeping data on local devices. However, we agree that discussing privacy implications further would enhance practical value. Our primary focus is demonstrating that collaboration can enhance personalization performance in language modeling, which is a significant contribution. We also conducted experiments in realistic language heterogeneity scenarios to empirically validate our strategies.

Regarding the ablation on the size of fine-tuning data, we conducted supplementary experiments, increasing the training tokens by 100 times, resulting in 84 million training tokens per client in the multilingual setup. The ranking of all methods remained consistent with our paper's reported results.

With regard to your raised questions, we answer them as follows:

• The AG News dataset is very small, and we do not have enough data to construct a shared dataset across the clients, which is why we left those entries empty. However, we explored other possibilities for obtaining a publicly shared dataset and generated some synthetic news articles using ChatGPT. With this synthetic shared data, we obtained the following results: our strategy 3 performs on par with strategy 2, and both outperform all other baseline methods by a large margin. This further illustrates the practical potential of strategy 3, as one can easily create such synthetic datasets.

• Yes you are right about the mistake in Figure 1. We will make sure this is corrected in future iterations.

• Thanks for pointing this typo out, it should be “ the weight of edge (i, j) should denote to what extent can user j ’s gradients help to facilitate user i’s learning progress.”

1. We acknowledge our test scenarios are limited due to computing resources. Regarding datasets, we argue our constructed scenarios reflect realistic language heterogeneity, encompassing variations in language usage, topic-specific wording, and data-source-specific wording.

Additionally, as suggested, we tested our methods on a larger dataset (84 million training tokens per client, 100 times larger than in the paper, 1st Table below). We also compared our strategies against baseline methods across different LoRA ranks (2nd Table). Both show consistent trends as in the paper, confirming the validity of our methods. (S denotes strategy in tables)

Clarifications on symbol ambiguity: The algorithm assumes an all-to-all communication graph. While Figure 1 illustrates one client for readability, communication occurs in parallel for each client. Symbol $s_i$ denotes the message client i sends apart from $\Delta \theta_i$, which varies by strategy. Their meanings and their extra computation costs can be found in Table 1 in the paper.

2. Methodologically, we acknowledge limited novel contributions. However, this is the first work demonstrating collaboration leveraged to improve personalization performance (each client has a personalized state) in language modeling, which is a significant contribution. Note that the theoretical baseline is not available in practice, as it requires information on underlying data mixtures on the device. The theoretical baseline helps to demonstrate how well our method can perform even without knowing data distribution a priori.

3. Regarding why strategy 1 underperforms, please refer to Figure 8 in the appendix: In strategy 1, the trust matrix converges to a uniform matrix during training, resembling FedAvg's performance.

Thanks for your helpful feedback.

Regarding the differences between strategies 2 and 3, we identified three major points: 1). Strategy 2 learns a sparser pattern than strategy 3, with many entries in the trust matrix being close to 0, unlike in strategy 3. See Figure 4 of the paper. 2). We observed that the trust matrix in strategy 3 is more stable across communication rounds compared to strategy 2, with fewer abrupt changes in values. 3). The trust matrices from strategy 3 have higher diagonal values than those from strategy 2.

We cannot definitively explain why strategy 2 underperforms, but we speculate that the more stable trust matrix and higher self-trust in strategy 3 contribute to stabilizing the learning process, leading to better results. We plan to include more visual evidence on the evolution of learned trust matrices in future iterations.

Yes, we acknowledge that the method cannot scale up very well to a large number of clients. Being difficult to scale up is a general problem in all collaborator-selection type personalized methods, including ours. However, we emphasize that in this project, we target stateful clients (each client has a personalized model state) instead of stateless clients as in cross-device setup[1].

Regarding the incremental contribution comment, methodologically, we acknowledge that we are not making novel contributions. However, to the best of our knowledge, this is the first work demonstrating that collaboration can be leveraged to improve personalization performance in language modeling. By personalization, we mean that each client has a personalized state. We consider this a significant contribution. In language modeling, the balance between personalization and collaboration is not straightforward. Despite the drastic differences between categories, there are notable similarities in sentence structure and word choices, alongside domain-specific word choices. We would also like to draw your attention to Table 4, which shows that in different scenarios, the performances of FedAvg and Local methods vary significantly, and our collaborator-selection-based method can give consistently good performances.

Thanks for the related work suggestions. We will make sure to include them in future iterations.

[1] Kairouz et al. Advances and Open Problems in Federated Learning. Foundations and Trends in Machine Learning Vol 4 Issue 1.

Thanks for your comments and feedback.

Regarding the motivation, we are not targeting cross-device federated learning; our goal is to achieve a personalized state for each client, resembling more of a cross-silo setup, which requires stateful clients. This paper focuses on understanding personalization and collaboration within the context of privacy-preserving language models. Our experimental setup is similar to [1]. Although we considered a fully connected setup here, partial participation per communication round is orthogonal to our scheme and can be integrated. However, investigating the effects of partial participation in detail is beyond the scope of this paper.

Regarding the term "trust," you raise a valid point. In this project, our use of the term is motivated by social science literature: clients can be seen as members of social groups who make decisions (personalized models) through interaction with other members. This interaction, performed via collaborator selection, depends on the trust between group members, with similarity serving merely as a metric. We will clarify this in future versions.

We want to clarify that we are not providing state-of-the-art personalized federated learning algorithms. While many methods are applicable, this is the first work showing that collaboration improves stateful personalization in language modeling. Moreover, we have baseline methods including: 1) FedAvg and local training, demonstrating the need for both personalization and collaboration for better validation perplexity; 2) a theoretical baseline using a collaboration graph based on underlying data mixture similarities.

The theoretical matrix assumes known data mixtures for each client. For instance, if client 1 has 1/4 German and 3/4 French texts, and client 2 has 1/4 German and 3/4 Italian texts, the theoretical weights are the dot product of [1/4, 3/4, 0] and [1/4, 0, 3/4], followed by row normalization. While not necessarily the optimal trust matrix, it uses underlying mixture information (unrealistic in practice), hence "theoretical". It is worth noting that our strategy can even outperform this baseline, emphasizing the importance of a dynamic collaboration graph.

Hope those clarifications clear your doubts. We welcome further discussions and would appreciate a score adjustment if we manage to address your raised concerns. Thank you.

[1] Sun et al. Improving LoRA in Privacy-preserving Federated Learning. ICLR 2024

Thanks for your response and score adjustment. We will include your helpful comments in future iterations.

We address your concerns as follows:

• We opted for a fully connected topology because we are focusing on a cross-silo scenario with a small number of clients. This is the case for the standard federated learning setup, where the server can easily relay meta-data between all clients (see Table 2.1 in [4]). In the more general decentralized case where a different connected topology is preferred, one can replace the current trust matrix W with a sparse version, where some entries are zero. Our setting thus includes both federated learning as well as arbitrary decentralized training. As long as the graph represented by W is strongly connected (i.e., every vertex is reachable from every other vertex), our algorithm should still function, though convergence will be slower.

We provide experimental results on the training time (X times the needed training iterations as in the fully connected case) required for a ring topology to achieve the same perplexity level as a fully connected topology in the table below. Note that NA indicates we did not reach the same perplexity after ten times the training iterations. In high heterogeneity scenarios with a very sparse ring topology, each client is allocated a specific category. This can result in cases where adjacent clients in the ring do not share any categories, leading to an oracle trust weight of 0 and hence the NA entries. Despite this, our algorithm can still learn, converging to a perplexity approximately one point higher than what is reported for the fully connected case.

2. We summarize the insights as follows:

• From strategy 1 underperforming the other two strategies, we conclude that predictions are more informative than model weights in identifying collaborators in language modeling.
• Both as predictions-based collaborator selection methods, strategy 3 outperforms strategy 2. We further investigated the evolution of learned trust matrices. We found the trust matrices in strategy 3 are more stable across communication rounds compared to strategy 2, with fewer abrupt changes in values, and the trust matrices from strategy 3 have higher diagonal values than those from strategy 2. We cannot definitively explain why strategy 2 underperforms, but we speculate that the more stable trust matrix and higher self-trust in strategy 3 contribute to stabilizing the learning process, leading to better results.

3. As we mentioned in an earlier response, we are not proposing a SOTA method, instead we aim to understand how to balance personalization and collaboration. Moreover, as we focus on a server-less setup, the standard way is to use gossip averaging algorithm (which does collaborator selection), without getting a global model state involved [1,2,3]. That is also a reason why we did not compare personalization algorithms where a server is assumed.

Thanks a lot and we welcome further discussions if things are unclear.

[1] Sui et al. Find your friends: personalized federated learning with the right collaborators

[2] Zhang et al. Personalized Federated Learning with First Order Model Optimization

[3] Fan et al. Collaborative Learning via Prediction Consensus

[4] Kairouz et al. Advances and Open Problems in Federated Learning