FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models

W1. Thanks for the great comment, and we will describe our setting more clearly in the revision. The LLM service providers are not the FL server. The LLM service providers are the APIs (e.g., ChatGPT) that the clients will continue utilizing for inference in the deployment phase. Thus, we have an assumption that if the clients do not have model access, it is acceptable to send only data with a prompt to the LLM service providers for inference, which is realistic since the clients do not have more risk of privacy leakage compared with the deployment phase. In our framework, the clients do not need to send labels to the LLM server, which may contain more sensitive information (e.g., the diagnosis of a patient), and preserve the privacy of the clients’ labels which constitutes an important asset for the clients. The clients use the inference results received from the server and the local labels to compute the loss. Furthermore, our work is not limited to the setting in which the devices have no model access. The LLM APIs are not necessarily on the cloud, our work is also motivated by the setting that the clients are the devices with model access but without enough memory and computational power to compute the back-propagation of LLMs. It is also notable that training DNNs on edge devices is always a problem in real life (e.g., pytorch-mobile and TensorFlowLite still do not support on-device training GPU acceleration). In these settings, the devices can only conduct inference rather than back-propagation, and our FedBPT allows them to still participate in federated finetuning.

W2. Thanks for this constructive comment. Evolutionary algorithms are typically slower than SGD under the central training setting. However, in our design, each client plays the role of a “search node”, which improves the parallelism and accelerates the search process. To achieve the accuracies under non-IID settings of SST-2 in Table 1, FedPrompt requires more than 300 communication rounds, while our FedBPT only needs less than 100 communication rounds, which converges in fewer rounds. We will include our clarification in our revision with the assessment of convergence time for all the settings and datasets. There are many techniques to reduce the communication cost of gradient-based methods, but in many settings, gradient-based methods are not feasible, and we can only apply gradient-free methods, which are the focus of our paper. For example, when you are using ChatGPT, you do not have access to the model parameters and can only get the inference results by sending the request to the model through the API provided by OpenAI. Gradient-based methods can achieve relatively higher accuracy, but they are unfeasible in such realistic settings where the devices have no access to model parameters or have no power to conduct back-propagation.

W3. The length of the prompt is related to the trade-off between the memory cost and accuracy. It is true that a longer prompt can perform better. However, a longer prompt also causes higher memory costs during inference. We follow previous works [1,2] to set the length of the prompt in our paper. Based on previous works [3], LLMs always have a low intrinsic dimensionality. Thus, we reduce the dimension of optimized parameters to $d$ through a fixed projector to reduce the communication cost of FL and the computational cost of CMA-ES. We will include this discussion in our revision.

W4. Even though the datasets are simple, we still outperform other gradient-free methods in terms of accuracy. We agree that the classification task is not too hard in real life, but it is notable that this is the first paper to finetune LLMs in FL using a gradient-free optimization method. We follow existing works [1] on gradient-free optimization of LLMs to set our experiment dataset. It is always challenging to finetune DNNs using gradient-free methods, let alone LLMs. We need to work hard with the whole community to improve the performance and scalability of our proposed framework. However, we believe that we show the community it is a promising direction to apply gradient-free methods in fine-tuning LLMs through FL, which pushes the forefront of deploying FL in real life significantly. Notably, LLMs are becoming larger, and the clients are less and less able to perform gradient-based optimization on devices (even larger ones). The newer and better pre-trained LLMs would be more generalizable to prompt learning, which makes our framework more promising. This concern may come from the lack of limitation discussion in the paper. We will include this limitation and future works in the revised version.

[1] Sun, Tianxiang, et al. "Black-box tuning for language-model-as-a-service." ICML2022.

[2] Li, Xiang Lisa, et al. "Prefix-tuning: Optimizing continuous prompts for generation."

[3] Armen Aghajanyan, et al. "Intrinsic dimensionality explains the effectiveness of language model fine-tuning."

W1. Thanks for the great comment. The LLM service providers are not the FL server, which is the APIs (e.g., ChatGPT) that the clients will continue utilizing for inference in the deploying phase. Thus, we have an assumption that if the clients do not have model access, it is acceptable to send only data with a prompt to the LLM service providers for inference, which is realistic since the clients do not have more risk of privacy leakage compared with the deployment phase. We will claim this point more explicitly in the revised version. In our framework, the clients do not need to send labels to the LLM server, which may contain more sensitive information (e.g., the diagnosis of a patient), and therefore preserve the clients’ label privacy which constitutes an important asset for the clients. In addition, the LLM APIs are not necessarily on the cloud, our work is also motivated by the setting that the clients are the devices without enough memory and computational power to compute the back-propagation of LLMs. It is also notable that efficiently training DNNs on edge devices is always a problem in real life (e.g., pytorch-mobile and TensorFlowLite still do not support on-device training GPU acceleration). In these settings, the devices can only conduct inference rather than back-propagation, and our FedBPT allows them to participate in federated finetuning.

W2. Thanks for the suggestion to include more baselines. Our goal is to design a framework that enables resource-limited devices without model access to finetune LLM in a federated fashion. For example, when you are using ChatGPT, you do not have access to the model parameters and can only get the inference results by sending the request to the model through the API provided by OpenAI. Gradient-based methods can achieve relatively higher accuracy, but they are unfeasible in such realistic settings where the devices have no access to model parameters or have no power to conduct back-propagation. We conduct experiments primarily to show that we can improve accuracy compared with the other gradient-free methods with extremely efficient communication. Even though it is not our focus, we will include more discussion around these related techniques in our revision.

W3. This is a great comment. Actually, there have been many papers [1,2] discussing data privacy leakage through the LLM gradients in FL, but there is no paper claiming that prompt updating can cause data leakage. There is no existing way to evaluate the relationship between the prompts and the training data. We conduct experiments to explore the similarity of the learned prompt embedding $Az$ to the word embeddings in the training data vocabulary. For SST-2_RoBERTa, the maximal cosine similarity between the prompt embedding and the training word embeddings is only 0.22, and the mean cosine similarity is 0.08, which is relatively low. We “translate” the learned prompt embedding on SST-2_RoBERTa to the natural language by replacing the prompt token with the word with the highest similarity in the vocabulary. The translated prompt is ['Ġcould', 'Ġ13', 'Ġpost', 'Ġmillion', 'Ġreal', 'Ġmen', 'Ġsuch', 'h', 'Ġwho', 'ie', 'ĠAmerica', 'Ġtotal', 'Ġ2015', 'z', 'Ġjust', 'ĠBut', 'Ġweeks', 'Ġteam', 'Ġformer', 'Ġopen', 'Ġmorning', 'ĠHouse', 'Ġcomes', 'Ġwomen', 'Ġformer', 'Ġ2015', 'Ġfinal', 'e', '2', '8', 'Ġof', 'Ġoil', 'Ġ9', 'ĠMr', "'m", 'Ġsays', 'Ġbut', ']', 'Ġfuture', 'ak', 'com', 'b', 'Ġsales', 'Ġc', 'ist', 'Ġplay', 'ĠNews', 'b', 'Ġnext', 'ĠS']. We compared it with the training samples and did not find a clear relation with the input. Thus, we believe that it is safe to transmit the prompt embedding to the FL server and update the global prompt.

Q1. A practical example is that the CMA-ES is deployed on the client’s device, and the LLM is deployed on an LLM service provider’s cloud (e.g., OpenAI). The clients conduct inference through the inference API (e.g., ChatGPT) and compute the evaluation results locally based on the inference results received from the API.

Q2. Please refer to our response to W3.

Q3. FedPrompt is actually a baseline that treats z as a model and applies FedAvg. We will add discussion around this in the revised paper.

[1] Melis, L., Song, C., De Cristofaro, E., and Shmatikov, V. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 691–706. IEEE, 2019.

[2] Fowl, Liam, et al. "Decepticons: Corrupted transformers breach privacy in federated learning for language models." arXiv preprint arXiv:2201.12675 (2022).

W1. Evolutionary algorithms are typically slower than SGD under the central training setting. However, in our design, each client plays the role of a “search node”, which improves the parallelism and accelerates the search process. To achieve the accuracies under non-IID settings of SST-2 in Table 1, FedPrompt requires more than 300 communication rounds, while our FedBPT only needs less than 100 communication rounds. CMA-ES algorithm only updates the multivariate Gaussian distribution of the prompt with 500 variables, which is super efficient and can be neglectable compared to model training with SGD. Thanks for this constructive comment. We will include our clarification in our revision with an assessment of convergence time in different settings.

W2. Thanks for the comment, and we are glad to introduce the contribution and novelty of our paper. First, we want to claim that we are motivated by a very realistic setting in which the clients use LLM APIs and have no access to the model parameters. For example, when you are using ChatGPT, you do not have access to the model parameters and can only get the inference results by sending the request to the model through the API provided by OpenAI. In this practical setting, no existing FL solution works, and our work is the first successful attempt. Technically, our paper is not simply combining BPT and FedAvg. We show that simply combining BPT and FedAvg cannot achieve good performance, as shown in our results (the “FedAvg-BBT” baselines). We explore in our paper why such a simple combination is suboptimal and provide an adaptive aggregation method with theoretical derivation, which is innovative. Furthermore, we expose and solve the problem of local overfitting when using BPT as local optimization in non-IID settings through input perturbation, which is also original and novel. In addition to reducing communication costs and supporting the clients to conduct optimization through inference only, a novel and effective aggregation with theoretical guidance is always thought of as a major contribution to the FL community. In addition, unveiling the optimization issue and improving the performance when applying BPT in non-IID settings is also a major contribution to FL.

W3. We appreciate that more advanced baselines could be provided. However, we want to claim that outperforming the gradient-based methods in accuracy is not our goal. Our goal is to design a framework that enables resource-limited devices without model access to finetune LLM in a federated fashion. For example, when you are using ChatGPT, you do not have access to the model parameters and can only get the inference results by sending the request to the model through the API provided by OpenAI. Gradient-based methods can achieve relatively higher accuracy, but they are unfeasible in such realistic settings where the devices have no access to model parameters or have no power to conduct back-propagation. We conduct experiments primarily to show that we can improve accuracy compared with the other gradient-free methods with extremely efficient communication. Even though it is not our focus, we will include the discussion around these advanced gradient-based prompt tuning methods in our revision.

Q1. A practical example of our setting is that the CMA-ES is deployed on the client’s device, and the LLM is deployed on an LLM service provider’s cloud (e.g., OpenAI). The clients conduct inference through the inference API (e.g., ChatGPT) and compute the evaluation results locally based on the inference results received from the API and the local ground-truth label. Thus, the clients do not need to have model access.

Q2. As claimed above, we focus on realistic settings. Additionally to the limited access to the model parameters and constrained computational power, it is also practical that the clients have limited labeled data points and different data distributions, i.e., non-iid settings. Fine-tuning the whole LLM with a limited number of samples in FL causes model divergence across clients and cannot achieve good performance after aggregation. Thanks for this question, and we will include this discussion in the paper.

W1. Thanks for this constructive comment. Evolutionary algorithms are typically slower than SGD under the central training setting. However, in our design, each client plays the role of a “search node”, which improves the parallelism and accelerates the search process. To achieve the accuracies under non-IID settings of SST-2 in Table 1, FedPrompt requires more than 300 communication rounds, while our FedBPT only needs less than 100 communication rounds to converge. We will include an assessment of convergence time for all the settings and datasets in our revision.

W2. As shown in Equation 7. Not only the noisy data is used, but also the clean data is used to evaluate the local prompt. Thus, utilizing samples with 80% of tokens masked does not mean that we lose 80% information of the samples. We want to find the prompts that make the LLM “confident” about the clean samples and “confused” about the noisy data. Thanks for the suggestion of applying FedProx. It is not straightforward to apply the proximal regularizer in FedProx to gradient-free optimization, and we will discuss this in the discussion section.

Q1. Gradient-free methods were applied to finetune LLM prompts recently [1,2]. However, we did not find any existing work applying gradient-free optimization in FL for LLMs. We believe this is the first successful attempt in this field. We appreciate it if the reviewer could provide a reference if there are similar works. Additionally, our paper is not simply combining BPT and FedAvg. We show that simply combining BPT and FedAvg cannot achieve good performance, as shown in our results (the “FedAvg-BBT” baselines). We explore in our paper why such a simple combination is suboptimal and provide an adaptive aggregation method with theoretical derivation. Furthermore, we expose and solve the problem of local overfitting when using BPT as local optimization in non-IID settings. In addition to reducing communication costs and supporting the clients to conduct optimization through inference only, a novel and effective aggregation with theoretical guidance is always thought of as a major contribution to the FL community. In addition, unveiling the optimization issue and improving the performance when applying BPT in non-IID settings is also a main contribution to FL.

Q2. Thanks for the comments. The reason why standard aggregation (e.g., FedAvg) is suboptimal when applying BPT is two-folded: 1. The clients locally optimize the CMA-ES parameters parameterized by multivariate normal distribution statistics. Mathematically, the arithmetic mean of the covariance matrices is not equivalent to the covariance matrix of our targeted optimal global distribution. Hence FedAvg is not effective under this setting. 2. CMA-ES, like other gradient-free methods, is a random search algorithm that cannot guarantee to achieve a local optimum as with gradient-based optimization algorithms. Therefore, directly averaging optimal and inferior local search results makes it difficult to achieve a global optimum. We will emphasize this more in our revision.

Q3. F^k(z) represents the average loss value corresponding to the prompt z. Transmitting the average loss value will not cause serious privacy leakage of the data since the FL server has no access to the loss value corresponding to each sample. Transmitting local loss was also utilized in other FL works such as [3]. Thanks for this constructive suggestion, and we will include this clarification in our revision.

Q4. We focus on the common FL setting rather than the personalized FL setting. Our goal is to train an optimal model with good performance on the global data distribution. In non-IID settings, the clients might have unbalanced training data distribution, but we should not train biased models for them since we cannot assume the data distribution after deployment. The samples of the dominant class are not “easy samples” within the global data distribution. The bias problem is caused by the unbalanced local training data in non-IID settings.

Q5. We propose the masking technique to improve the accuracy of non-IID settings and mitigate the accuracy gap between the non-IID setting and the IID setting. The masking method reduces the accuracy gap between the non-IID setting and the IID setting by nearly 60% on SST-2 and 70% on AG’s News. Such results prove the effectiveness of masking in solving the accuracy degradation caused by non-IID.

[1] Malladi, Sadhika, et al. "Fine-Tuning Language Models with Just Forward Passes." arXiv preprint arXiv:2305.17333 (2023).

[2] Sun, Tianxiang, et al. "Black-box tuning for language-model-as-a-service." International Conference on Machine Learning. PMLR, 2022.

[3] Tang, Minxue, et al. "FedCor: Correlation-based active client selection strategy for heterogeneous federated learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

W1. Thanks for the appreciation of our improvement in communication efficiency. Our goal is to design a framework that enables resource-limited devices without model access to finetune LLMs in a federated fashion. For example, when you are using ChatGPT, you do not have access to the model parameters and can only get the inference results by sending the request to the model through the API provided by OpenAI. Gradient-based methods can achieve relatively higher accuracy, but they are unfeasible when the devices have no access to model parameters or have no computational power to conduct back-propagation. We conduct experiments primarily to show that we can improve accuracy compared with the other gradient-free methods with extremely efficient communication. For Llama 2, FedBPT improves accuracy by more than 12%, 11%, and 13% for SST-2, AG’s News, and Yelp compared with the manual prompts under non-IID settings, respectively. Such accuracy improvement is significant.

W2. Thanks for this constructive comment. We will include experimental results by setting the population size as 40 and 100 in our revised version.

Q1. We use a Viridis color map to represent the counts in the confusion matrix. The revised caption will be: “Figure 3: Confusion matrix of a client holding data that more than 90% is in class one. Each rectangle's color represents the number of counts, and darker shades indicate lower counts .”

Q2. Our perturbation adds random information. It is interesting to change the words with opposite semantic information. In this case, the loss of the LLM inference results should be larger, which would boost the effect of our method formulated in Equation 7. We will leave it for future work.

Q3. The method of selecting a prompt from the dataset is the In-Context Learning baseline in the experiments in which we randomly select up to 5 training samples (input text and labels) and concatenate them with the input texts. The improvement of our method is primarily from the optimized prompt generated by our method FedBPT. But we agree that if we initialize with the In-Context Learning prompt, there should be more improvement. Thanks for the constructive suggestion, and we will include this in our discussion section.