FLAT-Chat: A Word Recovery Attack on Federated Language Model Training

The reviewer couldn't go through the comments thoroughly in the short time frame given. However, the admitted similarity of the method and results to prior work that is not even necessarily SOTA makes me reaffirm my initial grade. Further, the label recovery strategies of [1] and [3] are indeed relevant and can be applied with no modifications to the text setting. I will further read the rebuttal after the end of the discussion period.

Thank you for your comments and suggestions. We hope the following clarifications can address your concerns.

W1: The description of the proposed method can be hard to read at times, suggesting reorganizing paragraphs and the algorithm in section 3 and including some missing information (e.g. GMM fitting, regression prediction and training loss constitution).

A1: We sincerely appreciate these suggestions regarding rearrangements and supplements. We will incorporate them in our revision.

W2: Citation and comparison to related work: [1-4].

A2: Thank you for pointing out the literature.

[1] and [3] are for image reconstruction, focusing on image data in a continuous space while our work focuses on text data in a discrete space.

[2] shares similar insights to our Lemmas 1 and 2. We have compared this baseline (coined Abs.) in Table 4. We will acknowledge our Abs. is equivalent to the suggested work. We would like to highlight our progress compared with [2]: (1) we have extended the theory and method (see GMM score and its results) and (2) our approach manages to estimate the number of tokens used in the recent training batch, which is often omitted in many related works.

[4] FILM targets sequence recovery while our work focuses on industrial-scale word recovery attack. Nonetheless, our work could serve as the first Bag-of-Words Extraction step of FILM.

W3: Considering not only recovering the unique tokens but also counts (frequency) in the attack setting from a practical perspective.

A3: We argue recovering the appearance of tokens is the first and the most essential step for information leakage. The frequency of a token is arguably less important than know it is used or not. Nonetheless, we appreciate the suggestion and would like to leave it to future work.

W4: Bad evaluation results, compared with RLG in its applicable scenarios, or methods [1-3] with less restrictions. Besides, some presented precision-recall results.

A4: Actually, the out-of-scope size of training data is quite common, imagining industrial FL scenarios using large training batches. DLG is not aware of the potential issue of a numerical value exceeding the range of $D$ which is usually several hundred. Our estimation model based on the parameters from GMM could solve the issue very well, according to Figure 3.

S5: Adapt modeling the probabilities pi,j with a Log-Gaussian distribution (log pi,j).

A5: Thanks for the suggestion, this method may serve as a better estimator. We will consider adding this to our future work.

S6: Can the approximation of $|T|$ be improved by using $\frac{|B|}{(|V|-1)|B|}$ or some of the methods in [1-3]?

A6: There is an issue in the suggested method to estimate $|T|$: the $|B|$ is the suggested formula, $\frac{|B|}{(|V|-1)|B|}$, will be canceled. We did not find promising methods in predicting $|T|$ in [1-3], especially under the setting of large batch size and large vocabulary size.

N7: Eq.13, Eq.6, Eq.9 and Eq. 7 modification suggestions.

A7: We sincerely appreciate your suggestions, and we will incorporate them into our revision.

Q8: What auxiliary data was used to obtain the parameters of FLAT-CHAT in Table 2 (Machine Translation) experiments?

A8: The auxiliary data used in Table 2 MT experiments for fine-tuning is the News-Commentary v15 de-en dataset. We will include an explanation in our revision.

Q9: Can you explain why the precision and recall numbers between Tables 2 and 3 differ that much?

A9: The results in Table 2 and Table 3 are consistent. We assume you are asking the reason why RLG may have significantly different precision and recall when # Tokens is large. It is because RLG (1) fails the predict the correct size of the batch and (2) fails to predict the `long-tail’ tokens.

Q10: Can the authors explain the reasoning behind the Abs baseline in Appendix C? the authors propose there is very similar to [1]

A10: Actually this methodology is similar to the one in [1], we will include the comparison and citation to our revision.

Q11: Can you provide precise runtimes of the proposed method and baselines?

A11: The runtime depends on the batch size of the training data, and we have provided the averaged running time for comparing the efficiency of RLG and our method in Table 3. Considering some cases of 1.5 seconds vs 34.3 hours, the significance of our approach is clear.

Q12: Run experiments on a newer open-source LLM like Llama [6] or Chinchilla [7].

A12: Thanks for your suggestion. We assume our method is tested under general settings for LM which should also work for other transformer-based models including Llama and Chinchilla.

Thank you for your comments and suggestions. We hope the following clarifications can address your concerns.

W1: Attack significance is low because the order of tokens cannot be recovered.

A1: Revealing the training word types could also infringe serious risk for the scenarios requiring brief context, like healthcare diagnosis or personal identifiable information (PII), including credit card numbers and social security numbers. The significance of our work lies in that we are the first work that scaled up the word reconstruction attack to industrial settings, such as more than 10 K tokens in a batch, vocabulary size larger than 50 K and extremely fast inference speed. Furthermore, consolidated word reconstruction could serve as the first step of sequence reconstruction attacks, followed by iteratively swapping tokens and training them within a language model, as adapted in the prior work FILM and LAMP [1].

W2: The technical challenge is not clear, as prior work iDLG also leverages the last layer's gradient to infer the labels of trained samples in classification tasks.

A2: The focus of the iDLG is to infer the input data X, and it requires gradients from all layers. The usage of last layer gradients is just to infer the classification label for better gradient matching. However, our method focuses on inferring the output data Y from a single last layer's gradients.

W3: Please illustrate potential FL applications for LM training.

A3: The LM training can be under the scenarios like training LM for medical reports, financial reports, keyboard as mentioned etc. In many scenarios, the text corpus would be kept in clients for privacy concerns and FL will be their first choice in training shared powerful LMs (which could be used for many purposes, such as classification, QA, NLI, NLG, etc.).

W4: Including FILM as baseline for performance comparison.

A4: FILM targets sequence recovery while our work focuses on industrial-scale word recovery attack. Nonetheless, our work could serve as the first Bag-of-Words Extraction step of FILM.

W5: The attack significance is downgraded as it can be defended by DP-SGD.

A5: The significance of our work is (1) We propose a new word recovery attack which works on industrial-scale, with theoretical proof and extensive empirical evidence; (2) We demonstrate the risk could be mitigated by DP-SGD. Our work enhances the motivation of using a proper security schema in FL. It is unfair to discredit a novel attack merely because it could be potentially defended, e.g., the many attack methods, FILM and LAMP [1], proposed by recent literature could also be defended by DP-SGD to some extent.

Q6: Used learning rate and the privacy budget in DP-SGD.

A6: Our experiments use a learning rate of $5 \times 10^{-4}$, as detailed in the footnote of Section 4.1, which introduces the experimental setup. We will include the privacy budget in our revision.

Q7: What does the 'Loss' in Figure 4 mean?

A7: The ‘Loss’ in Figure 4 denotes the averaged cross-entropy loss, indicating the utility (how efficiently it operates in a language prediction task) of a language model. We will include an explanation in our revision.

[1] LAMP: Extracting Text from Gradients with Language Model Priors (NeurIPS 2022)

Thank you for your appreciation and suggestions.

W1: more exploration (e.g. of larger batch sizes) on the failure mode of the approach.

A1: (1) We have covered some experiments with the failure performance of RLG, such as 1,500 and 2,000 tokens in Table 2. (2) When scaling the batch size, the attacker will still be valid as we find Prec. score of FlatChat is kept stable or even grows in some cases (see Tables 2 and 3), which benefits from a relatively conservative batch size predictor.

W2: Does the frequency of each word in the batch impact the quality of the fit?

A2: We consider it as a question out of curiosity instead of an argument of weakness. It is implicitly encoded in our Theorem 3, as scores of high-frequency words will be at the `centre’ of the positive cluster.

W3: further experiments which show performance of freeze/dp-sgd for language modelling

A3: Yes, we have demonstrated the performance of both freeze and dp-sgd on Machine Translation task in Figure 4.

Q4: why results for Scratch with the task of Large Language Modelling are not included

A4: In many LLM application scenarios, we directly use well-trained LLM instead of LLM from scratch. Additionally, attacking Scratch is a relatively easier experimental setting for attackers as demonstrated in Machine Translation tasks.

Q5: Is the gaussian mixture model accurate at every epoch of fine-tuning? Or is it only at the first epoch?

A5: GMM could be easily generalized to various settings:

1. The feasibility of the Gaussian Mixture Model is proved by our Theorem 3 based on Theorem 4 (CLT)
2. FlatChat empirically works on well-finetuned models, such as MT’s FINETUNE and GPT-2.
3. Because the data leakage in the later epochs is of less interest given the data has already been disclosed in the earlier epochs, we focus on the token recovery attack in the early training epoch.
4. Any gradient-based reconstruction attacks will fail in an extreme case, say the model overfits a batch of data leading to zero loss and zero gradient vectors.

Thank you for your comments and suggestions. We hope the following clarifications can address your concerns.

W1: Some writings can be simplified, e.g. the lemmas and their proofs in Section 3.2 can be condensed in Equations and will not impact their readability.

A1: The proof of the properties is one of the key findings in this work as they ground the success of our word recovery attack. We sincerely appreciate your suggestions, and we will rearrange this content in our revision for better readability.

W2: It would be better to consider the common practice of the embedding layer and the last layer sharing the same parameters in a language model.

A2: Yes, we have conducted tests on this setting in our preliminary study on MT tasks and our method has demonstrated successful outcomes. However, it is important to note that the share-embedding setting exposes both embedding and the last linear layers to data reconstruction attacks.

W3: The attack is limited to inferring the bag of words while the order of the words cannot be recovered.

A3: Word reconstruction could also pose a privacy risk in instances like exposing trade secrets, healthcare patient diagnosis, or personal identifiable information (PII) such as credit card numbers. Furthermore, consolidated word reconstruction could serve as the first step of sequence reconstruction techniques, followed by iteratively swapping tokens and training them within a language model, as adapted in the prior work FILM and LAMP.

Q4: How would the performance change when tying the weights between input embedding and output layer?

A4: Our implementation is in the context of weight tying and has demonstrated successful outcomes. We will add such an explanation to our revision.

Q5: Will the inference attack still be effective if using secure aggregation as defense? where the server can only observe the aggregated gradients instead of individual’s.

A5: Thanks for your question. That is why we need to consider the setting with more training tokens involved. It is worth noting that aggregating (say averaging) gradients from multiple batches (say from various clients) is equivalent to the gradients acquired from the `aggregated' (say concatenating all samples in these batches as a single batch) larger batch.