Beyond Gradient and Priors in Privacy Attacks: Leveraging Pooler Layer Inputs of Language Models in Federated Learning

Thank you for your constructive feedback. Below are our responses to your main concerns. If these address your initial queries, we'd be grateful if you could revise your evaluation accordingly.

Q1: Clarification of Threat Model

---

We are thankful for the reviewer's astute observation that our discussion has been primarily focused on the optimization perspective without a high-level exploration of our threat model. This approach is indeed a result of deep contemplation.

1. Classification of Attacks: Following previous work, we categorized attacks as Malicious and Eavesdropping (or Honest-but-Curious Server) Attacks, but this oversimplifies the issue. Traditional Malicious Attacks involve altering server states or model architecture, rendering gradient information useless. However, our method, which only modifies the Pooler layer of the model, doesn't affect the original training purpose, thus differing from typical Malicious Attacks.

2. Wider Pooler Layer: The width of the Pooler Layer depends on the recovery needs. For a small input portion (X' with d_r << d_in), the required width reduces significantly, as detailed in Section 4: "Flexibility of Recovered Dimension". With the growth of Large Language Models like GPT-3, which already exceeds the necessary dimensions (12288 for d_r = 50), this width requirement becomes less vital. Moreover, our attack focuses on wide models, aligning with previous studies [Geiping et al.2020, Fowl et al.2022], revealing more risk in our settings.

3. Activation Functions Change: We've identified risks with activation functions like Selu, whose higher-order derivatives are neither odd nor even. Our ongoing research explores activation functions with strictly odd or even derivatives, like tanh or relu. While many previous studies have required specific activation functions [Zhu et al., 2019, Flow et al., 2022], we view this kind of discovery as a contribution, not a drawback

4. Redefining 'Honest-but-Curious': Compared to Eavesdropping (Honest-but-Curious) attacks, our method does involve some modifications to the model. However, we propose that 'honest' should not be narrowly defined as the server's absolute fidelity to clients. Instead, 'honest' should be understood as the server's adherence to its primary role of obtaining effective gradients and minimizing the training loss. By this definition, our approach falls under this new category.

5. Clarifying Our Method's Positioning: In summary, our method cannot be classified as a Malicious or Eavesdropping Attack unless we redefine the Honest-but-Curious category. Our method, compared to Malicious Attacks, is more difficult to detect and does not violate the fundamental objective of federated learning. In contrast to Eavesdropping Attacks, our approach achieves more effective outcomes. Therefore, we mainly discuss our method from the perspective of optimization.

6. Elaborating on the Threat Model:

   • We do not fine-tune the model's word and positional embeddings. The gradients for these embeddings are non-zero for words in the client's training data, enabling easy recovery of client sequences.
   • The server operates under a redefined 'honest but curious' model, aiming to learn maximally from gradients without altering the learning protocol.

7. Enhanced Understanding: To aid the reviewer's understanding, we will revise our manuscript and include a table in the appendix. This table will delineate the differences between Malicious Attacks, Eavesdropping Attacks, and Honest-but-Curious Server attacks, with examples from previous work (LAMP, Wang et al., 2023).

Q2: Novelty Concern

---

Thank you for your comments on our work's novelty. We want to clarify a few points:

1. Contribution Clarification:

   • Our contribution isn't about identifying gradient-based attack issues, which was just an analysis foundation. Our innovation is to breach textual data privacy using the recovered information of intermediate features.

2. Differentiation from Prior Work [1]:

   • We recognize similar use of intermediate features in previous studies [1] on image data. However, our work pioneers this concept for textual data attacks, facing unique challenges not seen in image data due to differences in data representation and model architectures.
   • Our technique method is totally different from [1]

Q3: Tone and Subjectivity

---

Thank you for pointing out the subjective tone. We will revise this. The term "revolutionary" was specifically for our new metric on textual data information leakage, not our method (Contribution 4).

Q4: Optimization Objective

---

I appreciate your insight on our optimization objective. We aim to recover the directional component of the Pooler layer's input using decomposition, acknowledging challenges in reconstructing its magnitude. Our solution employs cosine similarity loss to address this limitation.

Thank you again!

Dear Reviewer,

Thank you for your initial review of our manuscript and your subsequent response.

We apologize for the delay in uploading our revised version, which might have led to it being overlooked. Based on your valuable feedback, we have significantly revised our manuscript. We would greatly appreciate it if you could take the time to review our updated submission. We believe these changes comprehensively address your concerns and hope that you might reconsider your assessment in light of these updates.

Again, Thank you for your valuable insights and the time you've dedicated to reviewing our work.

Best regards, Authors

Q1: The reviewer disagrees with the author's characterization of "honest-but-curious" to qualify this work. Honest is a qualifier used primarily for data holding clients not for the aggregating sever. Q2: While applying a finding from a modality to another can qualify as a valid contribution the text should not be misleading. Q3: The reviewer disagree with the "revolutionary" qualifier even applied to the metric the authors found

The reviewer stand with their original scoring.

Dear Reviewer,

Thank you for your subsequent feedback. Below are our responses to your new concerns. If these address your initial queries, we'd be grateful if you could revise your evaluation accordingly.

1. "Honest-but-Curious" Definition Misunderstanding: We noticed a misinterpretation of the term 'honest' in the context of federated learning. In your review, 'honest' is associated with data-holding clients. However, in federated learning, as supported by a breadth of recent literature [1][2][3][4][5][6][7][8], 'honest' traditionally refers to the server. This distinction is crucial for understanding the methodology and assumptions of our study. Misinterpreting this term could lead to a skewed evaluation of our work, and we hope this clarification helps better understand our research approach.

2. First Utilization of Intermediate Features for Privacy Text Attacks: We wish to clarify our claim about pioneering the use of intermediate features in privacy attacks within NLP for federated learning again. The ICML 2023 paper [8] you referred to focuses on image data. However, our work is based on the domain of NLP and presents more complex challenges not encountered in CV, for example, the discrete nature of token ID and gradient average on the dimension of sequences. These challenges differentiate our approach from those applied to image data, underscoring the novelty and specificity of our work in the context of federated learning within NLP. Moreover, the methods used in these two works are different. (We have consistently specified that our contributions are within NLP and federated learning, and at no point have we attempted to mislead regarding the scope of our work.)

3. Terminology Adjustment: We would like to draw attention to a key revision in our manuscript that may have been overlooked. The term initially described as Revolutionary Evaluation Metric has been thoughtfully changed to Semantic Evaluation Metric to reflect its nature and scope more accurately.

In summary, we hope these clarifications address the concerns raised and provide a clearer picture of the contributions and contexts of our work. We appreciate your attention to these matters and look forward to any further feedback you may have.

Sincerely, Authors

[1] Gupta, Samyak, et al. "Recovering private text in federated learning of language models." Advances in Neural Information Processing Systems 35 (2022): 8130-8143. (NeurIPS 2022)

[2] Balunovic, Mislav, et al. "Lamp: Extracting text from gradients with language model priors." Advances in Neural Information Processing Systems 35 (2022): 7641-7654. (NeurIPS 2022)

[3] Geiping, Jonas, et al. "Inverting gradients-how easy is it to break privacy in federated learning?." Advances in Neural Information Processing Systems 33 (2020): 16937-16947. (NeurIPS 2020)

[4] Fowl, Liam, et al. "Decepticons: Corrupted transformers breach privacy in federated learning for language models." arXiv preprint arXiv:2201.12675 (2022). (ICLR 2023)

[5] Boenisch, Franziska, et al. "When the curious abandon honesty: Federated learning is not private." 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P). IEEE, 2023.

[6] Huang, Yangsibo, et al. "Evaluating gradient inversion attacks and defenses in federated learning." Advances in Neural Information Processing Systems 34 (2021): 7232-7241. (NeurIPS 2021)

[7] Fowl, Liam, et al. "Robbing the fed: Directly obtaining private data in federated learning with modified models." arXiv preprint arXiv:2110.13057 (2021).(ICLR 2022)

[8] Kariyappa, Sanjay, et al. "Cocktail party attack: Breaking aggregation-based privacy in federated learning using independent component analysis." International Conference on Machine Learning. PMLR, 2023.(ICML 2023)

[9] Zhu, Ligeng, Zhijian Liu, and Song Han. "Deep leakage from gradients." Advances in neural information processing systems 32 (2019). (NeurIPS 2019)

Thank you for your constructive feedback. Below are our responses to your main concerns. If these address your initial queries, we'd be grateful if you could revise your evaluation accordingly.

Q1: Clarification on the Novelty of Using Intermediate Features

---

We sincerely appreciate your reference to the paper [HKHG+], which shares a similar intuition of shifting the focus from model input to intermediate features but in a completely different context.

1. Different Context and Application: Our work uniquely focuses on recovering textual data in LLMs within the context of federated learning (FL). The cited work [HKHG+] primarily examines the use of intermediate feature perturbation to enhance the transferability of adversarial inputs in the context of black-box adversarial attacks.

2. Totally Different Methods: Our method does not know the true intermediate feature, instead utilizing gradient information and tensor decomposition technique to estimate the feature. The reference [HKHG+] can easily calculate the intermediate features with the model and input.

To better respond to the reviewer's concern, we will present a discussion that delves into the underlying intuitions of both approaches, highlighting the differences and acknowledging the similarities where applicable.

Q2: Risks in LLMs within FL

---

Thank you for inquiring about the privacy risks of textual data in LLMs within FL. To safeguard user privacy, users' textual data is typically stored on local edge devices. These devices compute gradient information and upload it to the central server. Ideally, the server doesn't access the original text, but studies have shown it's possible to reconstruct training text from gradient information and prior knowledge. We hope this clarifies the challenges in LLMs within FL.

Q3 & Q5: Clarification on Notations

---

1. [CLS] Token: This special token, added at the start of each input sentence, is selected to represent the sentence for classification tasks.

2. $\hat{x}$ and $\hat{y}$ in Equation 3: These symbols are the optimization goal that represents the estimated input and label, respectively.

3. B in Equation 2: Refers to Batch Size.

4. $\nabla^{3}g(w)$ and $x_{i}^{\otimes3}$: $\nabla^{3}g(w)$ is the third derivative of $g(w)$, and $x_{i}^{\otimes3}$ signifies the tensor product of vector $x_{i}$ with itself three times.

Q4: Clarification on security model

---

1. Adversary in the Attack: The server in FL is the adversary, operating under a redefined 'honest but curious' setting. It aims to learn as much text as possible from the gradients without altering the learning protocol. (The honest here refers to the server always minimizing the training loss without other objectives)

2. Adversary's Ability: The server can modify the Pooler layer of the model, allowing it first to recover the intermediate features (No other prior knowledge). However, the server does not finetune the word and positional embeddings (The gradients for these embeddings are non-zero for words in the client's training data, enabling easy recovery of client sequences).

3. Action of Participants: Clients compute the gradient and upload it to the server, which is benign.

4. Security Implication: This approach challenges traditional security models in FL. It's less detectable than Malicious Attacks and more invasive than typical Honest-but-Curious attacks, aiming to recover textual data from shared gradients. (The honest here refers to the server not changing model architecture and parameters)

5. Attacking Goal: The primary goal is to recover client sequences or information from the gradients without deviating from the fundamental objectives of federated learning, like effective gradient aggregation and model loss minimization.

Q6: Theoretical Discussion.

---

We have addressed the reviewer's concern in the following sections:

1. Bridging Experimental Findings with Theory: In Section 5.3, "Impact of Recovery Dimension" and Appendix Section A.4, "Influence of Data Dependence" we have thoroughly discussed how our experimental findings correlate with the theoretical insights presented in (Wang et al., 2023).

2. Discussion on Optimization Signals: We have also explored the impact of different optimization signals in the second and third paragraphs of Section 5.3. Additionally, our ablation experiments in Section 5.4, “Impact of Feature Match in Different Optimization Phases”, further elaborate on how varying optimization signals influence attack performance.

Q7：Comparison with Previous

---

Please refer to Section 5.3, "Results and Analysis", for a detailed explanation. The performance degradation observed in previous works when increasing the batch size is attributed to the gradient being averaged over more tokens and sentences. Our method introduces an additional supervisory signal that mitigates this issue.

Thank you again!

Thank you for your constructive feedback. Below are our responses to your main concerns. If these address your initial queries, we'd be grateful if you could revise your evaluation accordingly.

Q1：Performance on Other Types of Tasks

---

Thank you for highlighting the importance of evaluating our proposed method's effectiveness in other natural language processing tasks, such as translation or generation tasks. We acknowledge that our approach might face greater challenges in these contexts. In such tasks, the loss generated by different tokens is typically summed or averaged before computing gradients, in contrast to classification tasks that usually produce a single loss.

1. Acknowledging Shared Challenges in the Community:： We recognize that these challenges are not unique to our method but are common obstacles faced across the community in privacy attack research. This issue underscores the complexity and variety of tasks in natural language processing and their implications for privacy attacks.

2. Emphasizing the Practical Importance of Privacy Attack Research:： We wish to emphasize that our primary goal is not to develop a universal privacy attack method but to highlight potential threats of LLMs within federated learning. It's essential to understand these vulnerabilities of the current protocol to prevent potential risks.

3. Future Research Directions:: Your concern aligns with our future research direction. We are motivated to explore how our attack methodology can be adapted or improved to address the unique challenges in tasks like translation and generation.

Q2: Cost Analysis

---

We appreciate your attention to the cost implications of our method and would like to clarify two key aspects regarding this:

1. Comparable Cost to Traditional Optimization-Based Techniques:: Our method incurs almost the same computational cost as other traditional optimization-based techniques. This is primarily because the recovery of intermediate features in our approach involves a tensor decomposition technique, which is a one-time process. The computational effort for this part is negligible compared to the subsequent optimization steps. Moreover, during the optimization phase, our method only requires the additional computation of a cosine similarity loss, which also does not significantly increase the computational burden. Based on our assessments, our method does not add substantial extra computational overhead (less than 1.1 times the usual cost).

2. Context of Attack Scenarios in Practice:: It's important to consider the practical context of attack scenarios. Attackers often prioritize the effectiveness of the attack over the cost, implying that computational expense is not the primary concern in such research. This perspective is crucial in understanding the feasibility of privacy attacks, including ours, in real-world scenarios.

Q3: Possible Countermeasures

---

We are grateful for your attention to both the attack methodologies and potential defense techniques in our study. We recognize the importance of defenses in the context of privacy attacks and appreciate this opportunity to discuss them.

1. Effectiveness of Attack in Certain Settings: Our research acknowledges that honest-but-curious type attacks on LLMs, especially when freezing the token and positional embeddings, tend to be effective primarily in settings with smaller batch sizes. While our method significantly improves the effectiveness of attacks for B > 1, we observe a notable decline in effectiveness when B > 8. This suggests that in federated learning scenarios with large batch sizes and a benign server, textual data privacy can still be reasonably safeguarded. (The 'honest' here refers to the server's goal to recover client sequences or information from the gradients without deviating from the fundamental objectives of federated learning, like effective gradient aggregation and model loss minimization.)

2. Decision Against Including Defense Techniques: Based on this observation, we chose not to focus on defense techniques within the scope of this paper. Our primary objective was to reveal the potential risk of privacy breaches in LLMs within federated learning settings, particularly those with moderate batch sizes.

Thank you again!

Thank you for your constructive feedback. Below are our responses to your main concerns. If these address your initial queries, we'd be grateful if you could revise your evaluation accordingly.

Q1. Constraints of the Proposed Attack:

---

We recognize some limitations to a certain extent, but not entirely, as real-world attacks are often more extreme than this scenario. To provide further clarity, we compare our methodology from an attack perspective with prior research.

1. Wide Pooler Layer Constraints: The width of the Pooler Layer is adaptive to recovery needs. As we detailed in Section 4, "Flexibility of Recovered Dimension", for smaller input portions (X' with d_r << d_in), the required width is significantly reduced. With advancements in Large Language Models like GPT-3, which already surpass necessary dimensions (12288 for d_r = 50), this width requirement becomes less critical. Furthermore, our focus on wide models aligns with previous studies (e.g., Geiping et al., 2020; Fowl et al., 2022), highlighting increased risks in our settings.

2. Activation Functions Change: We have identified risks related to using activation functions such as Selu, whose higher-order derivatives are neither odd nor even. Our ongoing research extends to activation functions with strictly odd or even derivatives, such as Tanh or Relu. While many previous studies have focused on specific activation functions (Zhu et al., 2019, Flow et al.2022), we view this discovery as a contribution, not a drawback.

3. Comparison with Previous Studies:

   • Previous works [e.g., Flow et al., 2022, 2023; Boenisch et al., 2023] have implemented attacks capable of handling very large batch sizes and long sequence text recovery. However, these methods, called Malicious Attacks, have critical flaws. Firstly, they violate the federated learning protocol, as the gradients returned by the client during an attack are meaningless. Secondly, they involve multi-layer modifications to model parameters or even insert new modules, which are far more aggressive than our modifications. In contrast, we only modify the Pooler Layer and consistently adhere to the federated learning protocol. Nevertheless, we recognize the value of these works in advancing privacy attack research.
   • Works like LAMP can extract private text information using only optimization-based techniques and prior knowledge without any model modifications. However, this is a self-imposed limitation. In reality, servers can undertake actions well beyond this scope. Although such actions may be more easily detectable, our method achieves less detectability while providing additional optimization signals, marking a significant innovation.

4. Specificity of Attack Scenarios: Research in privacy attacks holds a unique importance in practice. This is because every discovery in this area, even with certain constraints, can cause tremendous destruction once happens. The attacker will try their best to satisfy the constraints because of the hidden massive profit. Therefore, the risks uncovered by these studies are often difficult to measure, making them crucial for understanding security vulnerabilities.

Q2. Architecture of Evaluation Baselines

---

We are grateful for your observation regarding the architecture used in our baseline evaluations, and we would like to clarify this aspect to dispel any confusion.

In our comparative analysis, we maintained the original model structures of our baselines without altering their activation functions. This decision was driven by our goal to uncover potential private text attack methods, where the choice of activation function plays a crucial role. In our research, we discovered that activation functions with neither purely odd nor even higher-order derivatives, such as Selu or $x^3+x^2$, pose significant risks in our setting compared to other functions like Relu/Tanh.

We recognize the discovery regarding the choice of activation functions is a part of our contribution. It highlights the vulnerabilities in real-world applications in this setting. Therefore, we do not consider the selection of activation functions as a distilled item for comparison with other methods. Instead, it is more pertinent to internal comparisons within our own methodology. We have included a detailed discussion on this topic in section A.3 "Impact of activation function".

Q3. Clarification on Equations 5 & 6

---

We appreciate your request for clarification regarding Equations 5 & 6, and we will provide more explanation in the appendix of our revised manuscript.

Regarding the uniqueness of $x_i$'s recovery, it largely depends on the correlation among different data points within the batch. In the appendix, we have a dedicated discussion section titled "Influence of Data Dependence", where we delve into how the cosine similarity between recovered features and actual features gradually decreases as batch size increases.

Dear Reviewer:

We extend our sincere thanks for your latest response. Your insights, particularly from the standpoint of research that makes life better, are well-token.

However, we kindly urge the reviewer to consider our work from the perspective of research that makes life worse for the last time. Our study specifically focuses on identifying and understanding critical risks associated with Large Language Models (LLMs) in the context of federated learning. There may exist certain limitations in our methodology, but the significant risks and dangers that our research uncovers are both actual and consequential.

Thank you for your reinvested time and comments

Best, Authors.

Thank you for your constructive feedback. Below are our responses to your main concerns. If these address your initial queries, we'd be grateful if you could revise your evaluation accordingly.

Q1: Contribution and Novelty of the Proposed Method

---

We appreciate your acknowledgment of our work's structure and ideas. However, it's crucial to emphasize that our approach is not a mere combination of existing methods; it includes several unique contributions:

1. Theoretical vs. Practical Application: Wang et al.'s work, while theoretically effective, encounters practical challenges due to various constraints that limit its effectiveness in real-world scenarios. In contrast, our approach adapts these theoretical insights to practical federated learning settings.

2. Limitations in Wang et al.'s Work: Wang et al.'s method is most effective in a two-layer neural network context. Extending it beyond this depth requires customizing model parameters, like identity matrices, which contradicts the definition of 'honest but curious' in federated learning. Our method maintains this 'honesty' by focusing on the top two layers of deep neural networks. (The 'honest' here refers to the server's goal to recover client sequences or information from the gradients without deviating from the fundamental objectives of federated learning, like effective gradient aggregation and model loss minimization.)

3. Focus on Intermediate Feature Recovery: Unlike Wang et al.'s work, which is limited to directly recovering model input, our method explores the potential of recovering intermediate features, adding a new dimension to attack strategies.

4. Flexibility in Recovery Dimension: Our approach offers more flexibility in optimizing recovery dimensions, as discussed in Section 4.2 "Flexibility of the Recovery Dimension." We can adaptively try different dimensions to meet specific needs, which is not the case in Wang et al.'s work.

5. Strategic Weight Initialization: We address the issue of gradients approaching zero due to weight randomization, as often happens in Wang et al.'s method. In Section 4.2 "Strategic Weight Initialization," we innovatively propose a coexistence of original and random weights, effectively overcoming this challenge.

6. Addressing Data Point Order in Batches: Wang et al.'s method struggles with determining the order of data points in the same batch. Our method, from an attacker's perspective, compares with the true features and adopts a greedy strategy based on the highest cosine similarity for one-to-one correspondence, effectively resolving the data point order issue.

Q2 & Q3: Batch Size and Defense Methods

---

We recognize the importance of defenses in the context of privacy attacks and appreciate this opportunity to discuss them.

1. Effectiveness of Attack in Certain Settings: Our research acknowledges that honest-but-curious type attacks on LLMs, especially when freezing the token and positional embeddings, tend to be effective primarily in settings with smaller batch sizes. While our method significantly improves the effectiveness of attacks for B > 1, we observe a notable decline in effectiveness when B > 8. This suggests that in federated learning scenarios with large batch sizes and a benign server, textual data privacy can still be reasonably safeguarded.

2. Decision Against Including Defense Techniques: Based on this observation, we chose not to focus on defense techniques within the scope of this paper. Our primary objective was to reveal the potential risk of privacy breaches in LLMs within federated learning settings, particularly those with moderate batch sizes.

Q4: Knowledge of Model Architecture

---

Thank you for raising the question about the attacker's knowledge of the model architecture; we clarify this as follows:

1. Knowledge of Model Architecture in Security Settings:

   • In the security model we've considered, whether the server setting is honest or malicious, the model's architecture is always known to the server. This knowledge is an inherent aspect of the federated learning setup, where the server orchestrates the learning process and, therefore, has access to the model architecture.

2. Implications for Potential Hackers:

   • Furthermore, even in scenarios where a hacker might steal model parameters, the limited variety of commonly used language models today often makes it possible to infer or guess the model architecture.

5：Definition of Private Information in Text

---

Thank you for your question. In your example, while 'delicious' isn't inherently sensitive, the key point is that attack methods don't distinguish between words like 'delicious' and genuinely sensitive information like personal health data. If our attack can recover something as benign as 'delicious,' it also has the potential to recover more sensitive information. Therefore, any potential data recovery is considered a privacy risk