Detecting and Perturbing Privacy-Sensitive Neurons to Defend Embedding Inversion Attacks

Q1: What if the privacy neurons were chosen based on a threshold of m, such as 0.5?

We appreciate the reviewer’s suggestion regarding threshold-based neuron selection. To address this, we conducted experiments with varying thresholds $h$, where the privacy neurons $\mathcal{N}_t$ are selected if the mask score $m_i$ for the $i$-th neuron is larger than $h$. The results, obtained using the STS12 and FIQA datasets under a perturbation level of $\epsilon = 2$, are summarized in the tables below. For comparison, we also include the performance of unprotected embeddings and the Laplace Mechanism (LapMech) baseline.

Results and Discussion. From the results summarized below, we make the following observations:

• Using a threshold-based neuron selection method proves to be an effective strategy. When setting the threshold $h \leq 0.6$, DPPN consistently outperforms the LapMech with lower privacy metrics (Leakage & Confidence) while maintaining higher downstream utility.
• Optimal threshold selection. We notice that most of the mask score $m_i$ for the $i$-th neuron falls into the range of 0.4~0.7. According to the experiment results setting the threshold to 0.5 or 0.6 seems to be a robust choice. We hope these results could address the reviewer’s concerns regarding threshold-based selection.

STS12 dataset:

FIQA dataset:

[1] Louizos, Christos, Max Welling, and Diederik P. Kingma. "Learning Sparse Neural Networks through L_0 Regularization." International Conference on Learning Representations. 2018.

We thank the reviewer for their feedback and will respond to the raised questions below.

W1: The paper does not provide sufficient detail on how parameters $\xi$ and $\gamma$ in formulas 3 and 5 are selected during the experiments.

Thank you for pointing out the missing details regarding the parameter selection of $\xi$ and $\gamma$ in Formulas 3 and 5. We followed the optimal settings established in prior work [1], using $\xi = 1.1$ and $\gamma = -0.1$. We will incorporate this clarification into the revised manuscript to enhance transparency and reproducibility.

W2: What if privacy neurons were selected based on their top dimension-wise sensitivity?

We appreciate the reviewer’s suggestion and agree that selecting privacy neurons based on their top dimension-wise sensitivity could serve as a meaningful baseline for neuron selection methods. To evaluate this approach, we conducted experiments using sensitivity-based selection and compared the results with DPPN and random selection. Here we report the privacy metric (Leakage) and utility metric (Downstream) by varying the top-$r$% privacy neurons. The results are presented in the tables below.

The results indicate that while sensitivity-based selection is effective in mitigating privacy leakage compared to random selection, DPPN demonstrates significantly superior performance. Specifically, the privacy neurons identified by DPPN result in lower leakage and higher downstream utility across different settings. This improvement can be attributed to DPPN's ability to assign higher weights to critical neurons via the neuron mask learning process. We hope the results clarify the reviewer's concern.

STS12 dataset

FIQA dataset

W4.1: The real-time performance and computational cost of DPPN in practical applications are unclear.

We appreciate the reviewer for pointing out this important concern. In fact, the proposed DPPN defense method is designed to be efficient in both complexity and scalability. Specifically, the data owner can precompute and store the privacy neurons for each token in advance. During inference, the system only needs to sample noise and add it to the corresponding precomputed privacy neurons, which is computationally lightweight. As a result, the computational complexity of DPPN is identical to all of the baseline methods.

W4.2: The interpretability of the DPPN method is relatively low, potentially limiting its use in scenarios requiring high model interpretability.

We believe this may be a misunderstanding. Interpretability is actually one of the core contributions of the DPPN method. Specifically, the detected privacy neurons offer a meaningful interpretation of the embedding dimensions. While individual embedding dimensions may not have explicit predefined meanings, our findings suggest that the embeddings themselves store token information in distinct, specific dimensions. As illustrated in Figure 6, we observe that semantically similar words tend to share the same privacy neurons, which can be leveraged for implicit privacy protection. We hope this clarifies the reviewer’s concern.

References

[1] https://learn.microsoft.com/en-us/azure/ai-services/language-service/overview

[2] Morris, John, et al. "Text Embeddings Reveal (Almost) As Much As Text." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

[3] Kim, Donggyu, Garam Lee, and Sungwoo Oh. "Toward privacy-preserving text embedding similarity with homomorphic encryption." Proceedings of the Fourth Workshop on Financial Technology and Natural Language Processing (FinNLP). 2022.

Q5: Only two datasets' results are shown in the main text. Could the author explain in more detail the reasons for choosing these two datasets?

We selected the STS12 and FIQA datasets for inclusion in the main text for two primary reasons. First, these datasets are widely used benchmarks in prior work on embedding attacks and defenses [2,3]. This allows us to maintain consistency with established studies and provide a meaningful comparison. Second, these datasets represent distinct downstream tasks for a comprehensive evaluation of defense utility: the STS12 dataset assesses performance on the semantic text similarity task, while the FIQA dataset is used for retrieval tasks. Additionally, we emphasize that two other datasets, PII-Masking300K and MIMIC-III clinical notes, were employed to assess privacy risks in real-world threat scenarios. This ensures that our study not only aligns with prior research but also explores the implications of our methods in practical, high-risk applications.

Q6: Lack of in-depth analysis of the results in Table 1.

We appreciate the reviewer’s feedback regarding the analysis of results in Table 1. The primary goal of this experiment is to highlight the superior performance of DPPN in achieving an improved privacy-utility tradeoff compared to baseline methods. As demonstrated in Table 1, DPPN consistently outperforms the baselines by achieving lower privacy leakage and better downstream performance. This improvement stems from DPPN’s selective perturbation paradigm, which focuses on perturbing only the top 20% most informative neurons. In contrast, baseline methods perturb all dimensions uniformly, which compromises utility while providing a general defense. We believe this selective approach is a key contribution of our work. Furthermore, we validate the effectiveness of DPPN’s perturbation function in Section 5.2 and analyze the quality of neuron detection in Section 5.1 to support our findings.

W1.1: The DPPN method may not explicitly detail how to accurately match and process personal information.

We thank the reviewer for highlighting this point. To extract sensitive information, we utilize named entity recognition (NER) models to identify and extract named entities effectively. The detailed process for matching and processing personal information using the DPPN method is thoroughly described in Appendix C. We would be glad to address any further questions or provide additional clarifications if needed.

W1.2: The effectiveness of DPPN relies on accurately identifying and selecting neurons related to privacy information. Inaccuracies in this process may lead to sensitive information leakage.

We agree that the effectiveness of DPPN heavily depends on the accurate identification of privacy-related neurons. To address this, we conducted a thorough investigation into the quality of our neuron detection methods in Section 5.1. Figures 4 and 5 also demonstrate that DPPN achieves performance comparable to white-box defenses. Specifically, on the STS12 dataset, DPPN exhibits only a 3–6% absolute difference in privacy leakage metrics and less than a 5% relative difference in downstream task performance. For a more comprehensive analysis, we encourage the reviewer to refer to Lines 358–369.

W2: Lack of Theoretical Guarantees.

We agree that it is important to develop a formal privacy guarantee for a defense method. As outlined in the general response, we propose a potential direction to extend DPPN with formal differential privacy definitions. While we acknowledge the technical challenges involved in formalizing such a theoretical framework, we believe that the key insights provided by our work—particularly in identifying privacy neurons—lay a strong foundation for future research to develop more robust defense strategies in this domain.

We thank the reviewer for their feedback and will respond to the raised questions below.

Q1: What are the benefits of DPPN compared to matching and transforming PII in the corpus?

We acknowledge that matching and transforming PII is an effective approach to prevent privacy leakage, as it removes sensitive information entirely from the original text. However, this method has significant drawbacks, such as the loss of critical contextual information, which can substantially degrade the performance of downstream tasks. For instance, in medical retrieval tasks, removing details such as disease names or patient descriptions can severely impact the utility and effectiveness of the system.

To empirically evaluate the impact of removing PII on downstream tasks, we implemented a text anonymizer based on the state-of-the-art tool provided by Azure Language Service [1], where all privacy tokens were replaced by the symbol ‘*’.

We compare the privacy leakage and downstream performance of four methods: unprotected embeddings, embeddings with PII removed, and DPPN at two perturbation levels ($\epsilon=2$ and $\epsilon=1$). The results are summarized in the tables below, leading to the following key observations:

1. Removing PII significantly impacts downstream utility. Comparing the unprotected embeddings with the RemovePII method, we observe a substantial drop in downstream performance—from 74% to 59% on the STS12 dataset, and from 33% to 21% on the FIQA dataset. While RemovePII effectively mitigates privacy leakage, it suffers from severe information loss, resulting in low utility.

2. DPPN provides an adaptive privacy-utility trade-off. As shown in the tables, DPPN allows for balancing privacy and utility by adjusting the perturbation level $\epsilon$. For instance, with $\epsilon=2$, DPPN achieves better downstream performance than RemovePII while maintaining a relatively low privacy leakage (13.44% on the STS12 dataset). This adaptive mechanism enables data owners to fine-tune the balance between protecting privacy and maintaining utility according to their specific requirements.

Results on STS12 Dataset

Results on FIQA Dataset

Q2: What are the benefits or advantages of the author's method compared to traditional DP-based methods?

The advantages of our proposed DPPN over traditional DP-based methods are twofold:

1. Enhanced Privacy-Utility Tradeoff: Unlike traditional DP methods that perturb all dimensions indiscriminately, DPPN selectively perturbs only the most informative dimensions. This approach significantly reduces the risk of privacy leakage while preserving utility for downstream tasks. Table 1 and our experimental results demonstrate that DPPN consistently achieves a better balance between privacy and utility.

2. Robustness Against Diverse Attack Models: We evaluated DPPN against three strong attack models. The results show that DPPN provides better defense performance than other DP-based methods. This robustness validates its effectiveness in handling real-world scenarios with varied attack strategies.

Q3: What are the challenges faced in this research?

This research encounters two primary challenges. First, as noted in the general response, extending the current framework to satisfy formal differential privacy guarantees remains an ongoing effort. While we recognize that ensuring these guarantees is critical for developing a robust defense framework, we are still addressing the curse of dimensionality and working to derive a formal proof for our perturbation function. Second, learning a high-quality neuron detection method is inherently challenging due to the lack of explicit semantic meaning in embedding dimensions. Empirically, token information can only be inferred by observing changes in embeddings with and without specific tokens. Eventually, we found that adopting a neuron mask learning scheme proved to be an effective approach, and achieves performance comparable to white-box neuron detection scenarios.

Dear Reviewer 8zq5,

Thank you once again for your insightful comments and suggestions, which have been immensely helpful to us. We have posted responses to the proposed concerns.

We understand this may be a particularly busy time, so we deeply appreciate any time you can spare to review our responses and let us know if they adequately address your concerns. If there are any additional comments, we will make every effort to address them promptly.

Best regards,
The Authors

W4: Experiments with two baselines are not very convincing.

We thank the reviewer for raising this concern regarding the selection of baseline methods. As discussed in the related works section (Section 8), there are two main approaches for privacy-preserving text embeddings: noise injection and adversarial training. Since DPPN achieves privacy preservation by selectively injecting noise into text embeddings, it is most appropriate to compare DPPN against noise injection-based methods.

While there is an extensive body of research on privacy-aware word embeddings, there are significantly fewer works focused on sentence-level embeddings. As a result, the two baseline methods used in our experiments—Laplace Mechanism (WSDM 2020) [2] and Purkayastha Mechanism (WWW’23) [3]—are currently the most theoretically formalized and publicly recognized methods for this problem. This is further supported by Table 1 in the survey paper [4], which identifies [3] and [5] as the only two perturbation-based methods for sentence-level privacy, with [2] being a general DP mechanism. It is worth noting that we excluded [5] from our experiments due to its additional requirements, including fine-tuning the embedding model and reliance on an external corpus. Given the scarcity of prior research in sentence-level privacy-preserving text embeddings, our choice of baselines reflects the state-of-the-art approaches available.

Experiment results with additional baseline. To address the concern and further enhance our evaluation, we implemented an additional baseline method that perturbs text embeddings by adding Gaussian noise. This method was studied in the Vec2Text [6] paper as a promising privacy defense mechanism. However, it is important to note that this method does not offer formal guarantees of differential privacy (DP). The results of this additional baseline are presented in the tables below. Our analysis shows that the NormalNoise baseline exhibits similar performance to other baseline methods. Notably, our proposed DPPN consistently outperforms NormalNoise across all evaluation metrics. These results reinforce the robustness of DPPN and further validate its effectiveness compared to other noise injection-based approaches. We hope this addresses the reviewer’s concerns and demonstrates our rationale for baseline selection.

STS12 dataset

FIQA dataset

We thank the reviewer for their feedback and will respond to the raised questions below.

W1: Is LapMech a variant of DP? What is the performance comparison with standard DP?

LapMech is indeed based on the Laplace mechanism, which is one of the foundational approaches in differential privacy (DP) and provides pure $\epsilon$-DP guarantees. Both baseline methods in our evaluation, LapMech and PurMech, are DP-based approaches tailored for privacy preservation in text embeddings. Among them, PurMech represents the state-of-the-art defense against inversion attacks on embeddings.

Regarding performance comparisons with standard DP mechanisms, we believe that the evaluations presented in Table 1 and Table 7 of our manuscript offer a comprehensive analysis. These results demonstrate that DPPN significantly outperforms the DP-based baselines in both privacy and utility metrics.

W2: What is "Downstream" in the utility metric?

Downstream refers to the subsequent tasks that utilize the text embeddings as input features. Since text embeddings serve as general-purpose representations, their utility is often evaluated through their performance on various downstream applications, including text classification, information retrieval, and semantic similarity assessment. Details of the downstream tasks and their evaluation metrics for each dataset could be found in Table 8 of our manuscript. In our evaluation, we specifically follow the comprehensive evaluation protocol established by the Massive Text Embedding Benchmark (MTEB) [1]. This benchmark provides standardized metrics for assessing embedding quality across multiple downstream tasks, allowing for systematic comparison of embedding utility.

W3: It seems that DPPN does not outperform other defense methods in some datasets. How do you explain the results in Table 7?

We acknowledge that DPPN does not consistently outperform all baseline methods across every dataset and perturbation level. However, it is important to emphasize that DPPN demonstrates superior performance in the majority of cases. Specifically, in Table 7, DPPN outperforms the baseline methods in 48 out of 60 cases, achieving an 80% win rate.

For the minority of cases where DPPN underperforms, the relative difference compared to the best-performing baseline is only 8.05%. In contrast, when DPPN outperforms the second-best baseline, it achieves a substantial gains with average improvement of 31.21% in these scenarios. We also observe that DPPN’s underperformance typically occurs at lower perturbation levels (e.g., $\epsilon = 6$ or $\epsilon = 8$). Since the injected noise is minimal, the advantage of DPPN is reduced. This limitation is common to noise-based privacy schemes, where the trade-off between privacy and utility diminishes with lower noise levels.

In summary, while DPPN may not always achieve the top performance in every setting, we believe its overall performance and significant improvements in the majority of cases could validate its effectiveness and robustness compared to baseline methods.

[1] Muennighoff, Niklas, et al. "MTEB: Massive Text Embedding Benchmark." Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.

W5: It seems that only one attack model is evaluated in the whole paper, making the defense performance not convincing again. Are there any results for other attack models?

We believe there is a misunderstanding regarding the evaluation of attack models in our paper. As clarified in Table 3 and Section 6.1, our manuscript evaluates three attack models: Vec2text [6], GEIA [7], and MLC [8]. Due to page constraints, we only presented experimental results on the STS12 dataset with $\epsilon = 1$ and $\epsilon = 2$ in the main text. To address the reviewer’s concern, we include additional experimental results using the GEIA and MLC attack models below. For brevity, we report only privacy-related metrics (Leakage & Confidence), as the utility metric remains consistent with those in Table 1 where DPPN outperforms the baseline methods.

The results in the tables below demonstrate that DPPN consistently outperforms baseline methods (LapMech and PurMech) across various perturbation levels ($\epsilon$) and attack models. We hope this provides sufficient evidence to validate the effectiveness of DPPN and addresses the reviewer’s concern.

Results using MLC[8] as attack model:

STS12 dataset

FIQA dataset

Results using GEIA[7] as attack model:

STS12 dataset

FIQA dataset

W6: Only Table 3 shows the privacy metrics results for different attack models. How about the utility metrics?

The experiment presented in Table 3 specifically focuses on evaluating privacy vulnerabilities under different attack models. The utility metrics are not included in Table 3 because they remain constant across all attack models for a given defense method. These utility results are identical to those reported in Table 1 for the STS12 dataset, as the defense mechanism's impact on utility is independent of the attack model being evaluated.

References

[1] Muennighoff, Niklas, et al. "MTEB: Massive Text Embedding Benchmark."

[2] Feyisetan, Oluwaseyi, et al. "Privacy-and utility-preserving textual analysis via calibrated multivariate perturbations."

[3] Du, Minxin, et al. "Sanitizing sentence embeddings (and labels) for local differential privacy."

[4] Hu, Lijie, et al. "Differentially Private Natural Language Models: Recent Advances and Future Directions."

[5] Meehan, Casey, Khalil Mrini, and Kamalika Chaudhuri. "Sentence-level Privacy for Document Embeddings."

[6] Morris, John, et al. "Text Embeddings Reveal (Almost) As Much As Text."

[7] Li, Haoran, Mingshi Xu, and Yangqiu Song. "Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence."

[8] Song, Congzheng, and Ananth Raghunathan. "Information leakage in embedding models."

Dear Reviewer QCCL,

Thank you once again for your insightful comments and suggestions, which have been immensely helpful to us. We have posted responses to the proposed concerns.

We understand this may be a particularly busy time, so we deeply appreciate any time you can spare to review our responses and let us know if they adequately address your concerns. If there are any additional comments, we will make every effort to address them promptly.

Best regards,
The Authors

Q2: Is it possible to design an adaptive attack scenario to demonstrate that this defense method remains effective even if the privacy protection scheme is exposed to adversaries.

We appreciate the reviewer highlighting this scenario to evaluate DPPN’s robustness when adversaries are aware of the protection scheme. To address this, we consider an adaptive attack where the adversary knows which embedding dimensions (i.e., the privacy neurons $\mathcal{N}_t$) are perturbed.

The research question is: Can the adversary extract private information from the “unperturbed” embeddings? As defined in Definition 1, text embeddings decompose into privacy-sensitive embeddings and privacy-invariant embeddings. Since the privacy-sensitive embeddings are perturbed through noise injection, the adversary would resort to train their attack model on privacy-invariant embeddings.

Adaptive attack experiment setup:

The attacker is given the full text embedding and access to the indices of the top-10% privacy neurons. The attacker builds a threat model by excluding these privacy neurons from their training (i.e., training is performed on $d*0.9$ dimensions). We term the scenario as oracle since the attacker has additional knowledge on the defense mechanism. We also report the attack performance when the defense mechanism is unknown to the adversary.

Results and discussion:

• With external knowledge on the defense mechanism, the adversary did attack better compared to the unknown scenario. As expected, an adversary with external knowledge of the defense mechanism performs better than one without it. On the STS12 dataset, privacy leakage increased to 25% in the oracle scenario, compared to 13% in the unknown scenario. A similar trend is observed in the FIQA dataset.
• The attacker cannot inference private information from privacy-invariant embeddings. Despite oracle knowledge, the adversary’s ability to infer private information is significantly reduced compared to unprotected embeddings. For example, in the STS12 dataset, leakage drops from 60% (unprotected) to 25% (oracle), demonstrating that excluding the privacy neurons significantly limits the adversary’s success. A similar reduction is observed in the FIQA dataset, where leakage drops from 77% to 28%. These results indicate that even with full knowledge of the protection scheme, the adversary cannot reconstruct private information from the privacy-invariant embeddings.

STS12 dataset:

FIQA dataset:

We thank the reviewer for their feedback and will respond to the raised questions below.

W1 & Q1: The paper lacks a formal privacy guarantee. Is it possible to formulate privacy realized by this framework?

We agree that it is important to develop a formal privacy guarantee for a defense method. As outlined in the general response, we propose a potential direction to extend DPPN with formal differential privacy definitions. While we acknowledge the technical challenges involved in formalizing such a theoretical framework, we believe that the key insights provided by our work—particularly in identifying privacy neurons—lay a strong foundation for future research to develop more robust defense strategies in this domain. Additionally, we conducted an experiment (addressed in the following question) to demonstrate that our method remains effective even when the defense mechanism is disclosed to adversaries. We hope this clarifies and addresses the reviewer’s concern.

W2: This defense method requires two datasets, which necessitates labeling what information as private information. The proposed defense method involves two datasets to evaluate the embedding differences with and without private information. However, we argue that this does not impose an additional labeling burden, as the datasets can be constructed using publicly available or generated resources.

For instance, if the private information to be protected pertains to medical data, the data owner can utilize publicly available medical-related articles, such as those on Wikipedia, or curated medical datasets. Additionally, synthetic privacy-related texts can be generated using generative models like ChatGPT, which can simulate relevant private information. This approach enables the data owner to construct the positive dataset $D^+$. The negative dataset $D^-$ can then be derived from $D^+$ through simple modifications, as outlined in Lines 205–206 of the manuscript. In summary, the requirement of two datasets does not require additional manual labeling.

Dear Reviewer GaoG,

Thank you once again for your insightful comments and suggestions, which have been immensely helpful to us. We have posted responses to the proposed concerns.

We understand this may be a particularly busy time, so we deeply appreciate any time you can spare to review our responses and let us know if they adequately address your concerns. If there are any additional comments, we will make every effort to address them promptly.

Best regards,
The Authors