Evaluating Privacy Risks of Parameter-Efficient Fine-Tuning

continues from the previous comment...

---

[Minor]

We carefully proofread our manuscript again and ensured that all typos mentioned in the minor section have been corrected, along with others.

---

[Summary]

We appreciate the reviewer’s feedback on the clarity and presentation issues. We kindly request the reviewer to reconsider the rating if our answers clarify existing concerns. If there is/are any remaining concerns, we will be happy to address them during the discussion phase.

We thank the reviewer for the time and effort in evaluating our work. Below, we answer the reviewer’s concerns and questions. We also updated our manuscript to reflect the feedback.

---

[Weakness 1 and Question 1: Claims about 100 Language Models]

We acknowledge the confusion caused by our claim of "100 language models." To clarify, we have revised our manuscript to state: "We evaluate two pre-trained language models fine-tuned on 2 datasets, repeated 5 times with different random seeds, resulting in a total of 100 variations."

[Weakness 2: Claims about 5 Distinct Fine-tuning Datasets]

We clarify that the phrase “5 distinct fine-tuning datasets” in Line 184 does not imply that we constructed per fine-tuning method. It refers to the creation of five datasets by randomly inserting secrets into the original dataset, each based on a distinct random seed.

[Weakness 3: Generalization of Our Results]

We thank the reviewer for suggesting a way to make our results more convincing. We run an additional evaluation with a Pythia-2.8B model on the MIMIC dataset. We use the same secret “mary smith” for ease of comparison. Our result is shown below:

Our findings were consistent with the results shown in our manuscript. While base fine-tuning easily memorized the secret, prompt tuning and other PEFT methods demonstrated significantly lower memorization. Similar to the GPT2-XL results, Adapter and LoRA showed comparable exposure levels, with prompt tuning yielding far lower exposure. Interestingly, prefix tuning lagged slightly behind Adapter and LoRA in Pythia-2.8B. We have included this additional result in our main table in the revised manuscript.

---

[Questions 3 and 5: Correcting Typo Leading to Misinterpretation]

• (Q3) We apologize for any confusion. We intended to convey the opposite: memorization (or attack success) can be mitigated even with a large epsilon value, which helps preserve reasonable performance.

• (Q5) We thank the reviewer for pointing out this typo. We indeed meant “decrease” in this line, and this has been fixed in our revised manuscript.

[Question 4: Motivation for Our Adaptation to the Memorization by Carlini et al.]

We adapted the memorization definition of Carlini et al. to examine the impact of secret positions within a training record. Carlini et al. focus exclusively on scenarios where a secret appears at the end of a training record, e.g., “Please contact me at john.doe@gmail.com” It does not consider many real-world cases where secrets appear in the middle of a context, such as “... since the prior John Doe radiograph, and a very small left apical…” Our evaluation reveals that depending on the design of the PEFT algorithms, a secret in certain positions is more likely to be memorized. This contrasts with the prior work, which shows that secrets at the end of a sentence are more likely to be memorized.

[Question 6: Clarification of Our Results Shown in Figure 3]

We thank the reviewer for suggesting a way to make our results more convincing. We first clarify that we emphasize the results from prefix-tuning because it causes secrets located at the beginning of a sentence to be memorized more, which is the opposite of all the other cases. Secondly, we included the results from full fine-tuning, prompt tuning, lora, and adapter in the Appendix on Figures 10, 11 and 13 at the time of our submission (9, 10, 13, as of our revision). But, we do acknowledge the importance of clearly referencing the materials in the Appendix. We have added in the main text to refer to these supporting data clearly, located in the Appendix.

[Question 7: PEFT Hyper-parameter Selection]

We acknowledge that we were not explicit about our hyper-parameter choices. We selected these hyper-parameters based on the recommendations in the original studies [1, 2, 3]. We have revised the experimental setup section to ensure that this is clearly described in detail.

[1] Housby et al. Parameter-Efficient Transfer Learning for NLP In International conference on machine learning, ICML, 2019.

[2] Li et al. Prefix-tuning: Optimizing Continuous Prompts for Generation. ACL-IJCNLP 2021

[3] Lester et al. The Power of Scale for Parameter-efficient Prompt Tuning, EMNLP 2021

[Qusetion 8: Concurrent Work]

We thank the reviewer for pointing out this concurrent work. We found from the OpenReview that the concurrent work is currently under submission to the same conference, ICLR 2025. We thus plan to include a discussion of this work later on when preparing our camera-ready version.

---

continues to the next comment...

Dear Reviewer Wqun;

As the deadline for the discussion period approaches, we kindly ask if our response adequately addresses the reviewer’s concerns or if further clarification is needed. We look forward to hearing from the reviewer soon.

Thank you!

Dear Reviewer Wqun,

We sincerely thank the reviewer for engaging in the discussion and providing additional reminders about the minor issues. Below, we outline our plan and actions to address those minor concerns:

---

[Pythia-2.8B on Enron]

We agree with the reviewer that Pythia-2.8B is required for the completeness of our results in Table 1. We are currently running the evaluation of Pythia-2.8B in the background. Meanwhile, we want to share the results from the MIMIC-III evaluation, which demonstrate that our observations also hold for commercial-scale models.

Due to the large model size, the evaluation on MIMIC-III took 60 hours on 2 Nvidia GPUs with more than 60GB memory (H100 or A100) and we expect the same for Enron. In an academic setting, it was challenging to secure such GPU resources within the short rebuttal period. However, we would like to officially state that, once the evaluation is complete, we will include the results for Pythia-2.8B on the Enron dataset in the camera-ready version of our paper.

[Related Work]

Our initial plan was to include a discussion of the concurrent work in the camera-ready version, as it is currently under review and the BibTeX entry appears as “Anonymous” authors. However, we agree with the reviewer that it is helpful for our discussion to outline how we will include the concurrent work. We have updated the Related Work section of our manuscript, accordingly, and we will ensure the references are corrected once the concurrent work is “unanonymized”.

---

We thank the reviewer again for their constructive comments. We hope that our response has addressed all the concerns, including the minor issues. If so, we kindly request the reviewer to reconsider the rating.

Sincerely,

The Authors of the Submission 11024

Dear Reviewer Wqun,

This is a kind reminder that the additional result for Pythia-2.8B on the Enron dataset is now included in Table 1 of our revised manuscript. We keep the secret the same ("Leo.Moreno@gmail.com") for ease of comparison.

We observe that the memorization pattern remains consistent with our results on other models, across fine-tuning algorithms: base fine-tuning readily memorizes the secret, whereas the other PEFT mechanisms exhibit significantly lower levels of memorization, often by orders of magnitude. Among the PEFT methods, prompt tuning shows lower exposure than LoRA and prefix tuning. Notably, we find that Adapter achieves the lowest exposure.

Thank you,

The Authors of the Submission 11024

Dear Authors,

Thanks for your response. Although I was referring to this related paper [a], as the rebuttal for the exclusion of the concurrent work is acceptable. Nonetheless, thanks for acknowledging that.

About the new experiments, it is interesting to see that the Pythia-2.8B model achieves lower exposure and perplexity in some cases. Can the authors give a reason / justification why this is the case?

While I understand things can easily be overlooked due to responses to different authors, I would encourage the authors to have a general pass on the paper for the camera-ready version to ensure consistency. For instance, including the Pythia model as part of the considered models. Also include the Pythia model in all the experiments instead of just the GPT2 and GPT2-XL models.

Again, thanks for the effort you put into the paper. With this, I have no further questions.

Minor:

Your previous response on the new result should be Enron (Pythia-2.8B) and not MIMIC (Pythia-2.8B).

[a] "Open LLMs are Necessary for Private Adaptations and Outperform their Closed Alternatives" Vincent Hanke et al. (Neurips 2024)

Dear Reviewer Wqun,

Thank the reviewer for the continuous engagement in the discussion.

We attribute the lower perplexity to the larger model size (~2.8B). The original studies of PEFT methods we examine [1, 2, 3, 4] demonstrate improved task-specific accuracy when applied to larger pre-trained models.

We observe lower levels of exposure in our results for Adapter and LoRA. We hypothesize that for memorization, the Adapter’s rank should be proportional to the model size. We maintain the Adapter rank to 16 across all models, while our baseline models (GPT-2 and GPT-2 XL are 2–20x smaller than Pythia-2.8B). This may make the Adapter less likely to memorize secrets in fine-tuning datasets. We believe that investigating the interaction between model size and memorization in-depth is an interesting future work, and we will include this in our conclusion.

We will ensure that our final version is thoroughly reviewed to confirm that everything is correct.

Sincerely,

The Authors of the Submission 11024

---

[1] Housby et al. Parameter-Efficient Transfer Learning for NLP In International conference on machine learning, ICML, 2019.

[2] Li et al. Prefix-tuning: Optimizing Continuous Prompts for Generation. ACL-IJCNLP 2021.

[3] Lester et al. The Power of Scale for Parameter-efficient Prompt Tuning, EMNLP 2021.

[4] Hu et al. LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022.

continues from the previous comment...

---

[Questions 2 and 3: Impact of the Dataset Size]

As Reviewer yauG pointed out, the dataset size does not have a substantial impact on privacy leakage. Similarly, in our initial investigation, we did not observe any substantial impact of dataset size on our findings. A quantitative illustration would make our results more convincing. Below we summarize the findings from our initial investigation, where we varied the size of the MIMIC-III dataset by increasing it by 100% (2x) and also decreasing it by randomly choosing 50% and 25% of the MIMIC-III dataset for our experiments.

The table above summarizes our results, which are also consistent with the findings reported in our manuscript. Models fine-tuned using PEFT methods are less likely to memorize the secret. Prompt-tuning and LoRA achieve the lowest exposure, while the other two methods also reduce exposure to levels similar to the observation made in our manuscript.

---

[Summary]

We appreciate the reviewer’s feedback on the clarity. We kindly request the reviewer to reconsider the rating if our answers clarify existing concerns. If there is/are any remaining concerns, we will be happy to address them during the discussion phase.

We thank the reviewer for the time and effort in evaluating our manuscript. We answered the reviewer’s concerns and questions below and have also updated our manuscript to reflect them.

---

[Weakness 1 and Question 1: Generalization to Larger Models]

We thank the reviewer for suggesting a way to make our results more convincing. We run an additional evaluation with a Pythia-2.8B model (which is larger than LLaMA2-1.5B) on the MIMIC dataset. We use the same secret “mary smith” for ease of comparison. Our result is shown below:

Our findings were consistent with the results shown in our manuscript. While base fine-tuning easily memorized the secret, prompt tuning and other PEFT methods demonstrated significantly lower memorization. Similar to the GPT2-XL results, Adapter and LoRA showed comparable exposure levels, with prompt tuning yielding far lower exposure. Interestingly, prefix tuning lagged slightly behind Adapter and LoRA in Pythia-2.8B. We have included this additional result in our main table in the revised manuscript.

[Weakness 2: Choice of Our Task]

We clarify that our auditing task is based on the standard practices established by prior work for evaluating privacy risks [1], which are inarguably related to memorization, and this mechanism is based on information retrieval.

[1] Carlini et al., Quantifying Memorization Across Neural Language Models, ICLR 2024

There are two distinct tasks: (1) Membership inference-style auditing: Given a list of candidate secrets and a context, we check if the model produces the candidate (tokens) that was inserted into the training data. (2) Data extraction-style auditing: we query the model with various prompts and collect a set of tokens, which may or may not include the secret present in the training data.

We chose membership inference-style auditing as it represents a stronger version of data extraction-style auditing. It means that even if both auditing methods are performed, the results from data extraction-style auditing will be sub-linear compared to those obtained from membership inference-style auditing. No need to measure the sub-optimal attack success, as stated in our threat model section.

A separate approach, proposed by Nasr et al., involves auditing the privacy guarantees provided by differentially-private (DP) training of a model. But we do not believe this falls within the scope of our work, as no empirical attacks have matched the strength of the theoretical adversary assumed by DP (and also shown in our results).

[Weakness 3: Theoretical Analysis]

To the best of our knowledge, the theoretical analysis framework available from prior work is “differential privacy (DP).” Our work avoids an analysis with DP because the definition is agnostic to the training data, models, and the choice of secrets. Hence, conducting this analysis would offset the variations we observe across datasets, models, and PEFT algorithms against practically the strongest adversaries considered in prior work [1]. Developing a new theoretical framework is beyond the scope of our current work. We will include this as a potential direction for future research in the conclusion of our revised manuscript.

---

continues to the next comment...

Dear Reviewer Vj3z;

As the deadline for the discussion period approaches, we kindly ask if our response adequately addresses the reviewer’s concerns or if further clarification is needed. We look forward to hearing from the reviewer soon.

Thank you!

We first thank the reviewer for the time and effort in evaluating our work. We answer the reviewer’s concerns and questions below. We also updated our manuscript to reflect the reviewer’s valuable and constructive feedback.

---

[Weakness 1: Limited Scope of Privacy Attacks]

We thank the reviewer for their valuable comments on improving the accuracy of our claims within the scope of the privacy attacks. We are considering clarifying the focus of our work on data extraction attacks, which are arguably the most important concern in this context. Here we listed potential titles we can use, and we will also be happy to incorporate more suggestions from the reviewer:

Evaluating Memorization of Parameter-Efficient Fine-Tuning

Evaluating Membership Risks of Parameter-Efficient Fine-Tuning

[Weakness 2 and Question 2: Performance Metrics]

We first clarify that we adhere to standard practices for evaluating fine-tuned language models, as outlined in prior work [1]. In fine-tuning scenarios we employ, conventional evaluation metrics are not straightforward to apply. Metrics such as BLEU scores or accuracy on benchmarking datasets like Alpaca are designed to assess a model’s few-shot generation capabilities. On the other hand, task-specific accuracy can only be measured when we “fine-tune” pre-trained language models on specific benchmarking tasks (QA or sentiment analysis), not the tasks we use like MIMIC-III and/or Enron. However, we also acknowledge the importance of developing a universal metric for evaluating fine-tuned models across different datasets. We discuss this as a potential direction for future work in our conclusion.

[1] Wen et al., Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models, NeurIPS 2024.

[Weakness 3, 4, and Question 1: Presentation Issues]

We thank the reviewer for pointing out mistakes we made in our manuscript. We first reviewed the sections presenting our main results to ensure there are no dangling references to the Appendix (e.g., we corrected the reference in Sec 4.4 to direct readers to the additional results on prompt-tuning, which are located at the end of the Appendix). Moreover, we proofread the manuscript to fix any typos (e.g., in Line 376, we intended to write "without DP," not "with DP").

---

[Summary]

We appreciate the reviewer’s constructive comments. If there is/are any remaining questions and concerns, we will be more than happy to address them during the discussion phase.

Dear Reviewer CXB3;

As the deadline for the discussion period approaches, we kindly ask if our response adequately addresses the reviewer’s concerns or if further clarification is needed. We look forward to hearing from the reviewer soon.

Thank you!

We first thank the reviewer for the time and effort in evaluating our work. We answer the reviewer’s concerns and questions below. We also updated our manuscript to reflect the reviewer’s valuable and constructive feedback.

---

[Main Weakness: Generalizability]

We acknowledge the importance of evaluating the generalizability of our results across different secrets. In our preliminary investigation, we observed minimal variations as long as the secrets had not been encountered during pre-training. To provide more concrete evidence, we conducted the following additional experiments:

(Points 1 and 3: Less common names; other secret types)

We use the secret “clary zakharchuk” for fine-tuning GPT-2 models on the MIMIC-III data.

The table above presents our results, which are consistent with the findings reported in our manuscript. Models fine-tuned using PEFT methods are less likely to memorize the secret.

(Points 1 and 2: Secrets from a completely different domains)

We also test with a secret that is unlikely to naturally occur in the fine-tuning dataset. We insert the secret “Leo.Moreno@gmail.com” into the MIMIC-III dataset composed of medical records.

The table above summarizes our results, which are also consistent with the findings reported in our manuscript. Models fine-tuned using PEFT methods are less likely to memorize the secret. Prompt-tuning and LoRA achieve the lowest exposure, while the other two methods also reduce exposure to levels similar to the observation made in our manuscript.

---

[Other Weaknesses 1: Impact of the Dataset Size]

The reviewer is correct. We did not find any substantial impact of the dataset size on our findings in our initial investigation. We also agree with the reviewer that a quantitative illustration would present our results in a more convincing manner. Below we summarize the findings from our initial investigation, where we varied the size of the MIMIC-III dataset by increasing it by 100% (2x) and also decreasing it by randomly choosing 50% and 25% of the MIMIC-III dataset for our experiments.

Overall, the results remain consistent with those observed when we use the full dataset. Models fine-tuned with the PEFT mechanisms achieve lower memorization. Prompt-tuning and LoRA are the lowest, while Adapter and Prefix-tuning show slightly higher levels than the first two.

[Other Weaknesses 2 and 3: The Use of Other Epsilon Values]

• (W2) We acknowledge the reviewer’s concern about presenting only two epsilon values. Instead of presenting a figure like Figure 4, we decided to summarize our results in a table (see Table 2 in our revised manuscript) to clearly illustrate the interaction between exposure, perplexity, and various epsilon values across the different PEFT methods we examined.

• (W3) We agree with the reviewer that evaluating GPT2-XL at a reasonable epsilon value (1.0), as suggested, could strengthen the validity of our results. Accordingly, we will conduct this additional experiment and make sure to add this to Figure 12 (Figure 11 in our revised manuscript) in our camera-ready version.

---

continues to the next comment...

continues from the previous comment...

---

[Question 1: Information Bottleneck Hypothesis for LoRA]

We thank the reviewer for the suggestion. To evaluate the information bottleneck hypothesis for LoRA, we first ranked the perplexities of all candidate names used for MIMIC to identify the one that the model already exhibits a bias toward due to its pre-training procedure. We selected the name with the highest exposure without context in the pre-trained GPT-2 model. This name was then inserted into the fine-tuning dataset, and the model was fine-tuned with LoRA.

Our findings show that the exposure is significantly higher when using this alternate name as the secret—up to 7.13, compared to 1.88 when "mary smith" is used as the secret. This supports the hypothesis that the biases of the pre-trained model and its dataset play a critical role in determining whether LoRA can memorize secrets in the fine-tuning dataset. These results are included in Appendix section B.12.

[Question 2: The Choice of Delta]

We use a record-level delta, calculated as the inverse of the dataset size. For both MIMIC and Enron, this delta is $\sim$7.4$\times$10$^{−7}$ (1/13.3k), following standard practices in prior work and the original study [1].

[1] Abadi et al., Deep Learning with Differential Privacy, ACM CCS 2016

---

[Minor Comments]

We thank the reviewer for pointing out the presentation issues. We will also thoroughly review our manuscript again to ensure there are no remaining typos.

---

[Summary]

We appreciate the reviewer’s constructive comments. If there is/are any remaining questions and concerns, we will be more than happy to address them during the discussion phase.

We thank the reviewer for the time and effort in reading and evaluating our manuscript. Below we answer the reviewer’s concerns and questions. We also updated our manuscript to reflect the constructive feedback from the reviewers.

---

[Weakness 1: Improving Presentations]

We thank the reviewer for the suggestions. We have made the following updates to our manuscript to improve clarity further:

• Point 1: We revised Figure 1, 4, and 5 to use a log-scaled axis.
• Point 2: We included titles to all the figures highlighting their distinguishing features.
• Point 3: Instead of presenting our results in a figure (Figure 4, originally), we now use a table (Table 2) to clearly illustrate the interaction between exposure, perplexity, and epsilon across the PEFT methods we examined.

[Weakness 2: Additional Details about Our Results]

We first thank the reviewer for pointing out the typo in lines 246-247, where we indeed meant ‘decrease.’ We fixed it in our revised manuscript.

We clarify that a reduction in exposure across our PEFT-trained models is not due to the inability to generalize well. To evaluate, we followed the reviewer's suggestion and measured the exposure of three undertrained models with perplexity similar to that observed in the PEFT-trained models. As shown below, all these undertrained models achieve an average exposure of 5.20–5.60, at least an order of magnitude greater than shown in Table 1 of our manuscript. We included the result in the Appendix and referenced it in Sec 4.4 to improve clarity.

[Weakness 3: Motivation for Our Adaptation to the Memorization by Carlini et al.]

We adapted the memorization definition of Carlini et al. to examine the impact of secret positions within a training record. Carlini et al. focus exclusively on scenarios where a secret appears at the end of a training record, e.g., “Please contact me at john.doe@gmail.com” It does not consider many real-world cases where secrets appear in the middle of a context, such as “... since the prior John Doe radiograph, and a very small left apical…” Our evaluation reveals that depending on the design of the PEFT algorithms, a secret in certain positions is more likely to be memorized. This contrasts with the prior work, which shows that secrets at the end of a sentence are more likely to be memorized.

[Weakness 4: Combination of Empirical Privacy Defenses and PEFT]

We believe that this topic is beyond the scope of our work and could constitute a separate study. Hence, this wouldn’t be our weakness. For instance, prior work [1] has examined the impact of a subset of such heuristic protection methods, such as data augmentation, on membership inference attacks against computer vision models. However, we acknowledge the importance of future research in empirically evaluating heuristic protection mechanisms for privacy across different PEFT methods and have included this as a potential future work in our conclusion.

[1] Kaya et al., When Does Data Augmentation Help With Membership Inference Attacks, ICLR 2021

[Weakness 5: Illustrating Example Records in Evaluation]

We have included examples of secrets inserted into training records in the Appendix of our revised manuscript.

---

continued in the next comment

continued from the previous comment.

---

[Questions 1 and 2: Clarification of Our Results in Figure 3]

We admit our mistake in including an incorrect figure: the left figure represents an average exposure computed across different seeds, while the right figure is from a single observation. We fixed the right figure. The revised figure now shows that exposure is lower when trained with DP and that 500 insertions lead to higher exposure values compared to a single insertion.

[Question 3: Number of Insertions in (1, 500)]

We excluded results for insertion counts between 1 and 500 because we did not observe any meaningful variations in exposure. For instance, with 5 insertions, both standard fine-tuning and PEFT methods typically reached the maximum exposure in most cases. The only case where we observed a variation was when the secret was located at the first token position. To help understand, we provide an example of this variation using standard fine-tuning on GPT-w with the MIMIC dataset:

[Question 4: Impact of PEFT Hyper-parameters Under DP]

We could not find any consistent relationship between PEFT hyper-parameters and the perplexity under DP. The first observation we had is that the trend differs from the datasets we use. In the MIMIC-III dataset, larger hyperparameters generally result in higher perplexity (except for Adapter with $r=32$). In contrast, we observe a reduction in perplexity for the Enron dataset under the same conditions. One possible explanation could be that there are optimal PEFT hyper-parameters required to achieve reasonable performance (e.g., in MIMIC-III, the rank $\sim$4--8 and the prefix tokens $\sim$16). Increasing those parameters beyond the optimal range can increase the noise added by DP-SGD and make the performance fluctuate.

---

[Summary]

We appreciate the reviewer’s feedback on the clarity and presentation issues.

The current rating for our work is 3, which, according to the review guidelines of other premier ML conference venues, typically indicates "a paper with technical flaws, weak evaluation, inadequate reproducibility, and/or incompletely addressed ethical considerations." We see the weaknesses in the current review do not fall into these categories; thus, we kindly request the reviewer to reconsider the rating if our answers clarify existing issues. If there is/are any remaining concern(s), we will be happy to address them during the discussion phase.

Dear Reviewer wAdk;

As the deadline for the discussion period approaches, we kindly ask if our response adequately addresses the reviewer’s concerns or if further clarification is needed. We look forward to hearing from the reviewer soon.

Thank you!

I thank the authors for the detailed response. Most of my concerns have been addressed, and I have increased the score to 5. However, I still have several remaining concerns:

Regarding Weakness 3:

My concern was that introducing a Definition for a term that is barely used later in the paper (e.g. not used in the experiments) somewhat overloads the paper. In my opinion, describing the differences between the experimental setup and Carlini et al. could have been sufficiently covered in just one or several paragraphs.

Regarding Figure 3:

If the experiments in Figure 3 and Table 2 were conducted under the same setup (I couldn't find specific model and dataset details for Table 2, though I might have missed them), then Table 2 suggests that at $\varepsilon=10.0$, the model's perplexity was already above 10, indicating severely degraded model quality. I request that the authors clarify the setup for Table 2 and specify the model's perplexity in Figure 3 at $\varepsilon=10.0$, as I believe examining other metrics (such as exposure) is not meaningful when the model's performance is unsatisfactory.

Regarding Question 3:

I believe such an ablation study could be beneficial to demonstrate how exposure deteriorates with increasing insertions. If exposure reaches its maximum value at 5 insertions (as stated in the authors' response), it would be valuable to examine the behavior for insertions between 1 and 5. Furthermore, it seems illogical to conduct most experiments with 500 insertions when exposure reaches similar values at just 5 insertions.

I thank the authors for the detailed response. I acknowledge that I have read it.

A few concluding comments:

• I don't see a description of the dataset and model in lines 424-425; there is only a discussion on the PEFT hyperparameters and the $\varepsilon$ value.

• I also wasn't aware that the setup with 500 follows prior work; it would be beneficial to specify this in the paper.

• Additionally, I find some of the authors' arguments regarding my previous concerns not convincing enough.

Therefore, I am maintaining the score of 5.

Dear Reviewer wAdk,

We sincerely thank the reviewers for the continuous engagement in the discussion.

[Memorization Definition (Weakness 3)]

We acknowledge the importance of clarifying the connection between our memorization definition and our experimental designs. We will make sure to spare one (or a few) paragraphs in the experimental setup section of our camera-ready version to address this.

---

[Prefix-tuning Results Shown in the Left-figure of Figure 3]

We clarify the experimental setup produced the results in Table 2 and Figure 3. In Table 2 (for prefix-tuning), we set the number of prefix tokens to 64. But for the analysis shown in Figure 3, we consistently set all the PEFT hyper-parameters to 16, including the number of prefix tokens. The dataset we use in Table 2 is the same as for our main results in Table 1. We stated this setup in lines 424–425; but, we acknowledge that improving the clarity of this description will enhance the readability of our results. We will make this clear in our camera-ready version.

We acknowledge that prefix-tuning is not particularly compatible with differential privacy. In Figure 3, while the model perplexity does not reach as high as 10, it remains relatively high at 5.0–5.2. To address this concern, we ran prefix-tuning with an epsilon of 160, which ensures the model perplexity of 1.5, comparable to other PEFT methods at an epsilon of 10. Even in this setup, we find a trend consistent with our results in Figure 3: the first few tokens are more likely to be memorized when prefix-tuning is applied. Near position 0, we observe exposure of ~2.5, while near position 50, exposures are ~1.2. We will make sure to present our results when the prefix-tuned models achieve a reasonable perplexity in the final version.

---

[Impact of the Number of Secret Insertions (Question 3)]

We clarify that this evaluation aims to determine “whether the trend observed with a single insertion holds when the number of insertions increases to 500—an extreme case tested in prior work [1], where memorization of a secret can almost always be ensured.” One insertion represents the trend likely to be observed in practice (as the number of duplications is unlikely to be 500). But with 500 insertions, following the upper bound of the worst-case duplication of a secret as studied in prior work [1], we ensure that the observation from a single insertion is not coincidental but is a consistent pattern. Tightening the number of insertions based on empirical observations risks the evaluation of our hypotheses above.

We acknowledge that the evaluation suggested by the reviewer could address a different research question: “How do the memorization behaviors of PEFT-ed models vary with different numbers of insertions?” Our answer from the evaluation with the number of insertions {1, 10, 50, 100, 500} is “We do not observe any significant variations.” But examining with more fine-grained intervals such as in (1, 4) to answer this question is beyond the scope of this work.

---

We hope our response has addressed the reviewer’s concerns. We know that the deadline for the discussion phase is approaching, but If there is/are any remaining questions and concerns, we will be more than happy to address them.

Sincerely,

The Authors of the Submission 11024

---

[1] Carlini et al. Quantifying Memorization Across Neural Language Models, ICLR 2023