Exploring Memorization in Fine-tuned Language Models

We are grateful for the insightful feedback and constructive critiques provided during the ICLR 2024 review process. We appreciate the opportunity to address the concerns raised and enhance the clarity and impact of our work. We have thoroughly addressed each point in our rebuttal and hope these improvements justify a reevaluation of our score.

W1：The paper does not control for differences in output lengths of different tasks. Therefore, the results about differences in memorization behavior for different fine-tuning tasks are unreliable without controlling for this confounder.

Response: Thanks for your comment. In our experiment, we did not standardize the output length; rather, we provided the model with a sequence prefix and allowed it to autonomously determine the appropriate output length. The generation process ceases upon encountering a designated stop symbol. It is accurate, as noted, that certain tasks—such as summarization and dialogue—tend to produce longer outputs in comparison to tasks like sentiment classification or question-answering.

To isolate the impact of the output lengh, we augment the answer of sentiment classification task. Specifically, we change the label "Negative" to "According to the review, this audience gave negative feedback about the movie, indicating a general dissatisfaction with its overall quality and appeal. So the sentiment of this review is negative.", the label "possitive" to "According to the review, the audience shared positive feedback about this movie, highlighting its overall appeal and entertainment value. So the sentiment of this review is positive."(40 words) We call this task "Sentiment-Aug". In this case, the output lengh is comparable to tasks like summarization and dialog. However, we can find that Sentiment-Aug still represents very low memorization ratio and is much lower than high-memorization tasks. This underscores the nuanced relationship between output length and memorization, which is dependent on the intrinsic demands of the task at hand.

W2: The results on using attention maps as indicators of memorization appear similarly unreliable because they also do not control for important potential confounders. Summarization and dialog (presumably high memorization tasks) are shown to have more uniform attention maps, compared to QA and sentiment classification, where attention is more concentrated. However, for the latter two tasks, it may be sufficient to pay attention to a few keywords in the input, whereas summarization and dialog presumably require a more comprehensive understanding of the entire text. Therefore, the differences in attention patterns might simply be due to the nature of the different tasks, and not due to different degrees of memorization. For an apples-to-apples comparison, the authors should compare attention maps of instances with different degrees of memorization on the same task.

Response:

• Thank you for your insightful feedback. Indeed, as you have discerned, different tasks exhibit varying dependencies on specific informational elements, a characteristic intrinsic to the nature of the task itself. Our findings corroborate your observation:

  • Tasks that necessitate a comprehensive attention to detail, such as Summarization, demonstrate dense attention maps and a higher memorization ratio.
  • Conversely, tasks like sentiment classification, which inherently require only a subset of the available information, show markedly lower memorization.
  • This suggests that the disparity in memorization across tasks is fundamentally linked to the volume of input information that the task demands for successful completion.

• Furthermore, in response to your suggestion, we conducted a comparative analysis of attention scores and memorization ratios within the domain of summarization. By examining T5 models fine-tuned on distinct datasets—Multi-news and CNN_dail—we discovered that the Multi-news T5 model exhibits both a higher memorization ratio and a greater CAD. Conversely, the CNN_Daily T5 model displays a lower memorization ratio and reduced CAD. These observations are in alignment with our conclusion, reinforcing the premise that the memorization disparity is attributable to the essential nature of the fine-tuned task.

How is fine-tuning for the different tasks performed? E.g. for sentiment classification, what is the target output() that the models are fine-tuned to predict? A single word/token, or a short sentence?

Response: For sentiment classification, we use a single word, either positive or negative as output. During finetuning, we add a prefix for the input text: “Please classify the sentiment of the following paragraph: ” .For extractive QA, the output data which are the corresponding answer of the questions from the input data, is usually several words or a short sentence. We finetune this task in a sequence-to-sequence manner thus use the answer term as output. For other tasks like summarization, translation, and dialogue, we directly use the sentences presented in the original dataset as output. In our experiment, we consider all tasks as the format of generation tasks.(Like instruction tuning)

In Section 6, how do you use the base/non-fine-tuned models for summarization?

Response: In our experiment, the test procedure is the same for both base models and finetuned models. It means that we input the prefix $p$ of finetuning data to models and compare the $f(p)$ with suffix $s$. For base model, we still input the finetuning data prefix for test, even if the model is not finetuned on the dataset.(So if the model remember the data means the data also presents in the pre-training set)

I have difficulty interpreting the memorization examples in Table 11 in Appendix D. Which prompts were used to generate the machine-written text?

Response: In our experiment to examine memorization, we don't have a task-specific prompt during the test time. The Machine-written text is the output of the model when we input the prefix of the original input. We follow the memorization inference setting from previous work[3].

[3] Quantifying Memorization Across Neural Language Models

Q1: Is there only one split of each input into prefixes and suffixes? If yes, the memorization behavior of the model might differ, based on the length of prefixes, so it might be worth testing memorization for different prefix lengths. If not, the memorization ratio definition seems difficult.

Response: Thanks for your question, we conduct ablation studies on Dialog,Summarization, and Sentiment Classification with differnt prefixs k. and reported the result on Appendix B.3. And the results show that:

• The length of prefix tokens can affect memorization.
  • The length of prefix tokens does indeed impact memorization. Specifically, for summarization and Dialog tasks, the memorization ratio generally increases with the length of the prefix. This finding aligns with previous research on pretrained memorization[^2]. However, for sentiment classification, changing the prefix does not result in significant changes, and increasing the prefix length does not necessarily lead to an increase in the memorization ratio.
• The task disparity still exists when using different prefix lengths.
  • Furthermore, it is worth noting that despite the influence of different prefixes on memorization, there still exists a noticeable disparity in memorization across tasks. Therefore, our conclusion remains even using different prefixes.

We are grateful for the useful comments provided by you. We hope that our answers have addressed your concerns. If you have any further concerns, please let us know. We are looking forward to hearing from you.

Thank you for the extensive response and for responding to all of my questions and concerns! Your answers address a lot of the clarity issues I had with the work, and it's good to see the additional results you provide. My main objections regarding soundness remain, however, so I will maintain my score.

Here are my thoughts on the response:

• W1: Sentiment-Aug task does not really address the problem of controlling for output length. The additional output tokens are just due to a fixed template that the model regurgitates, whereas in summarization and dialog the model produces potentially completely different token sequences depending on the input. Overall I'm not convinced that the notion of memorization based on whether sequences from the finetuning data are repeated is the right one for a task like sentiment classification. A more appropriate notion would seem to be based on detecting whether the assigned label strongly depends on certain substrings that also appeared in the finetuning data. It may be worth taking a look at work that proposed label memorization metrics for classification tasks, although it has been done in the vision domain and would have to be adapted to the NLP setting: https://arxiv.org/abs/1906.05271. I would consider developing separate approaches to memorization detection in tasks that require text generation vs classification or localization, since both the output formats as well as what constitutes memorization at a conceptual level differ significantly.
• W2: Thank you for your clarification. The results shown are a good first step, but I think it is necessary to investigate whether the attention-memorization trend holds more broadly to infer a statistically significant correlation.
• W4: It's good to see these additional numbers. I think the post-finetuning scores should be normalized by the pre-finetuning scores to make sure the measure is reporting the right results.
• Q1: Thank you for running the additional analysis. It is interesting to see that different prefix lengths recall memorized text to different extents.

We appreciate your reviews. We hope that our responses have adequately addressed your concerns. As the deadline for open discussion nears, we kindly remind you to share any additional feedback you may have. We are keen to engage in further discussion.

Q3:The experiment involving Flan T5, where the memorization ratio decreases after fine-tuning, lacks a clear explanation. The authors should provide a rationale for this unexpected result or revisit their analysis.

In our study, we conducted memorization tests on the instruction-tuning set of Flan-T5, which encompasses a variety of tasks. Our findings reveal a notable decrease in the memorization rate when compared to models fine-tuned on single tasks. A plausible explanation for this observation is the enhanced generalization capability of the model in a multi-task setting. Exposure to a broader array of tasks and data appears to facilitate this improved generalization[1]. Consequently, the model's ability to execute tasks is not solely dependent on memorization but also on its refined generalization skills. This explanation has been incorporated into our manuscript to elucidate the observed decrease in memorization rates. Nevertheless, we acknowledge that the underlying causes of this intriguing phenomenon are likely complex and multifaceted and requires further investigation.

Q5. The authors' choice of focusing on the last layers of encoder-decoder attention (as seen in Figure 2) appears somewhat arbitrary. An explanation for this choice and its potential impact on the correlation between attention scores should be provided.

In our T5 architecture study, we examined three types of attention layers: encoder, decoder, and encoder-decoder attention layers. Our focus on encoder-decoder attention layers stems from their ability to capture the attention distribution across input features for each output, unlike encoder and decoder attention layers, which track internal attention within inputs and outputs, respectively.

This choice is based on the hypothesis that memorization behavior differences are linked to the specific information requirements for task completion. Encoder-decoder attention scores are thus more aligned with our research objectives.

We observed consistent patterns across various layers of the encoder-decoder attention mechanism, with high memorization tasks showing dense attention and low memorization tasks focusing attention on fewer positions.(And we add the attention maps of other encoder-decoder layers in Appendix-F.2) Due to this consistency, we report primarily on the final layer, closest to the output, for clarity and relevance. Detailed attention maps across different layers are now available in the Appendix-F.2 for further reference.

Q7. The concept of discriminating between different types of memorization: I believe idea memorization which encapsulates the fundamental sense of the initial information, should be a superset of all other forms of memorization. So, I do not understand this whole concept of categorizing memorization and what are we trying to achieve with this.

Thank you for the question. We want to clarify that these three types of memorization cases are non-overlapping and idea memorization is not a superset of others. Here we will distinguish 3 types of memorization. First, verbatim memorization means exact copies of words or phrases without transformation. In the cases of paraphrase and idea memorization, the output are not identical to the original text but share similar meanings. While paraphrase plagiarism targets sentenceto-sentence transformations, idea plagiarism reads a chunk of the content and condenses its main information into fewer sentences(or vice versa). It is crucial to report and distinguish these types of memorization, as each poses unique implications for privacy concerns. By categorizing and examining these memorization behaviors, we aim to deepen the understanding of language models' capacity for retaining and reproducing information.

Examples:
Verbatim:
Text A: My name is Jack
Text B: My name is Jack

Paraphrase: 
Text A: My name is Jack
Text B: Jack is my name 

Idea plagirism:
Text A： A boy tell me in the class that his name is Jack
Text B: A boy is Jack

In practice of the PAN2014-detection, It starts by identifying closely matched short document fragments (referred to as 'seeds') and then expands these seeds into longer text segments. This is achieved by clustering these fragments based on their separation and employing a 'maxgap' threshold parameter to form coherent clusters.They experimentally find out the most suitable threshold for different plagiarism datasets so that those parameters could be used for detection of a specific type of memorization. In another word, each memorization cases will be count only once and there will not be overlapping across different catagories.

Q8. Please use % and not %. the latter is very non-standard for ML papers.

Thanks for your comment. We have changed the symbol in the revision.

[1]: Scaling Instruction-Finetuned Language Models

Thank you for your valuable feedback and constructive comments in the review of our submission for ICLR 2024. We appreciate the opportunity to address the concerns raised and enhance the clarity and impact of our work. We believe that the revisions and additional experiments we propose, as detailed in this rebuttal, will significantly strengthen our paper. Therefore, we respectfully request the reviewers to reconsider the scoring in light of these improvements and our responses to the feedback provided.

Q1: Need for a more rigorous analysis of task specificity: The authors should consider potential confounding factors,especially the influence of the pre-training data used for the model, as this could impact memorization ratios.

Q4: To draw meaningful conclusions, the authors need to compare the memorization scores of the T5 base model before and after fine-tuning

Response: Thanks for your valuable questions. we add experiment on Appendix B.1 to compare the memorization difference between T5-base and fine-tuned T5 to verify our conclusion.

Memorization of T5-base

Memorization Difference after fine-tuning

As you correctly mentioned, some of the fine-tuning data may also be present in the pre-training data, as T5 also shows some cases of memorization of the fine-tuning data. However, after fine-tuning, we can clearly see that the fine-tuned model's memorization for summarization and dialogues improves significantly. It is obvious that the model has memorized a large number of fine-tuning data samples during the fine-tuning stage. However, for other tasks such as sentiment classification, the memorization ratio does not change much, meaning that the model does not remember much information from the fine-tuned data.

Q2: The rationale behind introducing attention discrimination as a metric is unclear

The reasons we introduce attention scores as an indicator are:

• Intrinsic Task Properties:

  • Attention scores inherently represent the unique characteristics of each task. Diverse tasks have varying requirements regarding the extent to which they necessitate focus on the entire set of presented information. These requirements are aptly mirrored in the attention heatmap patterns.
  • We visualize the attention heatmap of the T5 base (non-finetuned) when we use the non-finetuned model to perform different tasks. We were surprised to find, in Appendix F.1 and Figure 5, that the patterns still differed significantly across tasks (high memory-intensive attention). It further verifies that attention distribution is an intrinsic property of the task itself. In other words, taking summary as an example, no matter what model we use (even the human brain) to complete the task, we need to pay attention to the details of the input to complete the task. If we train the model on this "high memory" task, the model tends to remember more.

• Robustness of Attention Scores: Compared to memorization ratios, attention scores are less susceptible to external variables such as model type, training duration, and hyperparameters. This robustness makes them a more reliable indicator of a task's intrinsic characteristics.

• Practical Implications: One use case might be: before fine-tuning a model for a specific task, the model builder could first test attention patterns for that task by inferring other models (e.g., pre-trained models). Based on the attention pattern, the model builder can know whether the training data will be easily remembered if we fine-tune it for this task.

We are grateful for the useful comments provided by you. We hope that our answers have addressed your concerns. If you have any further concerns, please let us know. We are looking forward to hearing from you.

We appreciate your reviews. We hope that our responses have adequately addressed your concerns. As the deadline for open discussion nears, we kindly remind you to share any additional feedback you may have. We are keen to engage in further discussion.

Thank you for your valuable feedback and constructive comments in the review of our submission for ICLR 2024. We appreciate your positive feedback and the opportunity to address the concerns raised and enhance the clarity and impact of our work. We believe that the revisions and additional experiments we propose(as detailed in this rebuttal) will significantly strengthen our paper . We hope this rebuttal demonstrates the advances and merits a higher score.

(a) What does x% memorization mean in the experiments? It would be great to demonstrate perfect and no memorization baselines to get a sense of the numbers.

Response: In our experiment, we input prefix $p_{i}$ to the model and compare $f(p_{i})$ with the all suffixes $\{s\}$ to identify whether it constitutes memorization. A memorization rate of x% indicates that an equivalent proportion of the prefixes induces the model to produce memorized content. A rate of 0% implies no memorized content is produced, whereas a rate of 100% indicates that all prefixes lead to memorized output. This metric is essential for assessing the model's propensity to replicate known information.

(b) How do the authors measure Idea memorization? How do they differentiate Idea memorization from summarization?

Response: In our research, although summarization and idea memorization share similarities, they actually refer to different concepts. Summarization is a task that the model conducts. The user inputs x(e.g. news), and the model should output the summarization of the input(e.g. The title). However, the idea memorization in our paper means a different thing. We input the part of information of x(x_prefix) to the model, if the model outputs some key information of the remaining part of x(x_suffix), we regard it as Idea memorization. E.g. We input the beginning of the news and the model tells us information of the remaining part of the news.

(d) How much do the observations vary with varying "k"?

Response: Thanks for your question, we conducted ablation studies on dialog, summarization, and sentiment classification with different prefixes k and reported the result on Appendix B.3. And the results show that:

• The length of prefix tokens can affect memorization.
  • The length of prefix tokens does indeed impact memorization. Specifically, for summarization and Dialog tasks, the memorization ratio generally increases with the length of the prefix. This finding aligns with previous research on pre-trained memorization[1]. However, for sentiment classification, changing the prefix does not result in significant changes, and increasing the prefix length does not necessarily lead to an increase in the memorization ratio.

[1] Quantifying Memorization Across Neural Language Models

• The task disparity still exists when using different prefixes.
  • Furthermore, it is worth noting that despite the influence of different prefixes on memorization, there still exists a noticeable disparity in memorization across tasks. Therefore, our conclusion remains even using different prefixes.

We are grateful for the useful comments provided by you. We hope that our answers have addressed your concerns. If you have any further concerns, please let us know. We are looking forward to hearing from you.