Rethinking LLM Memorization through the Lens of Adversarial Compression

We appreciate your positive feedback on our proposed metric for defining memorization in LLMs and its practical implications. We're glad you found our examination of unlearning methods using ACR insightful and our paper clear and well-organized. We acknowledge your concerns and attempt to address them below:

Re: Adversarial View for Compression Argument

We understand the need for further justification of the adversarial view in our compression argument. Here are a few points addressing this concern:

1. Necessity of Adversarial View: The adversarial perspective is crucial because it robustly challenges the model’s misuse of training data. This view ensures that the model cannot evade detection of memorization by merely altering its output slightly. We will expand our discussion to better justify the assumption that phrases stored within the LLM's weights can be reproduced with fewer input tokens than output tokens.

2. Threshold Analysis: We will provide a more detailed analysis of the threshold parameter used in our definition. This will include a discussion on how different threshold values affect the detection of memorization and the rationale behind our chosen threshold.

Re: Efficiency Analysis of Algorithm 1

We agree that an efficiency analysis of Algorithm 1 is important. Here are our points addressing this concern:

1. Efficiency Analysis: We will include an efficiency analysis of Algorithm 1 in the revised manuscript. This analysis will provide insights into the computational requirements of the algorithm and its scalability across different datasets and model sizes.

Re: Alternative Optimization Methods

We appreciate your interest in alternative optimization methods. Here are a few points addressing this concern:

1. Alternative Methods: Besides the Greedy Coordinate Gradient (GCG) method, we have explored other optimization methods such as Random Search. This is presented in Algorithm 3 in our appendix. Additionally, we can move the discussion of other discrete optimizers (like those popular in the LLM Jailbreaking literature) from where they are mentioned in the additional related work section of our appendix to the main body.

Re: Completion as a Metric for Memorization

We understand the need for further clarification on why completion alone is insufficient as a metric for memorization. Here are a few points addressing this concern:

1. Completion Limitations: Completion is too conservative to capture memorization when model owners may take steps to make it look like their model has not memorized data it shouldn’t have. For example, unlearning methods (Sections 4.1, 4.2, 4.3) can be used to obscure memorization. Our definition, which relies on optimization and not completion alone, is not fooled by these minor tricks to appear compliant.

2. Clarification: We will further clarify this point in the next version of the paper, explaining how our method provides a more robust detection of memorization compared to completion-based metrics.

Re: Limited Conclusions

We acknowledge that our initial conclusions were based on experiments with two specific LLMs. Here are a few points addressing this concern:

1. Broader Experiments: Our experiments to include four different models in the Pythia family, which are open-weight, open-source models available at the time of writing. Additionally, we conducted experiments with LLaMA-2, a LLaMA model tuned on TOFU, and a version of LLaMA that does not know Harry Potter. These models are characteristically different, providing a broader evaluation of our metric.

2. In-Context Unlearning: We also performed experiments with in-context unlearning, demonstrating the applicability of our method across a wider range of scenarios. These additional experiments expand the breadth of our evaluation and further validate the robustness of the ACR metric.

Conclusion

Once again, we thank you for the constructive feedback on our work. We believe that these clarifications and additional analyses will significantly strengthen our paper, addressing your concerns and providing a more comprehensive and robust evaluation of the ACR metric. We are confident that these enhancements will improve the quality and impact of our work.

Thank you for your positive review! We are pleased that you found our approach innovative and recognized the significance of our contribution to defining and measuring memorization in LLMs. We acknowledge your concerns and attempt to address them below:

Re: Favoring Longer Sequences

We understand the concern that the ACR metric may favor the detection of memorization in longer sequences due to the potential for a higher compression ratio. Here are a few points addressing this concern:

1. Performance Across Different Lengths: To address this issue, we have included a plot with sequence length on the x-axis to illustrate how ACR performs across different lengths. This analysis shows that while longer sequences can achieve higher compression ratios, the ACR metric remains effective and meaningful for shorter sequences as well. We have added this plot and corresponding discussion to the revised manuscript.

2. Balanced Evaluation: Our experiments were designed to include a balanced mix of both short and long sequences, ensuring that the evaluation of the ACR metric is comprehensive and unbiased. This helps mitigate any potential bias towards longer sequences.

Re: Introduction to MINIPROMPT and GCG

We appreciate the suggestion to provide more details about the MINIPROMPT and the Greedy Coordinate Gradient Descent (GCG) optimizer. Here are our points addressing this concern:

1. Detailed Introduction: We have expanded the introduction to MINIPROMPT and included a brief overview of the GCG optimizer in the main text. This additional context will make the paper more reader-friendly and accessible to those unfamiliar with these concepts.

2. Clarity and Readability: These enhancements ensure that readers have a clear understanding of the methodologies used in our experiments, improving the overall clarity and readability of the paper.

Re: Accounting for Variations in Expression Length

We understand the importance of addressing how the ACR metric accounts for variations in the length of expressions conveying the same knowledge. Here are our points addressing this concern:

Variations in Expression Length:

The ACR metric focuses on the compression ratio, which inherently normalizes the length of the input prompt and the generated sequence. This allows the metric to be robust against variations in expression length, as it measures the relative information content rather than the absolute length.

Disentangling knowledge from verbatim memorization:

The analogy of birds can fly is a great one. In general, facts are not copyrightable, and a desirable measure would be one that can disentangle knowledge from verbatim memorization. Toward this end, we conducted a new set of experiment. In particular, we paraphrased the 100 famous quotes used in our paper with ChatGPT and checked their ACR values. The results are as follows:

Our findings suggest that paraphrases of memorized content generally do not get flagged by our method unless the paraphrase itself is memorized (some paraphrases were also available on the internet based on our cursory search). This supports the idea that "facts are not copyrightable," and our method aligns with this principle.

Re: Interpretation of Metrics

We appreciate the question regarding the interpretation of models A and B, where A has a larger portion memorized, but B has a smaller average compression ratio. Here are our points addressing this concern:

    Portion Memorized vs. Compression Ratio: We believe the portion memorized is a more crucial metric in practical applications. For instance, in legal contexts, the binary test of specific samples is more likely to be applied, making the portion memorized a more relevant measure of memorization.

Conclusion

Once again, we thank you for your valuable feedback. We believe that these clarifications and additions will significantly strengthen our paper, addressing your concerns and providing a more comprehensive and robust evaluation of the ACR metric. We are confident that these enhancements will improve the quality and impact of our work.

Thank you for your comprehensive review and valuable feedback.

We appreciate your recognition of the strengths in our paper, particularly the well-positioned and motivated overview, exhaustive experimental setup, and the efficacy of our proposed metric in detecting memorized content despite unlearning efforts.

Regarding the weaknesses you identified, we conducted additional experiments to address your points.

Limited budget for GCG The idea is interesting, however, we found that controlling the budget does not significantly change things because our code is designed to exit early once a successful prompt (one that elicits an exact match) is found. Therefore, increasing the command line argument for num_steps does not provide additional benefits, as GCG is already optimized to use the necessary compute efficiently. And decreasing the number of steps shows many samples for which GCG fails to find a successful prompt altogether.

Paraphrases We paraphrased the 100 famous quotes used in our paper with ChatGPT and checked their ACR values. The results are as follows:

We conducted a cursory Google search and found that some paraphrased versions are present on the internet. Our findings suggest that paraphrases of memorized content generally do not get flagged by our method unless the paraphrase itself is memorized. This supports the idea that "facts are not copyrightable," and our method aligns with this principle.

Thank you again for your insightful comments. We believe these additions and clarifications will strengthen our paper.

Thank you for your thorough and thoughtful review. We are pleased that you found our definition of memorization consistent with common notions and that you appreciated the clarity and depth of our paper. We acknowledge your concerns and attempt to address them below:

Re: False Positives

We understand the importance of considering false positives in our method. Here are a few points addressing this concern:

1. False Positives with GCG Algorithm: The concern about the GCG algorithm eliciting generations that were never seen by the model during training is valid. Our experiments, including those involving random strings, have shown that while the GCG algorithm can elicit any generation, we have never observed compression of non-training data. This suggests that our method is robust against false positives. However, we will add further discussion and conduct an additional experiment to explicitly address this point. This experiment will aim to determine the likelihood of false positives by testing the ACR metric on a controlled set of non-training data.

2. Reference to Related Work: We will incorporate findings from relevant literature (e.g., Geiping, Jonas, et al. "Coercing LLMs to do and reveal (almost) anything." arXiv preprint arXiv:2402.14020 (2024)) to strengthen our discussion on false positives and demonstrate the robustness of our method.

Re: Membership Inference Attacks (MIAs)

We acknowledge the potential role of MIAs in determining training data usage. Here are our points addressing this issue:

1. Limitations of MIAs: While MIAs can be used to test for training data usage, they have significant limitations, as highlighted in recent work (e.g., https://arxiv.org/abs/2402.07841, https://arxiv.org/abs/2406.06443, https://arxiv.org/abs/2406.16201). Further, MIAs typically rely on scalar-valued losses, which are not easily interpretable in regulatory or legal contexts. This complicates their applicability in such settings and makes conclusive findings difficult to reach.

2. Context of Our Claim: Our claim in Line 125 refers to the broader challenges of proving training data usage without access to the training data itself. Even if a publisher can run MIA, the interpretability and conclusiveness of the results remain challenging. We will revise this section to provide more context and clarity on this point.

Re: Clarity on Line 313

We appreciate your request for clarification on Line 313. Here are our points addressing this query:

1. Clarification on GCG and Gradient-Free Search: One might think that our findings are the results of some peculiarity in GCG or some bias/preference GCG has for finding short prompts on some types of data. We establish that the same general trends in memorization can be observed with a gradient-free search algorithm, and thus conclude that we are not mistaking a GCG bias for some other signal. We will provide more details on this random search experiment and explain how it supports the robustness of our findings.

Once again, we thank you for the constructive feedback on our work. Working on the pointers has helped us improve the quality of our analysis. We look forward to further discussions and improvements in this evolving field.

Thank you for your insightful feedback. We appreciate your recognition of our innovative approach to memorization testing. We acknowledge your concerns and attempt to address them below:

Re: Comparisons with Other Methods

We understand the importance of comparing our Adversarial Compression Ratio (ACR) with other existing methods for testing memorization. Here are a few points addressing this concern:

1. Verbal and Experimental Comparisons: In Section 2 of our paper, we verbally compare and contrast ACR with existing methods. Additionally, our section on in-context unlearning includes some experimental comparisons. However, direct experimental comparisons are challenging due to the different contexts in which these methods operate. For instance, methods like Extraction (https://openreview.net/forum?id=TatRHT_1cK) can't make conclusive claims about short strings, whereas we can handle strings of any length. Also, the adversarial nature of our approach, where the model owner might take minor steps to change the extractability, also differentiates our method. The in-context unlearning section serves as an example where other tests might be tricked, but our approach remains robust. More concretely, a direct comparison with the paper requested can be found in Section 4.1 of the paper, and also in Section 4.2 (Figure 3) on compression versus completion. These results clearly demonstrate how compression is a desirable metric to uncover seemingly hidden memorization.

2. Effectiveness of ACR: ACR introduces a new paradigm of memorization detection from an adversarial perspective. This approach has significant implications for discussions on copyright and intellectual property, where model providers may take measures to hide memorization. We believe that this perspective enriches the discourse and offers new insights for policy-making and regulation. It is more effective in the sense that it pierces the illusion of compliance, showing memorization where other methods might fail to, for example because of system prompts.

Re: Suitability of "Number of Tokens" as a Complexity Measure

We acknowledge the concern regarding the use of "number of tokens" as a measure of complexity. Here are a few points addressing this concern:

1. Compression Ratio: The compression ratio, where both the numerator (length of the generation) and the denominator (length of the shortest prompt) must be in the same units, is helpful in enforcing the constraint that the prompt has less information than the output. While other notions of information content could theoretically be used, we find the ratio of token lengths to be practically useful in defining memorization.

2. Empirical Utility: Empirically, we have found that using the number of tokens as a measure of complexity works effectively in our experiments. The simplicity of this metric allows for a clear and consistent definition of memorization across different contexts.

Once again, we thank you for the constructive feedback on our work. We hope we were able to clarify your concerns. Please let us know if there are any lingering concerns.