Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs

We thank the reviewer for their valuable time and effort in providing this review. Following is our response.

Weaknesses

1. It is a bit unclear to me why it is necessary to mix in the Wikipedia sequences at such a high rate. RedPajamas is intended to provide a reasonable version of a training corpus, why not just train on that alone? Mixing in data with a repetition rate of 50 times is quite unusual, I wouldn't call that 'normal training conditions'.

   • In our experiments, we run two cases – stress testing goldfish loss to evaluate worst-case memorization and another in a relatively normal scenario. We inject different numbers (100 for stress testing and 2000 for normal case) of wikidocs as canaries at different upsampling (500 and 100 respectively) rates.
   • From prior work [1], it is known that some samples are more memorable than others. This can be due to multiple reasons from being duplicated multiple times, perplexity of text, or other unstudied reasons.
   • In our setup, we upsample/duplicate specific docs (Wikipedia) in order to observe memorization effects and measure mitigation by goldfish. Note, that the 2000 wiki docs after sampling 50 times only represent 1% of the total tokens model trained on with goldfish loss. Other tokens come from the the redpajama dataset. We use this controlled setup with wikidocs datapoints as canaries to simulate normal training scenarios which may have duplicated samples at unknown rates and corresponding memorization [2].

2. Several of the benchmark scores are barely above chance level

   • This is a good point. We directly discuss this in global rebuttal point 2 (Figure 2).
   • With the means of using the Control model (no continued training) and Standard loss (training w/o goldfish loss), we isolate the impact of using goldfish loss and report the eval figures as is.
   • In the global rebuttal point 2 (Figure 2), we point out that goldfish loss yields higher validation perplexity than standard loss. This suggests that for larger models, we'd see a benchmark performance drop in comparison to standard training (with identical hyperparameters).
   • In summary, goldfish loss observes a validation-loss gap in comparison to standard loss indicating some loss of performance. This suggests that for larger pretraining runs (>1B for >100B tokens, beyond our academic lab budget) and under identical settings, we would observe a decrease in benchmark scores for goldfish loss in comparison to standard loss. From our new experiments, we also show that this can be alleviated by matching standard loss on a number of supervised tokens by either of the above-mentioned configurations.

---

Questions

1. On normal training conditions
   • Our response is above.
2. Why the goldfish loss (per supervised token count) would be lower than the regular loss?
   • In Figure 5, the goldfish validation loss curve (solid) line is above the standard loss curve. This suggests that the model learns slower when matching standard loss on input-token by input-token count. Since the standard loss optimizes on all input tokens, it is intuitive to see it attain better validation through training.
   • Moreover, we also replot the validation curve across supervised token count (dashed line in Figure 5) i.e. tokens optimized upon calculated by multiplying the input token count by 3/4 (as on average, three out of four tokens (k=4) are used directly in the loss computation during training). This is an estimation of the validation loss when seen across the supervised token count.
   • We hypothesize that this validation loss gap (goldfish with higher loss than standard) is because of a lesser number of tokens supervised, i.e. optimized upon, during training. To better understand this issue using a more sensitive experiment, we pre-trained an LLM on proportionally increased tokens (to attain same number of supervised token count as standard loss).
   • In rebuttal pdf Figure 2 (right), we run goldfish loss with a proportionally increased supervised token count by (i) increasing batch size in one experiment and (ii) training for more steps in another. Both configurations alleviate the validation loss gap and achieve final validation loss nearly identical to standard loss.
   • We see that only the goldfish run with more steps (ii) achieves lower validation loss and not one with increased batch size (i) through training; albeit all 3 checkpoints end up with near identical validation loss. This is expected as across the same supervised tokens count, goldfish model (ii) has undergone more gradient steps than either standard loss or increased-batch size goldfish configuration.
   • Moreover, under the goldfish setup, we directly mutate the gradients by dropping a subset of (pseudorandomly chosen) tokens. This inherently changes gradients qualitatively. Prior work [3] has shown that this could speed up training and could be beneficial for selected downstream tasks.

---

Limitations

Chance level performance

• Please find our response above (and global rebuttal point 2). We have additionally added a chance level cut-off for each benchmark in our draft for transparency and completeness.

---

We thank the reviewer for their insightful and detailed questions that resulted in bettering our draft and analysis of goldfish loss. We further welcome any follow-up questions.

---

[1] Nicholas Carlini, et al., "Quantifying Memorization Across Neural Language Models," 2023.

[2] Katherine Lee, et al., "Deduplicating Training Data Makes Language Models Better," 2022.

[3] Zhenghao Lin., et al., "Rho-1: Not All Tokens Are What You Need," 2024.

We thank the reviewer for their time and for highlighting the real-world application of goldfish loss. Following is our response.

Weaknesses

Lacking theoretical guarantees

• As mentioned in our limitation (Section 7.1), our method is derived from first principles only and our strong results are empirical; thus comes no theoretical guarantees. We also discuss adversarial attacks on goldfish loss and perform experiments to provide empirical results (Section 6.2). We leave it for future work to provide memorization bounds for goldfish and standard loss.

On the computational complexity introduced by the hashing mechanism used for robust duplicate handling

• In short, the computational complexity of the integer hashing mechanism is entirely negligible, compared to the amount of compute required for a single training step. In our implementation, we project the integers defining tokens in the local context c onto a finite field with a fixed, pseudorandom map to the unit interval, but any integer hash implementation, such as common avalanche schemes, can be used. For us, the projection is a lookup operation after 2c integer multiplications and additions. For avalanche schemes that avoid the lookup, 10 to 30 int32 operations are generally sufficient for workable pseudorandomness. In contrast, the rest of the model training step requires 10*9 * 2048 * 3 ~ 10*12 floating point operations, for the 1B model.
• For more details, the exact Python implementation can be found in our supplementary material, and we'd be more than happy to add a paragraph in the appendix talking about hashing function implementations.

---

Questions

Impact of different values of k (drop frequency) and h (hash context size)

• The impact of different values of k can be found in Figure 3, Figure 4, Figure 6.
• We discuss the impact of different hash context window sizes in Section 3.1
• Finally, in Appendix B (Figure 9), we showcase memorization and benchmark evaluation across each values of k and h in our work.

---

We further welcome any questions or concerns that the reviewer may have.

We thank the reviewer and below is our response.

---

Weaknesses

1. Impact of goldfish-loss on downstream performance.

   • This is a good point. We directly discuss this in global rebuttal point 2 (Figure 2).
   • To reiterate, using goldfish loss is not a free lunch and see observe degradation in terms of validation loss (if not benchmark numbers at 1B scale). This infers that some loss of benchmark scores is expected (more pronounced at larger-scale training runs than our lab budget allows).
   • We add results to empirically validate the hypothesis that the validation loss gap is due to a lower number of supervised tokens i.e. tokens you compute loss on. We find that this gap can be alleviated by matching the number of supervised tokens to standard loss case (where input tokens == supervised tokens) either by (i) increasing batch size or (ii) training for more steps and yields near identical validation loss to standard loss model.

2. Model might generate paraphrased slightly paraphrased way...If this is really the case, the impact of this paper seems limited.

   • We address this in global rebuttal point 1 (Figure 1). Based on your feedback, Figure 1 (in rebuttal doc) now includes Rouge1, Rouge2, and BERTScore [2], which represent unigram overlap, bigram overlap, and embedding-based scores (a higher score suggests a closer semantic similarity to the ground truth), respectively.
   • As shown, while goldfish training prevents regeneration of exact training sequences, the increased BERT embedding-based BERTScore and small n-gram-based Rouge scores (in comparison to Control) suggest that the model does indeed sometimes use similar phrases or repeat the same factual content.
   • We think of this as a feature and not a bug: the model still retains and learns knowledge from the underlying data. If this were not the case, there would be no purpose in training on these documents.
   • This entails that goldfish loss is suitable to use when regenerating exact sequences from the training set is problematic (for copyright, private data, etc.) and not when generating paraphrased training data is problematic (although the utility of training on such datasets is limited). We note this in the discussion section of our manuscript.

---

Questions

1. Why are some of the experiments conducted on llama-2 and others on tinyllama?

   • We divide our testbed into two cases. 1) stress testing memorization (and its mitigation) in an extreme case (Section 4.1) and 2) a run emulating standard training (Section 4.2). In the former, we pretrain LLaMA-7B (the largest model we can fit, as larger models memorize more) on 100 wikidocs for 100 epochs. For standard training, we use TinyLLaMA-1B [1] (the largest model we can train for longer) to train 20B tokens where we oversample 2000 wikidocs into the normal RedPajama data mix, to emulate standard training with repeated documents that are easily memorized.

2. Hash function implementation

   • In practice, we project the integer sequence to be hashed into a finite field with a fixed, pseudorandom mapping to the unit interval, but any commonly used integer hashing scheme, for example through common avalanche strategies, can be used. We'd be more than happy to add additional details in the appendix. The exact implementation is available in supplemental material (line 32:97).

3. Do you also apply the goldfish loss on RedPajama?

   • Yes, the loss is applied uniformly throughout training on all tokens, simulating that at training time, the locations of repeated text fragments are unknown.

4. Can you explain more about why the goldfish loss works even when sampling with zero temperature (i.e., greedy decoding)? In such a case, the toy example from lines 104-113 will not work, as the token with the maximum probability remains the same. Is it the case that the generalization probability on those tokens is in fact quite small?

   • Yes, the value of q the Remark was chosen in a bit of a dramatic way to highlight the compounding effects of autoregressive sampling, even when q is large. For greedy decoding it is necessary that q is smaller, often enough. This is also where the simplicity of the toy model is too limiting. In reality, the values of q are certainly not uniform. There are tokens that are masked, but easily guessable by the model, and there are surprising tokens that determine the direction of the original sentence - and if masked break the generation of this sentence.
   • In practice, Figure 2 actually shows a real example of a sequence with greedy sampling where the excerpts generated with goldfish and standard loss differ. The argmax prediction at the key position ". [They ..]" differs from "very" which the model is conditioned on, but does not learn to predict.
   • A related finding that might shed some additional light is that we also attacked the model using beam-searches (Section 6.2, Figure 8) and did observe an increase in exact match rates for models trained on the target data. This implies that if, with sufficient guesses, the correct token is inserted, the model returns to the original path.

---

We thank the reviewer for their detailed and constructive criticism allowing us to enrich our manuscript. We further welcome any questions the reviewer may have.

We thank the reviewer for highlighting simplicity, efficiency, and performance as strengths. Below is our response:

---

Weaknesses

Results on benchmarks seem to show that while verbatim repetition is avoided, knowledge is still retained (Fig. 4). Is the output a paraphrase of learned text? My worry is that the strict "exact copy" metrics used to evaluate the loss might be too artificial, and might not give a complete idea of data leakage. While this deviates from the definition of memorization in 2.1, motivation in the introduction would push for analysis in this direction.

• We discuss this directly in our global rebuttal point 1.
• Furthermore, while an exact match is indeed a quite strict metric, the Rouge-L histograms in Figure 3, do provide a more nuanced picture (especially comparing the area from 0.4-0.6 between the control and the 3-GL model). In addition, we have now run additional tests and in rebuttal pdf Figure 1, we measure Rouge1, Rouge2, and BERTScore [1], which represent unigram overlap, bigram overlap, and BERT embedding-based scores (a higher score suggests a closer semantic similarity to the ground truth), respectively.
• Despite the goldfish model's deterrence to regenerate the exact sequences seen during training, in both Figure 3 and the Rebuttal Fig 1, the increased BERT embedding-based BERTScore and small n-gram-based Rouge scores (in comparison to Control) suggest that short phrases, vocabulary, and information are learned.
• This observation implies that while the model does not memorize, it still retains and learns knowledge from the underlying data. We argue that this is an inherent feature of using goldfish rather than a weakness. If a LLM is unable to even paraphrase its training documents, it means it did not learn.

---

Questions

Do you have any idea of the impact on the model’s generalization capabilities? Similar to the reasoning behind dropout, the proposed loss seems to avoid a form of overfitting and might refocus learning on higher-level representations.

• In terms of direct observations of generalization, we showcase validation curves in Figure 5 in Section 5.2 and observe lag behind standard loss during training. In the rebuttal pdf (Figure 2), we also see that this lag can be mitigated by matching the supervised tokens (i.e. tokens one optimizes on) by proportionally increasing batch size or training longer.
  We tentatively share the same hope that, in addition to mitigating memorization, this objective might prime the model towards more generalizable features, but could not focus on measuring that in this study (and at this model scale). The fact that when controlling for supervised tokens, the goldfish loss is ahead of standard training, may be a first tiny indication that there is something beneficial going on, but we would really need to run an entirely new controlled study on effects on generalization to say anything concrete.

• For completeness' sake a key feature in comparison to dropout is further that operates at feature level to combat overfitting, while goldfish operates at data (i.e. token) level to combat memorization. The goldfish setup allows the model to condition on all tokens for its forward pass (as opposed to dropout) and only mutates the gradient signal back propagated (by dropping tokens from loss computation).

• Further, a key feature of our approach is that the dropped tokens are pseudorandom based on local context, instead of random as features are in dropout, This is critical for memorization, as the model may still fit certain information, even with dropout (or any randomized regularization), if trained long enough on the same data.

• We also note an increase in the BERTScore [1] from the non-trained Control to the goldfish model, signifying the model's capacity to generate semantically consistent text and thus, its ability to generalize. This is further corroborated using a membership inference attack in Figure 7 employing loss and zlib [2], where the attacks sustain substantial performance, albeit with slightly reduced efficacy; indicating goldfish loss primarily discourages direct or extractable memorization.

---

Limitations

Motivations for goldfish loss

• We thoroughly resonate with the stance of the reviewer and strongly argue for rightful data ownership and equitable existence for data and model owners. The main motivation behind using goldfish is exactly what we measure i.e. mitigating verbatim reproduction of private data present in a large pre-training corpus at an unknown rate. We do not endorse usage or collection of data without proper attribution and credit to data owners. We have already stated this conversation in our conclusion, which we would be glad to extend, and we are happy to stress this stance further in the introduction of the (potential) camera-ready version as well.
• The goldfish loss should only be used in situations where training on a data source is permissible, but distributing verbatim copies is not. We would note that the issue of whether web data (and what kind) can be used for training purposes is still unresolved. In the US, there has been high-profile litigation (e.g., New York Times vs. OpenAI, and others), but these cases have yet to go to trial. Several GDPR complaints have been filed in the EU (and an Italian court issued a preliminary ruling) but these legal disputes are also ongoing. We have tried to avoid making specific claims about copyright issues in our paper as we are not attorneys, and any statements we make may be invalidated by upcoming court decisions, or by the ongoing process of implementing the EU AI act.

---

We again thank the reviewer for their time and stance on rightful data ownership. We further welcome any questions that the reviewer may have.