Emergent properties with repeated examples

We thank the reviewer for their time and comments and your appreciation of the clarity of presentation and the training paradigm.

Weaknesses: This paper is making very big claims (e.g. of "emergent properties" or "emergent learning") based on very synthetic settings. [...]

As noted in the beginning of our response to reviewer knen, we believe that we have aimed not to make general claims: all the claims on emergence or learning with repetition are associated to specific tasks. Please also kindly see our General Response and the discussion on "emergence" with reviewer yjc8, which might help to ground the discussion. This said, we believe our observations on modular multiplication are a clear case of emergence: without repetition, nothing is learned. With repetition, the model learns to performn the task. The GCD experiment indicates that this is not limited to one task. We make no claim about the applicability to very different fields (e.g. vision) or architectures (MLP or CNN); we are focusing on transformer architectures as they are used on contemporary NLP. On the other hand, the tasks considered are typical of the reasoning tasks which are a major field of current research on LLM. As we argue in our response to reviewer uAYC: Recent work on compute-optimal language models [Hoffman et al. 22] shows that many previously trained large language models could have attained better performance for a given compute budget by training a smaller model on more data. Most prior large language models have been trained for a single epoch [Komatsuzaki et al.'19 "One epoch is all you need"] and some work explicitly advocates against reusing data [Hernandez et al. 22]. [Muennighoff et al. NeurIPS 2023 Best Paper] undertake an extensive study of multi-epoch training for LLMs on natural data. They find that, while models trained for a single epoch consistently have the best validation loss per compute, differences tend to be insignificant among models trained for up to 4 epochs and do not lead to differences in downstream task performance (but surely not to any improvements). There is thus a deep-rooted belief in the transformer community that single-epoch training yields the best performance, and only when data is constrained some studies show that for a limited number of epochs (up to 4) can still yield some benefits (though not comparable to training on more data). Multi-epoch training is viewed as a poor proxy in attempts to attain the performance of single-epoch training if data was abundant. Repetition of training sets for transformers is viewed as a bug, not a feature, only to be employed when data is scarce.

In controlled transformer experiments we challenge the single-epoch paradigm: not only is repeated training on smaller data budgets powerful competition to single epoch-training (for the same number of training steps); in several cases repeated training on a smaller set allows to unlock capacities that are unattainable with single epoch training on a much larger dataset. In some cases, increased repetition of a smaller set leads to emergent phenomena of learning.

There was the classical decomposition of test accuracy to training accuracy + generalization gap. [...] Many of the observations might be more intuitively explained by undertraining [....]

We respectfully disagree. See Section 3 and especially Figure 1: models trained on very large datasets do not learn, even after more than a billion examples. Models trained on small datasets learn better, for the same training/compute budget. If undertraining was the problem it would affect small and large training sets. The generalization bounds you refer to also tend to improve with training data set size (though as we know, they tend to be vacuous for DL models and transformers anyway). Please let us know if we might have misunderstood your question.

We thank the reviewer for their time and effort and for acknowledging the importance and thoroughness of our work.

Weaknesses: From my point of view, this is not a particularly new idea. [...]

Our observation are made on reasoning tasks, like mathematics, that are solved with transformers. Allow us to provide some more context here: Recent work on compute-optimal language models [Hoffman et al. 22] shows that many previously trained large language models could have attained better performance for a given compute budget by training a smaller model on more data. Most prior large language models have been trained for a single epoch [Komatsuzaki et al.'19 "One epoch is all you need"] and some work explicitly advocates against reusing data [Hernandez et al. 22]. [Muennighoff et al. NeurIPS 2023 Best Paper] undertake an extensive study of multi-epoch training for LLMs on natural data. They find that, while models trained for a single epoch consistently have the best validation loss per compute, differences tend to be insignificant among models trained for up to 4 epochs and do not lead to differences in downstream task performance (but surely not to any improvements). There is thus a deep-rooted belief in the transformer community that single-epoch training yields the best performance, and only when data is constrained some studies show that for a limited number of epochs (up to 4) can still yield some benefits (though not comparable to training on more data). Multi-epoch training is viewed as a poor proxy in attempts to attain the performance of single-epoch training if data was abundant. Repetition of training sets for transformers is viewed as a bug, not a feature, only to be employed when data is scarce.

In controlled transformer experiments we challenge the single-epoch paradigm: not only is repeated training on smaller data budgets powerful competition to single epoch-training (for the same number of training steps); in several cases repeated training on a smaller set allows to unlock capacities that are unattainable with single epoch training on a much larger dataset. In some cases, increased repetition of a smaller set leads to emergent phenomena of learning.

We can contemplate how our observations carry over to LLMs trained on natural language data (NLP), and how they translate to actionable insights. While they seem at odds with the current practice of seeing training data only once, they might indicate that under the hood duplication in the training corpora mimics our two-set approach. If this is the case, intentional scrutiny of the training corpus to identify how to deliberately enact our observations could be beneficial for learning efficiency. And fine-tuning corpora, are often curated and feature less repetition. We believe two-set training, and associated methods, may directly prove beneficial for fine-tuning LLMs. However, the focus of our work is to study these phenomena in a controlled setting, not directly in the wild. Please kindly see our General Response on this as well.

In terms of related work, it may be interesting to investigate the connection to continual learning and catastrophic forgetting. This is more geared towards learning new tasks, and not about revisiting old examples from the same distribution. However, some of the work done here may provide valuable insight and would be nice to have some discussion comparing the methods in this area with yours.

There is a deep difference between continual learning and our approach. Continual learning focuses on distribution shift. Our setting, on the other hand, always remains in distribution. This said, our experiments in Section 6 and Appendix D.1 suggest that using selected examples in the repeated set, in a manner of curriculum learning, does not improve model performance. Please see also our General Response.

Moreover, your idea of two-set learning sounds similar to spaced repetition in human learning -- being shown old instances right before you are about to forget them. There is not much work in this direction for ML training applications, although I did find this paper [1] which seems to have similar ideas to yours, proposing an order and frequency to the examples being trained on.

[1] Repeat before Forgetting: Spaced Repetition for Efficient and Effective Training of Neural Networks, Hadi Amiri, Timothy A. Miller, Guergana Savova.

As in the General Response, we would like to point out that the work [1], and a large body of interesting works on catastrophic forgetting deal with the scenario of distribution shift (or multiple consecutive tasks in the training set) - which is not the scenario of our work. Indeed, our observed phenomena are all the more baffling as they appear without any distribution shift.

We thank the reviewer for their extensive and thoughful comments, and in particular the strong and concise summary of the strengths of our paper.

I believe the main limitation of this work lies in the limited number of tasks it considers, undermining the generality of its conclusions. In particular, the authors repeatedly make statements such as ‘models trained on smaller sets of repeated examples outperform models trained on larger sets of single-use examples’ (abstract); ‘smaller data budgets and more frequent repetition allow for faster learning, but also for much better performance’ (p. 5); ‘learning emerges through repetition’ (p. 5) etc. Based on the presented results however, it appears that these claims should be slightly tempered: there is a tradeoff between the repetition and the diversity that is necessary to avoid overfitting, as clearly illustrated in Fig. 2. While the authors show that repetitions at a given data budget may be beneficial, this relation is strongly non-monotonous, and in the greatest common divisor task one notably still requires a large number of unique samples to perform correctly (and as apparent in Fig. 4 the balance between repetition and diversity is hard to strike in this problem!). Clarifying this point and emphasizing that it is a tradeoff would make the paper more convincing, and would add to the relevance of the two-set procedure which precisely tries to provide the ideal tradeoff.

Please see our comments in the General Response that address the generality of our conclusion. There we provide arguments and ample precedence on doing experiments in a controlled (and thus necessarily limited) setting.

As for the formulation of the claims in our paper, we were aiming to present the facts and not to overclaim. As such, we had hoped that our writing was measured and truthful:

• In the abstract, the sentence you quote begins with "on three problems of mathematics (...) we show..."
• On page 5, these sentence appear while summarizing the result of a specific task, we make no claim about their generality.
• In the discussion, we devote a paragraph to explain that extending our findings to LLMs is the subject of further study

Please let us know which claims you are uncomfortable with and we will do our best to present them in a measured way with appropriate qualifications.

It also remains unclear whether the two-set procedure is relevant in settings where the transformer must not necessarily learn a clear-cut algorithm. For instance, could such a procedure be effective in image classification and more generally vision-related tasks? It seems plausible that the need for repetition is related to the hardness of the task at hand, which is not discussed in the paper. In that respect, the difference in hardness of the three tasks exposed is not straightforward to understand: the authors notably state that the computation of eigenvalues is the hardest task they consider, yet the smallest model (in number of parameters) is used to tackle it. Therefore, it is not evident how the eigenvalue problem contributes to the clarity of the paper and its conclusion, also considering the relatively poor results that the authors find (4/30 as the maximum trained model success rate if I understand correctly) and the lack of clear plots dedicated to this third problem, despite it being expected to be the most difficult to tackle.

This points to a curious fact about math transformers: the tasks they struggle to learn are not necessarily the hardest from a mathematical point of view. For instance, Charton (Linear Algebra with Transformers, in our references), shows that even small transformers have no difficulty learning to compute eigenvalues or vectors for real symmetric matrices of dimension up to 8x8. On the other hand, modular multiplication, a much easier mathematical task, cannot be learned without repeated samples, therefore proving hard for transformers.

We agree that the eigenvalue experiments are less spectacular: repetition only brings a small improvement in performance. We believe this is because this task is, in fact, easier for transformers to learn.

Still, we decided to add it to the paper, because it is very different from the two others:

• it features real/floating point numbers instead of integers
• it is a more complicated mathematical task, featuring an approximate algorithm
• it uses noisy data (due to the rounding of real numbers into floats)

We chose to inlcude these experiments, because they suggest our conclusions extend beyond arithmetic on integers. We will clarify this in the final version.

(1/3)

which claims you are uncomfortable with I realize that my phrasing was perhaps a bit unclear since I combined two different remarks in a single comment. I agree with the authors on their general response regarding the ‘issue’ of generality. Rather, and this was the second point of my original comment, I did get the feeling that their conclusions were, where I quoted them, perhaps a bit too schematic and simplified, and that they might not sufficiently emphasize the fact that there is always a tradeoff between diversity and repetitions. To be specific, I did not get the feeling that the phrase ‘smaller data budgets and more frequent repetition allow for faster learning’ was sufficient to summarize the situation that they present: this is only true when there is already a very significant number of unique examples and overfitting is an issue that they discuss. As mentioned in my original comment, I believe that Fig. 4 for example is particularly interesting in that aspect, as the balance between repetition and diversity is evidently hard to find on some problems. Note that I find this subtle balance most interesting, which is also why I believe it is detrimental to not highlight it more clearly in the abstract, introduction and summaries.

inlcude these experiments...they suggest our conclusions extend beyond arithmetic on integers I thank the authors for the clarification, and encourage them to highlight this curious fact about math transformers in addition to stating their motivation for the eigenvalue experiments more clearly.

(2/3)

However, we are happy to include what you might consider relevant I thank the reviewers for their complete answer. I believe that mentioning several possible explanations, including this one, and how likely they believe them to be could only improve their work. In my opinion including the complete discussion on this point in the appendix would be a welcome addition.

We have not extended our analysis to vision tasks... I definitely think that including something like ‘We believe our findings apply to reasoning tasks, with deterministic answers’ explicitly in the paper would be beneficial. Also mentioning the fact that transformers learn the GCD task in such a way could be helpful to provide further context for readers unfamiliar with these type of mathematical tasks in transformers.

Please let us know if this answers your question. I thank the authors for their answer, which I believe clearly addresses the question. Do they have an understanding of why these two approaches cannot be combined and how they may coexist? Is there any chance that the transformer implementation ends up being significantly different between the two training strategies, which could explain why they are mutually exclusive? Overall I encourage them to suggest some scenarios if they have any, even without evidence, as I think these observations are most interesting but have a hard time understanding how they may emerge. I do appreciate the authors explicitly stating the counter-intuitive nature of this observation.

(3/3)

we are happy to include the above experiments in the paper I thank the authors for running this experiment given the limited timeframe of the rebuttal period. I think it could be interesting to include it in the appendix where the Dandi et al. scenario is discussed, at least stating that this was attempted.

We believe this is characteristic of deterministic tasks. I thank the author for their reply. Again I think that mentioning this interpretation (even e.g. in a footnote) could be valuable, as I had not seen this behavior as clearly before.

We do not have a substantiated running hypothesis on how this happens... I am still confused on this point. To me, the results of Sections 4 and 5 do not necessarily require the transformer to act differently at the level of individual inputs, which would be the case if the transformer ‘cared’ about deja vu vs jamais vu if I understand correctly. To make this statement, I feel like further investigation on input-specific properties of the model would be required. I understand that ‘we learn in Section 4 is that there is a qualitative difference between model trained on repeated examples (even with low repetition) and models trained on unique (or almost unique) examples’, but I still don’t see why this could not be due only to some difference emerging in the learning dynamics and not in the transformer representation per se.

All in all, I thank the authors for their very complete response and for performing the experiment I had suggested — I also apologize for this quite late response on my side. I look forward to their revised version implementing the feedback from all reviewers and I believe that I should be able to raise my score appropriately.

We thank the reviewer for taking time to read our paper, for their valuable comments and the overall positive reception of our work. We would like to address your concerns below.

Weaknesses:

1. My major concern is that the experiments are only conducted on algorithmically generated synthetic data, instead of real world datasets. This makes me not fully convinced that the experimental findings are universal and can be transferred to more realistic setups. Specifically, I am wondering whether the main findings, especially the success of two set training, also applies to real language datasets. Do you have empirical results to support that? Besides, the three dataset are all math problems. Do you think that this specific type of training data might have a positive influence on the performance of repeated data?

Please see our General Response that addresses your concern - thank you!

2. Does the emergent property in the title mean that the observed phenomena only happens for training at scale? Do you have found any criteria that can indicate under what circumstance can repeated data or two-set training be beneficial?

Thank you for pointing out that "emergence" is an overloaded term. In this paper, we define emergence as the fact that a property is only learned in the particular case, i.e. with repeated examples. We do not discuss the impact of scaling (on the GCD, Charton 2024 mentions that scaling has a very limited impact on model performance). But we agree that there is another conotation of "emergence" in the community that refers to phenomena appearing only at scale. We will disambiguate this in the next version of our paper; thanks for pointing this out.

We hope we have adressed your questions to your satisfaction.

2/2

Relation to Continual Learning/Catastrophic Forgetting: Two reviewers have also asked about the connection to continual learning. We would like to point out that continual learning refers to a setting with distribution shift : the learner is supposed to adjust to new distributions in the input - and to avoid catastrophic forgetting of earlier tasks. This is an extremely interesting research area - but it is not what we study. In our setting the distribution is always the same, both for repeated examples (Section 4) and for the new phenomenon of two-set training (Section 5). Both the more frequently sampled and the less frequent larger set come from exactly the same underlying distribution - which is what makes this so baffling to us. We hope this clarifies why we have not drawn more parallels to continual learning/catastrophic forgetting.

Curriculum Learning: Several reviewers brought up curriculum learning, where training data is presented in a particular order (usually from easy to hard). Our work is different from curriculum learning: We show that randomly selecting a small subset of the training data, and repeating them more often can significantly enhance performance or even overcome learning bottle necks. We discover a synergistic effect: neither training on the small set alone, nor training with unlimited data budget in one epoch would allow any learning at all - it is the combination of both that makes two-set training powerful! The fact that the repeated set can be chosen at random, and that curating repeated examples brings no improvement in performance sets it aside from curriculum learning and suggest that what matters, here, is seeing the exact same example several times.

Additional Experiments In response to a request by reviewer knen, we have performed additional experiments for a different curriculum learning variant in connection with two-set sampling: we combine (some form of) curriculum learning with two-set training to show that training on the small, frequent set first before switching to the larger set does not provide any benefits (at best) and slows down learning (at worst).