Regularity explains emergence

• Response to Q1 : We prove that the optimal policy for the model is to give up difficult tasks. So it is "Prediction vs No prediction" instead of "Good prediction vs Bad prediction". Another way to put it is that the universal approximation works cited assume bounded Sobolev norms; the proposal of this paper is that the model cuts off unbounded Sobolev norms at a suitable threshold depending on model capacity to become bounded.

• Response to Q2: We believe for optimization of mathematical tasks, it should be true for well-trained embeddings in a quasi-isometric sense that $1$ is closer to $2$ than to $9$. We will verify it in future versions by inspecting the embedding.

• Response to Q3: We will pay attention to more detailed experiment results in future versions.

• Response to Q4: The overline stands for decimal concatenation. By $\overline{456}$ we mean the decimal integer 456.

• Response to Q5: Chain Rule

• Response to Q6: 1D is enough. We used 2D to better demonstrate the phenomenon that the prediction is at the origin when the model doesn't have enough capacity to predict.

Many thanks for your careful reading and detailed comments. Your opinions are greatly helpful for us to better approach the problem. Thank you for appreciating the point of using a ResNet setup to demonstrate the phenomenon before generalizing to the setting of an LLM. We agree with you that a general justification of the theory would be a high burden, and will work on it, especially in the natural language domain.

We also thank you for finding typos and grammatical errors. We will correct them in future versions.

Response to Q1 : Thank you for this question which hits right to the point. It is theoretically possible to make this estimate by going through the proof of Theorem 2.5 and the main ingredient of the resulting estimate will be the tail shape of the distribution of derivatives at all input-output pairs in the training dataset of the LLM. The fact that this distribution is not publicly available and expensive to compute, even if the training data is fully accessible, makes it hard to make a precise estimate. We believe that the fact that accuracy saturates is explained by the fact that when difficulty (regularity) of task increases, that is, decreases, (1) the amount of available training data at that level of difficulty dramatically decreases, (2) the scaling law requires N to scale polynomially to adapt. Combining the two means a moderate drop in will require a significant upscaling of to compensate, i.e. the scaling law ends and one needs other strategies such as CoT as alternatives. We will spend time to work on a new version and are not uploading a new version this time, but can confirm that a plot as what you suggested does show such saturation.

Response to Q2 : We hypothesize that beyond a regularity threshold, all models give up and provide random answers, resulting in similar but noisy error levels, which would be consistent with our interpretation. The standard error of error are roughly proportional to mean error and displays the same saturation pattern. For example, in experiment from 3.4, for the digit k=3, the standard error is almost constant, very close to 2 across all models and all number of steps >= 3, implying the the models are returning the randomly distributed responses beyond that regularity level.

Response to Q3 : The reason of using summation of single digits is that the regular digit addition is easier from our regularity view point, in the sense that a digit in the answer likely only depends on adjacent digits from the inputs, while the summation of multiple single digit numbes has multiple dependencies. For the two examples you mentioned, the first one is a very restrictive subset of summation of multiple single digit numbers and below the bar to see interesting emergence. For the second one, the carry matters only slightly and regularity still grows fast with number of digits and we still see the U-shape.

Response to Q4 : Thank you for suggesting the contrast, the "linear trend" type represents insufficiency of training data and the "breakthrough" type represents high regularity of optimal response and is closer to our framework. This aligns with our hypothesis in the following way: Problems that need logical reasoning/sequential steps can be viewed as a composition of problems. The optimal function is thus a composition of a sequence of optimal functions. By chain rule, the derivative of a composition of functions is equal to the product of individual functions' derivatives. Thus, the magnitude of its derivative is much bigger than that of a single function. Knowledge-based questions are either an individual function or a sum of these functions. Therefore, its derivative is of the same magnitude as that of an individual function.

Thank you very much for the insightful comments.

-Response to "I would like to see a much more rigorous development of the main theorem with specific definition and analysis of the derivative for token valued functions which are the main object of study for LLMs."

We appreciate this question and wish to hear more of your insight. Our assumption is that an appropriate tokenization is an optimally smoothified embedding of a discrete natural dictionary, in the sense that words with similar meaning are closer to each other. And there are many dimensions to represent similarity in different aspects. In fact, the subsequent layers of an LLM are smooth neural networks that treat the tokenized inputs continuously. The success of these models itself implies that the smoothness of the tokenization. Please share your opinion on this view.

-Response to "the numerical experiments in the paper are very limited"

The phenomenon of the numerical experiments in the Resnet sheds light on the fundamental philosophy of the issue that we are discussing. While we agree it is a toy experiment, we would like to include it to provide the first evidences. We plan to design more experiments in an LLM setting.

-Response to "The language model experiment is limited"

We agree that the experiments only covers a very special type of tasks and are short of showing the full landscape of the function $f$. The main obstruction is, indeed as you ask in the next question, the difficulty in systematically evaluate the derivative of $f$.

-Response to "How do the authors define the derivative over token valued neural networks?" As mentioned above, we think token valued inputs should be viewed as continuously valued once they are embedded as vectors in the sense that nearby vectors have similar meanings. But we are open to hear different opinions on this issue.

-Response to "Can the authors systematically evaluate the derivative and inferred the smoothed input-output function on a more general class of language models?"

We acknowledge that this is hard and in fact it is what limited us to arithmetic tasks whose derivatives are more explicit. We can think of evaluation approaches on some other task classes but they are still partial and not systematic enough. Any advice or references will be appreciated.

-Response to "can the authors analyze models of increasing size showing convergence to their central claim with model size?"

We are not sure if we understand this question but assume that you are asking about making the convergence more effective, or the relations between $\epsilon(N)$ and $N$. The short answer is yes on the theoretical level. But the longer answer is that the quantitative estimates requires knowledge about the tail distribution of derivative norms of training data used by the LLM, which is not accessible.

Thank you very much for the insightful comments.

-Response to ".. highlight exactly what constitutes the list of ``quantitative/concrete" predictions proposed by the theory."

The way to impose threshold of emergence is when the slope of accuracy curve is changing. There are significant slope change in the plot. In a future version, we will add a description to address this point.

-Response to ".. The toy model experiments using ResNet don’t closely match the large language models (LLMs) setup. "

We are optimistic that our interpretation is universal but open to other possibilities, and we accept the fact that ResNet is very different from LLMs. We would prefer to think this is rather a limitation in our experiments rather than of the theory, but would love to learn about why a priori LLMs and ResNet would require intrinsically different mechanisms for emergence.

-Response to "Choosing tasks more closely aligned with the theory would make the paper’s ideas clearer and more applicable."

We chose arithmetic tasks as it is mathematically possible to compute derivative of the target function for them. For a real-world question, for instance, "What is Shakespeare's most famous masterpiece?" it would be very difficult to analyze or even estimate the size of derivative of the target function. With tokenization and embedding, the function may be distorted in a very complicated way, and the solution's implicit relationship to the variation of the input is hard to measure. We would appreciate insights on this obstruction to the implementation of more profound experiments.

-Response to "Can the theory predict specific model sizes or conditions where emergent behavior happens?"

This is a very relevant question and we appreciate it. We believe this question is closely related to the next question you ask, as well as the relation between $N$ and $\epsilon(N)$ that reviewer 4byG asks about. It is theoretically possible to make this estimate by going through the proof of Theorem 2.5 and the main ingredient of the resulting estimate will be the tail shape of the distribution of derivatives at all input-output pairs in the training dataset of the LLM. The fact that this distribution is not publicly available and expensive to compute even if the training data is fully accessible makes it hard to make a precise estimate.

-Response to "Could you design an experiment with an autoregressive transformer model that would produce results more relevant to the theory?"

As we mentioned above, the main difficulty lies in determining the derivative norms of the ground truth target function, which made us stick to ResNet and arithmetic tasks in current experiments. We would appreciate any advice on the design of that part of the experiment.

-Response to "Can the theory’s error bounds predict error rates for specific tasks at different model sizes?"

In a perfect world where the tail distribution of ground truth derivatives is known, the proof of the main theorem would allow to predict the relation between $N$ and $\epsilon(N)$, namely at what difficulty level the model starts to give up tasks, which would cover part of the error rate. The remaining error rate, i.e. the chance that the model does make an effort but fails, is separate and can be estimated by the Siegel-Xu bound more directly.}