The Role of Task Complexity in Emergent Abilities of Small Language Models

One concern here is that the results are purely empirical. The relationship between the number of parameters necessary to solve a task and the Kolmogorov complexity is rather weak based on what the results demonstrate, in particular as Figure 2 actually demonstrates quite a few tasks to clutter around the same number of parameters.

Additionally, the point that joint training improves task performance is not very surprising, as the model must learn to adapt to different settings rather than a single one and therefore the fact that the embeddings express some more meaningful embeddings is hardly a new discovery.

This leads to my largest concern which is that the main claim of the paper (that jointly training on different tasks of varying complexity and diversity leads to more robust models/representations) is something that has been explored rather frequently at nauseam, in particular with respect to language models [1, 2, 3]. This, combined with the fact that the authors only provide fairly incomplete justifications of how Kolmogorov complexity affects the ability to learn tasks leaves this rather lacking and I'm not convinced that the arguments made for such as specific setting such as ListOps is exactly relevant to general use cases of language models.

[1] Xie et al., Doremi: Optimizing data mixtures speeds up language model pretraining. NeurIPS 2023.

[2] Xie et al., Data Selection for Language Models via Importance Resampling. NeurIPS 2023.

[3] Miranda et al., Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Data. arXiv, 2023.

1. Similar approaches have been proposed by others, such as https://arxiv.org/pdf/2405.16684 -> also using gzip to characterize task complexity

2. Proposed task complexity measure only works for ListOps. It is not scalable or generalizable. For example, how do you define the "minimal python implementations" for a large python codebase with multiple files?

3. In Figure 1, right panel, you plotted a line fitting only 3 algorithms. 3 algorithms is too small for any kind of curve fitting that indicates any scaling laws.

4. You characterized the algorithms that the neural network could develop to solve MAX, MED, MIN, and SUM as only 2 categories: symbolic and numeric. However, there is no guarantee that the neural network could discover some new algorithms that does not fit in your categories, such as a mix of two. The way to figure out what the neural network actually learned is to find a circuit systematically, with the approach like this paper

5. Proposed task complexity measure has 2 parts "symbolic KC" and "numerical KC". The measure is not consistent, and also due to the points above, not generalizable to other tasks.

6. In section 3.2, first paragraph, you assumed "MAX+MIN+MED" has a higher task complexity than "SUM". However, there is no proof showing that the combination of these tasks has a higher task complexity.

7. In section 3.3, you only inspected the learned embeddings, then concluded that the learned "SUM" algorithm "has not developed a structured representation of numbers". However, you should inspect other parts of the model before concluding that. Similarly, there is no evidence concluding that the "SUM" algorithm memorizes the numeric table.

1. Although the connection between Kolmogorov complexity and the emergence point is novel, the underlying mechanism in the other part of the paper appears to lack significant novelty. In other words, it does not provide substantial additional information to our current understanding of large language models (LLMs). For instance, it is evident that the SUM-only operation will not have a numeric representation as the basis of this operation, regardless of how many times it is composed, is to map 100 classes to ten classes. It is considerably easier to memorize than to understand. Given that the model must first have an understanding of operation composition, the remaining task is to deterministically map 100 instances to ten classes. Therefore, it is indeed far easier to memorize than to understand. Regarding the max, min, and median operations, for example, if the length of the array is 3, it will map 1000 to 10 classes, making it easier to understand than to memorize.

2. Kolmogorov complexity is extremely difficult to compute for any practical tasks. Consequently, the practical implications of this scaling law are limited.

3. There appears to be a substantial amount of content in the appendix. However, the current version of the main body contains verbose sentences and large spaces. Some of the appendix content should be incorporated into the main content to make the experiment setting more explicit.

4. It is contended that the support for the claim "multitask learning accelerates the learning of more challenging tasks" is not adequately supported by the experiments. Since SUMUP is accelerated by MAX, MIN, and MED only because the latter tasks assist in learning a numeric representation. Multitask learning in this context is merely a confounding factor. To support this claim, experiments should be conducted that demonstrate something like: MAX only does not help MIN, MED, or SUMUP, but MAX and MIN together can help MED and SUMUP.

5. I suspect some sentences are written by LLMs, or are polished by LLMs. Although it is not a big issue, the author should make such sentences more natural. E.g., “As we push the boundaries of AI capabilities, understanding these nuanced relationships between task complexity...”

1. While the Kolmogorov complexity is extremely interesting to study, using compressed size of minimal python code is somewhat problematic as a proxy, particularly for simple arithmetic operations. I appreciate the authors listing out the limitations of this approach and discussing its precedent but the rules specified in Appendic C seem somewhat arbitrary and I expect the compressed sizes of the 3 algorithms to be fairly similar.
2. The accuracy definition which only tests the first character after the first = symbol seems fairly forgiving. Do the authors have a measure of what the "strict" accuracy would be? This is assuming that the model is expected to also output e - the end token.
3. I would appreciate if the authors edited the plots to make them easier to read/parse. The font sizes are far too small and the legends are fairly difficult to parse.
4. The fitted curve that results in the cubic scaling law (KC $\approx 18n_p^{0.34}$) seems to be based on too few datapoints to rely on.

Typos:

1. SUM vs $'$sum$'$. I recommend the authors pick a standard notation and stick to it.
2. Missing parenthesis after Appendix I on page 10.
3. [insert number]x on page 10.
4. Page 3 line 135: "representing the "$\rightarrow$ "representing them"