Thank you for the feedback. We believe there has been some misunderstanding in how our paper is being interpreted. Moreover, we think our positions are more aligned than they may initially appear.

The research question is somewhat artificial in the sense that the concept of readability (in humans) concerns the cognitive load of interpreting a text, which is not the same thing as learning a statistical language model from a text.

This argument is actually one of the main motivations for our paper. There is an assumption being made about the positive influence of simple, child-directed language (which we measure with readability) on LM training and we are correcting it. While it might appear obvious that readability should not influence LM training, developmentally plausible pre-training is an active area of research [9]. Moreover, the existence of LMs that can reduce tasks that were formerly thought to require human intelligence to next-token prediction and improve their performance simply by being instructed to "think step by step," makes it highly tempting to anthropomorphize these models. The findings of [1] can be easily misinterpretted in ways that encourage the anthropomorphization of LMs. Our objective is to prevent such misunderstandings. A more thorough discussion of why our findings matter can be found in our general response.

In particular since readability is usually defined in terms of features related to frequency and length of individual tokens, but the paper does not discuss the influence of tokenization on the learning abilities of language models.

While we agree that it could be interesting to analyze the influence of tokenization on LM training, we have already dedicated Section 3 to establishing a measure of readability that strongly correlates with human experts (LLM-as-a-Judge). Recall that we also found that this measure correlates better with human experts than classic readability formulas, which are based on word (token) length and sentence length (and some measures like Dale-Chall and Spache implicitly consider word frequency). Therefore, we did not further analyze tokenization given the objectives of our paper.

The paper also contains an experiment that shows that the concept of readability and the concept of text quality (as interpreted in terms of perplexity and coherence) are unrelated, which is exactly what you would expect given the definition of these concepts.

The experiment aimed to validate our measures of readability and text quality (PPL and coherence). A key requirement for these measures is that they remain uncorrelated (this is by definition, as you noted). However, since we instruct an LM to generate these scores, we cannot simply assume the outputs will exhibit these properties. This is why we devoted significant portions of our paper (Sections 3 and 4) to validating these measures before conducting our main experiments.

When measuring quality, you use a set of open models. Why not simply use a state of the art model such as GPT-4o or Claude3.5 instead?

We chose to use open models to ensure reproducibility, a key tenet of good scientific research. While GPT-4o or Claude3.5 offer advanced capabilities, they are accessed through APIs, which can change over time without notice. This lack of transparency can make it challenging to replicate results consistently.

Dear Reviewer,

As the rebuttal period is drawing to a close, I would appreciate your response to the authors' rebuttal at your earliest convenience.

Best Regards,

Area Chair

Thank you for your response. Could you elaborate on why the contribution is not considered substantial enough? Without this information, it is challenging for us to address your concerns.

Thank you for the constructive feedback. We address your points below.

W1 + Q2 We believe that the argument boils down to whether one believes that anthropomorphizing LMs is problematic and that misinterpreting the findings of [1] facilitates the anthropomorphization of LMs. We have included a more thorough discussion under "Why do our results matter?" in the general response, and are happy to discuss further.

W2.1 We actually agree with your recommendation and initially organized the paper that way, but changed the order based on early feedback on our draft. Readers expressed that transitioning from dataset construction to two sections on metric validation before returning to experiments using the datasets we constructed earlier felt disjointed. We will carefully consider both your feedback and that of early readers and explore alternative options to improve the flow of the next revision.

W2.2 In our revised draft, we restructured the paper such that figures and tables are better aligned to their contexts. Additionally, we added bidirectional links between the main text and appendix figures/tables for easier navigation.

W3.1 + Q1

• I should see that in (b), the green dots (SLMs trained on LlamaTales-Jr) are approximately as high as the best gray dots (LLMs), and in (c), the blue dots (SLMs trained on LlamaTales-GRE) are ALSO approximately as high as the best gray dots (LLMs).

  This is the correct interpretation. What we were trying to convey is that SLMs trained on either simple language (TinyStories, LlamaTales-Jr) or complex language (LlamaTales-GRE) generate coherent text comparable to much larger models, which we did not observe for SLMs trained on FineWeb (representative of data used to train real-world LLMs). The red dashed line shows the coherence of the training split of the data used for prompting, i.e., what is achievable if the SLM perfectly reproduces its training data.

  We have included additional textual explanations to help interpret Figure 2.

• Your suggestion to move the plots for TinyStories and FineWeb to the appendix makes sense, given that they are not core to our main claims, and we have implemented this change. We have also aligned the y-axes for Figures 2 and A1-9 to make comparisons easier.

• Figure 3 (perplexity vs. parameter count) has been moved to the appendix.

W3.2 We have moved most of the bottom section of Table 1 (statistics on $n$-grams, words, and syllables) to the appendix.

W3.3 We have included examples from both datasets in Table A5 of the appendix and ensured that a reference to them is included in the caption of Table 1 and Section 5, where we introduce the datasets.

Dear Reviewer,

As the rebuttal period is drawing to a close, I would appreciate your response to the authors' rebuttal at your earliest convenience.

Best Regards,

Area Chair

I raised my score because the authors addressed my concerns and also presented new results following suggestions from other reviewers, which I believe make the paper stronger.

Yes, the paper is narrowly scoped, reassessing the interpretation of a single paper. However, I agree that the TinyStories paper has been widely misinterpreted, and the misinterpretation is harmful, leading to misunderstandings of how LMs learn, misleading verbiage for the general public, and potentially more human-inspired model development that is "on the wrong track."

If the paper is not accepted at ICLR, I encourage the authors to try an *ACL venue, where I believe the audience would be more receptive.

Thank you for the constructive feedback. We address your three points below.

1. In our revised draft, we repeated our experiments with alternative measures of quality (clarity and fluency) and observed that the results are consistent with our original findings: SLMs are competitive with much larger LMs when trained on either the simple language of TinyStories/Llamatales-Jr or the complex language of LlamaTales-GRE.

2. We also repeated our experiments for the metrics used in [1] (grammar, consistency, and creativity) and again observed that, with the exception of creativity, the results are consistent with our original findings. We found that creativity was the only metric uncorrelated with all other quality measures, including our non-LLM measure (model ranking), suggesting it might not be a good measure of quality.

Experimental results for points 1 and 2 are shown in Figures A4-8, and the prompts are shown in Figures A16-20.

3. Our revisions demonstrates a correlative (though not necessarily causal) relationship between $n$-gram diversity (unique $n$-gram counts) and dataset learnability. For a more thorough discussion, please refer to "What explains our results?" in the general response.

Dear Reviewer,

As the rebuttal period is drawing to a close, I would appreciate your response to the authors' rebuttal at your earliest convenience.

Best Regards,

Area Chair

We also notice a different but related issue in Figures A8c and A8d, where the black dots represent prompt completions for general fiction stories by publicly available LMs. The creativity scores among these LMs are nearly indistinguishable (except for the smallest LMs, but even then, their scores are still high). We also see that creativity does not exhibit a dynamic range comparable to any other text quality measure. Below is an example generated by pythia-70m.

In the quaint town of Ravenswood, nestled between the rolling hills of the countryside, a sense of tranquility reigned supreme. The residents, with their warm smiles and gentle mannerisms, embodied the very essence of bonhomie – an urban legend – which felt deeply left by the neighbours’ general expectation of human nature.\n\n“A world of hand flowers, fair girls, gulls, checkered lichen shoes,” is professor of social history and the former yearning to eradicate the dinosaur remains.\n\nThe New World has been seen as a ‘feral character’ by the biologist Marc Orr and an expert on perseverance and habit throughout life – in the case of sorts, their significant figures, they often sink into the tough old quarrels and feats of humankind from parochial matron after farm-like periods. Now an obvious museum says that, before people can recreate the glory in dying civilisations (whose hallowed living-rooms make much of this movie – there is a pun – Cream of the Fairness trilogy – and are a wonder for themselves), they al

This pythia-70m generation received a creativity score of 80 and a coherence score of 10. In contrast, here is a generation by Llama-3.1-70B.

In the quaint town of Ravenswood, nestled between the rolling hills of the countryside, a sense of tranquility reigned supreme. The residents, with their warm smiles and gentle mannerisms, embodied the very essence of bonhomie – an unspoken camaraderie that bound them together. The chimneys of each house billowed with smoke, a testament to the cozy hearths that illuminated the otherwise cold evenings. As the golden autumn leaves cascaded down from the trees, the townsfolk reveled in the simple pleasures of life, united by their shared appreciation for the beauty of their surroundings. Little did they know, beneath this picturesque veneer, a sinister secret lurked, threatening to shatter their idyllic existence.\nI. The Whispering Winds of Ravenswood\nAs the sun dipped beneath the horizon, casting a scarlet hue across the sprawling landscape, the winds began to stir. At first, it was a gentle whisper, a soft caress against the cheeks of those who strolled along the cobbled streets. However, as the night deepened

This Llama-3.1-70B generation received scores of 85 for creativity and 95 for coherence. As before, we think it is reasonable to say that the pythia-70m generation is of lower quality.

In summary, we agree that creativity can be a useful dimension of text quality, but our findings suggest that it should not be considered independently of the other measures we have examined. Given that creativity is a highly subjective concept, even more so than text quality, it can be interpreted in such a way that even nonsensical text might be considered highly creative from certain points of view.

We thank the reviewer again for their constructive feedback and openness to discussing our work. We are happy to provide further clarification on this topic or any other concerns the reviewer may have.

Thanks the authors for the detailed responses and additional experiments! I believe most of my questions have been addressed. However, I am not fully convinced by the reasoning for the statement that creativity "might not be a good measure of quality". Specifically, creativity having no correlation with coherence and perplexity doesn't mean it is unimportant to quality -- it can be an orthogonal dimension that is still valid. Do authors have stronger evidence to support that statement?

Thank you for the constructive feedback. We address your three points below.

1. If we understand correctly, the reviewer has two questions: Why do our results matter? And what explains our results?

   To address the first question, we believe that the argument boils down to whether one believes that anthropomorphizing LMs is problematic and that misinterpreting the findings of [1] facilitates the anthropomorphization of LMs. We have included a more thorough discussion under "Why do our results matter?" in the general response.

   In response to the second question, our revised draft demonstrates a correlative (though not necessarily causal) relationship between $n$-gram diversity (unique $n$-gram counts) and dataset learnability. For a detailed explanation, please refer to "What explains our results?" in the general response.

2. We agree with your points regarding clarity and made the following changes in our revision:

   • We reorganized the paper to better align figures and tables with their relevant sections. Additionally, we added bidirectional links between the main text and appendix figures/tables for easier navigation.
   • In addition, we have added references to Tables 5-16 (now A6-20), specifically in Section 6.
   • We have added reference to Figure 16 (now A37) in the caption for Figure 4. In short, the figure is meant as a companion to Figure 4 to allow the reader to compare $n$-gram novelty across our datasets which is difficult to see in Figure 4.

3. We share your concern regarding potential bias when utilizing LLM-as-a-Judge, particularly in the choice of the LLM and prompts. We also want to emphasize that our goal is not to find the optimal prompt. We aim only to identify a prompt that is adequate for advancing our experiments. That said, we have taken several steps to validate our use of LLM-as-a-Judge:

   • Recognizing that LLM-as-a-Judge might be influenced by the choice of LM, we evaluated a wide range of LMs to see how their scores aligned with non-LLM assessments of readability and text quality. For readability, we tested 21 LMs against human evaluations, as detailed in the correlation matrix in Figure A33. We found several LMs with high human correlation, indicating that LLM-as-a-Judge is reliable for readability assessment, provided the LM is recent and has over 70B parameters. For coherence, we examined 6 LMs, all of which showed strong correlation with our model ranking (Figure A31).

   • We explored different prompt variations for assessing coherence and readability. One variant instructs the LM to generate an analysis before scoring, unlike our original prompt, which asks for an immediate score. Another variant provides examples of low and high-rated texts with explanations, guiding the LM before scoring. For coherence, all three variants showed a similar correlation with our model ranking and each other. For readability, while all variants correlated strongly with human experts, the ones prompting immediate scoring had the highest correlation. The prompts and correlation matrices are detailed in Figures A10-15 and Figures A35-36, respectively.

   • In response to reviewer wvMq's feedback, we introduced five new prompts for an LM to evaluate clarity, fluency, creativity, consistency, and grammaticality, detailed in Figures A16-20. All metrics, except for creativity, supported our original findings (Figure A9), showing that our results were not unique to the coherence prompt. Creativity is uncorrelated with our non-LM measure (model ranking), indicating it may not be a reliable text quality indicator.

   • We used three new prompts to generate three 1B token datasets: LlamaTales-{History, Sports, News}. Similar to LlamaTales-GRE, these new datasets exhibit substantially higher language complexity (as measured by readability) than the children's stories in LlamaTales-Jr. Yet, SLMs trained on these datasets are competitive with much larger models in terms of coherence. The specific prompts are shown in Figures A23-25. This demonstrates that our original findings are not unique to the prompt used to generate LlamaTales-GRE.

Thank you for providing further explanations and editions. Based on the paper and the comment made during the rebuttal, I can see that the result matters but is not significant. The authors presented n-gram diversity correlating with learnability to support their hypothesis, but they did not explain the underlying reasons for their observations. The results show that complex language data neither helps nor harms the coherence of SLMs. From the appendix, the model output is coherent with complex and simple language. The discussion of how much the new proposed research direction might be helpful to future works in this area is unclear. While this paper identifies interesting problems, the paper’s contribution to the broader community is limited. I decided to keep my original score.

What explains our results?

In our original draft, we observed that SLMs trained on datasets with either simple language (TinyStories, LlamaTales-Jr) or complex language (LlamaTales-GRE) could generate coherent text comparable to much larger models. However, this was not the case for SLMs trained on FineWeb, our representative of data used to train real-world LLMs.

We hypothesize that the primary difference between the datasets is their diversity. FineWeb, derived from web crawl data, encompasses a much broader range of language than our other datasets, which were generated by a LM using template-based prompts with minor variations. To quantify diversity, we propose measuring unique $n$-gram counts. We find that FineWeb contains significantly more unique $n$-grams than TinyStories and LlamaTales. However, this leaves us with only four data points to draw conclusions from. In our revised draft, we present additional correlative evidence suggesting that $n$-gram diversity explains our findings.

First, we generate three new synthetic datasets, each containing 1 billion tokens: Llamatales-{Sports, History, News}. These datasets are created using the same generation process as LlamaTales-GRE, but with the prompts shown in Figures A23-25 to produce documents outside the domain of short fiction stories. Similar to LlamaTales-GRE, all three datasets feature significantly more complex language than LlamaTales-Jr (as measured by readability, see Table A1), yet they produce coherent SLMs when trained on these datasets. We then examine two new examples of "real" training data for LMs (in addition to FineWeb): 1 billion token samples of Dolma and SlimPajama. Consistent with our experiments with FineWeb, SLMs trained on these datasets are incoherent. Examples from the new datasets are provided in Table A21, and the SLM coherency results are shown in Figure A9. We have results for 33M and 9M parameter SLMs; the other model sizes are still being trained and will be included in the next revision. We also present example generations by SLMs trained on the new LlamaTales datasets in Table A22.

An examination of the $n$-gram diversity across our nine datasets reveals that easy-to-learn datasets (TinyStories, LlamaTales series) have significantly lower $n$-gram diversity compared to hard-to-learn datasets (FineWeb, Dolma, SlimPajama) for small values of $n$ (see Figures A26 and A29). This observation is intuitive. Easy-to-learn datasets contain fewer statistical patterns than hard-to-learn datasets, and SLMs have limited capacity to encode these patterns. Thus, our evidence suggests that the child-directed language of TinyStories is easy to learn because the data contain fewer patterns (in the form of unique $n$-grams), rather than due to any human concept of language complexity (e.g., grammatical, lexical, syntactic, or conceptual complexity).

We will include this expanded discussion in our next revision.

We have included a summary of our new experiments, along with the corresponding tables and figures, to give reviewers a clear overview of our updates.

1. Five new datasets (3 synthetic: Llamatales-{History, Sports, News} and 2 human-authored: Dolma, SlimPajama)
   • Goal: Demonstrate more correlative evidence for why $n$-gram diversity, rather than child-directed language, may better explain our findings.
   • Table A1: Coherence and readability of training splits (measured with LLM-as-a-Judge)
     • Takeaway: Similar to LlamaTales-GRE, our new synthetic datasets score high on coherence and low on readability (relative to TinyStories and Llamatales-Jr).
   • Figure A9: Coherence of SLMs trained on new datasets versus public models.
     • Takeaway: Consistent with our original findings, SLMs trained on our new synthetic datasets are competitive with much larger models. This is not true for SLMs trained on our new human-authored datasets.
   • Figures A26-28: Histograms of $n$-gram diversity (unique $n$-gram counts for various choices of $n$) across our datasets.
     • Takeaway: Easy-to-learn data (TinyStories, LlamaTales) exhibit far less $n$-gram diversity than hard-to-learn data (FineWeb, SlimPajama, Dolma).
   • Figure A29: Unique 3-grams (training data) versus learnability (coherence of text generated by a 33M parameter SLM divided by the coherence of the training data used for that SLM).
     • Takeaway: Low $n$-gram diversity strongly correlates with high learnability.
   • Figures A23-25: Prompts used to generate our synthetic data.
   • Table A21: Random examples from our new datasets.
   • Table A22: Random generations by SLMs trained on our new synthetic data.
2. Two new prompt variants for LLM-as-a-Judge
   • Goal: Address reviewer r6Wo's concern that

     The prompt used for LLM for evaluation and data generation is consistent but I am concerned if such a prompt would lead to bias. There’s no discussion of other prompts being used in the experiment and no comparison of the result with other prompting methods.

   • Figures A12-15: Two new prompt variants: (a) instructing the LLM to generate an analysis before assigning a score, and (b) providing the LLM with positive/negative examples in its instructions. We apply both variants to measure coherence and readability.
   • Figures A35-36: Correlation matrices of the prompt variants.
     • Takeaway: When measuring coherence, we observed no differences among the variants; all were strongly correlated with our model ranking (Section 4.2). When measuring readability, all variants correlated strongly with human experts, but generating an analysis before assigning a score was the weakest of the three.
3. Five new metrics: fluency, clarity, consistency, grammar, and creativity

   • Goal: address reviewer wvMq's concerns:

     The ignorance of other dimensions of quality (for example, as authors also mentioned, clarity and fluency) makes any statements about "generation abilities of SLMs" an overclaim.

     The quality measurement doesn't use any metrics from the original TinyStories paper: grammar, creativity, consistency with the beginning of the story.

   • Figures A4-A8: Results of evaluating SLMs (trained from scratch) and public LMs with the new metrics.

     • Takeaway: All new metrics (except for creativity) strongly correlate with our original metric: coherence. This supports our original finding that SLMs trained on complex language (instead of the simple language of TinyStories and LlamaTales-Jr) can still be competitive with much larger LMs. Creativity did not correlate with any of our quality measures and often ranked generations by toy models such as GPT2-small and Pythia-70M as being highly creative, suggesting this is not a good measure of text quality (Figure A8).

   • Figures A16-20: Prompts for our new metrics.