Fractal Patterns May Unravel the Intelligence in Next-Token Prediction

Thank you for your comments and the time you have taken to review our manuscript. Please see our response below.

Figure 1

Addressing your concern about the value of Figure 1, we would like to emphasize that the purpose of this figure is to visually demonstrate what a self-similar process looks like. We never used a figure to prove self-similarity. The correct and well-established approach in the (extensive) literature is to show that the auto-correlecation function decays algebraically (i.e. with a power law), giving rise to the Hǒlder exponent. This is precisely what we establish! The fact that a power law holds is shown in Figure 3 (top).

We elaborate on this in detail in Section 2, where we discuss the nature of self-similarity in language​​. The burstiness observed in Figure 1 is an intuitive and useful representation of self-similarity. Additionally, we provide extensive discussion on the significance and context of the Hǒlder exponent, particularly in Section 2.2, where its calculation and relevance to language are explained in detail​​.

PaLM-8B

Regarding your second point about the generalization of our findings from PaLM-8B to other models and language families, our claim is that we can obtain reasonable probability scores using LMs, such as PaLM-8B. There have been prior works that show this (we cite examples in the paper and we illustrate this calibration property in Figure 2). The reason we use PaLM-8B is merely because of the amount of compute required to conduct all the experiments.

However, we did acknowledge this concern in the paper and that is precisely why we expanded our analysis to PaLM-540B, as detailed in Section 2.5 in one task. This was done to verify the robustness of our findings, and the results affirm the presence of self-similarity and long-range dependence (LRD) in language, irrespective of the model size​​. We believe this approach underscores the general applicability of our findings across different model architectures and scales.

Connection to Zipf's Law

Your suggestion to explore the connection between fractal structures in language and the works of Zipf and Shannon is discussed in Section 4. As we mention there, Zipf’s law and Shannon’s method of estimating the entropy of English only use first order statistics so they cannot be used to answer questions about long-range dependence or self-similarity. If there is anything missing that needs to be clarified, please let us know and we would be happy to elaborate.

Drawing links between language and natural phenomena

We only mention that the Hurst parameter of 0.75 is quite common in natural phenomena and the fact that it also holds for language is interesting. Regarding the relation to learning processes, we have explored this in Section 3 in which we show a connection between scaling law parameters and fractal dimensions. For example, we find that when the Hǒlder exponent and the Hurst parameter are large in a particular domain, language models would benefit from having longer contexts during inference. We also studied the connection to the context length at training time but with no conclusive results, as we discussed in Section 3 using the Big Bench benchmark. There are many other similar questions of how fractal dimensions relate to learning and we believe we are only scratching the surface here.

Answers to the Questions

• LLMs have been used to create creative content. One example is asking GPT4 to write a proof that there are infinitely many primes in the form of a poem (https://arxiv.org/pdf/2303.12712.pdf ).
• We do provide a reference in the paper for that particular statement. It’s in Page 9 in https://arxiv.org/pdf/2303.12712.pdf.
• In the notation, there is $w_{t-1}$, which is a single token, and there is $w_{[t-1]}$, which is the entire sequence of length t-1 tokens. So, both statements in the question are incorrect. When we query the LM to predict the probability of token $w_t$, we condition on $w_{[t-1]}$ not $w_{t-1}$. We will make the distinction easier by using boldface for $\bf w_{[t-1]}$ .
• The lengths are random variables. Since the process is stationary, those lengths have definite expected values, but the value of course varies from clauses to sentences and sections. How the expected length changes will determine the level of self-similarity. These are standard definitions as we state in the paper and we provide references for them. We do not invent them.
• Thank you. We will fix the missing fields in those references.

If there are any other questions or concerns, we would appreciate it if you let us know so we can respond to them during the discussion and improve the manuscript.

Thank you

Dear reviewer,

We would like to kindly inquire if our responses have satisfactorily addressed your concerns and questions. We are open to further clarification if needed. If we have addressed your concerns, we would appreciate it if you consider revising your review/score accordingly. Otherwise, please let us know so we can respond to your inquiries during the discussion period.

Sincerely

Thank you for your valuable insights and the careful review of our manuscript. We are grateful for your recognition of the novelty of our approach as well as your positive comments on the clarity of our writing and the design of our experiments.

Addressing your first concern regarding the representation of language as a sequence of negative log probabilities (i.e. bits), we acknowledge that this approach may simplify the rich and complex nature of language. As you mention, it is unclear how the approach can be extended to take the identities of tokens into account. However, the approach can be interpreted as a lower bound to the complexity of natural language. In particular, long-range dependence on the sequence of bits implies the existence of long-range dependence on the sequence of tokens (by the data processing inequality).

Regarding the concern that our findings might not be perceived as surprising or novel, we appreciate this viewpoint. While the hierarchical and recursive nature of language is a subject of study in cognitive sciences and linguistics, our work aims to bridge these concepts with the computational perspective of language models, focusing on how these inherent properties of language influence model performance. This interdisciplinary approach, we hope, adds value to the existing body of research by providing a fresh perspective to understand LLMs.

We thank you for emphasizing on the distinction between “self-similarity” and "fractalness." It is not our intention to use these terms interchangeably, and we have acknowledged that they are different concepts in the paper. We will take care to avoid this in the revised version.

If there are any other questions or concerns, we would appreciate it if you let us know so we can respond to them during the discussion and improve the manuscript.

Thank you again for your careful and thorough critique.

Dear reviewer,

We would like to kindly inquire if our responses have satisfactorily addressed your concerns and questions. We are open to further clarification if needed. If we have addressed your concerns, we would appreciate it if you consider revising your review/score accordingly. Otherwise, please let us know so we can respond to your inquiries during the discussion period.

Sincerely

Thank you for your thoughtful and constructive feedback on our manuscript. We are grateful for your recognition of the novelty and technical soundness of our approach and its potential impact in advancing the conceptual understanding of large language models (LLMs).

Addressing your concern about the connection between fractal structure and the capabilities of LLMs, we acknowledge that our current work is an initial step in this direction. As you suggested, extending the analysis to other modalities and models would offer interesting insights, and finding further connections between fractal dimensions and learning could uncover other results that are useful for practitioners. We plan to explore these directions in the future.

In our work, we propose a precise mathematical formalism (using fractals and self-similarity) to quantify properties of language that may have been known intuitively but not formally shown, such as long-range dependence, and connected those to properties of learning. We hope this work lays the foundation for further explorations, highlighting the potential of fractal analysis as a lens for understanding and improving LLMs. We believe that additional research in this direction could provide substantial value to the field and enhance the applicability of our findings.

If there are any other questions or concerns, we would appreciate it if you let us know so we can respond to them during the discussion and improve the manuscript.

Thank you again for your constructive critique.

Dear reviewer,

We would like to kindly inquire if our responses have satisfactorily addressed your concerns and questions. We are open to further clarification if needed. If we have addressed your concerns, we would appreciate it if you consider revising your review/score accordingly. Otherwise, please let us know so we can respond to your inquiries during the discussion period.

Sincerely