We appreciate the reviewer's insightful questions and valuable critiques. To summarize, we have addressed concerns about a toy theoretical model, and typographical errors. We trust that our responses below will provide a more comprehensive understanding of our research and its implications.

---

I think the theoretical section could be improved by integrating the proofs, when given the extra page if accepted. Also I wouldn’t call the theoretical insights „theorems“ but rather propositions.

We agree with the reviewer's sentiment, and will rename the theorems to propositions instead, and will integrate the proofs given the extra page if accepted.

I think the paper would greatly benefit from a toy model which allows to study when transcendence is possible, theoretically and practically.

We agree that the paper can benefit from a toy model for studying transcendence. Following your suggestion, we introduce the following toy model, demonstrating transcendence in the case of Gaussian data and linearly separable classes. Specifically, our input data is $d$-dimensional Gaussian, and the target is classification with $n$ classes. The ground-truth is generated by a linear function, i.e. $y = \arg \max_i W_i^\star x$ for some $W^* \in \mathbb{R}^{n \times d}$. We sample $k$ experts to label the data, where the labels of each expert are generated by some $W \in \mathbb{R}^{n \times d}$ s.t. $W = W^* + \xi$, where $\xi_{i,j} \sim \mathcal{N}(0, \sigma^2)$, for some standard deviation $\sigma$. Namely, each expert labels the data with a noisy version of the ground truth separator, with noise std $\sigma$. We then train a linear model on a dataset with $m$ examples, where each example is labeled by a random expert. We observe the test accuracy, measured by the probability assigned to the correct class, for different choices of temperature, and compare to the best expert. As can be seen in our synthetic experiments (in the attached rebuttal PDF), we are able to observe transcendence in exactly the same situation as in our chess setting, i.e. - when the diversity of the experts is high (high std) and the temperature is low. This also aligns with our theoretical analysis.

I think the major weakness of the paper is the missing understanding when transcendence can occur and when it can’t.

We would like to clarify that our paper provides a thorough analysis of the conditions necessary for transcendence both theoretically and empirically. Specifically, we detail in Section 3 the theoretical conditions for transcendence, emphasizing the necessity of low-temperature sampling (Section 3.1) and dataset diversity (Section 3.4). Our empirical results in Section 4 further validate these theoretical insights.

• Theoretical Analysis (Section 3): We analyze the necessity and sufficiency of low-temperature sampling and demonstrate that without it, transcendence cannot be achieved (Theorem 1). We also explore conditions involving multiple experts and the role of data diversity.
• Empirical Validation (Section 4.2): Our experiments confirm that low-temperature sampling enables transcendence to occur on high-diversity datasets by denoising errors (Figure 2 and Figure 4). We show that models trained on less diverse datasets, such as ChessFormer 1500, fail to transcend, highlighting the importance of dataset diversity (Figure 5).
• Toy Model: We note that our new toy model (see Rebuttal Summary) also cleanly demonstrates that transcendence is possible only in cases where the expert diversity is high, and the sampling temperature is low.

See weaknesses

• Typo in line 100 we make assume
• Typo in line 123 outputs a correct but noisy prediction(s).
• Reference in line 201 - reference to the current section
• Typo in line 237: to the right?
• Figure 4, improve legend and generally readability/visualizations of the Figure, E[F] two times, not clear what the difference is
• Typo in line 249 This is matches
• Typo in line 265 We, we
• Reference Theorem line 142

We apologize for the typographical errors and unclear references you noted. We have corrected the following issues in our local version and will upload the revisions in the final version of our paper if it is accepted.

In response to the Figure 4 concerns, we would like to clarify that it presents the expected reward distributions at two different temperatures, specifically $\tau=0.75$ and $\tau=0.001$. The two instances of E[F] represent the expected reward under these two temperature settings. We will revise the figure legend and accompanying text to make this distinction clearer."

Looks fine, would be great to discuss future research directions.

We would like to kindly clarify that we do lay out future research directions in Section 6, Discussion and Future Work. Some possible future work may include seeking other causes of Transcendence, expanding our analysis to more domains such as NLP, CV, and RL, and finally potentially using Transcendence in an iterative fashion to achieve a stable RL-like algorithm without learning a critic function but purely through Imitation Learning. All of these directions would be quite interesting to pursue.

---

Given these clarifications and the thorough addressing of your concerns, we hope the reviewer may consider raising their score. We are confident that our detailed response and amendments underscore the rigor and scalability of our work, potentially driving future insights into generative models. We trust that these responses adequately address your concerns. Your insight has been highly constructive and we appreciate your time in this review process. We hope to hear from you soon.

We thank the reviewer for their perceptive questions and feedback on the choice of chess as a setting and clarifications about the theory. In the responses following, we have sought to address each point in detail. We hope our clarifications will elucidate a better understanding of our approach and its potential to influence this research field.

---

• Chess is the only experimental setting..

While chess provides a well-understood and controlled environment for initial validation, we agree that demonstrating our findings across diverse domains would strengthen our claims. Thus, we have run a new preliminary experiment on the Stanford Question Answering Dataset (SQuAD 2.0), testing the effects of temperature denoising performance on several LLMs of various sizes. The SQuAD task is a reading-comprehension question-answering task on Wikipedia articles. This setting highlights how denoising improves performance in language models as these transformers begin to make fewer errors in language generation. Empirically, we measure the exact-match and F1 score on the output generations of the LLM at different temperatures. Other prior work has also found ensemble models outperform humans on this task [1,2,3]. As we explain within our paper, ensembling and majority-voting can be thought of as another form of temperature denoising, as these methods all exploit the same underlying property of learner diversity. This expansion provides a broader validation of better performance through low-temperature denoising. We include these results in the rebuttal response.

In addition, we cite in Appendix D ("Further Related Work") that several prior works have also found that pure imitation learning can lead to generative models that outperform the dataset in both the domains of Reinforcement Learning [4] (Atari games) and Natural Language Processing [5] (next-token prediction task), although they do not elucidate the mechanisms under why such phenomenon may occur. In our work, we give one such mechanism of transcendence through temperature denoising and support it with rigorous theoretical and empirical evidence.

[1] Li, Shilun, et al. "Ensemble ALBERT on SQuAD 2.0." 19 Oct. 2021, arxiv.org/abs/2110.09665

[2] Zhang, Zhuosheng, et al. "Retrospective Reader for Machine Reading Comprehension." 11 Dec. 2020, arxiv.org/abs/2001.09694v4.

[3] Rajpurkar, Pranav, et al. "SQuAD Explorer." Stanford Question Answering Dataset.

[4] Figure 4, Section 5.2. Chen, Lili, et al. "Decision Transformer: Reinforcement Learning via Sequence Modeling." 24 June 2021, arxiv.org/abs/2106.01345.

[5] Shlegeris, B., Roger, F., and Chan, L. Language models seem to be much better than humans at next-token prediction. Alignment Forum, 2022.

• Theorem 2, basically easily follows..

Thank you for asking for further clarification. We kindly refer you to Section 3.2, where we elaborated further on the assumption that ${f_{max}}$ is better than the best expert. We discuss scenarios where this assumption holds true, particularly in contexts where the model benefits from aggregating diverse expert knowledge, leading to performance that exceeds individual experts. To be clear, this is not always realistic to assume. For instance, in the paper, we demonstrate that one cannot assume that $\hat{f}_{max}$ is always better than the best expert in the 1500-rating chess example. Due to the lower data diversity, aggregating the different experts does not result in better performance than the best expert. Another example is given in the new toy model that we propose (see the Rebuttal Summary), where training on data generated by non-diverse experts (low std) does not result in transcendence.

• The connection between majority voting and low temperature sampling..

We will give here a short proof for this result, and add a more detailed proof to the final revision of the paper. Let $z$ be the vector of outputs, i.e. $z = \hat{f}(\cdot|x)$, and denote $s = \text{softmax}(z|\tau)$, where $\tau$ is the temperature. Let $a = \max z$ be the maximal value in $z$, or the value of the logit(s) for the majority vote. Let $i$ be some coordinate of $z$ that is strictly smaller than $a$, namely $z_i < a$. Then, observe that $s_i = \frac{\exp(z_i/\tau)}{\sum_{j}\exp(z_j/\tau)} \le \frac{\exp(z_i/\tau)}{\exp(a/\tau)} = \exp((z_i-a)/\tau) \to_{\tau \to 0} 0$. Therefore, all the coordinates that are smaller than the maximal value will converge to probability $0$ as $\tau$ converges to $0$. It is therefore easy to show when $\tau \to 0$ the vector $s$ converges to a vector that is the uniform distribution over the maximal values of $z$ (which we denote by $\hat{f}_{\text{max}}$).

• To my understanding, in Section 3.4..

It is possible that our original notation was not clear enough. We denoted by $\delta(\text{condition}) = 1$ the function that outputs $1$ if the condition holds, and wrote $\delta(y \in Y^*_x, x \in X_i)$ to indicate that the function is $1$ if both $y \in Y^*_x$ and $x \in X_i$. Your suggested notation is equivalent but easier to understand, so we will update our paper to use this notation instead.

• In Eq.4..

Thank you for pointing this out. After correcting, the second expectation in Eq. 4 now correctly reads $y∼f(.∣x)$, which clarifies the calculation of the reward difference.

• In line 237..

We have made the suggested correction, replacing 'left' with 'right' in line 237 to accurately describe the direction in the context.

---

We appreciate the feedback and suggestions, which have significantly improved the clarity and rigor of our paper. We hope that these revisions address your concerns and help our paper meet the standards for publication. We believe that our work on transcendence in generative models has the potential to influence future research in this area, and we are grateful for your review and insights. We look forward to further feedback you may have.

We thank the reviewer for praising the novelty of the concept of transcendence and the rigor of our theoretical proofs while noting concerns about the limited scope of our experiments and the lack of exploration of the practical implications of transcendence.

---

Limited..

To help address this concern, we have run a new preliminary experiment on the Stanford Question Answering Dataset (SQuAD 2.0), testing the effects of temperature denoising performance on several LLMs of various sizes. For more details, please see the main rebuttal PDF and the first rebuttal to Reviewer Se7B.

Assumptions..

We acknowledge this both in the theory section itself and in the limitations section at the end. As with many theoretical works in the field, simplifying assumptions is necessary for proving strong results, and these may not always capture the complex nature of real-world settings. To alleviate this problem, we have performed thorough empirical experiments to show that our theoretical findings indeed apply to real-world settings.

Practical..

While we agree that practical implications and real-world applications are not thoroughly explored, doing so would exceed the scope of our work and lead to a less focused paper. The 9-page limit constrains us from thoroughly exploring theoretical validation, empirical analysis, and practical applications of Transcendence.

For preliminary discussion, we can identify tasks that benefit most from AI, like copyediting, by considering those with errors aligning with Transcendence's theoretical conditions. Generally, AI excels in situations where the "wisdom of the crowds" surpasses individual judgment, such as question-answer platforms like StackOverflow or detecting spam in Gmail. These domains already leverage AI effectively (e.g., LLMs for copyediting and QA, Naive Bayes for spam).

[1] "Studying the 'Wisdom of Crowds' at Scale." Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 7, no. 1, 2019, pp. 171-179.

Computational..

We would like to kindly clarify a misunderstanding here. We do not use significant computational resources, and training our model requires only a single consumer-level GPU. As detailed in section 4.1, our transformer model is only 50M parameters, which is very small compared to large transformers such as GPT-3 (175B).

How do..

As noted above, we additionally have new experiments validating that denoising improves performance in Natural Language Question-Answering and Reading Comprehension. Here, it is clear how evaluation is done as human labels are provided. A baseline can easily be measured by having humans perform the task and using their performance as the threshold needed for transcendence. In tasks where human labels cannot be assumed, Elo or Glicko-2 Ratings is a powerful metric for any task where two models can be compared against, which is a much weaker assumption than assuming gold labels or the ground truth is known. In fact, our experiments are run with Glicko-2 Ratings, demonstrating a clear path for measuring transcendence in future more complex tasks.

What are..

Relaxing some of the theoretical assumptions such as uniform sampling of experts, would in fact potentially enable other forms of transcendence. To give one example, imagine that each expert has a domain of expertise where they are sampled more often and are almost always correct, and outside that domain, they are almost always wrong. Thus, whilst we had $\overline f(y|x) = 1/k \sum_i^k f_i(y|x)$ before, we would now have $\overline f(y|x) = \sum_i^k f_i(y|x) p(i|x)$, where $p(i|x)$ is the prior likelihood. Intuitively, this would enable a new form of transcendence, as the mixture distribution would "choose" the correct expert given different domains, improving performance over any one expert. We leave a more formal exploration of this form of transcendence for future work.

What ethical..

It is hard to address ethical implications without speculating well beyond the empirical and theoretical results currently available within our paper and related work, and we would not feel comfortable making claims that cannot be backed with rigor or supporting evidence.

How can dataset..

There have been several works published on measuring and ensuring dataset diversity, using information-theoretic [1] and geometric approaches [2,3].

[1] Dieng, Adji Bousso, et al. "The Vendi Score: A Diversity Evaluation Metric for Machine Learning." 2 Jul. 2023.

[2] Han, Jiyeon, et al. "Rarity Score: A New Metric to Evaluate the Uncommonness of Synthesized Images." 26 June 2022.

[3] Sajjadi, Mehdi S. M. et al. “Assessing Generative Models via Precision and Recall.” 2018.

Can the authors..

Pre-trained language models have already demonstrated the potential of transcendence in providing significant real-world benefits, including already cited use cases such as copyediting, question-answering, and spam filtration. In addition, Khan Academy has launched Khanmigo [1], an AI-powered assistant that uses advanced language models to offer personalized education, enabling tailored learning experiences for students at scale.

[1] https://www.khanmigo.ai/

Limitations Section:

We have addressed all these limitations in the above responses to the weaknesses and questions you have raised.

---

We are grateful to you for your insightful queries and comments. We hope our clarification and comprehensive responses adequately address the concerns raised and encourage a review of the score assigned. We are confident that the transcendence phenomena and subsequent research agenda could have a significant impact on understanding the capabilities and limitations of future generative models and enable future research on this concept. We kindly request a reevaluation of our work, considering its potential contribution to this area, and thank you for your time and consideration.