Chain-of-Thought Reasoning is a Policy Improvement Operator

We thank the reviewer for their comments.

Regarding ablations on simplify-and-guess. In Figure 5, we run the experiment measuring the impact of using K=0 (no decomposition and just chess) as well as K=N-1 (break the problem all the way down) compared to a middle ground of K=5. We will run an ablation measuring the impact of other intermediate values of K shortly.

Regarding Figure 6. Thank you for pointing out the difficulty in reading accuracy in this figure. In the updated paper, we include a table of all accuracies in the Appendix for ease of reading. Our hypothesis for why there might be better generalization in the middle than at the ends is that the model may be reaching its capacity for learning (see general comment) as it allocates more and more of its “ability” to fast-adding larger and larger digit numbers.

Improving Data Efficiency. In Appendix I, we find that doubling the parameter count of the model leads to almost a 10x reduction in data needed. An interesting direction for future work is to see if this trend continues.

General Applicability of SECTOR. The reviewer expressed concerns about the general applicability of SECToR. In this paper, we merely wished to demonstrate a proof-of-concept for self-learning. However, we believe that the method is generally applicable to others tasks in which 1) models perform better after using chain-of-thought or another inference time decoding procedure, and 2) there is some form of self-consistency checking available.

Recent research has suggested that many benchmarks are improved via chain-of-thought reasoning and related ideas. In terms of the self-consistency check, while the use of commutativity is specific to addition and related arithmetic operations, many other tasks have other forms of self-checking. For example, one could imagine proving a theorem in several different ways or navigating through a city via multiple different paths as a self-consistency check.

Supervised training Questions. In Figure 2, we provide explicit examples of the data for both fast and slow addition. These examples can be easily generated programmatically and the script used to generate them is provided in the supplementary data in generate_data.py. N-digit addition problems are generated by randomly sampling two N-digit numbers. Appendix I provides a graph depicting how much data is used during the supervised/self-train period. In particular, Figure 10 shows that the 582M model used around 0.5M training examples during the supervised training period and self-generated an additional 1M training examples for supervised training. The number of tokens per example varies but is on the order of 100 for slow addition (per decomposition step) and 10 for fast addition.

Curriculum Learning. Appendix J states that: “We run an ablation where we train a 582M parameter ByT5 model on 1 through 6 digit addition in a single supervised fine-tuning step, instead of via the curriculum learning setup done in Section 3.5. We find that this model, when properly trained, generalizes to up to 9 digit (slow) addition perfectly.” Our conclusion is that curriculum learning is not necessary in the supervised training, but required in the self-training phase, as the model cannot generate the data for N+1 digit addition until it has learned to add N digit numbers. In the paper, we used curriculum learning in both phases because this provides that the unique difference between self-training and supervised training was the data generation.

Hypothesis on Emergence. The reviewer took issue with our hypothesis that a sufficiently large LLM might be able to forgo supervised training entirely. This was framed as a hypothesis, as we did not have the resources to run this particular experiment. However, we politely disagree that there is no supporting evidence in the paper. Specifically, the preceding sentence refers to experiments done in Appendix I in which we found that that doubling the parameter count of the pretrained model (from 300M to 582M) reduced the training data needed in the supervised training period by almost a factor of 10 (from 4M to 0.5M supervised training examples). We did not have the computational resources to do an extensive study of scaling laws, but believe it is a reasonable hypothesis that a sufficiently large model might reduce to only a few examples. In particular, the SOTA LLMs have 1000x (~2^10) the parameter count of the models which we used here. Additionally, the history of LLMs are filled with tasks that previously required fine-tuning but can now be done with mere prompting (including chain-of-thought reasoning itself!). However, given this feedback, we have phrased our hypothesis more cautiously in the updated version of the paper.

Failure at 29 Digits. In the general response, we described the results of a new ablation addressing this concern. This experiment sheds some light on the causes of the failure at 29 digits.

We thank the reviewer for their comments. Regarding the comments on formatting, these have been fixed in the updated version of the paper.

Fast adding / simplify-and-guess. Fast adding indeed means generating an answer without using chain-of-thought. Figure 2 contains specific examples for both slow and fast addition. To describe simplify-and-guess, it is helpful to walk through Figure 4. Given an eight digit problem, the model first breaks down the problem into a seven digit problem (slow chain-of-thought), before taking a guess for that seven digit problem (fast addition). This is guess #1. To obtain further guesses, the model breaks down the 7 digit problem it produced in the previous step into a 6 digit problem, before taking a guess to obtain guess #2. This is repeated K times and all K guesses are aggregated via majority vote.

Compared to simply decomposing the problem until it is 1-digit, simplify-and-guess is an additional error- checking mechanism in case there is error introduced in the decomposition step. Figure 5 describes the benefit that simplify-and-guess provides over just simplification or just guessing. Additionally, we hope to include an ablation explorating the impact of different choices of K shortly.

Data Prefix. Training examples were prefixed with either “Add fast.” or “Add slow.” However, since the model was fine-tuned (and not just prompted) on these examples, the precise prefix is unlikely to be important since the model can easily learn the mapping between the prefix and the task at hand.

Regarding dataset details. The reviewer claims that the paper lacks details about the dataset. We respectfully disagree. In Section 3.2, we state that “all training examples are generated programmatically by an external script during supervised learning.” This script is included in the supplementary data in the file called generate_data.py. In Figure 2, we provide explicit examples of the data for both fast and slow addition. Additional details about the dataset, including the precise size of the dataset and the split between validation and training, are given in Appendix B.

Ablation on Training on Oracle Data. In the general response to reviewers, above, we provide the results of a new ablation that continues the 29 digit self-training run, except with ground truth data after self-training has failed. As explained in the general response, we find that training can continue for a few more iterations, suggesting that SECToR’s self training performs worse, but not exceptionally worse than supervised fine-tuning with ground truth examples.

We thank the reviewer for their comments.

Regarding baselines. The reviewer asks whether SECToR will outperform a baseline where the model is trained only on programmatically generated data. It will not. Our claim is not that self-training will perform better than training on ground truth data. We would not expect this behavior even if the data generation process has no errors, since in that case, it would only match and not exceed the baseline of training on ground truth data. Rather, our claim is that self-training can occur in the absence of ground truth data. In the general response, above, we describe a new ablation in which we have replaced model-generated data with ground truth data after the self-training process fails (i.e., for digits 29+).

Regarding problem sources. In SECToR, the problem statements themselves were provided to the model and not self-generated. In future work, we will explore asking the model to generate the problems themselves as well as the candidate solutions.

Regarding loss of existing capabilities. We do not claim that SECToR produces an LLM that is superior to the original one -- rather, our goal is to establish a proof-of-concept that self-learning in language models -- for multiple iterations that extend far beyond the original distribution of training -- is empirically possible. As per our general response, above, the paper’s primary goal is not to teach an LLM to perform addition — one could simply use supervised fine-tuning for this. The goal is to provide a proof of concept for self-learning, with addition used as a testbed.

Regarding the importance of simplify-and-guess. The reviewer’s proposed method would correspond to using simplify-and-guess with K=1 (simplify once then guess immediately). The summary is that although decomposition and fast-adding are fairly reliable, even a tiny error incurred in the data generation process is amplified in future iterations due to error avalanching. We will perform an ablation on the precise importance of K and include it in the next draft of our paper.

Regarding related work. We don’t see much connection between the papers on distilling chain-of-thought utterances from a larger teacher model into a smaller student model and our work, as we focus on self-learning where a model learns to self-improve without access to any external data source (whether from a larger teacher model or ground truth data). However, the other suggested related work is very relevant and included in section in the updated version of the paper with references to self-improvement via prompting.