Making Large Language Models Better Reasoners with Alignment

Thank you for your valuable review comments. However, there are some misunderstandings regarding both the novelty and motivation of our work, we hope our explanation and responses can resolve your misunderstanding and concerns.

Q1: Question about the novelty: the proposed approach is very similar to Huang et al., 2022) [1], which severely hurts the paper in terms of novelty.

A1: It is essential to point out that the contributions between [1] and our work are totally different.

[1] aim to achieve self-improvement of LLMs with two steps:

1. Using LLM to generate solutions for a given question with chain-of-thought promoting and select high-confidence solutions with self-consistency techniques.
2. Fine-tuning the LLMs on the selected self-generated solutions with a next-token prediction objective to achieve self-improvement.

Totally Different from [1]:

1. We discover that LLMs fine-tuned by the vanilla next-token prediction objective suffer from an assessment misalignment problem.
2. We propose an alignment fine-tuning (AFT) paradigm to address our identified problem. Our proposed AFT method consists of novel and effective alignment loss to train LLMs, which is totally different from the next-token prediction objective used in [1].
3. We delve deeply into recent ranking-based alignment methods (which are truly similar works to ours) and discover that the constraint, which has been overlooked by these approaches, is also crucial for their performance.

Given such obvious differences in the contributions between [1] and our work. We can confirm that the contributions of these two works are orthogonal.

Q2: Question about the motivation: The results of our pilot experiment (Table 1) are trivial and not surprising. What are the expected results of our pilot experiment, and what is the motivation derived from the pilot experiment?

A2: The results of our pilot experiment inspired us to propose an effective alignment fine-tuning paradigm to improve the reasoning ability of LLMs, which is not trivial for the following reasons:

In this paper, we define two important abilities of LLM reasoners: 1) reasoning ability: LLMs can generate the right solution for the unlearned question; 2) assessment ability: LLMs can evaluate the quality of different candidate solutions for the learned question, i.e, they can assign higher scores (lower perplexity) to right solutions compared with wrong solutions.

Previous studies have shown that the vanilla chain-of-thought fine-tuning (VFT) can significantly improve the reasoning abilities of LLMs. However, is VFT sufficient for teaching LLM reasoning ability? We conducted a pilot experiment to explore this question.

If VFT is sufficient enough, we expect that LLMs trained with VFT will demonstrate strong assessment ability. This is based on a reasonable assumption that LLMs that truly learn to solve a reasoning question should be able to assess the quality of different candidate solutions to the learned question.

Contrary to our expectations, our pilot experiments show that VFT-LLMs lack this desired assessment ability: the assessment accuracy on their learned questions is just around 70% on GSM8K and 62% on ECQA.

Their assessment accuracy is surprising (NOT trivial) and far from our expectations because:

1. the assessment task in our pilot experiments is a binary classification task where a random selection baseline can have up to 50% assessment accuracy.
2. The questions of our pilot assessment task are from the training set, and LLMs have already been trained on these questions.
3. With the right training in problem-solving, we as humans can achieve near-perfect assessment accuracy. Take Figure 1 as an example, after learning to generate the reference solution for the given question, it becomes very easy for humans to evaluate the quality of two candidate solutions for the learned question. However, this level of assessment is currently beyond the capabilities of VFT-LLMs.

In addition, the pilot experiment also shows that the reasoning ability on unlearned questions has a strong positive correlation to the assessment ability of learned questions.

Therefore, our pilot experiences inspire us to improve the reasoning ability of LLMs by proposing an assessment alignment loss to improve their assessment and reasoning ability.

Dear Reviewer 9VoV,

Thank you again for your valuable feedback and comments! We would greatly appreciate it if you could let us know whether you are satisfied with our response, especially the response about the novelty and motivation of our paper. We will be happy to address any remaining concerns.

Sincerely, Paper7016 Authors

Dear Reviewer 9VoV,

Thank you again for your valuable feedback and comments!

We can confirm that there exist some misunderstandings regarding both the novelty and motivation of our work.

We greatly need and look forward to discussions with you to eliminate misunderstandings.

We would also be delighted to address any other concerns.

Sincerely, Paper7016 Authors

Dear Reviewer 9VoV:

Thank you again for your valuable feedback and comments! Please see the responses as follows:

Q1: Regarding the empirical comparison to Huang et al., (2022). — Your main concern

A1: The review writes:

The main concern that I have is the lack of conceptual and empirical comparison to Huang et al., (2022).

The fact that the loss proposed in this work is "different" is not sufficient; maybe the part of generating various CoT solutions (which as far as I understand, is common to the two papers) is the key, and then training on these generated solutions can be done even with the standard next-token loss?

I agree with you that generating various CoT solutions is important, and training LLMs on these generated solutions with a standard next-token loss is a necessary and strong baseline. It is essential to point out that we have compared our AFT with such baseline, i.e., Rejective Sample Fine-tuning (RFT, a concurrent work to ours). RFT further trains LLMs on the high-quality generated solutions with the correct answer using the next token prediction. Our experiments have demonstrated that AFT outperforms RFT, especially in the ranking situation.

Although Huang et al., (2022) focus on the self-improvement of LLMs on unlabeled datasets, which is totally different from ours, we have also followed your suggestion to replicate Huang et al., (2022)'s approach. This replication involves training LLama-7B on their generated CoTs with the majority voting answer.

As is shown, our AFT outperforms Huang et al., (2022). More importantly, from the principles of the methods, AFT can better utilize the feedback compared with RFT or Huang et al., (2022). Specifically, AFT has the ability to help LLMs recognize quality differences among any given COT pair in a ranking context, while the RFT or Huang et al., (2022)'s approach can not achieve this because they only try to optimize the probability of the chosen COTs.

Based on the experimental results and the underlying principles of these methods, it's clear that AFT is essential. We hope these results can address your main concern.

Q2: About the assessment ability for VFT-LLMs.

The reviewer writes:

sometimes the model assigns higher probability to the wrong output and low probability to the correct output? Isn't this the source of any kind of mistake in any machine learning model? isn't it trivial?

It is trivial for the model to assign a higher probability to the wrong output for their unlearned question. However, we believe doing so for their learned questions is not trivial. If we think VFT is good enough to teach LLM reasoning, it will be really surprising when VFT-LLMs stumble on extremely easy assessment tasks for their learned questions, such as Figure 1.

The reviewer writes:

Why are these numbers surprising? The number "70%" on its own is meaningless without comparison to any alternative (that is non-human and non-random). Did the authors expect 100%? What about 80% or 90%, are they close to expected or not?

In Table 4, we have compared the assessment ability of VFT with that of our AFT (a non-human and non-random alternative). We include the results here for your convenience:

Please note that the COTs used to measure the assessment ability here are newly sampled COTs, distinct from the COTs used to train AFT.

This result demonstrates that there is substantial room for improvement in the assessment ability of VFT, further affirming the validity of our motivation: Enhancing the reasoning ability of LLMs by addressing the assessment misalignment problem in VFT.

We would greatly appreciate it if you could let us know whether you are satisfied with our response. We will be happy to address any remaining concerns.

The authors write:

A3: The “reasonable scores” imply that VFT-LLMs should give lower perplexity (high score) to right COTs in GP compared to wrong COTs in GN for their learned questions. However, our pilot experiments reveal that VFT-LLMs fail to give such “reasonable scores”, i.e., they have poor assessment ability

It feels that the authors refer to relative properties with absolute and over-decisive conclusions. The authors write that "VFT-LLMs fail to give reasonable scores"; however 70% doesn't sound like a complete failure to me, and it doesn't sound like a "poor assessment ability" if the only comparison is to a human who can achieve 100%.

As you suggest, we conduct further experiments on the Chat version of LLM, i.e., LLama-2-7B-chat and LLama2-13B-chat on GSM8K

Thank you for these additional experiments, I think they are an easy experiment that strengthens the paper with more empirical evidence.

Conclusion

Although our main discussion here focused on the question of whether "70% assessment is high or low?", I think this issue can be easily solved by toning down this part.

The main concern that I have is the lack of conceptual and empirical comparison to Huang et al., (2022). I am not one of the authors of that paper, and it doesn't bother me personally that this paper was not cited. I honestly want to understand the difference and whether the simpler approach of Huang et al. may be sufficient, without introducing additional losses.

Thank you again for your valuable feedback and comments. Please see our responses as follows:

Q: Is our conclusion made from an absolute number?

The reviewer writes:

I still think that the question of whether the model's assessment accuracy "is high or low" is not really meaningful. It's a nice anecdote to show that it improves, but it's not really indicative on its own, and I'm not sure that great conclusions should be made from the absolute number of "70%".

The number 70% is not only an absolute number, it symbolizes a lot of surprising instances like Figure 1. The VFT-LLMs fail to give reasonable scores in such frustratingly easy assessment tasks for their learned question. If you think such stupid mistakes on their LEARNED question are trivial, let's leave aside the debate about this. At least, these results can fully demonstrate that VFT is insufficient to cultivate a real reasoner.

Our conclusion is not made from an absolute number. Our pilot experiment aims to point out that VFT is not enough, and thus we need a new fine-tuning paradigm.
How to design such a fine-tuning paradigm? Our pilot experiment reveals a strong positive correlation between assessment and reasoning accuracy (with Pearson Correlation Coefficients of 0.93 and 0.98 at GSM8K and ECQA, respectively). Therefore, the main conclusion in our pilot experiment is that we can improve the reasoning ability of LLMs by improving their assessment ability. The conclusion is derived from the correlation, not an absolute number. In addition, our experiments have confirmed our conclusion.

Q: Regarding the advantages of AFT.

The reviewer writes:

Theoretically speaking, yes, but in practice, the improvement is even lower than the standard deviation, while RFT is much simpler and does not require additional losses.

As previously agreed upon, in theory, AFT boasts a distinct advantage over RFT because of its ability to leverage ranking feedback, an attribute that RFT lacks. Beyond theoretical considerations, our empirical experiments have demonstrated the potential of AFT. Compared with binary feedback, the performance gap between AFT and RFT becomes more pronounced when using ranking feedback. The expanding performance gap demonstrates the theoretical advantage of AFT.

We concur with your assessment that RFT is a straightforward and effective strategy. Nonetheless, our AFT also boasts ease of implementation.

The theoretical advantage, practical experiments, and the simplicity of the implementation collectively underscore the necessity of AFT.

We will add a comparison with Huang et al (2022) in our paper, and we hope that our response can persuade you to buy our idea. Thank you for your consideration.

Thank you for your valuable review comments.

Q1: It would be great if the authors could provide some intuitions on their designed losses to address the corresponding constraint.

A1: In this paper, we find that the rank-based alignment loss without constraint will lead to the collapse of LLMs. We point out this is because the unconstraint alignment loss will over-punish the negative solutions that also express some reasonable reasoning process (Our analyses in Sections 6.1 and 6.2 substantiate this conclusion). Therefore, we design two constant alignment losses, detached constraint alignment losses (DC) and boundary constraint alignment losses (BC) to align LLMs without over-punishing the negative solutions.

Q2: It would be great if the authors could explain why the performance dropped for other baseline methods when compared to vanilla fine-tuning.

A2: As illustrated in Sections 6.1 and 6.2, the performance of other ranking-based methods drops because they lack constraint in their losses and over-punish negative COTs. After adding our constraint, they outperform the vanilla fine-tuning.

Q3: I also wonder how the quality of LLM-generated COTs impacts the performance of AFT. For example, how large is the variance using 3 generated examples?

A3: Our AFT enhances the reasoning capabilities of LLMs by improving their assessment ability on sampled candidate COTs. We discovered that the diversity of these sampled candidates is crucial. For instance, we found that data sampled from models fine-tuned over two epochs outperforms data sampled from models fine-tuned over three epochs, with a noteworthy accuracy difference approaching 1%.

On your suggestion, we sampled 3 groups of generated COTs (each group comprising 3 COTs per question) on GSM8K from the fine-tuned Llama-7B model. We then used this sample data to further fine-tune the Llama-7B model using AFT. The standard deviations measured were 0.42 and 0.53 for AFT-DC and AFT-BC, respectively. The results appear stable, which we believe can be attributed to the homogeneity in the quality of samples drawn from the same model.

The question of how to define quality, and how to sample high-quality solutions from fine-tuned LLMs, remains a valuable area of exploration, which we leave for future work.

Thank you for your valuable review comments.

Q1: Did you compare your methods to other chain-of-though reasoning approaches or other fine-tuning strategies for mathematical tasks?

A1: Our work is orthogonal to existing chain-of-thought reasoning approaches in mathematical tasks, which typically fall into two categories: chain-of-thought prompting and chain-of-thought fine-tuning.

1. Chain-of-thought prompting, which is orthogonal to our work, focuses on triggering the reasoning capabilities of Language Models (LLMs) without tuning them. It complements our proposed Alignment Fine-Tuning (AFT) methods. Our analysis, illustrated in Figure 2(d), investigates the integration of our AFT with Self-Consistency [1], a prevalent chain-of-thought prompting ensemble strategy, and demonstrates that AFT and Self-Consistency complement each other.
2. Chain-of-thought fine-tuning, which primarily involves collating chain-of-thought data (such as with Flan [2], Distill-Step-by-step [3], MetaMATH [4], MAmmoTH [5]) and training LLMs with standard next-token prediction objective. Our work highlights the limitations of this vanilla chain-of-thought fine-tuning (VFT) paradigm and introduces an alternative AFT method. Our experiments demonstrated AFT outperforms VFT.

To the best of our knowledge, VFT is the most widely used and effective fine-tuning strategy to improve the reasoning ability of LLMs [2] [3] [4] [5], and we are the first work to point out the assessment misalignment problem of VFT for reasoning, and we proposed AFT to address this problem. Addressing the assessment misalignment problem is a new perspective to improve the reasoning ability of LLMs. There are no known exploration or fine-tuning strategies aimed at addressing this problem specifically within the context of mathematical tasks. Therefore, the most relevant approaches to ours are other alignment fine-tuning methods, such as RRHF and PRO. In this paper, we delve deeply into these methods, and our proposed constraint can also enhance these alignment-tuning methods.

We will enrich our paper by incorporating more discussion related to mathematical tasks within the section dedicated to related work.

Q2: Why are the chosen evaluation datasets sufficient for your claims? Are these the main datasets of other related works in the field or are other reasoning datasets "easier" than the chosen ones?

A2: We focus on two typical reasoning tasks, including mathematical reasoning and commonsense reasoning. For the mathematical reasoning task, we employ GSM8K and AQUA-RAT as our benchmarks, while we use ECQA for the commonsense reasoning task. All of these benchmarks are broadly recognized and used in relevant research works [1] [2] [3] [4] [5] [6] [7].

We observe the assessment misalignment problem of VFT on both mathematical and commonsense reasoning tasks, and our proposed AFT significantly outperforms VFT on all benchmarks. These results are sufficient for our claims.

Q3: Are the empirical results on-par with other referenced works in the field, such as Li et al., 2023?

A3: Li et al., 2023 [6] is a chain-of-thought prompting work, which is orthogonal work to ours. They find that better reasoning performance of LLMs can be achieved by a step voting strategy.

Our work focuses on revealing the assessment misalignment problem of the vanilla chain-of-thought fine-tuning (VFT) paradigm, and proposes an alignment fine-tuning (AFT) paradigm to address the find problem. Therefore, we choose VFT and recent alignment methods as our baselines. Our experiments demonstrate the importance of assessment misalignment problems and the effectiveness of our AFT.

Dear Reviewer rVEU,

Thank you again for your valuable feedback and comments! We would greatly appreciate it if you could let us know whether you are satisfied with our response. We will be happy to address any remaining concerns.

Sincerely, Paper7016 Authors

Dear Reviewer rVEU,

Thank you again for your valuable feedback and comments! Please see the responses as follows:

Response to WrtQ1/Q3

It's not straightforward to discern which direction yields more fruit as each path offers substantial and valuable insights. However, our experiments demonstrate that our proposed AFT can improve the chain-of-thought reasoning approaches in other directions:

1. For chain-of-thought prompting direction approaches, which aim to trigger the internal reasoning ability of LLMs. Our AFT actually improves the upper bound of these prompting approaches, because AFT can enhance the internal reasoning ability of LLMs. For example, as shown in Figure 2(d), our AFT can strengthen self-consistency, which is a prominent strategy in the chain-of-thought prompting approach.
2. For chain-of-thought fine-tuning direction approaches, which are fundamentally data-centric because they focus on gathering high-quality training datasets with the chain-of-thought process. All these works utilize VFT to train LLMs. As shown in our main results (Table 2), our AFT can enhance these data-centric approaches by better utilizing their gathered data compared with VFT.

Response to Wrt Q2

Q: Why do we choose GSM8K, AQUA-RAT, and ECQA as our evaluation datasets?

A: As you suggest, we add an explanation in the paper. Please refer to Appendix A.1. for details.

Q: Do other relevant competitors use this exact setup?

A: It may be more accurate to use relevant works instead of relevant competitors, because we are the first work to point out there exists an assessment misalignment problem in VFT, which proposes a new perspective to improve the reasoning ability of LLMs. The relationship between our work and relevant works is not competitive, because they are orthogonal and complementary.

We do not utilize an identical setup to our relevant works. Indeed, currently, there is no uniformity in the setups of other related works either. For instance, the training data, pre-trained LLMs, and data processing strategies employed in each work are nearly always distinct. In addition, different from previous works, our aim does not lie in achieving state-of-the-art results in mathematics or commonsense reasoning tasks. Rather, our primary focus is on identifying, analyzing, and mitigating the previously unidentified assessment misalignment problem of VFT.

Response to Wrt Q4

We have some confusion about the “standard RLHF”. In our opinion (as also described in our related work, Section 2.2), there exist three typical kinds of preference alignment methods that can alleviate our revealed assessment misalignment problem: (1) reinforcement learning; (2) rejective sampling; and (3) ranking losses.

Does the "standard RLHF" denote the “reinforcement learning” methods? In this paper, we do not analyze this type of method, because it is well known that the reinforcement learning methods are very difficult to tune and need much more computation resources. More importantly, our AFT belongs to ranking losses, and thus we have devoted a lot of space to analyzing other ranking losses.

We would greatly appreciate it if you could let us know whether you are satisfied with our response. We will be happy to address any remaining concerns.

Q1: The slightly risky assumption that generated outputs with the correct final answer should be assigned a higher score than those with the wrong final answer.

A1: Thank you for your valuable review comments. In this paper, we manually check 50 pairs of incorrect-correct pairs, discovering that in 48 out of these 50 pairs, the solution with the correct answer surpassed the one with the incorrect answer in terms of quality. In addition, our proposed alignment fine-tuning (AFT) paradigm significantly outperforms the vanilla chain-of-thought fine-tuning (VFT) paradigm. These evidences substantiate that it is reasonable to consider solutions with correct answers superior to those with incorrect ones.

Moreover, the main contribution of our paper does not reside in the development of the feedback signal. Rather, it fundamentally revolves around pinpointing the assessment misalignment problem within VFT and introducing AFT as a solution to this issue. We believe that with more precise feedback, our AFT will yield more significant improvement.