Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs

Thank you for your valuable feedback and questions. Below, we respond to the comments in Weaknesses (W) and Questions (Q).

---

W1: CPO performance on MATH benchmark.

Following your feedback, we included the performance of both CoT and CPO using the LLaMa3-8b-base model in $\\textrm{\\color{blue}Table B}$ of the Rebuttal PDF. For the MATH benchmark, we tested on a sample of 300 instances, as discussed in our response to W1 of Reviewer VaVd. As indicated in the table, CPO enhances performance by 1.1% on the MATH benchmark relative to CoT, demonstrating its effectiveness on more complex reasoning tasks.

---

W2: CPO's influnce on the diversity of generated paths.

In response to your insightful suggestion, we systematically evaluated the diversity of generated reasoning paths before and after fine-tuning, as measured by distinct-N [H]. The results are summarized in the table below:

The results indicate a decrease in distinct-N values after fine-tuning with CPO, suggesting a reduction in diversity. This phenomenon may stem from our preference optimization algorithm (i.e., DPO), which optimizes the reverse KL divergence and inherently exhibits mode-seeking behavior, thus reducing diversity in generation [I]. We will incorparate this evaluation and analysis in the revision.

[H] Li et al. A diversity-promoting objective function for neural conversation models, NAACL 2016

[I] Wang et al. Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints, arXiv:2309.16240

---

W3: CPO's effectiveness is heavily dependent on ToT's performance.

We acknowledge the concerns raised regarding the dependence of CPO on the quality of reasoning paths generated by ToT. While it is true that suboptimal paths from ToT could affect the overall performance, our results indicate that CPO has the capability to occasionally surpass the performance upper bounds of ToT. This is evidenced by the results presented in Table 1 of our paper.

Additionally, CPO's design is not inherently restricted to using ToT-generated paths. Our framework is adaptable and can integrate with other tree search methods that might yield higher quality reasoning paths, which we leave as our future work.

---

W4: CPO's performance on LLaMA3.

The release of LLaMA3 on April 18, 2024, just 28 days prior to our submission deadline, presented significant challenges in conducting comprehensive experiments across all seven tasks with this new model.

In response to your feedback, we have included additional results for the LLaMA3-8b-base model’s performance using both CoT and CPO on the ProofWriter and MATH benchmarks in Table B of the Rebuttal PDF. These results demonstrate that CPO enhances performance by 1.1% on both datasets when compared to CoT, affirming its applicability even on more advanced models.

---

W5: The interpretability of the optimized reasoning paths.

We appreciate your emphasis on the importance of interpretability in the reasoning paths optimized by CPO. To address this, we have included an analysis using two specific examples in $\\textrm{\\color{blue}Table E}$ of the Rebuttal PDF. These examples illustrate that the paths preferred by CPO closely resemble those selected by ToT rather than CoT, suggesting a relatively higher quality of reasoning in the paths chosen by CPO. A thorough analysis of these patterns and their implications will be provided in the revision to enhance the interpretability and trustworthiness of the model's decision-making process.

Thank you for your valuable feedback and questions. Below, we respond to the comments in Weaknesses (W) and Questions (Q).

---

W1: CPO is fundamentally still DPO.

Thank you for raising this point. We would like to clarify that while our approach adopts the DPO algorithm to fine-tune language models, our core contribution lies in different aspects compared to DPO.

DPO is an algorithm designed to align models (typically LLMs) with a target reward directly through pair preference data, avoiding the need for a surrogate reward model, and is widely used in RLHF for LLMs. The algorithm itself does not specify how to obtain the preference data, which can be sourced from human annotators [26,41,5,40], external reward models [43,1], or the model being fine-tuned itself [14].

Our work falls into a specific methodology for constructing preference data from the model itself. We find that time-consuming inference schemes such as ToT naturally produce thought-level preference data that are helpful for learning more time-efficient inference methods like CoT. This construction, although cost-effective (as it requires no external reward models or human annotation), has not been explored in previous research. Please also refer to our response to W1 of Reviewer WuBE for a discussion and comparison with closely related works.

Moreover, the DPO algorithm in our approach is not the only choice. Advanced preference optimization algorithms (such as SimPO [F]) could potentially yield better results, which we will leave as future work. Please also refer to our response to Q1 of Reviewer WuBE.

[F] Meng et al. SimPO: Simple Preference Optimization with a Reference-Free Reward, arXiv:2405.14734.

---

W2: Why not directly use token-level optimization.

We opted not to use token-level optimization primarily due to the following reasons:

1. Challenges in Reward Acquisition: CPO relies on constructing preference data from the model’s self-evaluation capabilities. Our preliminary experiments suggested significant difficulties in generating reliable token-level rewards using prompt-based LLM methods, which is crucial for token-level preference data construction.
2. High Computational Cost: The computational demands and associated costs of token-level ToT are substantially greater than those for thought-level ToT.

Considering these factors, chain-level optimization presents a balanced approach, mitigating both the computational overhead and the LCP gradient cancellation problem.

Thank you for your valuable feedback and questions. Below, we respond to the comments in Weaknesses (W) and Questions (Q).

---

W1: Comparision with ReST and self-rewarding baseline

We would like to clarify that the settings of ReST and self-rewarding are different from ours, as discussed in L66-L78 of Section 2. These methods rely on either external reward models (ReST) or labeled data (self-rewarding), which makes them not directly comparable to our approach.

However, following your suggestion, we have added a comparison with these two baselines in the table below. To ensure a fair comparison with our CPO method, we prompted the LLM itself in the same way as our CPO to serve as the reward model for ReST and as the labeled data annotator for self-rewarding, respectively. The results indicate that, on average, our CPO method surpasses both ReST and self-rewarding under this fair comparison setting.

We will include the two baselines in the revision.

---

W2: Dataset construction using ToT is computationally expensive.

The computational expense introduced by ToT is limited to the training stage. A key motivation of our approach is to transfer the computational burden from the inference to the training phase. This allows CPO to directly produce answers through greedy decoding during inference, significantly enhancing efficiency.

Moreover, by using only 200 samples to generate preference pairs, we were able to achieve performance improvements efficiently (Please also refer to our response to Q3 of Reviewer VaVd).

---

W3: Task-specific evaluation prompt templates

We apologize for any confusion. To clarify, the prompt template used in our evaluation consists of two parts: (1) the general guidelines, and (2) task-specific demonstration examples (L135-L137).

For example, here is one of the demonstration examples we used for QA datasets:

Question: When did the last king from Britain's House of Hanover die? Thought: Step 1, when did the last king from Britain's House of Hanover born? Evaluation Process: The thought process focuses on the birth date of the last king from Britain's House of Hanover. However, knowing the birth date does not directly help in determining the date of death, which is the actual question. The lifespan of an individual can vary widely and cannot be accurately inferred from their birth date alone. Therefore, this thought process is unlikely to lead to the correct answer without additional information. So the evaluation result is: this thought is impossible to help pariticially or directly answer the question. Evaluation Results: Impossible

We will clarify theis and provide task-specific demonstration examples in the revision.

---

Q1: Choice of DPO instead of other preference optimization algorithms.

Our contribution primarily focuses on the construction of preference data (Please also refer to our response to W1 of Reviewer ADqw). We chose to experiment with DPO because it is one of the most typical and widely used preference optimization algorithms. While we acknowledge that DPO is not the sole option available, advanced algorithms like SimPO [F] could potentially yield superior results, which we will leave as future work.

[F] Meng et al. SimPO: Simple Preference Optimization with a Reference-Free Reward, arXiv:2405.14734.

---

Q2: Lacking self-imporvement baselines.

Please see our response to W1.

---

Q3: Unbalanced preference dataset constructed by ToT.

In the constructed preference data, each positive sample is structurally paired with a corresponding negative sample, maintaining a one-to-one mapping at the level of individual pairs, thus achieving balance structurally. However, since ToT selects more paths than it rejects, the overall dataset exhibits an imbalanced distribution with more positive than negative samples. Despite this numerical discrepancy, our method oversamples positive instances by design, which could mitigate the potential negative impacts of an unbalanced dataset on the CPO model training. This strategy is also adopted by [G]. Figure 3(a) of our paper further substantiates this hypothesis by exploring various methods for selecting negative samples, which leads to differerent quantities of negative examples. The results in the figure suggests that the number of negative samples has minimal impact on the training outcomes.

[G] Pattnaik et al. Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences, arXiv:2403.07230.

Thank you for your valuable feedback and questions. Below, we respond to the comments in Weaknesses (W) and Questions (Q).

---

W1: Evaluation of the potential impacts of fine-tuning process on LLM's other abilities

In response to your suggestion, we conducted additional experiments using the LLaMA2-7b-base model to assess how fine-tuning on one specific task influences performance on other tasks. The results are presented in the table below:

In the table, '-' represents the base model without any fine-tuning. Bold indicates testing on the same task as the model was fine-tuned on.

The results reveal that while fine-tuning on specific tasks often improves performance on the targeted task, it can lead to decreased performance on other tasks. To address this, we are considering strategies such as using a more diverse mixture of data for fine-tuning and incorporating additional regularization techniques. Interestingly, the improved reasoning ability may generalize to other domains (e.g., the improved performance of SVAMP dataset after fine-tuning).

We will include discussions in the revised version. Thanks for your suggestions.

---

Q1: LN35, what is the advantage of CPO comared to existing MCTS methods?

Our CPO utilizes computationally expensive ToT to construct preference data for optimization during the training phase, and can efficiently generate answers through greedy decoding during testing. ToT is a method that augment the reasoning capabilities of LLMs by using tree search algorithms (i.e., BFS and DFS) to guide multi-step reasoning. While existing methods also explore using MCTS (Monte Carlo Tree Search) to guide LLM reasoning, our choice of ToT offers several advantages:

1. Self-Evaluation: ToT leverages the LLM itself to evaluate its generated reasoning paths, eliminating the need for labeled data to train an additional value function or reward model, which MCTS requires.
2. Performance: According to our preliminary exploration, MCTS guided LLM decoding does not consistently outperform ToT, which is also observed by [14].
3. Efficiency: MCTS is generally more resource-intensive than the search algorithm used in ToT (i.e., BFS and DFS), especially in terms of computational resources and time, because it involves a large number of random simulations. Moreover, ToT, with its pruning mechanisms, offers a more efficient tree search process by reducing unnecessary explorations.

---

Q2: LN117, considering online RL that regenerates the data using the updated LLM.

We chose not to employ online RL due to the significant computational overhead it incurs, as each update step involves sampling from the current policy model.

Instead, we opted for an iterative approach as a compromise between online and offline settings. This method reduces computational demands while still allowing for periodic updates from the policy model. Details on our iterative setting can be found in Appendix C of our paper.

---

Q3: Table1, CPO performance on GSM8K and ProofWriter.

Following your suggestions, we have included our results on GSM8K and ProofWriter in $\\textrm{\\color{blue}Table B}$ of the Rebuttal PDF. For the ProofWriter dataset, we sampled 300 instances for testing, as justified in our response to W1 of Reviewer VaVd. For the GSM8k dataset, we evaluated the models using the full test set. As shown in the Table, CPO enhances performance by 1.5% on GSM8K and 1.1% on ProofWriter compared to CoT, demonstrating its effectiveness on these common reasoning tasks.

References

[A] Chia et al. Contrastive Chain-of-Thought Prompting, arXiv:2407.03600

[B] Shi et al. Detecting Pretraining Data from Large Language Models, ICLR 2024.

[C] Wang et al. Causal-driven Large Language Models with Faithful Reasoning for Knowledge Question Answering, MM2024

[D] Yang et al. Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models, arXiv:2406.04271

[E] Zhou et al. LIMA: Less Is More for Alignment, arXiv:2305.11206

Thank you for your valuable feedback and questions. Below, we respond to the comments in Weaknesses (W) and Questions (Q).

---

W1: Evaluation using entire test sets

Thank you for your suggestion. We selected evaluation sets of 300 questions per dataset to manage the high computational demands of evaluating ToT (more than 230,000 A100 GPU hours across seven full test sets and three base models). Sampling a relatively small subset is consistent with common practices in the field; for instance, ToT by Yao et al. [46], Self ask by Press et al. [29], Verify-and-edit by Zhao et al. [50], and Contrastive CoT by Chia et al. [A] also utilize small evaluation sets or sampled subsets for similar reasons. We will release the subset we used to make our results reproducible.

We also acknowledge that using small evaluation sets could introduce variance in the results. Following your suggestion, we have included evaluations on the entire test sets in $\\textrm{\\color{blue}Table A}$ of the Rebuttal PDF. Currently, we are only able to provide results for CPO and two baselines (i.e., CoT and TS-SFT), as the computational cost for evaluating ToT is substantial.

Our findings is that the relative improvement of CPO over the baselines is consistent with our previous results. We will incorporate these expanded evaluations in our revised paper.

---

W2: Details about the baseline implementation & terminology

We apologize for the confusion. Throughout our paper, TS-SFT refers to the model fine-tuned on training examples with reasoning paths discovered by ToT, as described in L176-178. Note that these reasoning paths do not necessarily lead to correct answers, as we do not assume access to ground truth in our setting. The term TS-LLM in L206 was a typo and will be corrected to TS-SFT.

We will clarify these in the revision and consistently use the term TS-SFT to avoid misunderstandings. Thank you for your attention to detail.

---

W3: Choice of datasets and comparison approaches

We understand the concerns about our selections and the comparability of our results with established benchmarks.

1. HotpotQA is typically evaluated against wikipedia pages. We selected this dataset because LLMs are generally pre-trained on Wikipedia [B]. This aligns with prior work that tests LLMs on HotpotQA by relying solely on the information encoded in the model's parameters like [C].
2. Our choice of tasks was driven by the performance of the models using the ToT method, which showed improvements on QA and Fact Verification.
3. Based on your suggestion, we have added performance for the LLaMa2-7B base model on the full GSM8k test set in $\\textrm{\\color{blue}Table B}$ of the Rebuttal PDF. As shown in the Table, on LLaMA2-7b-base, CPO enhances performance by 1.5% on GSM8K, compared to CoT. Regarding the Game24, we find ToT can only achieve less than 3% with LLaMA3-8b-base (also observed by [D]). Our method, which solely relies on the inherent capabilities of the LLM for self-improvement without assuming access to human annonated data like [14], faces significant challenges in improving upon these figures. In addition, while ToT was tested on Game24 using GPT-4 in the original paper, GPT-4 does not currently offer an interface that supports DPO finetuning, precluding our ability to apply our methods in that context.

---

Q1: Metrics for QA datasets

We primarily reported accuracy as our metric (line 171) following [29]. Additionally, we added F1 scores for the three QA tasks in $\\textrm{\\color{blue}Table D}$ of the Rebuttal PDF. As shown in the table, the performance in terms of F1 scores is consistent with its corresponding accuracy.

---

Q2: Ablations in 5.3: Are the trends consistent across different models and datasets?

Following your suggestion, we have included ablations and analysis across different models and datasets in $\\textrm{\\color{blue}Figure A, B, C(a) and C(b)}$ and $\\textrm{\\color{blue}Table C}$ of the Rebuttal PDF. We find the trends are generally consistent across different models and datasets.

---

Q3: Limitation of 200 instances for constructing preference pairs

We construct preference pairs with up to 200 instances for the following reasons:

1. In our experiments, approximately 200 samples (e.g., questions in the QA task) on average could generate about 6,531 preference pairs, suggesting that our CPO requires only a small number of samples by design.
2. Constructing preference data is a time-intensive process. The choice of 200 samples represents a practical trade-off between efficiency and effectiveness, allowing us to manage resources effectively while still achieving noticeable improvements.

Following your suggestions, we added an analysis of performance trends from 0 to 3,000 samples in $\\textrm{\\color{blue}Figure C(c)}$ in the Rebuttal PDF. We observed that while performance increases with more samples, the preference data constructed from 200 samples already provides a significant boost in model performance.

---

Q4: Total number of preference pairs used in training

On average, we trained on 6,531 preference pairs across all three LLMs and seven datasets. We appreciate your suggestion and will ensure to include this specific number in the main body of the paper for clarity and completeness.

---

Q5: Stability of approach to different prompts for state evaluation

In our prior experiments, we tried different prompting templates for evaluation. We slightly changed the prompts by 1) adding more suggestive information, such as "a good annonator will classify it as"; and 2) replacing prompts with synonymous sentences. We find that the reward score returned by the LLM is robust to such kinds of changes. Balancing the performance and the length of different templates, we chose the current prompting template. We will include the discussion in the revision.