Self-Boosting Large Language Models with Synthetic Preference Data

We sincerely thank Reviewer d9yw for your review and are grateful for the time you spent on our submission. We're pleased you find our method effective and well-validated. Below, we provide a point-by-point rebuttal to clarify your concerns.

Q1: Ablation study on the effect of excluding noise keywords

As suggested, we conduct experiments on the effect of excluding noise keywords. We train and evaluate the Llama3 prompt generator with three random keywords, either including or excluding one noise keyword. We calculate the average similarity across the generated prompts using SentenceTransformer, as mentioned in Section 2.2, and employ GPT-4 Turbo for LLM-as-a-Judge evaluation of prompt quality (on a scale of 1 to 10, where 1 represents a prompt that is very unrealistic, unnatural, or unanswerable, and 10 represents a prompt that is very reasonable, realistic, and answerable).

( Evaluation run on 1k data for each setting, each example contains 3 keywords )

The results show that the inclusion of noise keywords improves the quality of the self-generated prompts (as LLMs learn to ignore unrelated words), but even without noise keywords, the self-prompt generator can still generate relatively high-quality prompts.

Q2: Analysis of the types of keywords

1. Extraction Form

Currently, our extraction form employs a rule-based random selection. Specifically, we use the NLTK toolkit to filter out stop words, extract all noun phrases from the sentences, and remove any preceding articles. We then randomly sample from these phrases. The decision to use noun phrases is based on following preliminary experiments:

(Note: "Words" refer to single-word keywords, while "Phrases" refer to each keyword can be a phrase comprising 1 to 3 words.)

As shown, using phrases rather than limiting keywords to single words generally results in better diversity and quality. This improvement may be due to phrases incorporating more inductive bias from the pretraining data for prompt generation. Specifically, while random sampling from all phrases or words without filtering is acceptable, it is not as effective as random sampling from nouns, as nouns contain the most important information needed to diversify a sentence.

Beyond these evaluation results, we have also manually observed the impact of different extraction forms and found that using phrases as keywords generally resulted in higher quality prompts. Therefore, our experiments utilize randomly sampled noun phrases. We will include these preliminary experiments, which support our choice of keyword extraction methods, in the Appendix to provide further insights into self-prompt generator training.

2. Order

Whether order influences is a very insightful point. In our current approach, we randomize keyword order. To systematically examine this, we compared results from maintaining the original keyword order (also in the pretraining data) versus randomizing it. The results are shown in the table below.

( Evaluation run on 1k data for each setting, each example contains 3 keywords )

The results indicate that keyword order has a minimal impact on prompt generation. Randomization enhances diversity, while preserving the original order retains some semantic context from the pretraining data, yielding slightly higher quality prompts. However, this effect is negligible.

Q3: Vary the number of keywords
Thanks for the great suggestion. As suggested, we have added experiments that vary the number of keywords, and the results are presented in the following table. The experimental findings indicate that for training the self-prompt generator, using 3-5 keywords strikes a good balance between the diversity and quality of the generated prompts. Including too many keywords can be detrimental to the quality of the generated prompts.

( Evaluation run on 1k data for each setting. )

Q6: Complementary references
Thanks for pointing out the related literature! Lin et al.'s work $1$ shares a similar spirit with our prompt generator in generating sentences from keywords, but they focus on explicitly testing machines' commonsense reasoning ability by generating a coherent sentence with a given set of common concepts. Seo et al.'s paper $2$ leverages Korean morphological variations to create synthetic data, which is then enhanced in quality using contrastive learning. We will add discussion on these related and insightful works in Section 5 in our final revision.

Thank you very much for the constructive comments, which really help us further improve our work. We hope our answers have addressed your concerns. If you have any further questions, we are happy to address them.

Reference
* $1$ Mukherjee, Subhabrata, et al. "Orca: Progressive learning from complex explanation traces of gpt-4." arXiv preprint arXiv:2306.02707 (2023).*
* $2$ Lin, B. Y., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y., & Ren, X. (2020, November). CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 1823-1840).*
* $3$ Seo, J., Moon, H., Lee, J., Eo, S., Park, C., & Lim, H. S. (2023, December). CHEF in the Language Kitchen: A Generative Data Augmentation Leveraging Korean Morpheme Ingredients. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 6014-6029).*

We sincerely thank Reviewer Ftfv for your review and are grateful for the time you spent on our submission. We are glad for the acknowledgment that our approach is novel, effective and practical to tackling a critical challenge in LLM development. Below we would like to give detailed responses to each of your comments.

Q1: Limited task scope

Except for instruction-following tasks, we have also evaluated the model performance on six diverse LM Evaluation Harness tasks (Table 5, including ARC challenge for reasoning and GSM8k for math domain), as well as six Open LLM Leaderboard tasks (Table 4, including PROST for reasoning and MathQA for math domain).

We would like to note that SynPO is not designed for complex reasoning but we agree that comprehensive evaluation across a broader range of tasks would strengthen our findings. To address potential concerns regarding task scope, we have extended our evaluation to include more complex reasoning tasks and specialized domains as suggested.

• For complex reasoning tasks, we have evaluated the model performance on seven additional reasoning-related tasks from LM Evaluation Harness. The results are presented below:

(Evaluation results on 7 LM Evaluation Harness tasks for reasoning, using Llama3)

• For specialized domains, we evaluated model performance on four additional tasks from diverse domains and reported the domain-specific results on the AGIEval benchmark:

(Evaluation results on 4 LM Evaluation Harness tasks for different domains, using Llama3)

(Evaluation results on AGIEval for different domains, using Llama3)

Specifics for AGIEval:

AGIEval is a comprehensive benchmark specifically designed to assess foundation models in the context of human-centric standardized exams across diverse domains, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests.

Other Task Specifics:

• LogiQA: Logical reasoning tasks requiring advanced inference and deduction.

• LogiQA 2.0: Large-scale logical reasoning dataset adapted from the Chinese Civil Service Examination.

• MMLU-Pro: A refined set of MMLU, integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options.

• SiQA: Social Interaction Question Answering to evaluate common sense and social reasoning.

• QA4MRE: Question Answering for Machine Reading Evaluation, assessing comprehension and reasoning.

• PIQA: Physical Interaction Question Answering tasks to test physical commonsense reasoning.

• NQ Open: Open domain question answering tasks based on the Natural Questions dataset.

• PubMedQA: Question answering tasks based on PubMed research articles for biomedical understanding.

• RACE: Reading comprehension assessment tasks based on English exams in China.

• SWAG: Situations With Adversarial Generations, predicting the next event in videos.

• EQ-Bench: Tasks focused on equality and ethics in question answering and decision-making.

Q2: Unexplained performance gaps
We'd like to note that the scoring method used in AlpacaEval is based on pair-wise comparison (as shown in Table 8). This approach reflects the relative performance improvement over the baseline model, which can result in a larger numerical scale of improvement (e.g., SimPO training on UltraFeedback improves AlpacaEval score from 6.2 to 22.0 $1$ ). When we consider absolute score changes on direct evaluation benchmarks (such as the MT-Bench results in Table 3) or accuracy-based benchmarks (like the Open LLM Leaderboard results in Table 4), the improvements are more moderate but still consistently positive.

Q3: Regarding why synthetic data works better and ablation studies

Regarding why synthetic data performs better and related ablation studies, we would like to clarify that our comparison with the SFT baseline does not involve contrasting synthetic data with real data. With limited SFT data as seed data, our goal is to construct effective preference data to continuously improve LLMs' instruction-following and end-task capability. The SFT baseline serves as the initial checkpoint for further self-boosting.

As for the reasons why synthetic data works well, there are several main reasons:

1. Compared to human-collected preference data (whose format is also preference pairs), SynPO is an iterative and nearly on-policy process. At each iteration, the current model state is considered for constructing synthetic preference data. As validated by Chen et al. $2$ and Yuan et al. $3$ , on-policy preference data are usually more effective than static data.
2. Compared to continuously training on seed SFT data (as shown in our ablation study in Section 4.2, Table 7), in scenarios where only a limited amount of seed SFT data is available, the number of SFT data is limited and the format of SFT is not as effective for alignment as preference pair format data. Preference pairs can lead to more effective learning of human preferences than SFT data, which may overfit to specific examples $2$ . Our approach of self-synthesizing effective preference pairs allows for iterative improvements in model performance.

In Table 6, we further compare the effects of synthetic prompts and manually collected prompts of the same scale. The results show that synthetic prompts perform slightly better due to two main reasons:

1. Synthetic data can better control for diversity (as shown in Figures 4 and 5).
2. Our method allows the model to gradually self-learn the distribution gap during the self-improvement training process, enabling it to address specific shortcomings at each stage rather than repeatedly learning from a fixed distribution as with real data.

Overall, we greatly appreciate your efforts for your thoughtful and insightful comments on our paper. We hope our answers have addressed your concerns.

Reference

$1$ Meng, Yu, Mengzhou Xia, and Danqi Chen. "Simpo: Simple preference optimization with a reference-free reward." arXiv preprint arXiv:2405.14734 (2024).*

$2$ Chen Z, Deng Y, Yuan H, et al. Self-play fine-tuning converts weak language models to strong language models $J$ . arXiv preprint arXiv:2401.01335, 2024.*

$3$ Yuan, Weizhe, et al. "Self-rewarding language models." arXiv preprint arXiv:2401.10020 (2024).*

$4$ Ouyang L, Wu J, Jiang X, et al. Training language models to follow instructions with human feedback $J$ . Advances in neural information processing systems, 2022, 35: 27730-27744.*

We really appreciate your effort for reviewing our paper and your acknowledgement of our paper’s contribution. We are also glad for the acknowledgment that the problem our experiments are extensive and the topic will be well-received. Below, we would like to give detailed responses to each of your comments.

Q1: Clarity of the initial paragraph
Thanks for pointing out the potential confusion. We will clarify them point by point below and update our expressions in paragraph 3 in our introduction for better understanding.

• The initial model, which undergoes preference optimization, is first trained to be a prompt generator and is used with the synthetic preference data.
• The "response" refers to the outputs generated by the initial model when given synthetic prompts. Only the prompts are generated by the self-prompt generator; the responses are the LLM's completions of these prompts.
• The self-prompt generator and the response improver are trained from the same base model.

Also, we present the SynPO algorithm in Appendix A for a more higher formulated explanation for reference. Thank you very much for the constructive comments again, which really help us further improve our clarity.

Q2: Small point for model name

Thanks for pointing out this! Yes, we agree that using ‘-base’ may bring some confusions, and we have replaced the model names from Mistral-base, Llama3-base to Mistral-base-SFT and Llama3-base-SFT, respectively. We’ll include this revision in our latest version.

Q3: Confusion for improvement on things learned in pretraining

As shown in Table 5, SynPO primarily enhances performance in commonsense reasoning tasks (such as ARC and HellaSwag) and model honesty (TruthfulQA). For knowledge benchmarks like MMLU, the focus is more on maintaining performance rather than improving it.

Regarding the observed improvements in certain knowledge-related benchmarks (e.g., OBQA), there are two main explanations:

1. Alignment Benefits: We agree that the knowledge contained within the model is acquired during the pretraining phase. However, instruction-tuning and alignment can enable the model to better induce and articulate this knowledge in its responses $1,2,3$ . For example, Self-Rewarding $1$ has improved Llama2's performance on Natural Questions.
2. Self-Refinement Gains: Our method involves a self-refinement process, which has been shown to be effective in various tasks $4, 5$ . In most scenarios, verification and refinement can be simpler than direct generation, allowing the model to correct inaccuracies and improve its performance $6,7$ . This process of pre- and post-refinement helps the model enhance its capabilities on these tasks.

Yes, our ablation study detailed in Section 4.2 examines the impact of seed SFT data. The results, presented as "Seed SFT results" in Table 7, demonstrate that Llama3 reaches optimal performance after the second epoch. Additional training epochs did not yield improvements and, at times, even diminished performance. This confirms that the self-refinement process effectively enhances the model's ability to follow instructions and perform tasks, rather than simply benefiting from GPT-4 input.

Again, we sincerely thank you for the constructive comments and positive feedback, which really help us further improve our work. If you have any further questions, we are happy to address them.

Reference:

$1$ Yuan, Weizhe, et al. "Self-rewarding language models." arXiv preprint arXiv:2401.10020 (2024).*

$2$ Chen Z, Deng Y, Yuan H, et al. Self-play fine-tuning converts weak language models to strong language models $J$ . arXiv preprint arXiv:2401.01335, 2024.*

$3$ Yin, Yueqin, et al. "Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment." arXiv preprint arXiv:2405.20830 (2024).*

$4$ Pan, Liangming, et al. "Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies." arXiv preprint arXiv:2308.03188 (2023).*

$5$ Weng, Yixuan, et al. "Large language models are better reasoners with self-verification." arXiv preprint arXiv:2212.09561 (2022).*

$6$ Kamoi R, Zhang Y, Zhang N, et al. When can llms actually correct their own mistakes? a critical survey of self-correction of llms $J$ . Transactions of the Association for Computational Linguistics, 2024, 12: 1417-1440.*

$7$ Madaan, Aman, et al. "Self-refine: Iterative refinement with self-feedback." Advances in Neural Information Processing Systems 36 (2024).*

We sincerely thank Reviewer EKkD for the positive recommendation as well as the valuable suggestions. We really appreciate your kind words that our work is intuitive and has good results. Below we would like to give detailed responses to each of your comments.

Q1: “Has the synthetic preference data been tested for benchmark data leakage? I didn't see anything suggesting that checks for data leakage were done.”

Thank you for this insightful point. As suggested, we conducted experiments to test for benchmark data leakage.

Specifically, we compared the n-grams of our training datasets (seed SFT data, synthetic preference data, and UltraFeedback for reference) with the n-grams of the test set data to identify any overlaps. If any n-gram from a test data entry appears in our dataset n-grams, that entry is marked as leaked, and we calculate the proportion of leaked data for each test dataset. For datasets with candidate answers, we concatenate the question with candidate answers for analysis; for those without candidate answers, we use only the question. Following HELM $1$ , we set the n-gram size to 13.

The results are as follows:

(Benchmark data leakage test for Open LLM Leaderboard tasks. UltraFeedback data for reference.)

(Benchmark data leakage test for instruction-following benchmarks. UltraFeedback data for reference.)

Overall, the overlap between our training datasets (seed data and synthetic preference data) and the test sets is very low, indicating that there is no data leakage issue.

Interestingly, synthetic preference data generally shows even less overlap with the test sets than other data.

We will incorporate these findings into the revised version of our paper. Thank you for highlighting this important aspect, which certainly enhances the robustness of our research.

Q2: “While there's novelty to the pipeline as a whole, the individual components of the pipeline have been known.”

The continuous improvement of LLMs has always been a challenging and complex process $2,3,4$ . In our setting, it mainly involves self-prompt generation, response synthesis, and optimization.

• Self-prompt generation: our self-prompt generation method differentiates itself from previous approaches that rely on seed data or larger models by enabling the model to self-generate high-quality, diverse prompts based on random keywords from pretraining corpora.
• Response synthesis: to the best of our knowledge, we are the first to utilize pre- and post-self-refinement responses to construct synthetic preference pairs.

While we adopt SimPO’s loss function during the optimization phase, the innovations in the first two parts are significant contributions to prompt generation and synthetic preference data.

Q3: Typos

Thank you for your meticulous review and for pointing out the typos. Yes, on L247, the (\sigma) should indeed be (\beta). We will update the mentioned typos in the latest version.

We sincerely thank Reviewer TAJC for the review and are grateful for the time you spent with our submission. We are glad for the acknowledgement that our approach is sound and provides transparency for control and analysis. We wish to address your concerns by giving detailed responses to each of your comments as follows:

Q1: The self-boosting mechanism

Yes, a small amount of seed data is required.

• Compared to "self-improvement," we use the term "self-boosting" to emphasize that the model's continuous iterative enhancement is self-driven (by the self-refinement process and pre- and post-refinement contrast) rather than claiming to be entirely independent. We aim to highlight that, unlike previous works that rely on external tools or teacher LLMs[1][2], SynPO is mainly a process of self-correction and self-driven improvement.
• Additionally, we would like to clarify that the external knowledge required by our method is minimal, similar to the self-rewarding approach[3], including a small amount of seed data.
• Besides, the selection of keywords is entirely automatic (simple rule-based) and does not require careful selection or human effort.

Q2: Concerns and questions about bad noising keywords

There seems to be some misunderstandings on the motivation of noise keywords introduction and our insights in creating the rejected samples.

• First, we would like to clarify that the introduction of noise keywords is merely an optional trick to enhance the robustness of the prompt generator (L155-158). For each prompt $x^*_i$ in the seed data, we randomly extract two keywords from $x^*_i$ and one noise keyword from $x^*_j, j ≠ i$. This ensures that the training data output excludes semantically irrelevant input keywords, guiding the model to generate prompts based on relevant keywords while disregarding unrelated words in the given list. Therefore, this strategy aims to ensure the naturalness and semantic coherence of the generated prompt, not to produce bad samples.
• Since the noise keywords serve only to introduce noise, there is no need for an additional strategy to ensure one keyword is more noisy than another. Simply sampling randomly from the entire keyword list, excluding the current prompt's keywords, is sufficient.

To further demonstrate that noise keywords are an optional strategy in our method, we have conducted experiments to show their impact. We trained and evaluated the Llama3 prompt generator with three random keywords, either including or excluding one noise keyword. We calculated the average similarity across the generated prompts using Sentence-Transformer, as mentioned in Section 2.2, and employed GPT-4 Turbo for LLM-as-a-Judge evaluation of prompt quality (on a scale of 1 to 10, where 1 represents a prompt that is very unrealistic, unnatural, or unanswerable, and 10 represents a prompt that is very reasonable, realistic, and answerable).

(Evaluation run on 1k data for each setting. Each example contains 3 keywords (3 keywords for the 'no noise' setting, 2 keywords and 1 noise keyword for the 'w noise' setting)

The results are listed in the following table. They show that the inclusion of noise keywords improves the quality of the self-generated prompts (as LLMs learn to ignore unrelated words), but even without noise keywords, the self-prompt generator can still generate relatively high-quality prompts.

Q3: Which type of keywords are more effective

As clarified, the noise keywords maintain the same type as the extracted keywords for better robustness. In our implementation, the keywords are randomly selected from rule-based filtered phrases. Specifically, we use the NLTK toolkit to filter out stop words, extract all noun phrases from the sentences, and remove any preceding articles. We then randomly sample from these phrases. The decision to use noun phrases is based on the following preliminary experiments:

(Note: "Words" refer to single-word keywords, while "Phrases" refer to each keyword and can be a phrase comprising 1 to 3 words.)

As shown, using phrases rather than limiting keywords to single words generally results in better diversity and quality. This improvement may be due to phrases incorporating more inductive bias from the pretraining data for prompt generation. Specifically, while random sampling from all phrases or words without filtering is acceptable, it is not as effective as random sampling from nouns, as nouns contain the most important information needed to diversify a sentence.

Beyond these evaluation results, we have also manually observed the impact of different extraction forms and found that using phrases as keywords generally resulted in higher quality prompts. Therefore, our experiments utilize randomly sampled noun phrases. We will include these preliminary experiments, which support our choice of keyword extraction methods, in the Appendix to provide further insights into self-prompt generator training.

Q4: New insights in creating the bad samples
As clarified in Q2, we do not regard prompts with noise keywords and their corresponding responses as bad samples. The synthetic prompts are the same for both chosen and rejected samples. Our main intuition in constructing good and bad samples lies in inducing pre- and post-self-improvement responses as natural rejected and chosen candidates, respectively (Section 2.2). The chosen responses provide clear guidance on what approximates a gold standard response through the iterative self-refinement process. This method of generating synthetic preference is fundamentally different from previous methods that sample multiple responses from a model and then score them as good or bad samples $1$ . We sincerely hope that this clarification addresses your concerns regarding the novelty of our insights.

Q5: Prompt generation capability of less capable LLMs
The prompt generation capability of an LLM can be influenced by the model's scale. However, since our method synthesizes training data through the construction of keywords to prompts, even a 0.5B model can be finetuned to be a good self-prompt generator. To further illustrate this point, we conducted experiments using the Qwen2.5 model with 0.5B and 1.5B parameters:

( Evaluation run on 1k data for each setting )

It can be observed that, consistent with characteristics of LLMs, the prompt generation capability is also affected by the model size. However, even a small model with 0.5B parameters can become a good self-prompt generator after training with our method (generating prompts with high quality and diversity).

On the other hand, we focus on the continuous improvement of LLMs. Thus, self-boosting primarily addresses the shortage of high-quality preference data. Our experiments mainly use 7B and 8B models, which are moderately sized. For very weak small models, further enhancement can be easily achieved by using a stronger teacher model to construct distillation data. Even a slightly larger teacher model can provide adequate supervision at a low cost.

Overall, many thanks for your insightful points and suggestions. These comments really help improve our work. We hope our answers have addressed your concerns. If you have any further questions, we are happy to address them.

Reference:
* $1$ Li, Haoran, et al. "Synthetic data (almost) from scratch: Generalized instruction tuning for language models." arXiv preprint arXiv:2402.13064 (2024).*
* $2$ Shi, Taiwei, Kai Chen, and Jieyu Zhao. "Safer-instruct: Aligning language models with automated preference data." arXiv preprint arXiv:2311.08685 (2023).*
* $3$ Yuan, Weizhe, et al. "Self-rewarding language models." arXiv preprint arXiv:2401.10020 (2024).*