Self-Alignment Optimization for Language Models

Thank you for your valuable feedback. We would like to address your concerns as follows:

Regarding the novelty of the method and its relation to RLAIF:

We respectfully disagree with the opinion that SAO is simply an extension of the existing RLAIF literature. The primary novelty of our method lies in the introduction of a new self-improvement framework that aligns LLMs with human performance, without the need for external datasets, human labor, or larger reward models that are generally used in RLAIF framework. This makes our approach highly cost-efficient and resource-friendly. As you noted in your "Strength" comment, this approach eliminates the need to collect external feedback or rely on external tools, which is a significant step forward in self-improvement for LLMs.

---

Regarding the Persona-based prompt space generation and comparison to existing prompts (e.g., UltraFeedback):

Thank you for suggesting this comparison. To explore the influence of persona role-playing, we included an ablation study in Section 5.4.3 (Figure 3e). Without persona-based generation, even with more repetitive prompts, the model still maintains performance. Additionally, we performed further experiments comparing different prompt sources using the Gemma2-9B-it model with a 10k dataset, shown in the table below:

This experiment demonstrates that directly using existing prompts (such as those from UltraFeedback, which include objective questions like math and code) can hinder alignment performance. We believe this happens because objective questions might lead the model in the wrong direction, whereas persona- and random-generated prompts (which are subjective) better align the model with human preferences.

---

Regarding the analysis of self-judge portion:

We appreciate your suggestion to compare self-judging against external reward models. However, in Section 5.4.4 (Figure 3f), we have compared self-judging with random judgment and an external reward model (ArmoRM). Our results show that self-judging is more effective at identifying the correct direction for preference optimization.

---

Regarding the performance difference between SAO on Llama-3 and Gemma-2:

The observed difference in performance between Llama-3 and Gemma-2 is due to the fact that the prompt, response, and judgment processes differ for each model within the SAO framework. A more "intelligent" model, like Gemma-2, is better equipped to conduct self-improvement, which may explain the better results seen with this model. Additionally, we believe that larger models (≥70B) have further potential in this self-improvement process. As for LLM-as-a-judge accuracy, since we focus on subjective prompts (e.g., preferences and opinions) rather than objective tasks like math or code, it’s difficult to define "accuracy" in the traditional sense. The effectiveness of the judge is more about guiding the preference optimization process rather than strict correctness.

---

We hope these clarifications address your concerns or misunderstandings. If you have any further questions or concerns, we would be happy to discuss them in the rebuttal stage.

Thank you for the response. The comparison against UltraFeedback prompts makes the person-based generation much more persuasive and convincing

However, the experiment setup for self-judge is still unclear, and the claim that the vanilla prompting result is much better than a well-trained reward classifier "ArmoRM" is quite difficult to believe without extensive study of the reward classifying performance. Like what are the areas the self-judge worked better than ArmoRM whcih resulted in the improved performance in the current setup.

The result of paper is promising, but the idea and scientific contribution of the paper needs better scoping. So, I tend to hold my current rating.

Thank you for your valuable feedback. We would like to address your concerns as follows:

Regarding the dependence on initial model performance for self-alignment (small LLMs):

We understand your concern about the generalization of the SAO framework to smaller models (1B to 3B parameters). To address this, we conducted additional experiments with the Llama-3.2-Instruct-3B model, and the results are shown below:

These results show consistent improvement with the SAO framework, even with smaller models. However, we acknowledge that larger and more capable base models tend to yield better performance, which leaves further potential for larger models (≥70B). This highlights the scalability of the SAO framework as we move to more powerful models.

---

Regarding challenges with output consistency in self-supervision:

Thank you for raising this important concern. We understand that output consistency is crucial for the success of the self-alignment process. To address instances where the model's output deviates from the desired format, we use regular expressions to extract the relevant output structure, as demonstrated in the code provided in our supplementary material. From our observations, almost all data pairs can be successfully extracted, and we encourage reviewers to run the provided code to verify this process.

---

Regarding the effectiveness of persona-based prompt generation compared with existing prompts:

Thank you for suggesting this comparison. To assess the influence of persona-style prompt generation, we included an ablation study in Section 5.4.3 (Figure 3e). The study shows that, even without persona-based generation, the model maintains a reasonable level of performance. Additionally, we conducted further experiments comparing different prompt sources using the Gemma2-9B-it model with a 10k dataset, as shown in the table below:

These results suggest that using persona-based prompts yields better alignment performance than using existing prompts like those from UltraFeedback, which include objective questions like math and code that may lead the model in the wrong direction using self-generation for optimization.

---

Regarding the use of different prompts for MT-Bench and AlpacaEval:

We follow the official evaluation scripts and prompts from the FastChat and AlpacaEval GitHub repositories, which are the most current works in the field like SimPO(NeurIPS24). This ensures consistency with standard evaluation practices and enables fair comparison to previous methods.

---

We hope these clarifications address your concerns. If you have any further questions or suggestions, we would be happy to discuss them in the rebuttal stage!

---

Thank you for your feedback. We would like to clarify some points as follows:

Regarding “the variation of existing approaches” of the paper and several related works

We appreciate your comments regarding the novelty of our work. However, we respectfully disagree with the suggestion that our proposed method, SAO, is merely a variation of existing approaches such as Self-Rewarding LMs. The overall workflow of SAO is significantly different, particularly with components like persona-based diverse prompt generation and pairwise self-judgment. Specifically, Self-Rewarding LMs rely on external datasets as seed datasets and require a few-shot setting to generate prompts. In contrast, SAO is fully dataset-free and annotation-free, which means it does not require external data for initialization or labeling. As shown in the Self-Rewarding LMs paper, their method performs poorly with only synthetic data, and combining it with external datasets yields better results. This highlights the distinction between our work and theirs.

We also thank you for pointing out relevant prior works such as SALMON, OAIF, and SAMI. We will include a discussion of these works in the Related Work section to better highlight how SAO differentiates itself. However, to better demonstrate the effectiveness of SAO, we focused on comparing it with modern baselines (e.g., Gemma-2-9B-it-SimPO and Gemma-2-9B-it-SPPO) in our main tables.

---

Regarding the use of GPT-4o-mini as a judge model for Arena-Hard and MT-Bench:

We appreciate your concern about using GPT-4o-mini as a judge model for Arena-Hard and MT-Bench. The primary goal of using GPT-4o-mini was to evaluate the models in a cost-effective manner. GPT-4o-mini has demonstrated reasonable judgment capability in many existing works. To address your concern, we also evaluated Arena-Hard using GPT-4-Turbo as a judge, and the results show consistent improvement: Gemma-2-9B-it: 40.8 vs. Gemma-2-9B-it-SAO: 54.3. We will include these results in the appendix for further clarification.

---

Regarding the reported improvements on the Open LLM Leaderboard:

We understand your concern that the reported improvements on the Open LLM Leaderboard appear minimal. However, we want to clarify that the primary objective of SAO is not to significantly improve downstream NLP performance but to enhance alignment performance without decreasing the performance of the model.

When comparing SAO to Gemma-9B-it-SimPO, SAO demonstrates comparable alignment performance while maintaining or slightly improving downstream task performance, as shown in Table 2. In contrast, Gemma-9B-it-SimPO, which relies on an external labeled dataset, sacrifices downstream performance for alignment improvements. This trade-off is a critical consideration for real-world applications, and SAO addresses this issue by maintaining performance across tasks.

---

Regarding the primary novel contribution of the work:

To clarify, the primary novel contribution of SAO is that it introduces a dataset-free and annotation-free framework for self-improvement in alignment, which incorporates persona-based prompt generation and pairwise ranking annotation-free approaches—both of which, to the best of our knowledge, have not been proposed previously in the field. The SAO framework can be applied to various preference optimization methods, such as SimPO, and achieves promising performance comparable to models trained on external datasets, while maintaining downstream task performance. This framework enhances the model’s intrinsic capabilities and ensures consistent performance across a wide range of tasks without significant trade-offs.

---

We hope these clarifications address your concerns and provide a clearer understanding of our work.

---

Thank you for your thoughtful comments. We would like to address some of your concerns and misunderstandings as follows:

Regarding the ability of LLMs to self-improve without external supervision:

We respectfully disagree with the assertion that LLMs cannot self-improve without external supervision. In fact, recent work, such as (1) boosting reward models with preference-conditional multi-aspect synthetic data generation and (2) Self-Rewarding Language Models, demonstrates that LLMs can indeed leverage self-generated signals for self-improvement. Our approach builds on this foundation, showing that self-improvement can be achieved through self-supervised signals rather than relying solely on external supervision.

---

Regarding the simplicity of the idea and the surprising improvements on AlpacaEval 2.0 and Arena-Hard:

While the core concept may appear simple, we believe the effectiveness of the SAO framework lies in its novel combination of dataset-free and annotation-free alignment techniques, which have not been fully explored before. The improvements seen on AlpacaEval 2.0 and Arena-Hard are significant, and we have provided a detailed analysis in Section 5.4 to explain the factors contributing to these results. This includes the influence of dataset size, different preference optimization algorithms, persona effectiveness, and self-judgment ability.

---

Regarding the experimental results and improvements on AlpacaEval 2.0:

We have included a detailed explanation in lines 427-431 to clarify this point. Our primary goal was not to generate task-specific data pairs (such as for math or code), but rather to maintain downstream NLP performance while improving alignment ability. While models like Gemma-2-9B-it-SimPO, which are trained on specific datasets, sacrifice some downstream performance to achieve better alignment, our method offers a more balanced approach. SAO demonstrates proactive value by improving alignment without sacrificing downstream task performance.

---

Regarding the use of few-shot settings in Table 2:

We follow the official few-shot evaluation settings used in the OpenLLM leaderboard, which are the standard in the field and have been adopted by works like SimPO (NeurIPS 2024). We understand that you are suggesting zero-shot settings; however, we believe that comparing with the few-shot setting is more representative of current practices, and we wanted to align with the prevailing methodologies used in the evaluation of models.

---

Regarding the need for larger LLMs (at least 70B) to support the claims of this paper:

While we acknowledge that larger models can offer potential benefits, it is important to note that in academic research, especially in alignment, working with models of less than 10B parameters is more common due to the constraints of training large-scale models. Despite this, the performance we demonstrate with smaller models has been shown to outperform some of the most advanced models, such as GPT-4. This highlights the practical value of our approach, which can achieve competitive alignment performance without the need for enormous model sizes.

---

We hope that these clarifications address your misunderstandings and concerns and provide a clearer understanding of our contributions.

---

---

Thank you for your constructive feedback. We would like to clarify some points as follows:

Regarding the novelty of SAO's components:

We appreciate your feedback. The motivation of our paper is to propose a novel self-alignment framework that is dataset-free and annotation-free. While we respectfully disagree with the assessment of novelty in terms of individual components, we also provide a detailed analysis of each in Section 5.4. Specifically:

• Persona-based prompt generation: As shown in Section 5.4.3, even without persona role-playing (i.e., using more repetitive prompts), the model still shows significant improvement based on the SAO framework. However, incorporating persona role-playing allows for more diverse prompts, which further enhances alignment performance.
• Self-judgment: While we adopt criteria from Shen et al. (2024) (shown in Fig.1), we introduce a novel pairwise self-judgment method in Section 5.4.4, where our results show that our model outperforms baselines like Random-Judge and ArmoRM-Judge.
• Preference optimization: As demonstrated in Section 5.4.2, SAO is flexible in improving vanilla models using various optimization algorithms, including DPO, ORPO, and SimPO, with SimPO providing the best results, which we adopt as the default in our framework.

---

Regarding the performance improvement compared to direct counterparts trained on external datasets (e.g., Gemma-9B-it-SimPO):

We understand your concern regarding the quality difference between self-generated datasets and externally labeled datasets. While human-level labels and curated datasets do have better alignment potential, as is commonly acknowledged, our paper introduces a self-improvement framework that operates without external signals. Therefore, it is more intuitive and reasonable to compare SAO against vanilla models (e.g., Gemma-2-9B-it), as the model improves itself in this case. When comparing our SAO method with Gemma-9B-it-SimPO, SAO demonstrates comparable alignment performance, while maintaining or slightly improving downstream task performance, as shown in Table 2. In contrast, Gemma-9B-it-SimPO, which relies on an external labeled dataset, sacrifices downstream performance for alignment improvements—this is a crucial consideration for practical applications. Additionally, as SAO is still in an emerging research area, we have included the limitations of our framework in Section 7, where we acknowledge that although our approach has shown promising results with simple prompt templates, exploring more complex templates may lead to further improvements.

---

Regarding the practical impact of being dataset-free:

We acknowledge the importance of improving RLHF pipelines; however, the primary focus of SAO is data-center efficiency during the pre-RLHF stage, where it eliminates the need for expensive human labeling or GPT-4-level labor and filtering processes. We believe this approach makes a significant contribution to the field of self-improvement through synthetic data. It offers a new perspective on how self-generated data can enhance LLM alignment with minimal external input. The importance of self-synthetic data is also highlighted in the works we compare to, such as Self-Rewarding and SPPO.

---

Regarding efficiency and cost concerns:

Thank you for raising this important point. We would like to clarify that SAO does not introduce additional training costs compared to existing RLHF methods (e.g., SimPO). Our focus is primarily on training dataset construction rather than increasing training costs. The techniques you mentioned like prompting, representation editing, and reward-guided test-time decoding are aimed at improving the decoding stage and are not directly comparable to SAO. These methods generally focus on training-free approaches, which still lag behind in performance compared to SAO and other mainstream RLHF methods. Furthermore, these decoding-optimation models are not as easily deployable as Hugging Face models.

---

We hope these clarifications address your concerns and provide a clearer understanding of our work! Please feel free to reach out with any further questions during the rebuttal stage.