Rule-Based Rating and Selection of LLM Training Data

We appreciate the valuable feedback and below are our responses (all the line numbers and sections are referred to the new pdf version):

1. (a) Paper too complex, and (b) Implementation:

(a) Because our pipeline is entirely automated and end-to-end, and it is a complete pipeline starting from automatic rule generation all the way to the final data selection for downstream tasks, it might look complex to some readers. Nonetheless, the idea is simple, within an LLM-as-a-judge setting (which means using powerful LLMs to rate data quality for selection, and is one of the most popular methods now for quality data selection [1][2][3][4][5][6][7][8][9]), we investigate what quality aspects we should consider for data rating. Our strategy is to first generate a large rule pool to ensure we cover enough aspects, and then remove the correlated ones to make the rules more even/balanced. We proposed the quantitative measure to assess this “correlation”, and used DPP as the algorithm to achieve this selection of uncorrelated rules. We hope this clarifies the essential idea of our method.

(b) Indeed our implementation needs the rating and selection process, but these are inevitable under the LLM-as-a-judge setting [1][2][3][4][5][6][7][8][9]. Our method only adds the rule generation and selection step to standard LLM-as-a-judge implementation.

2. Heavily depends on GPT-4:

For LLM-as-a-judge, powerful LLM such as GPT-4 is often used as the judge [1][2][3][4][5][6][7][8][9]. Moreover, due to efficiency, we use Llama3-8B-Instruct as the rater, which is a lighter model than GPT-4. We only used GPT-4 for rule generation, which is done by only 1 prompt. Our paper focuses on solving the issue within the LLM-as-a-judge setting, which is the challenge of designing and aggregating the rating aspects (rules). When the ratings are available, our DPP selection is extremely fast. For more details on the run time, please refer to the paper (Line 209-215 and 807-818).

If the reviewer worries about the reliance on GPT for rule generation: we have provided the comparison against using Claude as the rule generator in Section A.6.5(Line 988-1018). We use PCA to visualize the embedding (generated using sentence-transformer) of the two sets of rules. It is observed that the two clusters of rules generated by GPT and Claude completely overlap, demonstrating no distinct separation. Additionally, we have further computed the Wasserstein distance as a quantitative evaluation of the distribution differences between GPT rules and Claude rules. Specifically, we compute the distance between the GPT rules and Claude rules. Then we randomly split GPT rules into two parts and computed their Wasserstein distance and similarly for the Claude rules. By comparing these values we found no clear distribution bias when switching from one rule generator to the other. Please see Table 4 (Line 1022-1032 of most recent version).

Moreover, if the reviewer concerns about the general bias brought by LLMs (including GPT and Claude), to address the concern, in Line 1034-1042, compared to human-generated rules, we prompt GPT-4 to generate ethical and safety rules and compare them with the human-designed rules in Collective Constitutional AI[1]. We asked 3 authors who have not seen the constitutions to distinguish the rules blindly. We get an average accuracy of 20.7%, suggesting it is indeed hard to distinguish between rules generated by GPT and those designed by humans. All these discussions underscore the potential of GPT as a reliable rule generator that is capable of producing rules that are comparable to those crafted by human experts. We thank the reviewer for pointing this out and we hope these in some aspect address the reviewer’s concern about GPT-rule-generation bias.

3. Cost of human in the loop:

As mentioned above, the entire LLM-as-a-judge is proposed to replace human raters and has been commonly adopted by researchers. In our case, we further address the bias of the rules designed and aggregated by humans. Our fully automated method eliminates the need for repeated human intervention with each new task, which is not only costly but also difficult to integrate into certain workflows, such as composite reward models in RLHF. Human in the loop is usually more expensive than using automatic models [7][8][9][10]. Moreover, when humans attempt to balance rules to minimize bias, they depend on their semantic understanding, yet the same rules can exhibit different rating correlations across datasets. For example, in QuRating, the designed rules unexpectedly showed high correlations, potentially introducing bias and redundancy in data quality ratings, and obstructing rule customization or re-weighting. In contrast, our method uses to the score representations of rules, which is guaranteed to be data-dependent. As detailed in Section A.7.9 (Lines 1481) and A.7.10 (Table 19, Line1537-1541), we analyzed the correlation of the “uncorrelated” rules generated or selected by GPT and found that their actual rating scores were significantly more correlated than those selected via the DPP method.

In summary, we are grateful to the reviewers for highlighting these points, and we hope our detailed discussion and additional experiments potentially addressed the concerns. We would appreciate it if the reviewers could consider increasing the score based on these updates!

[1] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024

[2] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: harm-lessness from ai feedback. 2022. arXiv preprint arXiv:2212.08073, 2022.

[3] Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.09739, 2024.

[4]Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024.

[5] Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety.

[6] Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorod-sky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796, 2024.

[7] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. In Forty-first International Conference on Machine Learning

[8] Zheng, Lianmin, et al. "Judging llm-as-a-judge with mt-bench and chatbot arena." Advances in Neural Information Processing Systems 36 (2023): 46595-46623.

[9] Huang, Hui, et al. "An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task-specific classifiers." arXiv preprint arXiv:2403.02839 (2024).

[10] Ziegler, Daniel M., et al. "Fine-tuning language models from human preferences." arXiv preprint arXiv:1909.08593 (2019).

We sincerely appreciate the insightful comments provided by Reviewer 9u4d! To address the reviewer's concern, we have conducted additional experiments and added the corresponding analysis and discussions. This is a friendly reminder that the deadline of the rebuttal period is approaching. We would really appreciate it if the reviewer could take a look at our responses at their earliest convenience. We hope that the additional experiments and discussions address the reviewer’s concerns effectively and would be deeply grateful if the reviewer could consider raising the score!

We appreciate the valuable feedback and below are our responses (all the line numbers and sections are referred to the new pdf version):

1. Citation to P3:

We appreciate the reviewer for bringing this paper to our attention. According to the dates, this paper is considered a concurrent work. Moreover, the new version of the Arxiv is three weeks later than our ICLR submission. However, as suggested by the reviewer, we have added the citation and the discussion in Line 818-825 to compare this with our method. The only thing our work shares with theirs is using DPP as the diversity tool. Particularly, the authors in P3 also use DPP to perform data selection, but directly applied to the data itself. Nonetheless, the approach to directly perform data selection using DPP requires the computation based on the kernel matrix with a dimension equal to the number of samples, which is usually huge (could be millions) in the context of LLM data, where for our application of DPP on the rules, the dimension is the number of rules (typically dozens). Moreover, while DPP data selection inherently ensures data diversity, this is just one aspect of data quality. In contrast, our rule-based approach guarantees diversity and balance between the rules, i.e. the rating quality aspects. Our framework assesses multiple aspects of data quality, ensuring a more comprehensive and robust selection process. We think is it helpful to add the citation and add this discussion to compare the crucial differences, and thank the reviewer for this valuable suggestion!

2. (a) Rating prompts with task descriptions, and (b) Optimal r

(a) For the setting of LLM-as-a-judge, a rating prompt is inevitable [1][2][3][4][5][6][7][8][9]. There are no universally standard prompts and our rating prompt is already a refined version of the prompts used in previous works such as Self-Rewarding[1] and Constitutional AI[2], by adding data and task descriptions. We added the clarification in Line 1565 of the paper that the data and task descriptions are generated with the assistance of GPT-4. It is quite common to use human-designed prompts or GPT-generated prompts for LLMs (for example see [1][2][3][4][5]). We appreciate the reviewer for pointing this out!

(b) For the optimal choice of r, we first emphasize that studying the optimal r essentially is the question of how many aspects we should consider for data quality selection, which is a very challenging question. Our experiments in Section 5 (Evaluation B) explore the optimal r (please see Figure 3 and further Figure 7). We observed that optimal r is not near the boundaries but rather around 10-15. This motivates our choice of r=10 in the paper. For fine-tuning experiments, it is impossible for us to have enough GPT resources to investigate exactly the optimal r. However, our experiments of using all 50 rules, and variation of r=20 in A.7.5 (Line 1318-1330), all further confirm our hypothesis that the optimal r is not near the boundaries. We have emphasized the motivation for the choice of r=10 in Line 361.

3. Other rule evaluations:

Thanks for mentioning this insightful question. In fact, evaluating or determining the optimal choice of data quality aspects (rules) is extremely challenging[3][4][5]. As far as we know, all the previous works rely on human-designed rules but struggle with how to design, select, and weigh the rules. We are the first to propose a quantitative criterion for data quality aspects selection. While this criterion may not be comprehensive, it is essential. As discussed in the paper, rule correlations could disrupt balance and evenness, resulting in biased ratings toward certain aspects. In fact, in some domains, such as the IMDB, we have observed that the Pearson correlation of the rule orthogonality with the rule accuracy to be as high as more than 0.6 (please see Figure 6), indicating that the benefits of orthogonality may be greater than anticipated. We concur with the reviewer that this is just a starting point, and a more thorough evaluation of rules is necessary, pointing to future areas of research.

In summary, we thank the reviewer for mentioning the related paper and asking insightful questions! We hope our detailed discussion could help address the concerns and would appreciate if the reviewer could improve the score!

[1] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024

[2] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: harm-lessness from ai feedback. 2022. arXiv preprint arXiv:2212.08073, 2022.

[3] Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.09739, 2024.

[4]Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024.

[5] Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety.

[6] Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorod-sky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796, 2024.

[7] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. In Forty-first International Conference on Machine Learning

[8] Zheng, Lianmin, et al. "Judging llm-as-a-judge with mt-bench and chatbot arena." Advances in Neural Information Processing Systems 36 (2023): 46595-46623.

[9] Huang, Hui, et al. "An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task-specific classifiers." arXiv preprint arXiv:2403.02839 (2024).

We appreciate the valuable feedback and below are our responses (all the line numbers and sections are referred to the new pdf version):

1. Reliance on GPT for rule generation:

We thank the reviewers for the insightful comments! To maintain interpretability, our rules are set to be natural language forms. The reliance on power LLM (and the most common one is GPT) for such text rules is quite reasonable. In order to show that the GPT rules are generally reliable and unbiased, we have provided the comparison against using Claude as the rule generator in Section A.6.5(Line 988-1018). We use PCA to visualize the embedding (generated using sentence-transformer) of the two sets of rules. It is observed that the two clusters of rules generated by GPT and Claude completely overlap, demonstrating no distinct separation. This suggests that GPT-4 functions effectively as an unbiased rule generator. We appreciate the reviewer’s suggestion and hope adding this comparison experiment would help resolve some of the concerns on the bias of GPT.

2. Models within same family:

For previous such as QuRating[1], only one model architecture was used for the experiments. In our case, due to computing resource constraints and the comprehensive scope of our experiments which cover multiple domains and baseline methods, we were not able to repeat the finetuning experiments for the Llama2 family. However, in our case, we considered both Pythia and Llama. Moreover, in experiments in Section 4 (Evaluation A), we have explored the variation in the model size and we included the results for both Llama3-8B and Llama3-70B (please see Line 362-372 and Section A.6.4, Line 951-986). We observe very similar phenomena and this indicates that our results are generally applicable regardless of the model architecture. Currently, due to computing resource constraints and the comprehensive scope of our experiments which cover multiple domains and baseline methods, we are not able to repeat the experiments for further models. For the future version, we will try to add experiments for model architectures such as the Llama2 family.

3. (a) Ablation of DPP, and (b) Compare with GPT 10 uncorrelated rules:

(a) Within our framework, we compared setups using Random 10 rules and DPP 10 rules as part of an ablation study to assess the effectiveness of DPP. Please see Line 354-361.

(b) Following the reviewer’s suggestion, we also tested GPT-generated 10 uncorrelated rules to contrast with rule generation solely by GPT-4. Before, we mentioned in Section A.7.9 (Lines 1343-1409), we analyzed the correlation of these GPT-generated uncorrelated rules and found that, despite claims of being “uncorrelated,” their actual rating scores showed significantly higher correlation compared to the DPP method. This underscores the value of our Rule Correlation measure for quantitatively assessing correlations. We have highlighted this point to our readers in Section 5.3 (Lines 485-491). Now, we added new benchmark results presented in Tables 1-3 (Lines 458-525) confirm that our DPP method outperforms the GPT-Uncorrelated rules. We are grateful to the reviewer for this insightful suggestion!

4. Data distribution:

As suggested, we have included an analysis of the data distribution in Section A.7.7 (Lines 1323-1335 and Figure 10). Using the Code domain as an illustrative example, we demonstrated that DPP tends to select more domain-related data, such as from GitHub and StackExchange.

We are grateful to reviewer 1T3n for these valuable suggestions! We hope that these additional discussions and experiments in the new version will encourage reviewer 1T3n to consider a higher score!

[1] Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.09739, 2024.

1. Reliance on GPT for rule generation:

The selection bias in foundation models might be similar since they are trained on comparable data. However, upon reviewing Section A.6.5 and the PCA visualization (Figure 8), there doesn’t seem to be a clear overlap between the rule clusters produced by GPT and Claude, as described. It would be helpful if authors clarified how they are measuring this overlap and provided additional details on the metrics or methods used to support this conclusion.

2. Models within the same family:

While the Pearson correlation between rule correlation and MSE is not as high as 0.6 as mentioned in the paper(line 362-372 and figure 2), we can still observe that as rule correlation increases, MSE also increases due to the positive Pearson correlation.

3. GPT 10 uncorrelated rules:

It seems the approach taken does not fully align with the proposed method. The authors could consider using GPT-4 to replicate DPP's role by adding a refinement step where GPT-4 filters and selects a subset of r rules that are both highly relevant to the task and diverse. The resulting subset could then serve as a direct baseline for evaluating the effectiveness of DPP in comparison.

4. Data distribution:

DPP tends to select domain-relevant data. To better understand the data distribution, the authors could analyze the distribution of the original dataset and the distribution of selected samples resulting from different selection methods, including DPP. Metrics such as the length of training samples, cross-entropy loss before fine-tuning, and cyclomatic complexity for coding datasets could provide insights.

The authors have addressed my comments partially so I will maintain my current score.

We appreciate the reviewer for the reply and new insights!

1. Reliance on GPT for rule generation:

We have further computed the Wasserstein distance as a quantitative evaluation of the distribution differences. Specifically, we compute the distance between the GPT rules and Claude rules. Then we randomly split GPT rules into two parts and computed their Wasserstein distance and similarly for the Claude rules. By comparing these values we found no clear distribution bias when switching from one rule generator to the other. Please see Table 4 (Line 1022-1032 of the most recent version).

Moreover, to address the concern about the general bias of LLMs, in Line 1034-1042, compared to human-generated rules, we prompt GPT-4 to generate ethical and safety rules and compare them with the human-designed rules in Collective Constitutional AI[1]. We asked 3 authors who have not seen the constitutions to distinguish the rules blindly. We get an average accuracy of 20.7%, suggesting it is indeed hard to distinguish between rules generated by GPT and those designed by humans. All these discussions underscore the potential of GPT as a reliable rule generator that is capable of producing rules that are comparable to those crafted by human experts. We thank the reviewer for pointing this out and we hope this in some aspect addresses the reviewer’s concern (and maybe some readers’ concerns) about rule generation bias.

2. Models within the same family:

We agree that the Pearson correlation varies based on different domains/tasks, and indeed positive Pearson correlation is shown between MSE and RuleCorrelation, indicating that rule correlation indeed reflects the rule quality in certain aspects.

3. GPT 10 uncorrelated rules:

We understand that the reviewer would like to test the exact ability of DPP to select out uncorrelated rules, compared to GPT selection. We have now further added experiments for all the domain fine-tunings. The results are shown in Tables 21, and 22 (Line 1574-1590). We note that it generally underperforms compared to our methods. Moreover, we calculated the rule correlation again, and found it is a bit smaller than random 10 rule selections, but still generally higher than DPP (the rule correlation and rule indices are included in Table 20, Line 1566-1572). We thank the reviewer for this suggestion and our new experiments validate the important role of DPP in rule selection.

4. Data distribution:

We appreciate the detailed suggestions for analyzing the data distribution. We have provided a deeper exploration of the data distribution in A.7.7 (Line 1379-1479). For Code domain, we studied the metadata distribution and it shows DPP selects more GitHub and StackExchange data. (We also studied the text lengths but found no clear pattern for the Code data. Here we did not use the cyclomatic complexity because due to our selection of pretrain data, even if the texts contain some code, they also contain statements or questions.) Moreover, we added an exploration of the IMDB domain. First from the text length distribution, we see DPP generally selects longer data. Then we studied the bigram entropy distribution, which indicates diversity of texts, and found that DPP selects texts with higher entropy/diversity.

Again we thank the reviewer for the active discussion and insightful suggestions! We hope these new experiments would help address the rest of the reviews’s concerns and we would appreciate a raise of the score!

[1] Saffron Huang, Divya Siddarth, Liane Lovitt, Thomas I Liao, Esin Durmus, Alex Tamkin, and Deep Ganguli. Collective constitutional ai: Aligning a language model with public input. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 1395–1417, 2024.

We appreciate the valuable feedback and below are our responses (all the line numbers and sections are referring to the new pdf version):

1.Automatic v.s. Manual rule selection:

Using 50 rules is simply due to GPU resource constraints. We prioritized demonstrating the general applicability of our method by covering many domains. In fact, there are potentially hundreds of aspects to evaluate data quality[1][6] (for example [6] uses 133 rules). Also, our completely automated method is an advantage compared to rule selection by humans, where human intervention is needed again for each new task, making it costly and challenging to integrate into certain pipelines, such as composite reward models in RLHF. Moreover, varying preferences among individuals can inevitably introduce bias during rule selection. Additionally, human selection of uncorrelated rules is generally based on the rules' semantic meanings, but rules can show different rating correlations on different datasets, resulting in bias and redundancy on data rating and also hindering customization or re-weight of rules. In contrast, we choose to select orthogonal rules based on their score representations, which are guaranteed to be data-dependent. We have added a discussion in Section 5.3 (Line 485-491). We thank the reviewer for pointing this out!

2. (a) Performance improvement and (b) Multiple trials:

(a) Our GPU resources are not as extensive as groups like QuRating, which are able to pretrain LLMs. In contrast, we perform our fine-tuning with only 20K samples, yet we still achieve significant improvements and superior performance over baselines. For instance, in Medical domain, our DPP method improved the average accuracy of Pythia by about 5%, and it outperformed other baselines, such as achieving a 7.2% increase in accuracy over DSIR[5].

(b) We added the results to be 3 trials for Uniform Sampling and Random 10 Rules and reported their mean and standard deviation in Section A.7.3 (Line 1198 - 1227). Due to limited GPU resources, we were unable to perform multiple trials for all experiments, but it is quite common in LLM research to report one trial now, for example see [1][3][4].

3. More baselines: We have invested significant efforts to implement the suggested methods. Due to the time-consuming nature and unreliable codes of some methods, we eventually reproduced LESS and DiverseEvol. We added their results for all the 5 settings in Table 7-9 (Line 1163-1195) to further validate the advantages of our methods. We appreciate the reviewer’s suggestion and mention of these baseline methods!

4. GPT 10 uncorrelated rules:

We thank the reviewer for this great suggestion! The experiments for GPT 10 uncorrelated rules for all 5 settings are now added to Table 1-3 (Line 456-525). By comparing against this, the advantage of our methods is more obvious. Moreover, this is directly prompting GPT to generate 10 uncorrelated rules. We also added experiments to prompt GPT to select 10 uncorrelated rules from a 50-rule pool. Please see A.7.10 (Line 1528-1559) for the results.

5. (a) All 50 rules v.s. DPP 10 rules, and (b) Data distribution:

(a) In Section 4, we have demonstrated multiple plots showing that the optimal r is not near the boundaries (Figures 3c, 3f). The non-monotonicity in r is itself an interesting result reported by our paper. We conjecture that orthogonality is important: larger r covers more aspects but may introduce correlations that disrupt the balance or evenness, causing the rating biased toward certain aspects. When we mention the “diversity” brought by orthogonality, we mean under the condition of balance, or essentially a “balanced diversity” (Line 496). We appreciate the reviewer posting such an insightful question which highlights an interesting crucial part of our work. In fact, one of the contributions of our paper is to explore the reason behind the optimal r using rule correlation/orthogonality, proposing quantitative metrics and various experiments to support our hypothesis. We have added some discussion about these in Line 493-499.

(b) We have examined the data distribution and added discussion in Section A.7.7 (Lines 1352-1450 and Figure 10,11,12), using the Code and IMDB domains as illustrative examples. We demonstrated that DPP tends to select more domain-related data, longer texts, and texts with higher diversity (quantified using bigram entropy).

In summary, we appreciate reviewer 94XN for so many insightful questions and suggestions! We hope that our additional experiments and discussion bring more clarification to the concerns and would really appreciate if reviewer 94XN could raise the score.

[1] Together AI. Red pajama. Blog post on Together AI, 2023. https://www.together.ai/ blog/redpajama.

[2] Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.09739, 2024.

[3] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024

[4]Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024.

[5] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36, 2024.

[6] Huang, Saffron, et al. "Collective Constitutional AI: Aligning a Language Model with Public Input." The 2024 ACM Conference on Fairness, Accountability, and Transparency. 2024.

We sincerely appreciate the insightful comments provided by Reviewer 94XN! Following the reviewer’s valuable suggestions, we have conducted the recommended experiments and added the corresponding discussions. This is a friendly reminder that the deadline of the rebuttal period is approaching. We would really appreciate it if the reviewer could take a look at our responses at their earliest convenience. We hope that the additional experiments and discussions address the reviewer’s concerns effectively and would be deeply grateful if the reviewer could consider raising the score.