WizardArena: Post-training Large Language Models via Simulated Offline Chatbot Arena

Thank you for your meticulous review and insightful questions and the time you spent reviewing our work. We sincerely apologize for any confusion this may have caused. In the revised version of the paper, we will provide more detailed and clearer implementation in Sections 3 and 4. Below, we will address the specific issues you have highlighted:

Weaknesses-1. Regarding the data construction:

Question-1.1: The data processing steps need to be clarified. For instance, what are illegal conversations?

We employed the same method as LMSYS-Chat-1M to filter out illegal conversations. Specifically, we utilized the OpenAI Moderation API [1] to tag all conversations. If the model classifies the content as potentially harmful, the tag is set to True. If any instruction within a conversation is marked as unsafe by the model, we categorize the entire conversation as illegal and subsequently filter it out.

[1] https://platform.openai.com/docs/guides/moderation

Question-1.2: There might be redundancy when constructing the Diverse Subset and Hard Subset.

Thank you very much for raising these questions. In the revised version of the paper, we will combine the sections on constructing the diversity and hard test sets in Sections 3 and 4 into a single section, and provide a more detailed and clearer descriptions.

Question-1.3: How were the different stages of data (20.4K, 19.3K, 17.6K) obtained, as mentioned in lines 248-250?

For the i-th round of DPO data, the four models—WizardLM-SFT-Ii, Command R+, Qwen1.5-72B-Chat, and OpenChat-3.5—engaged in head-to-head pairwise battles. The response with the highest number of wins was designated as "Choose," while the one with the fewest wins was labeled as "Reject." Data where the gap between the Choose and Reject scores fell below the threshold K (K ≤ 1) were filtered out. Meanwhile, as the WizardLM-SFT-Ii model evolved and became more powerful, approximately 10k to 14k of the initial 30k seed data were eventually removed in each round. This process resulted in three rounds of DPO training data pairs(D2 20.4k, D5 19.3k, D8 17.6k).

Question-1.4: During the construction of the pair-judge training set, did all four models participate in generating outputs for each candidate training sample? And were the best outputs obtained from comparisons among the four models?

No. During the SFT stage, data from battles where WizardLM-β-Ii was defeated by competing models were selected, and the best-performing response from the winning model was chosen as the final response. For DPO and the Reward Model, pairwise battles were conducted among WizardLM-β-SFT-Ii, Command R+, Qwen1.5-72B-Chat, and OpenChat-3.5. The response from the model with the highest number of wins was designated as "Chosen", while the response with the fewest wins was labeled as "Rejected", thereby constructing the <choose, reject> data pair.

Weaknesses-2. Regarding the data construction:

Question-2.1: The data in Table 2 needs to be clarified regarding its source model. How many models’ results were compared (are the 15 models from Table 1 involved)?

Thank you very much for your careful review. The consistency metrics in Table 2 are calculated based on the results of the 15 models listed in Table 1 across LMSYS-ChatBot-Arena, MT-Bench, WizardArena, and Arena-Hard-V1.0. Future revisions of the paper will will emphasize the source of these models in detail.

Question-2.2: The effects of DPO and PPO are similar, as shown in Table 1 and Figure 5. Are their differences limited to the use of different reinforcement learning algorithms? In Figure 1, PPO seems to involve more rankings.

Yes. In our study, DPO and PPO achieved similar performance on both WizardArena and MT-Bench. As illustrated in Figure 2, PPO was involved in more ranking processes, primarily for two aspects: We apply PPO for post-training on the basis of SFT and SFT+DPO. And DPO is only trained on the basis of SFT.

Weaknesses-3. Regarding the analysis:

Question-3.1: The comparison in Table 3 is unfair (e.g., IFD and INSTAG) ...

We greatly appreciate your insightful observations and valuable questions. To ensure fairness in comparison, the Pair-judge method in our paper involved only battles between WizardLM-β-SFT-I0 and Command R+ as the reference model, focusing on data where WizardLM-β-SFT-I0 was defeated, without incorporating information from other advanced models. Both IFD and INSTAG methods selected instructions based on calculated instruction complexity and utilized the corresponding responses from Command R+. Thus, the responses for Pair-judge battles, IFD, and INSTAG were all derived from Command R+. Consequently, this ensures that the comparison of different data selection methods in Table 5 is fair. We will further emphasize this in future versions of the paper.

Question-3.2: Table 4 shows the consistency between GPT-4 and Llama3-70 as the judge, which is good. However, it does not provide a complete comparison of all models; are the rest of the models consistent with the models displayed?

We greatly appreciate your valuable suggestion. The table 9 in newly uploaded WizardArena_rebuttal PDF presents the ELO rankings for 16 models evaluated using Llama3-70B-Chat and GPT-4 as judge models in WizardArena-Mix. Using GPT-4 judge's ELO as the reference benchmark, the Spearman correlation coefficient between Llama3-70B-Chat judge and GPT-4 judge is 97.42%, and the Human Agreement with 95% CI is 95.58%. The overall average consistency between the two judge models is 96.50%. Consequently, employing Llama3-70B-Chat as a cost-effective evaluation model achieves high consistency with GPT-4, ensuring the reliability of the evaluation and training with WizardArena.

Question-1.5: the lmsys-chat-1m includes both single-turn and multi-turn dialogues, but the handling process for multi-turn dialogues, including Arena testing data construction, is not clear.

We describe the processing of multi-turn dialogue data in detail involving two aspects:

• Testing Data Construction:

  For the diversity test set, we concatenated all instructions into a single string from the multi-turn dialogues in LMSYS-Chat-1M. MinHashLSH was employed for deduplication, followed by generating 1024-dimensional text embeddings using the gte-large-en-v1.5 model. These embeddings were then reduced to two dimensions using T-SNE and clustered into 500 categories via the K-Means algorithm. Two samples were selected from each category to construct the 1k diversity test set.

  For the hard test set, we used GPT-4-1106-preview to score each instruction on a scale of 1 to 10 in the multi-turn dialogues, based on the scoring prompt detailed in Appendix B. The scores were averaged to determine the overall difficulty of the multi-turn dialogues. Dialogues were then ranked by their average difficulty scores, and the top 1000 most challenging dialogues were selected as the hard test set.

• Training Data Construction:

  For constructing the SFT, DPO, and Reward Model datasets, Llama3-70B-Chat was used to score each instruction in the multi-turn dialogues, and these scores were then averaged. Dialogues where WizardLM-β underperformed were included in the SFT training set. The highest and lowest average scoring dialogues were used to construct the <Choose, Reject> pairs for DPO and the Reward Model.

• Evaluation Phase:

  Llama3-70B-Chat was used to score each instruction in the multi-turn dialogues in WizardArena. These scores were averaged to reflect the model's overall performance across the dialogues, which was then compared to other models to calculate the ELO score.

Dear Reviewer B73Q,

We would like to thank you for your detailed reviews. We genuinely appreciate the time and thoughtful consideration you have dedicated to our work.

Since the discussion period is coming , we would be grateful if you could let us know whether our response has addressed your concerns or if you still have any other questions. We are looking forward to receiving your any additional feedback you may have.

We would be happy to do any follow-up discussion or address any additional comments. Thank you once again for your valuable contributions to our work.

Respectfully,

Paper 9817 Authors.

Dear Reviewer, we thank you for your valuable comments and the time you spent reviewing our work!

Please find below a detailed discussion of the points you have raised:

Weaknesses-1: The iterative training data generation needs multiple training and generation rounds... It can be costly and hard to tune, e.g. how to choose training algorithms, the reference model, the generation rounds, etc.

Thank you very much for your valuable questions. Below, I will provide detailed answers concerning the choose of training strategies, reference models, and the number of generated rounds.

1. training algorithms choose:

Table 5 (lines 316-328) of our paper discusses the selection of training strategies for the first round of three batches of data. In the updated table 4 in newly uploaded WizardArena_rebuttal PDF, initially continuing with DPO or PPO training based on SFT resulted in significant improvements in the WizardArena-Mix ELO score by 135 and 142 points, and in MT-bench scores by 0.37 and 0.31 points, respectively, outperforming the SFT-only training (SFT+SFT).

Further training with PPO after SFT+DPO yielded a modest 0.05-point increase on the MT-bench, but a notable 21-point improvement in the WizardArena-Mix ELO score. Specifically, on WizardArena-Mix, the SFT+DPO+PPO strategy outperformed the SFT+SFT+SFT, SFT+SFT+DPO, SFT+SFT+PPO, SFT+DPO+DPO, and SFT+PPO+DPO strategies, with ELO score increases of 44, 15, 12, 10, and 6 points, and MT-Bench score improvements of 0.12, 0.08, 0.07, 0.02, and 0.06, respectively.

Therefore, employing the SFT+DPO+PPO training strategy in iterative training achieved relatively optimal results for the first round of three batches of data. It shows that continuously applying RL training strategies on top of SFT can further enhance the model's intrinsic capabilities. Consequently, our study adopted the SFT+DPO+PPO training strategy in each round for iterative training.

2. reference model choose:

The core idea of our paper is to enhance the weaker aspects of the WizardLM-β model by learning from the strengths of different reference models through the Judge-Pair Battle method. If a model’s generated response is selected as training data following the Judge-pair Battle with other reference models and subsequently improves WizardLM-β's performance, then that model can be considered a reference model.

For instance,in the first round of SFT data, OpenChat-3.5 contributed 4.3k data, Qwen1.5-72B-Chat provided 7.1k data, and CommandR+ contributed 8.6k data. Generally, larger model capacities correspond to a greater proportion of the data. Additionally, Table 5 in newly uploaded WizardArena_rebuttal PDF demonstrates that adding Qwen1.5-72B-Chat and OpenChat-3.5 on top of Command R+ significantly enhances WizardLM-β's performance in WizardArena-Mix (ELO +32). Therefore, the choice of CommandR+, Qwen1.5-72B-Chat, and OpenChat-3.5 as reference models in our study is both reasonable and effective.

3. The generation rounds choose:

Our study conducted a total of three rounds of iterative training. By the fourth round, the improvement in WizardLM-β’s ELO score on WizardArena began to slow growth, approaching performance saturation. As shown in the table 6 in newly uploaded WizardArena_rebuttal PDF, WizardLM-β-I4’s ELO score in WizardArena-Mix increased by only 7 points compared to WizardLM-β-I3. This is primarily because WizardLM-β-I3 had already evolved to a high level of performance, approaching the capabilities of Command R+ and Qwen1.5-72B-Chat, which resulted in a gradual decrease in the availability of effective training data. Therefore, our paper employed three rounds of iterative training.

Weaknesses-2: Since the method uses a powerful LLM as the judge (Llama3-70b), it is questionable whether the judge can still correctly evaluate models that are more capable than the judge. This can affect the evaluation and the training data generation when we want to scale the proposed framework to obtain more powerful LLMs.

The following discussion elaborates on the use of Llama3-70B-Chat as the judge model to evaluate more capable models than the judge from three perspectives:

1. Due to time constraints and the cost of manual annotation, we randomly selected a subset of the WizardArena-Mix, comprising 50 diverse samples and 50 hard samples, totaling 100 test cases. We chose four models: WizardLM-β-PPO-I3, GPT-4o, GPT-4-1106-Preview, and Claude 3.5 Sonnet, with WizardLM-β-PPO-I3 serving as the Battle model and the others as reference models. We used Llama3-70B-Chat and a human annotator for anonymous evaluations. The results, presented in Table 7 in newly uploaded WizardArena_rebuttal PDF. It demonstrates a high level of consistency between the outcomes of Llama3-70B-Chat and human evaluations.

2. As shown in Table 2 in newly uploaded WizardArena_rebuttal PDF, we used more advanced models (i.e., GPT-4o, GPT-4-1106-Preview, Claude 3.5 Sonnet) into WizardArena and conducted Battles with other models, using Llama3-70B-Chat as the Judge Model to calculate ELO scores. The rankings align closely with those observed in the LMSYS ChatBot Arena.

3. Learning from more advanced models during training stages using Llama3-70B-Chat as the judge model. Table 8 in newly uploaded WizardArena_rebuttal PDF illustrates the performance impact of utilizing more advanced models in battles against WizardLM-β-7B when Llama3-70B-Chat is used as the judge model. Leveraging the $M_1$ models improved the ELO score from 875 to 1265, outperform battling with the $M_0$ models

In conclusion, Llama3-70B-Chat has proven to be a highly reliable and scalable tool for evaluating more powerful models, making it an effective solution for advanced model assessment scenarios. Therefore, Llama3-70B-Chat as the judge model can correctly evaluate models that are more capable than the judge.

Dear Reviewer tJho,

We would like to thank you for your detailed reviews. We genuinely appreciate the time and thoughtful consideration you have dedicated to our work.

Since the discussion period is coming , we would be grateful if you could let us know whether our response has addressed your concerns or if you still have any other questions. We are looking forward to receiving your any additional feedback you may have.

We would be happy to do any follow-up discussion or address any additional comments. Thank you once again for your valuable contributions to our work.

Respectfully,

Paper 9817 Authors.

Dear Reviewer, thank you for your valuable comments and the time spent reviewing our work! Your feedback is invaluable for improving the quality and competitiveness of our paper.

Please find below a detailed discussion of the points you have raised:

Weaknesses-1: The approach of this work is to create pairs of data for reinforcement learning using a game of multiple models. However, similar work, such as HH data, dedicated extensive sections to explaining the data generation process... This is lacking in this paper.

The primary distinctions between our study and the HH Data approach are as follows:

1. Our study employs open-source models(i.e.,Llama3-70B-Chat) as the Judge model. The Judge-pair Battle method is used to select high-quality data where WizardLM-β underperforms, which then serves as the training data for SFT . Additionally, preference data pairs are constructed for DPO and Reward Model training. In contrast, HH Data depends on crowdsourcing for manual annotation, which is time-consuming, costly, and lacks scalability. Our method is more cost-effective, scalable, and efficient.

2. Our study introduces an offline WizardArena, which closely aligns the online LMSYS ChatBot Arena. The Judge-pair Battle method is employed to identify high-quality SFT training data, making it inherently suitable for generating high-quality preference data pairs and prompting iterative SFT-DPO-PPO training. Conversely, HH Data utilizes the Reject Sampling strategy, where the model produces multiple responses, and collects human feedback data for preference modeling. Then, RLHF is applied to train a relatively helpful and harmless AI assistant.

Further versions of our paper will provide a more detailed description of the response generation process and sample analysis and will cite and discuss HH data in more detail.

Weaknesses-2: The paper mainly focuses on the evaluation capability of the model fine-tuned on synthetic data... However, it lacks a comparison with GPT-4, which would have relevant and valuable for comprehensive evaluation.

We sincerely appreciate your valuable suggestions. The updated Table 2 in newly uploaded WizardArena_rebuttal PDF below presents a comparative analysis of WizardLM-β's performance against other prominent models, including GPT-4o, GPT-4-1106-Preview, and GPT-4-0613. The results show that GPT-4o achieves the highest rank in WizardArena-Mix with an ELO score of 1397, holding its superior performance. However, as WizardLM-β performs the SFT-DPO-PPO iterative training via the Judge-Pair Battle method, the performance gap between WizardLM-β and GPT-4o progressively narrowed.

Specifically, the ELO score gap decreased from 524 points (873 vs. 1397) to 123 points (1274 vs. 1397), and eventually to a 60-point difference compared to GPT-4-0613. It shows that iterative training through the Judge-pair Battle method can markedly enhance a model’s ability to manage hard and diverse tasks, while steadily improving its performance in weaker areas. ( Note: The WizardArena-Mix ELO scores may exhibit slight fluctuations as the addition of new models.)

Question-1: In your study, you have chosen to evaluate the performance of a fine-tuned Llama 70B model. Have you considered including a comparative analysis with the GPT-4 model to enhance the credibility of your findings?

Yes. In Table 4 in our paper (lines 309-315), we present the evaluation results of using GPT-4 as the judge model in WizardArena. Taking LMSYS ChatBot Arena as a reference benchmark, GPT-4 as the judge model in WizardArena achieves a Spearman coefficient of 95.81%, the Human Agreement (95% CI) of 95.65%, the Differentiation (95% CI) of 86.84%, and an overall average consistency of 92.77%. When Llama3-70B-Chat is employed as the judge model, the Spearman coefficient is 98.32%, the Human Agreement (95% CI) is 96.54%, the Differentiation (95% CI) is 83.18%, and the overall consistency is 92.68%. These findings indicate that WizardArena exhibits a high level of consistency with LMSYS ChatBot Arena, thereby ensuring the reliability and accuracy of the offline WizardArena.

Furthermore, when using GPT-4 and Llama3-70B-Chat as the judge model in WizardArena, the Spearman coefficient reaches 95.81%, with the Human Agreement (95% CI) at 88.46%, and the overall average consistency at 92.14%. However, due to the substantial cost associated with GPT-4, this study employs Llama3-70B-Chat as a more cost-effective alternative in WizardArena.

Question-2: While employing the fine-tuned Llama 70B judge model for automated evaluation reduces costs and enhances efficiency, it might not capture all the nuances and subjective assessments that human evaluators offer. Introducing some human evaluations or qualitative analyses could make the results more convincing.

We sincerely appreciate your valuable suggestions. Due to time constraints and the manual annotation costs, we randomly selected a subset of WizardArena-Mix, comprising 100 diverse and 100 hard samples, totaling 200 test cases. We chose four models for evaluation: WizardLM-β-PPO-I3, OpenChat-3.5, Command R+, and Qwen1.5-72B-Chat. WizardLM-β-PPO-I3 served as the reference model, while the others acted as battle models. The evaluations were conducted using Llama3-70B-Chat and professional human annotators, with the win/loss/tie results presented in Table 3 in newly uploaded WizardArena_rebuttal PDF.

Specifically, when using Llama3-70B-Chat as the Judge Model, the win rates for WizardLM-β-PPO-I3 against Command R+, Qwen1.5-72B-Chat, and OpenChat-3.5 were 34.1%, 41.3%, and 79.7% respectively. When evaluated by human annotators, the win rates for WizardLM-β-PPO-I3 against Command R+, Qwen1.5-72B-Chat, and OpenChat-3.5 were 31.8%, 37.7%, and 82.1% respectively. Therefore, the high consistency between human evaluations and Llama3-70B-Chat further confirms the reliability and accuracy of Llama3-70B-Chat as the judge model in WizardArena.

Dear Reviewer cCkS,

We would like to thank you for your detailed reviews. We genuinely appreciate the time and thoughtful consideration you have dedicated to our work.

Since the discussion period is coming , we would be grateful if you could let us know whether our response has addressed your concerns or if you still have any other questions. We are looking forward to receiving your any additional feedback you may have.

We would be happy to do any follow-up discussion or address any additional comments. Thank you once again for your valuable contributions to our work.

Respectfully,

Paper 9817 Authors.

Dear Reviewer cCkS,

We would like to thank you for your detailed reviews. We genuinely appreciate the time and thoughtful consideration you have dedicated to our work.

Since the discussion period is coming to an end today, we would be grateful if you could let us know whether our response has addressed your concerns or if you still have any other questions.

We are looking forward to receiving your post-rebuttal rating and your any additional feedback you may have. Please feel free to reach out if you suggest any additional modifications or require further information.

Thank you once again for your valuable contributions to our work.

Best regards,

Paper 9817 Authors.

Dear Reviewer, we thank you for your valuable comments and the time you spent reviewing our work! Your professional feedback provides valuable guidance for writing a more comprehensive and competitive paper.

Please find below a detailed discussion of the points you have raised:

Weaknesses-1: Training Data Quality and Transparency. The authors should provide more clarity regarding the quality of the training dataset. As this is a resource paper, the quality of these datasets is the most important part. The supplementary materials contain only 100 examples, which seems insufficient for a comprehensive evaluation....

We sincerely appreciate your attention to our work and your careful and responsible review. We greatly apologize for any inconvenience caused. Regarding the issues related to the data and code, due to strict legal regulations within our company, we are unable to publicly upload the internal code and data without prior review, as doing so could result in severe legal consequences. We kindly ask for your understanding in this matter.

Therefore, the nips_code_WizardArena/dpo_train/dpo_train/data/sample_data.json provided in the appendix is derived from our initial early model generation phase, which did not involve any filtering or post-processing, and thus includes some low-quality and multilingual data. However, it is not our final training dataset. We employed Llama3-70B-Chat to score the model responses and filtered the data when the score difference between "Choose" and "Reject" was below a threshold K (i.e., K <= 1). Additionally, we utilized Polyglot [1] for language category labeling to exclude non-English data. The final training data in the paper were primarily obtained through the judge-pair battle with multiple advanced models.

Furthermore, we have submitted the code and data for company review, and once approval is granted, we will immediately open-source the entire code and data to support further research in the LLM community. We appreciate your continued interest and once again apologize for any inconvenience caused.

Weaknesses-2: The supplementary materials lack proper organization and documentation. Many folders are empty, and there is no README file or documentation explaining how to use the provided resources. As this is a resource paper, clear instructions and explanations are crucial for the research community to effectively utilize the proposed methods.

We sincerely appreciate your valuable suggestions. Once we receive company approval for open-source release, we will promptly release our data and code, accompanied by well-organized and comprehensive documentation. This will include detailed implementation descriptions of each training and testing step to enable the research community to effectively utilize our proposed methods. We look forward to your continued interest and thank you for supporting our work.

Weaknesses-3: Multilingual. A significant portion of the training examples appears to be multilingual, as evidenced by samples in the nips_code_WizardArena/dpo_train/dpo_train/data/sample_data.json file.

LMSYS-Chat-1M includes over 150 languages, with English, Portuguese, and Chinese being the most prevalent. However, LMSYS-ChatBot Arena does not offer multilingual leaderboards, instead focusing on individual languages. To thoroughly assess the performance of our proposed algorithm in a multilingual context, we selected one language (i.e., Chinese) for detailed evaluation. Due to time constraints, we constructed a test set of 500 multilingual instances based on the Offline Diverse & Hard WizardArena test set described in Section 4.1. This test set comprises 250 diverse and 250 hard samples, sourced from LMSYS-Chat-1M and WildChat[2], with strict deduplication between training and test sets.

For training, we randomly selected 30k original Chinese corpus and divided them into three equal parts: SFT 10k, DPO 10k, and PPO 10k. After conducting Judge-pair Battles with Command R+, Qwen1.5-72B-Chat, and OpenChat-3.5, these sets were reduced to SFT 9.1k, DPO 8.4k, and Reward Model 8.1k. The SFT, DPO, and PPO models were then trained in one round, with the results summarized in the Table 1 in newly uploaded WizardArena_rebuttal PDF. The Spearman correlation between WizardArena-CH and LMSYS-ChatBot-Arena-CH reached 98.68%, the Human Agreement with a 95% confidence interval (CI) was 96.45%, and the Differentiation with a 95% CI was 91.17%, indicating a high consistency of 95.43% between the two, thus demonstrating the accuracy and reliability of WizardArena-CH.

Furthermore, after training SFT, DPO, and PPO using the judge-Pair Battle method, the ELO score of WizardLM-β in WizardArena-Mix-CH increased from 808 to 1288 (+480), surpassing OpenChat-3.5(+20) and approaching GPT-3.5-Turbo-0613. This outcome suggests that the Judge-Pair Battle method is also highly effective for multilingual tasks. Meanwhile, in future versions of our paper, we will further supplement additional multilingual tasks.

[1] https://github.com/aboSamoor/polyglot

[2] Lin B Y, Deng Y, Chandu K, et al. WILDBENCH: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild[J]. arXiv preprint arXiv:2406.04770, 2024.

Dear Reviewer vqCh,

We would like to thank you for your detailed reviews. We genuinely appreciate the time and thoughtful consideration you have dedicated to our work.

Since the discussion period is coming , we would be grateful if you could let us know whether our response has addressed your concerns or if you still have any other questions. We are looking forward to receiving your any additional feedback you may have.

We would be happy to do any follow-up discussion or address any additional comments. Thank you once again for your valuable contributions to our work.

Respectfully,

Paper 9817 Authors.

Thank you for your response.

While the dataset seems to be an important contribution to this work, the code and full datasets are not currently available for review. The 100 samples provided appear to have serious quality issues.

I wonder if this paper might be more suitable for the NeurIPS 2024 Datasets and Benchmarks Track, which requires that A key criterion is accessibility: datasets should be available and accessible [1].

I am inclined to maintain my score, with a low confidence rating.

Reference:

[1] https://neurips.cc/Conferences/2024/CallForDatasetsBenchmarks