WildChat-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training

We thank the reviewer for the thoughtful response. To the best of our ability, given the 5000 character limit, we address your comments and questions below.

We agree that our claim about SDQ depending primarily on prompt diversity could be worded more carefully. In the camera-ready draft, we will re-scope the claim to be more in line with our findings, e.g., by inserting phrases such as “in our experimental setting”.

When we wrote that LLMs “do not share pre- and post-training data”, we meant that they do not entirely share it (that at least some data differs between the models trained); we will add this qualifier to the camera-ready.

To your point about being upfront about the performance of our mix in the text, the caption of figure 1 reads: “RE-WILD outperforms strong baselines, on average, across nine benchmarks. In particular, it exhibits strong performance on generalist chat and instruction following benchmarks.” We are glad your assessment that this is a strong result agrees with ours, hence, we refer to it as a strong mix.

In Sec 2.2, we state: Sometimes we will not specify the SFT target model name; in that case, it will always be Llama-3.1-8B-Base := L8B. If you would like us to repeat this information in another section, we can.

In Appendix D, we write “we have attached several relevant artifacts to this submission which would not have incorporated well into the body of the paper.” The artifacts in question are in the Supplementary Material ZIP attached to our submission. How does Re-Wild do for other base models compared to existing SFT datasets? This would be an interesting ablation – however, to conduct it, we would have had to retrain not only our own model but also every baseline, because the prior work does not. How we trained our baselines could in turn raise its own set of methodological questions. With limited resources, we feel consistency with prior work is sufficient.

How does it interact with scale? We provide scaling experiments at 100k, 250k, and 500k in our paper (Fig. 2) and we have more in our repository, which we will share as part of the camera-ready.

What is the performance improvement over simply using WildChat-1M? The improvement is significant; the results are available in our artifacts. We did not include this in the main paper out of respect for the WildChat-1M authors, who felt it would not be a fair comparison unless we improved their work by sampling every response from GPT-4, which was prohibitively expensive. However, we are prepared to reverse that choice and include it in the appendix if you feel it is vital.

We cannot release all configs and logs because some of them were corrupted on our cluster. We will release all uncorrupted ones. In the camera-ready draft, we will be happy to remove the word “may” from line 117-118l, change the citation for Cohere command-r plus to the model announcement the Cohere website, and highlight the best score for each group of base LMs in Table 6.

Regarding Figures 3 and 4, we will make available the detailed responses from the LLM judges in our repository associated with the camera-ready release – it is, therefore, a straightforward matter to recover the context around the words, should anyone wish to do so. We note, however, that many of these words are easily interpretable even out of context.

To clarify the pretrained versus post-trained model variants distinction; all of our DGMs were post-trained. However, many of the post-trained variants started from the same pretrained checkpoint. Therefore, we have 19 unique pre-trained models (each of which is post-trained) and 35 post-trained model variants (with non-unique pre-trained models). We will add this clarification to our camera-ready.

Why is DGM Q72 outperforming others on Math (table 2) but in Figure 1 the model trained on Re-Wild is not? This is a good question – the reason is that the baselines in Figure 1 are trained on different data than the models compared in Table 2. In particular, the Tulu 3 SFT mix, which outperforms ReWild SFT mix on MATH, contains significantly more math data in the mix.

We will update the camera-ready draft to link to Table 3, rows 1:4; thanks for this correction.

Thanks again for your time. In the event that you now feel more positively about our paper, we would appreciate it if you updated your score.

We thank the reviewer for the thoughtful response. To the best of our ability, we briefly address each of your comments and questions below.

To your point about confidence intervals; we agree that these would be valuable. Therefore, if our paper is accepted, we will add 95% confidence intervals using the normal approximation method to the main paper, where they are absent. We hope that this resolves your concern in this area.

We agree that direct size comparisons with other datasets should be included. The largest such datasets we are aware of are WildChat-1M and LMSys-Chat-1M. Our dataset is more than 50x larger than either of those datasets. We will include this information in our camera-ready.

We will improve our explanation of the derivation of chat transcripts by more thoroughly explaining the contributions of the original WildChat-1M, which included the acquisition of the human multi-turn conversations used in this work.

We agree that our claim about models learning better from same-family DGMs would benefit from more extensive cross-family experiments; we will be happy to reword the claim to be more modest in scope in the camera-ready draft.

The raw quantitative data used to generate Figures 3-4 will be made available in our repository for the camera ready release. To your point about the selection of which judgments to analyze potentially introducing bias, we agree, which is why we report results over all of MTBench without subselection.

Our heuristic choice of SFT mix is consistent with prior work (Tulu 3); as of right now, automated methods for subselection trail human performance (Tulu 3), and it is a relatively low-cost annotation task for human experts.

We agree that the response similarity analysis would benefit from a small-scale human study; if our paper is accepted, we will conduct such a study.

We did conduct experiments varying only the DGM while keeping dataset size fixed; these can be found in our supplementary material ZIP attached to the submission.

Sadly, at this time we can offer no theoretical basis for our findings; they are purely empirical. We consider this a useful direction for future work.

We did perform experiments with Qwen as well as Llama; the results can be found in Appendix Table 6.

To your point about RLHF; we agree with you that it would have been a valuable contribution, and we note as much in our limitations section; we consider this a very interesting and important direction for future work, and thank you for highlighting it. However, the paper, even in its current form, required considerable resources, and we hope you will take this into account.

Thanks again for your time. In the event that you now feel more positively about our paper, we would appreciate it if you updated your score.

We thank the reviewer for the thoughtful response. To the best of our ability, we briefly address each of your comments and questions below.

Regarding experiments to determine if synthetic data can surpass the performance of smaller-scale, higher-quality human-written chat responses: unfortunately, we know of no such publicly available chat dataset. The original WildChat-1M dataset contained GPT 3.5 and GPT-4 synthetic responses (we did train models on that data; the results of those experiments are available in our artifacts). The closest comparisons of which we are aware are the large-scale post-training runs exclusively on FLAN data described in (https://arxiv.org/abs/2409.15268). FLAN has human prompts and human responses, but is not a chat dataset; that work found that FLAN-trained models significantly underperformed WildChat-trained ones, controlling for dataset size. We agree that more research needs to be done in this area.

As to whether the primary benefit observed in our mix is due solely to the strong performance of a single DGM (Qwen 2.5 72B), our key contributions are (1) discovering which DGMs tend to produce higher quality responses and (2) providing experimental evidence on why they perform better.

We agree that we could have more thoroughly explained the contributions of the original WildChat-1M paper, which included the acquisition of the human multi-turn conversations used in this work. We will add more information on this to our camera-ready draft.

For the camera-ready version, we will also correct the issue of too much white space on page 4.

We thank the reviewer for the thoughtful response. To the best of our ability, we briefly address each of your comments and questions below.

You are correct that "wehow" is a typo and should read “we show”. We apologize for the inconvenience, and will remedy this in the camera-ready draft if we are accepted.

You are also correct that our benchmark suite contains no examples of highly specialized coding or legal reasoning tasks. This exclusion was motivated by our findings about the limited usefulness of generalist chat data for such tasks (see our conclusions at the end of 3.3).

To your point about our inability to include other post-training approaches in this work, we agree with you that it would have been a valuable contribution, and we note as much in our limitations section; we consider this a very interesting and important direction for future work, and thank you for highlighting it. However, the paper, even in its current form, required considerable resources, and we hope you will take this into account.

Thanks again for your time. In the event that you now feel more positively about our paper, we would appreciate it if you updated your score.

We thank the reviewer for the thoughtful response. To the best of our ability, we briefly address each of your comments and questions below.

To your point about our inability to include RLHF in this work (and therefore to answer your question about whether stronger SFT leads to stronger post-trained models), we agree with you that it would have been a valuable contribution, and we note as much in our limitations section; we consider this a very interesting and important direction for future work, and thank you for highlighting it. However, the paper, even in its current form, required considerable resources, and we hope you will take this into account.

We agree that a section documenting common pitfalls and best practices in synthetic data generation, including configurations that might ruin the entire dataset, would be valuable; we do not include it here because of space constraints, however, if the paper is accepted, we will add such a section to the appendix in the camera-ready version.

We are happy to confirm that Table 1, Row 1 is in fact not a typo – the name refers to samples with WildChat-1M prompts and Qwen 2.5 72B Instruct responses, following the convention introduced in Sec. 2.2. However, as we only explicitly define naming conventions for models and here implicitly use it to name a dataset, there could be confusion -- we will clarify that the names can apply to datasets generated using DGMs in the camera-ready draft.

Thanks again for your time. In the event that you now feel more positively about our paper, we would appreciate it if you updated your score.