OpenChat: Advancing Open-source Language Models with Mixed-Quality Data

[Q1] Would the proposed method work if the mixed-quality data are generated by open-sourced (and potentially smaller) models, such as LLaMA-13b vs LLaMA-7b?

Response:

• Thanks for your comment. It's not appropriate to train the base model (llama-2-13b) on datasets generated with models weaker than the base model (LLaMA-7b, LLaMA-13b), as they have worse capabilities and are unlikely to provide useful information for model fine-tuning.
• However, if the reviewer wants to look into the performance of C-RLFT on datasets other than ShareGPT (GPT-4 + GPT-3.5), we have added an experiment using the OpenOrca dataset [1]. OpenOrca is a reasoning dataset that is generated by human-written and GPT-4. We treat human data as $\mathcal{D}_{exp}$ and GPT-4 data as $\mathcal{D}_{sub}$, respectively. The results are shown in the following Table, where C-RLFT outperforms SFT on both datasets as well as the combined dataset.

---

[Q2] I'd like to see a discussion of Xu et al. (I understand that Xu et al. came out probably after this work, but it seems to me that there are some similarities in methodology between the two papers).

Response:

Thanks for your question. Our C-RLFT is fundamentally different from Xu et al., as their proposed method is still a preference-based method, like RLHF, while our method simply uses two different quality datasets.

• Our C-RLFT method has extremely relaxed requirements for the training dataset. It only requires to have a small set of expert data and a larger set of medium-quality data, without the need for preference labels. Meanwhile, C-RLFT reformulates the RL objective to a weighted supervised loss, which is more stable compared to the RL training in existing RLFT methods.
• Xu et al., like all RLHF methods, require pairwise preference labels for every question $x_i \mapsto (y_i^+, y_i^-)$, i.e., for every question $x_i$, it needs a preferred answer $y_i^+$ that is better than the worse answer $y_i^-$. Moreover, Xu et al. need to use two different quality LLMs to generate positive and negative answers, and then run DPO for final policy fine-tuning. The entire learning process is very heavy and inherits the potential instability and hyperparameter sensitivity of DPO-style RL training. Moreover, the positive and negative answers generated by LLMs could also introduce potential model-specific biases to the preference data. Nevertheless, we have cited this paper in our revision. We thank the reviewer for providing this reference.

References:

[1] Orca: Progressive Learning from Complex Explanation Traces of GPT-4. arXiv 2023.

[W1] My biggest concern is that while the paper claims their approach to be based on single-stage RL-free supervised learning, they do use data generated by API models (GPT-4 & GPT-3.5), and these models are believed to have gone through multi-stage training with RLHF. If the C-RLFT method leverages these kinds of data for their training procedure, it's not very convincing to me that the method really works in a fundamentally different way from prior approaches (SFT + RLHF). It also makes the comparisons to baselines not fully fair, since many baselines (under the 13b model scale) being compared to have not leveraged training data generated by (much larger) API models.

Response:

• Thanks for your question. Here we summarized the fine-tuning data sources of OpenChat and other popular open-source language models. Actually, most open-source SFT models leverage training data generated by (much larger) API models, except for llama-2-chat-13b. It should be noted many existing models have more stringent requirements on the finetuning data, either data quantity or data quality. By contrast OpenChat enables to use the minimal amount of potentially expensive expert data as well as more easily obtainable medium-quality data to achieve improved LLM finetuning performance.

References:

[1] Orca: Progressive Learning from Complex Explanation Traces of GPT-4. arXiv 2023.

[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.

[3] Learning from Human Feedback without RL. arXiv 2023.

[4] LIMA: less is more for alignment. arXiv 2023.

[Q4] How does a supervised learning approach with data weighting, predicated on the assumption that GPT-4 provides higher quality data than GPT-3.5, compare in performance to the RL framework, and can the simplicity of supervised loss be reconciled with the advantages of an RL-based approach?

Response:

• Note that C-RLFT is not a typical supervised learning approach, but solving a KL-regularized RL problem: $J\_\text{C-RLFT}(\theta)=\mathbb{E}\_{y \sim \pi\_\theta}[r\_c(x,y)]-\beta D\_{\mathrm{KL}}\left(\pi\_\theta, \pi\_c\right)$. Its optimal policy can extracted using a weighted supervised learning loss, but its nature is RL.
• Additionally, if the reviewer checks the llama-2-chat-13b baseline in the paper, it has the same base model as our openchat-13b, but is tuned using SFT+RLFT with ~27k high-quality SFT data + ~2.9M preference, but its average score is still worse than our openchat-13b. This shows the strength of our proposed C-RLFT.

[Q2] What strategies can be implemented within the proposed framework to mitigate the transfer of potentially negative behaviors from GPT-4 examples to the base model, and how can such behaviors be identified and counteracted effectively?

Response: It should be noted that our proposed C-RLFT framework is not bounded with GPT-4 and GPT-3.5 data. If the user is concerned with potential negative behaviors in the GPT-4 examples, one can simply replace the expert dataset $\mathcal{D}\_{exp}$ with a small set of carefully curated human expert data, like used in LIMA [4].

[Q1] One of interesting question when handling the mixed data quality problem is on the contribution of data with various quality. In this paper, one research question is what the performance will be if only GPT-4 samples are used even the size is ten times smaller than GPT-3.5. As a follow-up, how does the quality of data from different versions of language models, like GPT-4 and GPT-3.5, impact the performance of fine-tuned models, and what is the optimal quantity ratio of high-quality to lower-quality data necessary for achieving equivalent performance enhancements?

Response:

• Thanks for the reviewer’s comment. It should be noted that if only GPT-4 samples are used, then C-RLFT actually reduces to perform SFT on a single source of data. To fully address the reviewer’s concern, we conducted additional experiments to compare C-RLFT on GPT-4 + GPT-3.5 against SFT methods tuned on GPT-4 and GPT-3.5, as well as their combination. The results are presented in the following table.

• Regarding the question of the reviewer on the impact of training C-RLFT from different versions of the language model, please refer to our additional results on C-RLFT with Human + GPT-4 data in our response to W1. It can be shown that as long as we can have two datasets with different quality levels, C-RLFT can still achieve better performance as compared to performing SFT on such datasets.
• Regarding the optimal quantity ratio of high-quality to low-quality data, we actually have ablation results (Figure 7) in our main paper. We sub-sample one class in GPT-3.5 or GPT-4 with the ratio varying from 60% to 100%, while keeping the other class unchanged. The results show that the average performance is relatively stable, demonstrating that our method is robust to variation in dataset ratios, as long as the expert data size is not overly small. For details, please check Section 5.5 in our paper.

[W1] The paper does not adequately address the challenge of estimating data quality when leveraging mixed-quality data sources. The method presented seems to oversimplify this estimation and the associated reinforcement learning (RL) reward mechanism: The process of data quality estimation may be too simplified and not easily generalizable to realistic settings involving human feedback. The estimation relies on the assumption that GPT-4 provides better quality outputs than GPT-3.5, and GPT-3.5 surpasses the base model. Although this may be true in some instances, it's not universally applicable across all domains and might not hold as the base model evolves. This assumption could be problematic if the alignment is based on a superior base model or a larger one. Furthermore, presuming that the GPT series invariably delivers superior samples can inadvertently lead to the replication of any negative behaviors from these models, making it challenging to improve upon them with base model developments.

[Q3] In what ways can the proposed framework be adapted to more complex and realistic data environments, such as those involving human-generated data, which may exhibit significant variations in quality and where quality assessments may be more subjective?

Response:

• First, our proposed C-RLFT is not designed to "estimate data quality" as in typical RLFT methods, but rather provide the cheapest solution to bypass this step in LLM fine-tuning. In typical RLFT methods, we have to use lots of expensive human preference data to learn reward models, in order to explicitly estimate data quality. This poses major obstacles and instability in LLM fine-tuning, as human preference collection is very costly and the two-stage training process (learn reward first and then run RL algorithm like PPO) can be very unstable. By contrast, our method smartly avoids the explicit data quality estimation problem by simply leveraging the quality difference between two datasets to achieve the same policy fine-tuning goal.
• Second, our proposed C-RLFT is not bounded with GPT-4 and GPT-3.5 data. As clearly described in our paper, our method only needs an expert dataset $\mathcal{D}\_{exp}$ and a sub-optimal dataset $\mathcal{D}\_{sub}$. The GPT-4 and GPT-3.5 datasets are only for demonstrative purposes, and in principle, one could use any expert & sub-optimal datasets to fine-tune their models. To further address the concern of the reviewer, we conducted additional experiments to tune our model with human-written data (treated as expert data) + GPT-4 data (treated as sub-optimal data) from the OpenOrca dataset [1], the results are presented as follows. Our proposed C-RLFT still outperforms SFT models trained with Human or GPT-4 data, or a combination of both.

Table: Comparison of C-RLFT and SFT on Human & GPT-4 datasets (we report the accuracy of Big-Bench-Hard (BBH) as used in Orca[1] for comparison).

Thanks for adding the experiments. However, there's some confusion regarding why C-RLFT reduces to SFT when solely employing GPT-4 samples. RLFT remains applicable with GPT-4 data by considering all samples with r_c(x,y) as 1. As the authors highlighted, C-RLFT differs from standard supervised learning, focusing on a KL-regularized RL problem instead. Therefore, for a proper evaluation of data quality, it seems more appropriate to conduct an ablation study on the data alone, rather than altering the objective function.

Response:

Thank you for your comments. It seems there is some confusion regarding the reduction form of C-RLFT on the single dataset setting. We provide detailed clarification as follows:

• First, if we only have a single quality dataset, e.g., GPT-4, then the class-condition policy $\pi_{\theta}(\cdot|x,c)$ of C-RLFT will reduce to a non-conditioning policy $\pi_{\theta}(\cdot|x)$ since the conditioning term $c$ will remain constant.
• Second, the class-conditioned rewards $r_c(x,y)$ also becomes constant (denoted as $r$) since there is no quality difference in the data.
• Under such cases, the final form of the extracted optimal policy in Eq.(6) in our paper will become: $\pi\_{\theta}=\arg\max\_{\theta}\mathbb{E}\_{x,y,c\sim \mathcal{D}\_c}\left[\exp\left(r\_c(x,y)/\beta\right)\log\pi\_{\theta}(y|x,c)\right]$ $=\arg\max\_{\theta}\mathbb{E}\_{x,y\sim \mathcal{D}\_c}\left[\exp\left(r/\beta\right)\log\pi\_{\theta}(y|x)\right]=\arg\max\_{\theta}C\cdot\mathbb{E}\_{x,y\sim \mathcal{D}\_c}\left[\log\pi\_{\theta}(y|x)\right]$ where $C$ is a constant and the optimal policy extraction becomes exactly SFT on the single dataset $\mathcal{D}_c$ (e.g., GPT-4 dataset).
• To further address the reviewer's concern, we ran C-RLFT on 8x A100 80G GPUs with only the GPT-4 dataset, using the same base model (llama-2-13b) and hyperparameters as in our paper. The table below presents the results, demonstrating that C-RLFT and SFT exhibit similar performance on GPT-4 data, which supports our explanation. Moreover, C-RLFT with both GPT-3.5 and GPT-4 data outperforms the models trained with GPT-4 data alone by a significant margin, indicating that the model effectively learns from the quality difference between datasets.

[W2] It is not clear if the "low quality data" is even useful at all. (see my question below)

[Q1] What is the performance if the llama model is only trained on GPT-4 data? I feel it should be another ablated model to demonstrate the value of "low quality data".

Response:

• The "lower quality data" is important in our C-RLFT method, as it needs to provide contrast signals to guide policy learning. As shown in the previous Table R1 in our response to W1, simply performing SFT on GPT-4 data cannot guarantee the best performance. Naively tuning with mixed GPT-4 and GPT-3.5 data will lead to even worse results. In contrast, C-RLFT greatly outperforms SFT on the same datasets. This is primarily because C-RLFT is performing RL rather than supervised learning on data. It uses the quality difference between two datasets as an implicit reward signal to guide policy learning, making the LLM model capable of discriminating good vs bad samples during learning, thus leading to improved performance. Without the lower-quality data, such contrast learning signals may no longer exist.
• The performance llama-2-13b model finetuned on GPT-4 data using SFT is reported in the previous Table R1. We will also add the new results and related discussion in our revised paper.

[W1] The paper claims the superiority of C-RLFT over RLFT. But the claim is only supported by openchat-13b, which distilled its knowledge from two RLFT models: GPT-4 and GPT-3.5. So, it is only fair to say C-RLFT is better than SFT when distilling from GPT models, which is based on the fact the openchat-13b is better than vicuna-13b-1.5. However, one cannot say C-RLFT is better than RLFT, because openchat-13b learns from models trained by RLFT.

Response: We thank the reviewer for this thoughtful comment. Regarding the reviewer's concern, we provide detailed responses as follows:

• First, we'd like to clarify that our model is not simply distilling from GPT models. We have added the following experiments to compare C-RLFT vs SFT on different GPT data. As the results show, if we simply distill a pretrained model by performing SFT on GPT-4 or GPT-3.5 data, their performances are much worse as compared to our proposed C-RLFT. Moreover, SFT with both GPT-4 and GPT-3.5 (~70k) performs even worse than SFT tuned with only GPT-4 (~6k), suggesting that indiscriminatively training with mixed quality data is likely to negatively impact learning (also discussed in our paper). By contrast, C-RLFT can smartly leverage the quality contrast of two datasets and achieve much better performance, this cannot be explained by distillation from better GPT models.

Table R1: Comparative performance of C-RLFT and SFT tuned on different data sources.

• Second, note that vicuna-v1.5-13b and our openchat-13b uses the same base model (llama-2-13b), as well as ShareGPT datasets (both with GPT-4 + GPT-3.5 data) for finetuning, and vicuna-v1.5-13b uses even more data than ours (125k ShareGPT conversations vs 70k in ours), but our openchat-13b has achieved much better performance.
• We have also compared with LLM models that are trained with SFT+RLFT as shown in the following table. Specifically, llama-2-chat-13b has the same base model as ours, but is tuned using SFT+RLFT with ~27k high-quality SFT data + ~2.9M preference, but its average score is still worse than our openchat-13b. This shows the strength of our proposed C-RLFT.

Table R2: Comparing openchat with our C-RLFT to a series of fine-tuned LLMs with SFT + RLFT.

• Lastly, our main focus of this paper is to provide a much simpler and extremely lightweight alternative solution to the existing costly RLFT methods for open-source LLM fine-tuning, rather than developing the most powerful LLM. Our proposed method can be very useful for developers with budget constraints or unable to collect lots of human preference data.

[Q1] How sensitive is the results to the division of "good" data and "average" data? For example, how about if both sets of the data is of the same quality?

Response:

• Thanks for the reviewer's constructive comment. It should be noted that if the "good" data and "average" data in C-RLFT are of the same quality, it actually reduces performing SFT on a single source of data. To fully address the reviewer's concern, we conducted additional experiments to compare C-RLFT on GPT-4 + GPT-3.5 against SFT methods tuned on GPT-4 and GPT-3.5, as well as their combination. The results are presented in the following table.
• The results show that SFT tuned with only GPT-4 (~6k) even performs better than SFT with both GPT-4 and GPT-3.5 (~70k). This phenomenon confirms our statement in the paper that "it is not advisable to indiscriminatively feed all these mixed conversations to the base model, as the low-quality data are likely to negatively impact learning" in our Introduction. Furthermore, the performance of C-RLFT is still higher than SFT with GPT-4, proving the effectiveness of our proposed method. In our revised version, we will add this experiment to the ablation study.

Table: Comparative performance of C-RLFT and SFT tuned on different data sources.

---

[Q2] When training with Reinforcement Learning, how stable is the training process, since RL is usually different to train in most cases?

Response:

Thanks for your comment.

• It should be noted that although C-RLFT optimizes the KL-regularized RL objective, its policy optimization can be equivalently transformed into a simple weighted regression objective. This actually allows us to solve an RL problem instead of using a single-stage, supervised learning scheme to learn the optimized policy.
• Moreover, it should be noted that in our proposed C-RLFT, we no longer need to learn and optimize with a reward model, no need to involve the value networks as in PPO-based RLFT, and even not need to maintain the reference policy $\pi_0$ for policy constraint $D_{KL}(\pi_{\theta} \| \pi_0)$ as in most RLFT methods (also avoid related policy constraint weight parameter tuning). Everything we need is single-staged supervised learning, and thus is extremely lightweight and stable.

[W1] It requires the data quality to be divisible into two sets: good quality and average quality, which might potentially limit is applications.

Response:

• Thanks for your comment. It is worth noting that our method actually has extremely relaxed requirements on datasets and is very friendly for applications. It only requires to have a small set of expert data and a larger set of medium-quality data, without the need for detailed quality evaluation for each sample nor costly human preference labels. In our experiments, we use the widely adopted ShareGPT dataset (~70k conversations), with only 6k expert conversations generated by GPT-4. This finetuning dataset essentially is the same one used to finetune vicuna-v1.1-13b, but as shown in our paper, our openchat-13b achieved significantly better performance.
• We have also summarized the finetuning data sources of OpenChat and other popular open-source language models. It should be noted many existing models have more stringent requirements on the finetuning data, either data quantity or data quality. By contrast OpenChat enables to use the minimal amount of potentially expensive expert data as well as more easily obtainable medium-quality data to achieve improved LLM finetuning performance.
• Lastly, we'd like to mention that the expert vs sub-optimal datasets division in C-RLFT can also be conducted according to the downstream tasks, e.g., safe vs not-so-safe data, preferred role vs. less preferred role in role-playing tasks, etc. Our proposed C-RLFT framework provides a very flexible and performant approach to address a wide range of applications.

Table: The data source of the proposed OpenChat and other popular language models.