Unified Language Model Alignment with Demonstration and Point-wise Human Preference

Q3: "the numbers of win-rates in Tables 1 and 2 are somewhat weird and vague... The win-rates, especially in Golden HH, are close to 100 percents"

A3: The numbers reported in Table 1 and 2 are actually the win & tie rate [1, 2] of the harmful metric. We had made a typo in the original captions of Table 1 and 2. Sorry for causing your confusion! In the revised manuscript, beyond the win & tie rate presented in Table 1 and 2, we also report the detailed win, tie and loss rates respectively in Figure 1 and Figure 2 in Appendix D.

We also note that it is reasonable that the win & tie rates on Golden HH are close to 100%. Here we explain the reason. Since Golden HH is constructed based on HH, to make the results on Golden HH and HH comparable, the generated answers are compared to the chosen sample in the original HH dataset when evaluating the harmful score on Golden HH. However, the quality of the chosen samples in HH is not very high. When employing SFT on Golden HH where the new chosen data are of much higher quality, the answers generated on Golden HH is very likely to be at least no worse than the original chosen data in HH. As a consequence, the win & tie rates on Golden HH are close to 100%. Note that the above phenomenon actually implies that the quality of the chosen samples in Golden HH is much higher than that in HH (also see Table 5 in Appendix C for more detailed comparison of both datasets).

Q4: "It is also important to extend the evaluation to other benchmarks"

A4: We have added two more datasets QA-feedback and red-team in the revised manuscript. The first dataset QA-feedback is a point-wise QA dataset with binary labels, in which the perplexity and the helpful score are evaluated. The second dataset red-team is a point-wise dataset with continuous labels on robustness of LLMs to red teaming attacks, in which we report the perplexity and the harmful score. The empirical results are presented in Table 1 and Table 2 in the updated manuscript, from which we also observe that our proposed ULMA method outperforms other compared methods on both datasets. Please refer to the manuscript for empirical results and detailed discussions.

Q5: "there isn’t an official implementation of DPO... how does their result align with the reported one in the DPO paper"

A5: Indeed, in our experiments, we had followed the official implelentation of DPO in https://github.com/eric-mitchell/direct-preference-optimization. The reported numbers are a bit different from those in the DPO's paper, because we adopt the harmful score as the metric on the HH dataset, which is different from the helpful score adopted by the DPO's paper.

[1] Duan et al. BOTCHAT: Evaluating LLMs’ Capabilities of Having Multi-Turn Dialogues. ArXiv, abs/2310.13650, 2023.

[2] Xu et al. Superclue: A Comprehensive Chinese Large Language Model Benchmark. ArXiv, abs/2307.15020, 2023.

Thank you for the detailed review! In the following, we will address your questions and give clarification on various aspects of this paper.

Q1: "the binary signals could result in significant information loss, since the learning can only capture the data polarity, omitting the nuanced levels present in practice"

A1: Admittedly, as you have pointed out, many datasets on toxicity and verifiability are not binary labelled. Our original statement regarding binary labeling on toxicity and verifiability in Section 4.1 is only meant for providing an example to illustrate the limitation of pair-wise preference learning methods on point-wise preference datasets.

We shall clarify that such a statement does not mean that our proposed ULMA is restricted to binary point-wise datasets. In fact, we do not necessarily need to transform continuous labels into binary labels when applying ULMA. The core concept of ULMA, i.e., using a hybrid objective formulation to treat the high-quality demonstration data and possibly noisy preference data differently, is capable of generalizing to continuous labels to characterize the different levels.

Specifically, for these datasets, there is no direct separation of positive or negative samples as in the binary case. To apply ULMA, we can specify some sample as high quality data (e.g., the most helpful or harmless ones) and treat these high quality samples as "positive" demonstration data. Then we can adopt a hybrid loss by combining the SFT loss on these demonstration data with the original loss (e.g., MSE) on preference data.

In the revised manuscript, we add the above clarification and also provide a concrete example on the red-team dataset with continuous labels. Red-team is a point-wise dataset on LLM's robustness to red teaming attacks. In this dataset, each sample has a score ranging from 0 to 4, which is rated by human and indicates how successful the attack is to the LLM (in Likert scale; an attack with a higher level is more successful). Red-team is very similar to your mentioned datasets on toxic content generation or verifiability. However, this dataset only has a single sample for each prompt, hence no comparison can be made and pair-wise methods such as RLHF and DPO are inapplicable; we can only apply point-wise methods such as our proposed point-wise DPO.

To further improve the performance of point-wise DPO on red-team, inspired by the ULMA framework for binary cases, we treat the samples of level 0 as the high quality demonstration data. Then we introduce a hybrid loss by adding SFT loss of the samples rated 0 to the original MSE loss of all samples. As presented in Table 1, empirical results show that our proposed ULMA outperforms other methods on red-team. Please refer to the revised manuscript for more detailed description on the dataset and empirical results.

Q2: "lacks empirical analysis with limited experimental results to justify the design of each component"

A2: In the revised manuscript, we have conducted an ablation study to verify the design of the hybrid objective formulation in ULMA. Specifically, we first use positive samples as demonstration data to compare ULMA with point-wise DPO (which adopts KL regularization for these samples) to evaluate the effectiveness of ULMA for learning from high quality (positive) demonstration data. Then we use negative samples as dispreferred demonstration data to compare ULMA with the counterpart algorithm without KL regularization (i.e., Unlearning) to evaluate the ability of ULMA for learning from possibly noisy (negative) preference data. As presented in Table 3, the results show that ULMA can better exploit the high quality positive samples by applying SFT loss to these samples, while preventing from overfitting to possibly noisy negative samples via adding regularization to negative samples. Please refer to the revised manuscript for more detailed empirical results.

Thank you for the valuable review! We appreciate your positive assessment and will address your concerns in the following.

Q1: "The experiments section is not very detailed. Expanding the methodology to more datasets would be nice."

A1: We have added two more datasets QA-feedback and red-team in the revision. The former dataset is a point-wise dataset with binary labels, and the second dataset is a point-wise dataset with continuous labels. The empirical results are presented in Table 1 and Table 2 in the updated manuscript. Please refer to the manuscript for empirical results and detailed discussions.

In particular, on the red-team dataset, we give an example of expanding the methodology of ULMA, i.e., using a hybrid objective formulation for different types of data, to handle point-wise dataset with continuous labels. Specifically, red-team is a point-wise dataset on LLM's robustness to red teaming attacks. It consists of samples scoring from 0 to 4, among which those rated 0 can be considered as high quality demonstration data. To better exploit these data, we introduce a hybrid loss by adding SFT loss of the samples rated 0 to the original MSE loss of all samples.

Thank you for the insightful review! We acknowledge your valuable suggestions and concerns, which we will address in the following.

Q1: "In many cases, there can be a mapping function between pair-wise preference and point-wise preference"

A1: Admittedly, there exists some relation between pair-wise data and point-wise data as you have suggested, but we shall note that it is not a one-to-one correspondence, and the transformation may suffer from information loss. Now we explicate the above point in two folds.

(i) "Pair-wise to point-wise" transformation may lose some pair-wise comparison information. For example, consider two pair-wise datasets $\mathcal D_1=\\{x_1\succ x_2, x_2\succ x_3, x_3\succ x_4\\}$ and $\mathcal D_2=\\{x_1\succ x_3, x_3\succ x_2, x_2\succ x_4\\}$. Obviously, the two datasets differ from the pair-wise relation between $x_2$ and $x_3$. These two datasets will be transformed to the same point-wise dataset $\mathcal D_3=\\{(x_1,1), (x_2,0), (x_3,0), (x_4,-1)\\}$, in which $x_2$ and $x_3$ have the same score. In this case, the transformation drops the relation between $x_2$ and $x_3$, which incurs loss in information.

(ii) "Point-wise to pair-wise" transformation is unsuitable to some dataset where no comparison can be made, e.g., all samples have the same score or there is only one sample for some prompt. One example is the red-team dataset, which only has one sample for each prompt. This dataset cannot be transformed into a pair-wise dataset, and pair-wise methods such as RLHF and DPO are inapplicable. Please see the revised manuscript for more detailed descriptions and empirical results.

Therefore, the conversion between pair-wise and point-wise datasets is not always possible, and may suffer from information loss in many cases. In the revised manuscript, we have compared the performance of point-wise DPO and point-wise DPO on pair-wise dataset Golden HH and point-wise dataset QA-feedback, respectively. We observe that pair-wise DPO performs slightly worse than point-wise DPO on point-wise dataset, and vice versa, which supports our above claim.

In addition to the above difference, we shall emphasize that our investigation on point-wise dataset has motivated the design of more flexible tuning method. Specifically, the absolute labeling on point-wise dataset motivates us to treat different types of samples (e.g., high quality positive data and noisy negative data) in different ways, which results in the ULMA framework. Our investigation on point-wise dataset and proposed new dataset may motivate more elaborate algorithmic design for point-wise preference learning.

Q2: "more experimental results in terms of more LLM-based tasks"

A2: We have added two more datasets QA-feedback and red-team in the revision. The former dataset is a point-wise dataset with binary labels, and the second dataset is a point-wise dataset with continuous labels. In particular, red-team only has a single answer for each prompt and no comparison can be made, hence it cannot be transformed into a pair-wise dataset and pair-wise methods such as RLHF and DPO are inapplicable. The empirical results are presented in Table 1 and Table 2 in the updated manuscript. Please refer to the manuscript for more details.

Q3: "There is no significance test in the tables"

A3: In the revised manuscript, for each harmful/helpful score presented in Table 1-3, we report its 95-percent confidence interval after repeating the model training procedure for three times beyond its mean value. The results show that, on the red-team and QA-feedback datasets, ULMA performs significantly better than other methods; on the HH and Golden HH datasets, though the confidence intervals are overlapped, the mean value of ULMA is still better. Please refer to the manuscript for detailed empirical results.

Thank you for the helpful review! We now address your concerns and suggestions as follows.

Q1: "more human preference datasets to align LLM should be included"

A1: We have added two more datasets QA-feedback and red-team in the revision. The former dataset is a point-wise dataset with binary labels, and the second dataset is a point-wise dataset with continuous labels. The new empirical results are presented in Table 1 and Table 2 (together with the original results on HH and Golden HH). Please refer to the revised manuscript for detailed descriptions and empirical results.

Q2: "The proposed framework should be able to be generalized to more complex metrics (such as the discussion to handle continuous labels)... the dataset is not enough to support the capability of its generalization"

A2: In fact, the core concept of our proposed ULMA framework, i.e., using a hybrid objective formulation for different types of data, is capable of generalizing to more general metrics. On the new red-team dataset in the revision, we provide an example of how ULMA can be generalized to handle continuous labels.

Specifically, red-team is a point-wise dataset on LLM's robustness to red teaming attacks, which consists of samples scoring from 0 to 4. The scores are rated by human and indicate how successful the attacks are, among which those rated 0 can be considered as high quality demonstration data. To better exploit these data, we introduce a hybrid loss by adding SFT loss of the samples rated 0 to the original MSE loss of all samples.

In experiment, we observe that ULMA performs better than Unlikelihood and point-wise RLHF on red-team (note that pair-wise methods such as RLHF and DPO cannot be applied to this dataset since each prompt only has a single sample), which supports the capability of ULMA's generalization to more complex metrics.

Q3: "The generalization to other metrics with positive and negative samples needs further description in details"

A3: Good suggestion! We have detailedly discussed how to generalize ULMA to datasets with continuous labels in Section 4.3 in the revised manuscript (at the end of page 6). Specifically, for these datasets, there is no direct separation of positive and negative samples. To apply ULMA, we can specify some sample as high quality data (e.g., the most helpful or harmless ones) and treat these high quality samples as "positive" demonstration data. Then we can adopt a hybrid loss by combining the SFT loss on these demonstration data with the original loss (e.g., MSE) on preference data. A concrete example on the red-team dataset can be seen in A2.

Q4: "What’s the objective loss of ULMA for continuous preference labels? In this case, how does the framework deal with positive and negative samples?"

A4: The objective loss for continuous preference labels is a hybrid of SFT loss on high quality demonstration data and the original loss (e.g., MSE) on preference data. In this case, some samples in the dataset (e.g., the most helpful or harmless ones) are treated as the "positive" demonstration data and applied to the SFT loss. An example on the red-team dataset is discussed and empirical evaluated in the revised manuscript. Please refer to our response in A2 and A3 for more details.