Hummer: Towards Limited Competitive Preference Dataset

Thank you for reviewing our work! However, we respectfully believe that the essence and contributions of our work have been significantly misunderstood.

Response to Reject 1 (paper writing): Note that alignment dimensions $d _i=${$x^k, y _w^k, y _l^k$}$_{k=1}^{K _{i}}$ refer to the sub-set of the preference dataset: such as "Harmless" highlighting safety in Anthropic HH. For others, please see the new manuscript.

Response to Reject 2 (The practical usefulness of Hummer): We respectfully argue that the contributions of our work are:

1. We are the first to quantify the conflict in preference datasets and introduce a new statistical metric.
2. We introduce a new preference dataset to mitigate the conflict.
3. We develop a hybrid sampling strategy to train reward models to improve interested metrics.

The practical usefulness of Hummer includes (even without hybrid sampling):

1. Improved performance on RewardBench: We observe 2% improvement on RewardBench over UltraFeedback, despite Hummer originating from UltraFeedback. (Table 1).
2. Insights into dataset quality: we emphasize the importance of data quality in further improvements in ADC and RewardBench with Hummer-F (Table 1).
3. Boosts defense against jailbreak attacks: With further fine-tuning, beyond low ADC, we observe that HummerRM exhibits strong robustness against jailbreaks (Table 2).

Also, in related work, we provided a plausible explanation for safety compromises during further fine-tuning of policies in the new version: the emphasis on certain alignment dimensions and conflicts among them (Qi et al, ICLR (2024)). We are expanding our jailbreak experiments with further policy fine-tuning to reinforce our hypothesis. We plan to incorporate these results in a future version and, hopefully, share them by the end of the rebuttal, pending resource availability.

Q 1.1 (Comparison of ADC): Yes! it remains comparable due to the normalization (n-1 in the denominator) and expectation term, making it a "mean" metric independent of the dimension number.

Q 1.2 (Why not use shared dimensions?): Existing preference datasets and alignment dimensions are well-constructed. This work is motivated by conflicts between alignment dimensions, necessitating new dimensions and datasets. However, we recognize potential overlaps in alignment dimensions across different datasets.

Thank you for appreciating our work and highlighting the critical issues of generalization over various dimensions of the proposed dataset! We provide the detailed responses in the following:

Response to Reject Reason 1 (lacks a thorough discussion on the generalization of various dimensions): We respectfully argue that this is not feasible at the current stage due to the following reasons:

1. This requires a significant computational budget, and the evaluation pipeline is considerably long and time-consuming.
2. Prior work, such as Anthropic HH and UltraFeedback, designs the alignment dimensions through heuristics.
3. We acknowledge that there may be better approaches to constructing the desired limited-competitive preference dataset over dimensions through unsupervised learning or other methods.

Response to Reject Reason 1.2 (the paper would also benefit from a more detailed test of the weak-to-strong generalization in alignment.): We respectfully believe that there may be some misunderstandings regarding the relationship of this work to weak-to-strong generalization in alignment. Notably, weak-to-strong generalization [1] typically introduces a novel methodology for RLHF to improve the benchmark performance. Conversely, our work aims to identify and measure the competing dynamics inherent in preference datasets. However, we acknowledge the possibility of misunderstanding the question posed. If Reviewer gJMS is interested in specific experimental designs related to weak-to-strong generalization that could further enhance our work, we would be pleased to know. We are willing to undertake additional experiments within the constraints of our computational budget.

[1] Burns, Collin, et al. "Weak-to-strong generalization: Eliciting strong capabilities with weak supervision." arXiv preprint arXiv:2312.09390 (2023).

Thank you for appreciating our contribution, especially the significance of the ADC in measuring the conflict of preference datasets. We have updated a new version of manscript regarding all following concerns.

Response to Q1 (clarify for $u _{-}^2$): Yes. $u _{-} = \min${$u, 0$} operator has higher precedence than the square operator. Otherwise, ADC/ADC-B $\equiv 0$ as $u^2 \geq 0$.

Response to Q2 (ADC-B can only be applied to two (or more) datasets with data on the same alignment dimension?): We clarify that ADC-B is comparable with different preference datasets within different alignment dimensions. The critical distinction lies in the "dataset" for evaluation $n$ further-fined reward models on $n$ alignment dimension of the test part of preference datasets (Hummer) VS. external dataset with $m$ ability dimensions (RewardBench). After that, for both ADC and ADC-B, it results in $n$ average negative performance deviations. The final ADC-B metric is holistic, calculated as the average of these $n$ deviations from $n$ further fine-tuned reward models.

Response to Q3 (Figure 4 with further fine-tuning): We sincerely thank Reviewer 9qZi for pointing out this important typo in our work! The corssponding alignment dimensions in Hummer is {'accuracy', 'conciseness', 'depth', 'empathy', 'tone', 'specificity'}, and the 'tone' dimension in our prior draft is actually 'specificity' (#5) in our new updated version, detailed in Appendix Tab. 6. Fine-tuning on 'specificity' (#6) will exhibit higher conflict to other alignment dimensions, e.g., 'Accuracy' (#1) focusing on free-of-error, 'Empathy' related to safety. However, we observe a contradictory phenomenon: further fine-tuning on 'specificity' (#6) results in a negative performance deviation across all other dimensions; conversely, it reduces jailbreak rates by approximately 5%. We hypothesize that the datasets designed for attacks may contain a fraction that emphasizes 'specificity' (#6) and we plan to explore this in our future work.

Response to Typos/Minor Comments: Thanks for raising these concerns and please see our updated manscript.

Grateful to Reviewer Z9ad for expressing interest in the paper and constructive comments. We have updated a new version of manscript for all concerns.

Response to Reject Reason 1 (Notations can be significantly improved): We briefly clarify the notation one by one:

1. $D _n^p=${$d _1, \ldots, d _n$}: Yes, each $d _i$ is actually a set of tuples, for the specific alignment dimension preference dataset, such as (such as Anthropic HH with 2 dimensions: Helpfulness and Harmlessness).
2. the expectation operator operates in ADC: This expectation is over alignment dimension $(d _{i\in n})$. Say, we firstly obtain $n$ negative performance deviations via $(\overline{\mathbb{U}} _i - \mathbb{U})$ for all alignment dimensions averaged by n-1 (except the further fine-tuned one), and we take the expectation of $n$ alignment dimensions to obtain a scalar metric with $\mathbb{E} _i[\cdot]=(\sum _{i=1}^n[\cdot])/n$.
3. Priority of operators: $u _{-} = \min${$u, 0$} operator has a higher precedence. Otherwise, ADC/ADC-B $\equiv 0$ as $u^2 \geq 0$.
4. Relation between $m$ and $n$: We clarify that the number of $m$ and $n$ are indepedent. $m$ refers to an external evaluation dataset that encompasses $m$ ability dimensions; while $n$ indicates the total alignment dimension in the training preference dataset. However, we note that certain dimensions in the preference dataset may significantly overlap with abilities in Benchmark.

Response to Reject Reason 2 (Ablations study.): We have conducted the ablation on our proposed hybrid sampling (HS) strategy. We have three main takeaways: (1) For the mitigation of inner conflict (ADC) and jailbreak, the low-conflict dataset is the main contributor, while an observable extra margin with HS. (2) For the improvement of RewardBench, HS is the primary contributor. (3) Data quality is important for further improvements. (Tabb 1,2).

We also provide a plausible explanation for safety compromises with further fine-tuning on policies: the emphasis on certain alignment dimensions and conflicts among them (Qi et al, ICLR (2024)) (Section 2). We are expanding our jailbreak experiments with further policy fine-tuning to reinforce our hypothesis. We plan to incorporate these results in future versions and, hopefully, share them by the end of the rebuttal, given the constraints on computational resources.