SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks

Thank you for taking the time to review our paper and for the insightful comments.

• [W1] Thank you for this remark. We believe this is easily addressed by us going through the paper, and changing some expressions to be more precise. In this particular example you raise, we are referring to “moderate KL divergence” in comparison to the other PO methods we examine and not on its own. All our KL comparisons are relative and we do not rely on the actual values they obtain.

• [W2] First of all, as it has been shown in Rafailov et al. [2023], the reward-KL frontier should not be necessarily used to measure performance of a PO algorithm; it is mostly provided as an analysis of the algorithm’s expected behavior. Nevertheless, in Figure 2, the best reward frontier is achieved by MaPO which is our dense proposed approach for mask calculation. On the contrary, SparsePO achieves the best reward on high KL values compared to other methods. Finally, with respect to code, we acknowledge that in this particular domain the method is not performing adequately.

• [W3] To further strengthen the contribution of the paper we have included additional win-rates for HH and TL;DR datasets (please look at the general response). We are also in the process of performing evaluation on the OLLM-v1 framework, which might be more suitable for $>$2B parameter models. We will update our response with the results before the end of the rebuttal period.

• [W4] As mentioned in our responses to Reviewers 5gHZ and 6TmP, previous work in modeling preference at finer levels than complete responses has shown benefits in downstream tasks. In this context, our method presents the following advantages. First, we introduce an explicit way of controlling for the preference of each token during optimization. Intuitively, certain preference proxies (e.g. toxicity, sentiment) can be better modeled when the optimization signal is constrained to the most indicative tokens in the response instead of the entire response, potentially filtering out noisy signals. In our method, token preference is controlled by a specialized mask, having the option to leverage the model activations (MaPO, non-learned) or learning a sparse mask (SparsePO). When learned, the resulting masks achieve a level of sparsity optimal to a specific preference proxy. Please also refer to the general response to reviewers where we summarize our contributions and provide additional results.

• [Q1] In Figure 2 we report the trade-off between response-level KL divergence and reward aiming to visualize their Pareto frontier. In order to obtain this frontier, we use an existing sentiment classifier that acts as our ground-truth reward - a process that has been also used in the original DPO paper. Lower KL ranges indicate less divergence from the reference model and high rewards indicate that the method is able to generate more samples with positive sentiment. We find that MaPO is able to produce the most examples with positive sentiment while not diverging from the reference (high reward, low KL). On the other hand, SparsePO trades-off reward values for KL - meaning that when the model diverges from the reference model, SparsePO is still able to generate more positive reviews compared to other methods. This underlines the desired behavior of our method–by allowing more tokens to diverge from the reference we enable the policy to generalize to new answers. As mentioned in a previous response, this reward and divergence should be used to analyse the behavior of the approach, and not necessarily to evaluate its efficacy.

• [Q2] The comparisons we make for KL divergence are relative and not based on hard thresholds. We observe that TDPO-v1 overall does achieve lower KL compared to SimPO, DPOP and SparsePO, bringing to a middle point in regards with the other reported PO algorithms.

• [Q3] Thank you for pointing this out. This is not phrased in the best possible way, as we did not mean to claim any causal relation. We will rephrase to stress the correlation between KL divergence and obtained expected reward instead.

Thank you for taking the time to look at our responses. With respect to the points you raised now, please see below.

In the statement I highlighted in your reponse, could you point me to your experimental results for that? It is indeed good if SparsePO does that even at high KL divergence compared to other methods. Could you highlight this in your paper?

We have added an analysis in the amended paper, Appendix C.1, lines 1049-1069. In summary, we show that responses generated by SparsePO obtain a high ground-truth reward (as given by a sentiment classifier) whilst exhibiting higher KL divergence levels, compared to PO baselines.

I am not suggesting you should base your analysis on hard thresholds. I am suggesting that you could describe your results more quantitatively so that your point that "TDPO-v1 overall does achieve lower KL compared to SimPO, DPOP and SparsePO" is solidly supported.

Thanks. Could you provide your rephrased analysis?

Apologies for the misunderstanding, we have now included quantitative comparisons in our analysis (please see lines 232-244 in the updated manuscript).

We have amended this in the newest version of the paper that is now uploaded. Please refer to lines 232-244 for a re-phrasal.

Thanks for this and so empirical results suggest that lower sparsity leads to higher KL and Rewards. I would have expected based on this analysis, the paper would attempt to find good performing models at lower KL than other PO methods. Because otherwise the masks are not sparse (sparsity% < 20%) as suggested by the provided Table.

Thank you for this remark, please let us clarify. We investigated two SparsePO setups, one in which a common mask is learned for both U and D (SparsePO$[m_u=m_d]$) and another in which a separate mask in learned for each (SparsePO$[m_u \neq m_d]$). For $m_u=m_d$, it is indeed the case that a low sparsity leads to higher KL and rewards (reported results in the response table). However, for $m_u \neq m_d$, the sparsity in $m_u$ and in $m_d$ both interplay with KL, as seen in Figure 3 of the paper. Notably, high $\beta$ and high $m_u$ sparsity leads to low KL, whilst low $\beta$ and high sparsity leads to high KL. This last setup shows that SparsePO$[m_u \neq m_d]$ achieves high reward mask ($m_u$) sparsity, low KL mask ($m_d$) sparsity, high KL divergence and is still performant. This result also confirms insights in previous works investigating sparsity in the objective's reward term \citep{yang2024selective}.

Could you point me to specific systems in your results that have non-trivial level of sparsity and with substantial gains compared to other PO methods? I think these should be highlighted, but are not clear in the current version.

We provide sparsity plots for sentiment control (IMDB) and text-to-code generation (MBPP) in Section 3.2 and 3.5, respectively. As we observe, the sparsity of the masks changes very differently across tasks throughout training. In IMDB, we start with very high sparsity (80%) which eventually decreases whereas in code sparsity is overall stable between 20-26%. These results indicate that the amount of sparsity of each task significantly varies, making it `not trivial'. We also provide a complementary analysis, looking at the distribution of mask values for HH and TL;DR in Appendix C.3, which combined with our downstream task results suggest that sparsity does help achieve better performance.

Why can't other PO methods such as DPO achieve this by using a smaller beta value? And I am not sure what results in the paper specifically support the claim.

Please refer to our additional analysis in Appendix C.1, where we show that for a low value of beta ($0.1$), DPO and TDPO v1 are not able to achieve high rewards for high KL values.

What is the reference performance that the +10% is measured? It will be helpful to put together a table showing the best performing model produced by each method and controlled for sampling temperature. From my reading, the claim is based on comparing SparsePO to SFT rather than to DPO.

We apologize for the confusion, the win-rates of HH were accidentally mixed with the ones reported for TL;DR. We have updated the tables and we also present the correct results in Figures 5 and 7 of the latest version of the paper. Based on the highest score of each PO method, for HH we get: +6.8% over TDPO-v1, +12.6% over TDPO-v2 and +5.6% over DPO with SparsePO. For TL;DR we get: +1.8% over TDPO-v1, +4.6% over TDPO-v2 and +3.2% over DPO with MaPO.

Thank you for your time and feedback on our work.

• [W1] Thank you for your comments. Regarding validation for the core motivation of our work, we state the following. From a reinforcement learning perspective, Knox et al. [2022] lay down theoretical foundation on why taking into account the preference during each transition in a trajectory segment (e.g. explicitly modeling the reward of each part of a response) is better suited for human preference than modeling preference over the complete segment, a.k.a outcome reward modeling. This line of work has lately taken shape into ‘process’ reward modeling [Lightman et al., Wang et al., 2024], which has proven empirically beneficial for tasks in which the response can be divided into multiple steps, e.g. multi-step reasoning tasks. For the specific case of modeling tokens as individual steps, Zeng et al. [2024] further build on this line of work and laid theoretical and empirical validation, in which this work also builds on. Other work has also reported evidence of the importance of token-level modeling at different stages of training such as pretraining [Lin et al., 2024] and preference optimization [Yang et al., 2024]. From a decision making perspective, as studied in cognitive psychology and economics, human preference can be more precisely explained in realistic scenarios through dynamic decision making [Hotaling et al., 2015], in which a series of interdependent actions must be taken over time to achieve a goal. In such scenarios, preference is modeled at each time step, influenced by previous choices. In the context of natural language tasks, the need for non-uniform preference at the token level can become intuitive, e.g. particular words rendering a complete response toxic or positively sentiment laden. Figure 1 presented an example of such a case, where DPO implicitly learned preference at the sub-response level. In contrast, for formal language tasks such as code generation, such intuition is less clearly defined, since all tokens contribute to the executability of a function. Please refer to the General Response for more elaboration on this. Lastly, we would like to state that we will gladly present any additional validation you deem necessary, should you ask for it.

• [W2] We fear there might be a misunderstanding, we will try to make the wording in the paper clearer. We do not claim optimization equivalence with the TDPO paper, we simply follow a similar process to derive the final objective, as presented in Equation (6). Please check the Appendix for the detailed derivation steps. Our objective is equivalent to TDPO only when both mu and md masks are equal to 1.0 (as stated in lines 317-319).

• [W3] We understand that the choice of modeling the masks with a simple FFN might seem simplistic. We did some preliminary experiments with different architectures (e.g. 2 FFNs) but they exhibited worse performance overall. However, our goal in this work is to showcase the flexibility that our approach introduces in controlling the contribution of each token in the objective and not to investigate all potential masking strategies or architectures. We propose two different ways of modeling masks, first via model activations and second via a learnable component, but it is possible to learn a masking strategy using other methods, e.g. via external models or even manually creating masks for each example. We leave this exploration for future work.

• [W4] Unfortunately, we fail to understand how to calculate helpfulness and harmlessness metrics without having human judgments. Could you please elaborate? To circumvent this, we have provided win-rates with GPT-4 as a judge as the closest evaluation instead. Please refer to the general response for the numbers.

• [W5] Thank you for the suggestion, we are providing the corresponding results in the general response.

• [W6] As mentioned in previous responses, we acknowledge that performance on text-to-code is not ideal. However, we do not expect the method to work perfectly for everything and we provide a plausible explanation on why we believe our method is not as effective in this particular domain in lines 428-446 of the paper with additional arguments in the general response.

Thank you again for your valuable feedback on our submission! We have attempted to respond to your comments in our rebuttal and would like to kindly ask you if you feel we addressed your concerns or if anything needs further clarification.

Thank you for your time in reviewing the manuscript and for your suggestions.

• [W1] This is a quite insightful remark, thank you. Firstly, we should underline that we do not actually have access to token-level preference annotations and as such, we cannot make any claims that SparsePO matches human-level preference on tokens. What we claim instead, is that by adjusting the weighing over tokens we can better match human-level preference on the complete response. However, we could provide an analysis investigating the correlation between mask values and token ‘relevance’ to a preference proxy, further down during the rebuttal period. Regarding our qualitative analysis of chosen and rejected responses (Figure 4 and 9, respectively), we do observe that the masks are sparse and indeed only tokens that are important towards the preference are highlighted, though we do acknowledge that the coloring visualization makes distinguish relative differences difficult. To further support this analysis, we provide statistics of the distribution of mask values in the table below, showing quantiles at 0.05 (Q5) and 0.95 (Q95) as well as the mean (m) of the U term in Equation (6), for chosen responses. We will include this analysis as a histogram in the revised version of the submission. Finally, indeed, our method relies on the PO dataset at hand to model token-level supervision signals for ‘crucial tokens’.

• [W2 & W3] We acknowledge the low performance of our method on text-to-code generation benchmarks. We believe that this particular domain might not be the best for SparsePO for the reasons highlighted in lines 428- 445 of the paper and the general response. However, we do manage to achieve better performance based on the respective metrics on all the other benchmarks.

• [Q1] We have not tried to train the sparse mask via Gumbel softmax, though we agree that it is a valid option for achieving sparsity. In our approach, we managed to regulate sparsity via L1-regularization and consider different avenues as beyond the scope of our work.

• [Q2] Thank you for raising this concern. We delved further into the following indicative of reward hacking, reward ’accuracy’ (chosen reward > rejected reward), and report its progression during training for DPO, MaPO, and SparsePO models, in the Table below, for the sentiment control scenario. We observe that DPO quickly reaches a high accuracy, likely due to lowering the probability of rejected responses instead of increasing the probability of chosen responses. DPO’s accuracy of almost 100% is a clear indicative of reward hacking. For MaPO, we observe similar trends, albeit converging to a lower accuracy. However, SparsePO avoids reward hacking by managing to converge slower and to lower accuracies than the other models, whilst remaining performant in downstream tasks, as evidenced by our main results.

Thank you again for your valuable feedback on our submission! We have attempted to respond to your comments in our rebuttal and would like to kindly ask you if you feel we addressed your concerns or if anything needs further clarification.

Additionally, we added further evidence to the inquired points in the paper, please have a look at the latest version where additions/changes are highlighted with different colors. Specifically;

Learned sparse masks do not necessarily match human preferences: ...

[response] This is a quite insightful remark, thank you. Firstly, we should underline ...

We also added an analysis on the distribution of mask values and token-level KL divergence in Appendix C.3., for the tasks of summarization, dialogue, and code generation. In summary, we show that in cases where mask values highly concentrate around zero (and hence, high sparsity), KL divergence tends to be high throughout training, indicating that the few tokens that are allowed to diverge are diverging quite largely. Please also refer to Appendix C.6. for additional qualitative analysis on different tasks and the tokens that are crucial for them.

When the KL constraint is very small, a larger reward ...

[response] Thank you for raising this concern. We delved further ...

Summarization is another example where SparsePO avoids reward hacking, as illustrated in Appendix C.3. In summary, we find that most tokens in a summary response contribute to the reward whilst the KL divergence is higher than baselines.

We would like to thank you for your time and insightful comments!

• [W1] We understand that TL;DR can be problematic as a dataset and as such, to avoid any possible bias based on summary length, in the paper we filtered out training instances with chosen and rejected responses with a length difference greater than 100 words. From this resulting filtered dataset, we uniformly sampled 40k and 8k preference instances to form the training and test set, respectively (more information about this can be found in the Appendix). From this test set of 8k, we further uniformly sampled 100 unique prompts. We will extend this evaluation to include all the 1889 unique prompts of the test set. We will add these results in our rebuttal in the coming days. Regarding faithfulness, could you please elaborate on what analysis you would like to see? We can suggest to further filter out instances in which the gold summary is deemed as unfaithful, based on an existing faithfulness metric. Or to separate results of automatic metrics by faithfulness level of the system summaries. e.g. separated by low, mid, and high faithfulness. Regarding the suggested metrics we could report ANLI and SummaCSZ as well. Unfortunately, we cannot include Q2 easily, as no official implementation is available.
• [W2] We agree that OpenLLM-v1 might be a better choice for the model size that we report, hence we will also share the evaluation results on OLLM-v1 by the end of the discussion period. In the meantime we have expanded our HH analysis to include win-rates; please refer to the General response.
• [W3] Indeed we observe the results in text-to-code generation might not always be positive. We attribute this to the observation that for successful code execution, restricting learning to only a handful of tokens might not be adequate as mentioned in the general response and in lines 428-446.
• [Q1] Following previous work on offline preference optimization, in this study we chose to focus on offline algorithms and leave online ones to future work. Nevertheless, we certainly agree that PPO is not inferior to DPO, as prior work has shown. Actually, our approach is orthogonal to PPO, meaning that it can be easily extended to include masks.
• [C1] Thank you for pointing these inconsistencies (method/stratefy/objective/framework), we will make sure to address them in the manuscript.

[W1] Your analysis suggestion of filtering unfaithful data (as done here: https://aclanthology.org/2023.findings-acl.220.pdf) indeed makes sense and will increase my trust in your data and results. I think ANLI and SummaCSZ are sufficient.

[W2] I await your new results. I am unsure how you got the new HH win rates, and you got the 10%/3% improvement.

[W3] I find your response problematic. You state that your method is more capable in use cases that aim to "reduce toxicity" and "improve style", but less capable in code as all words are important in tasks that seek to "improve functional correctness". Given that statement, it is odd that you train on HH and evaluate on OpenLLM-2 which aims to provide a broad set of evaluations that don't relate to toxicity and style. Moreover, I think that math is very similar to code in that aspect, but yet training in HH improved it a bit. Still, you can suggest that training on the target task is different. So probably to convince me that your explanation is not just an ad-hock explanation of a bad result but a real failure mode of the approach it will be beneficial to show it has some predictive power. For example, you can train on math tasks and demonstrate lower results with your method.

Thank you for your comments.

Figure 6 is small, and 7 does not have a legend of its own making it hard to read.

We will make sure to add a legend to Figure 7. Since we cannot upload another paper version, please use any other legend (e.g. in Figure 6) as we made sure that the colors remain consistent across methods.

Results on OLLM-v1 make more sense. Are they significant?

All automatic metric results (in HH and TL;DR) are tested for statistical significance at the system level using Bootstrap resampling (Davison & Hinkley, 1997) with a 95% confidence interval (Appendix B.3, L949-L951). Such non-parametric test is most appropriate when the distribution of a metric value cannot be assumed to be Gaussian, as is the case for metrics employed in NLG tasks. Additionally, Bootstrap resampling has become the established method to report the significance of results in tasks such as summarization and QA.

Given the three versions of your method, the win rate seems mixed, and it is not clear which one is better, and if there is one that is better across the different experiments. SparsePO[mu = md] in HH, AND Mapo in TL;DR.

In the paper we have proposed 3 different strategies for applying masks over tokens to control the contribution of each token in the PO objective. Across experiments, the methodology itself clearly shows a consistent improvement in cases where indeed only a handful of tokens are important for steering the preference. As expected, which specific masking strategy is most effective ultimately depends on the preference proxy.

Okay. Nonetheless, you did not show that your explanation has a predictive power.

In the paper we report several cases that illustrate the predictive power of our method. In sentiment control, the Pareto frontier and our additional analysis in the previous round of responses illustrate that our method is able to generate more samples with the desired proxy. In summarization, existing metrics for summary quality and gpt-4 win-rates illustrate gains over other PO methods. In HH, OpenLLM v1 and v2 scores with gpt-4 win-rates provide evidence that our method improves over baselines. Could it be possible to clarify what do you mean by predictive power if the above response is not adequate?

The experiments do not match the expectations raised by the motivation, focusing on reducing toxicity or style adaptation.

Although we used some examples from HH and style adaptation to help illustrate our motivation, please note that by no means is our motivation limited to those tasks. Our motivation is that preference can be better attributed by weighting different tokens in a response, especially in natural language, an insight backed by previous work. Most notably, we presented evidence that our methodology leads to gains across challenging benchmarks beyond toxicity or style adaptation (OLLM v1 and v2).

Improvements are relatively small even for natural language tasks, and not systematic across the methods versions, and the different temperatures, making it hard to determine the nature of the improvement without significant testing.

We would like to point out that the relatively small difference in automatic metric values might be due to their inherent lack of sensitivity and their difficulty. For instance, a difference of half a percentage point (in average) over the OLLM leaderboard tasks is considered as a significant improvement in previous PO work. Additionally, as mentioned above, please note that all our results are tested for statistical significance. Regarding the observation of systematic differences across temperatures, it is a commonly observed behavior of PO methods to exhibit wider performance gaps at lower temperatures, with the gap completely closing at $T=\{0.75,1.0\}$. Our method exhibits this behavior too. Finally, the effectiveness of our methodology is reported across a varied set of tasks and discussed at length with dedicated experiments, as others reviewers kindly acknowledged.