Interpreting Language Reward Models via Contrastive Explanations

We thank the reviewer for the constructive feedback. Please see our common response to all reviewers above, for details of our updated revisions and responses to two specific questions. We provide detailed responses to the questions below:

Response to weakness 1:

Please note that the process for determining the attributes can be very flexible, and in our case, it is semi-automated. Indeed the attributes generated by the LLM are not always consistent. In our implementation, we first obtain a dictionary of the form {attribute name: number of times occurred}, then we manually group similar ones, e.g. harmless and harmful, too-long, too-short, and verbosity. We further clarify the meaning of each attribute by providing a description to the LLM (see Appendix A), which should mitigate the ambiguity issue you mentioned.

Response to weakness 2:

Our Step 2 prompt says “changes made to response A should be centered around the following words: {relevant words for this attribute identified in Step 1}”, but does not ask LLMs to only change those identified words. The LLM might change the whole response for two reasons. First, in Step 1, the LLM sometimes marks nearly every word in the sentence as relevant to the attribute, therefore in Step 2 the LLM could change the whole sentence. Second, in the implementation, we have an error-handling mechanism to ensure experiments execute at scale: if errors occur in Step 1, then skip Step 1 and directly execute Step 2 while allowing the LLM to change the whole sentence. The second case very rarely occurs. According to our observations presented in Appendix D, our current configuration in Step 2 (asking LLM to center the changes around relevant words identified in Step 1) works well because perturbations produced by such prompt are: (1) more semantically and syntactically closer to the original responses than the ones produced by the “pass” prompt (not limiting the extents of changes), and (2) more semantically and syntactically distant to the original responses than the ones produced by the “only” prompt *=(asking LLMs to only change those identified words).

Response to weakness 3:

Perturbing the original response to become better or worse will depend on whether this response is the chosen or rejected response. We perturb the chosen response to become worse and perturb the rejected response to become better, to hopefully flip the RM preference.

Response to weakness 4:

We have additionally included more qualitative results revealing a new failure case of larger 8B RMs. Please see the common author response and Appendix G in the paper for more details.

We thank the reviewer for the constructive feedback. Please see our common response to all reviewers above, for details of our updated revisions and responses to two specific questions. We provide detailed responses to the questions below:

Response to weakness 1:

Please see the common response about the list of attributes.

Response to question 1:

The main difference is the target of the research. These existing works aim at evaluating the RM’s ability to capture various types of biases by curating datasets and testing RMs’ performance on them, while our approach aims at explaining any binary comparisons made by an RM (although by using our explanation approach we could perform similar analysis to theirs). Another notable difference is in the methodology. They perturb one response along certain directions and compare between those perturbations, while we perturb two responses in a binary comparison and compare the perturbations of one response to the other original response. Also, please note that our explanation method (Figure 1) could be instantiated with any customised list of attributes. In that sense, our proposed method complements these existing works.

Response to question 2:

Automated text perturbation procedures are indeed sensitive to the LLM used. For example, [Bhattacharjeea et al., 2023] and [Sen et al., 2023] compare various LLMs for generating perturbations for counterfactual explanations (in classification tasks) and augmenting training data, and it is evident that their quality (label flip rate in classification, semantic and syntactic distances to original responses, diversity, etc.) are affected by using different LLMs. Overall, the finding is that less capable LLMs will generate perturbations that are more similar to the original text but less frequently change the model prediction result. Also, better LLMs will more likely generate perturbations that match the meaning of the specified attribute and are more natural texts. Therefore, we recommend a user of our method to use the most capable LLM available (in our case GPT-4o) to maximise the usefulness of our method. Additionally, the quality of any perturbation-based XAI methods will depend on the perturbation quality. In our case, the formalisations of the explanation method are agnostic to the generator of the perturbations.

Response to question 3:

Please see the common response about the list of attributes.

Response to question 4:

The quantitative evaluations in our work (Section 3) follow the standard practice in the contrastive explanations literature, and the coverage metrics aim to test an explanation generation method's ability to obtain valid explanations, instead of evaluating specific aspects of the RMs. Discovering the sensitivity of RMs to the list of attributes is a target of the following qualitative analysis in Section 4.

Response to question 5:

Thanks for this comment, we added references to them in the revised paper in Appendix A.

Response to question 6:

Yes there’s a reason for that observation. It could happen that for one comparison, CFs (SFs) are found among the perturbed chosen responses but are not found among the perturbed rejected responses, which contributes to the lower coverages when checking whether CFs are present for both original responses.

Response to question 7:

We have additionally included more qualitative results revealing a new failure case of larger 8B RMs, please see the common author response and Appendix G in the paper for more details.

Response to question 8:

Thanks for this comment!

Thanks for acknowledging our responses and, again, for the very detailed comments.

We agree that the sensitivity to the LLMs that generate the perturbations should be mentioned in the paper. We further added comments about this in lines 284-286 in the updated revision.

About Question 6, sorry if that was not clear. Here is a more detailed explanation: the "Both CF (SF) Cov." evaluation metric checks whether or not for each test comparison (x, y+, y-) at least one CF (SF) is found for both y+ and y-.

Assume we have a dataset with three test comparisons, for comparison 1 a CF is found for y+ but not for y-; for comparison 2 a CF is found for y- but not for y+; for comparison 3 a CF is found for both y+ and y-. Then, for this dataset, the Chosen CF Cov. is 0.67, the Rejected CF Cov. is 0.67, but the Both CF Cov. is 0.33.

The same reasoning applies also to evaluating SF.

We thank the reviewer for the constructive feedback. Please see our common response to all reviewers above, for details of our updated revisions and responses to two specific questions. We provide detailed responses to the questions below:

Response to weakness on related work:

The existing works mentioned in lines 179-181 target classification tasks in textual data. They either use a custom fine-tuned LLM (e.g. [Wu et al. 2021]), or directly prompt LLMs like GPT-4, to generate counterfactual texts for one piece of input text. Our key novelty is proposing the framework for explaining pairwise comparisons, which involves two pieces of text and is therefore substantially different to existing works. Although the part about generating perturbed texts is similar, no existing works directly apply to explain RM preferences. In fact, no existing works apply XAI methods for RMs. In our baselines, we adapt [Wu et al. 2021] (PJ) and naïve prompting for LLM (RP) into our explanation framework.

Response to weakness on conceptual 1:

Please see the common response about the list of attributes.

Response to weakness on conceptual 2:

We consider the flipping preferences as central to our definition of contrastive explanation, which aims to explain why an RM prefers one response over another. Analysing cardinal reward changes does not directly serve this explanatory purpose, but analysing flip rates does. Also, an RM's predicted reward for a response, if outside the scope of a comparison, could be difficult to normalise and interpret due to the pairwise training procedure. The notion of binary preference flipping is more straightforward to understand, especially for non-experts. Nonetheless, the presented analysis pipeline is independent from our core explanation generation method, so alternative analyses using the cardinal measures could be conducted in future work.

Response to weakness on significance:

We have additionally included more qualitative results revealing a new failure case of larger 8B RMs. Please see the common author response and Appendix G in the paper for more details.

Response to Nits 1:

This is part of obtaining the representative local example that matches the global pattern. In the scope of this paper, they are to produce Figures 4 and 5.

Response to Nits 2:

Indeed, we have added more descriptions in the revision to make the figures clearer.

As the discussion period is closing soon, we're wondering if the reviewer has any final questions or comments. Please note that we have updated the paper with revisions addressing the reviewers' initial comments, discussed in the "Common response to all reviewers". Thanks!

We thank the reviewer for the constructive feedback. Please see our common response to all reviewers above, for details of our updated revisions and responses to two specific questions. We provide detailed responses to the questions below:

Response to weakness 1:

In this work, our aim is to interpret the preferences made by RMs, which can be used for LLM alignment both at training time (in RLHF) or at inference time (e.g. best-of-N sampling). In the latter case, RMs are used standalone, and better-understood RMs make more trusted comparisons over given LLM responses. In the former case, if the final LLM produces poorly-aligned answers, with existing methods it would be difficult to know whether the LLM has correctly optimised a wrong reward, or it has wrongly optimised a correct reward (e.g. reward hacking - the LLM exploits loopholes in the RM to maximise reward). We see our proposed explanation generation method as one valid starting point towards investigating these problems. Meanwhile, more understanding of RMs has been strongly advocated in the literature (see lines 31-45 in the paper), and our method targets this direction.

Response to weakness 2:

Mentioning minimally and maximally was to introduce intuitions behind how contrastive explanations are traditionally useful in a tabular data domain, where explicitly computing a minimum or maximum distance (e.g. L1, L2 distances) perturbation of the input is possible. Our proposed way of generating perturbed responses is in line with existing works of generating counterfactuals in textual data for classification tasks, where directly computing high-quality explanations through minimising or maximising distances with gradient descent (as done in tabular and image data) is not applicable because the final perturbation would generally not be well-formed sentences. We agree that iteratively prompting LLMs to increase or decrease the extent of perturbation could be an interesting future extension of our method.

Response to weakness 3:

Please see the common response about the list of attributes.

Response to Question 1:

We have additionally included more qualitative results revealing a new failure case of larger 8B RMs; please see the common author response and Appendix G in the paper for more details. Apart from that, we see the qualitative result about model v2's sensitivity to harmful contents as a useful validation, because the fact that v2 was trained to detect harmful contents was not visible to our method. We also demonstrate the usefulness of our method for obtaining more insights into why an RM prefers one response over another in Figure 4 by validating desirable RM behaviours and revealing unexpected behaviours.

As the discussion period is closing soon, we're wondering if the reviewer has any final questions or comments. Please note that we have updated the paper with revisions addressing the reviewers' initial comments, discussed in the "Common response to all reviewers". Thanks!