We thank the reviewer for their thorough, and careful responses to our paper, and outline the changes we have made to address these concerns.

Regarding presentation and clarity

We are indeed focusing on the implicit reward models learned by our policy models, as the reviewer suggests. We now clarify this right from our abstract and do a thorough pass over the paper standardizing and clarifying where we are referring to:

• The desired reward model which is the objective of the RLHF training, or
• The implicit reward model which is learned by the policy model.

We have made an extensive pass clarifying language, notation and presentation throughout, and updating all of our figures.

Regarding unmotivated methodology

In response to the reviewer’s comment about better motivating our design choices, since our initial submission to ICLR and further spurred by the reviewer’s kind suggestion, we have made the following additions:

• We explore the efficacy of these different choices for our autoencoder setup via two different metrics namely:
  • The reconstruction error in the activation vector, and sparsity of the features that are extracted.
  • The average utility of the top-k feature descriptions, that we presented in the original draft of our paper.

Using the above two metrics as our objectives, we have added the following explorations to our paper:

• We add experiments for selecting layers to extract autoencoder features based on different choices, including:
  • Highest parameter divergences, as in the original draft of the paper.
  • Lower most layers, as suggested in the cited work of “Sparse autoencoders find highly interpretable features in large language models”, by Cunningham et. al.

We utilize these same metrics above to explore a sampling of choices in the autoencoder setup, namely:

• The choice of whether weights should be tied between the encoder and decoder of the autoencoder.
• The L1 “sparsity” loss on the feature dictionaries.

Regrettably, we are not able to engage in a deep study of the Bills et al methodology for auto interp, primarily due to the cost and latency involved with running high volumes of GPT-4 queries.

Regarding comprehensiveness of models and tasks

• We have added results for feature dictionaries on a GPT-NEO model. In addition, we commit that we will include autoencoders for GPT-J 7B in a camera ready version.

• Along with the results for Pythia in the original submitted draft of the paper, this would give us results for 3 different model architectures of varying sizes

• While we readily concede this is still not fully exhaustive, we note that for instance the original paper by Cunningham only studied one model size the Pythia architecture, and that we have been able to validate the general method works in a broader context.

• To show the utility of our method is not confined to sentiment based text generation, a camera ready version would include an appendix section with a small case study for the very widely used “Helpless and Harmless” hh-rlhf dataset from Anthropic.

For transparency, we clarify that the GPT-J and hh-rlhf case study would be done before the camera ready version, but are not likely to make the 22nd revised submission deadline.

Regarding limited experiments and evaluation

We extend our quantitative analysis by:

• Demonstrating correlation of the summed reward utility of top-k extracted features, with the original utility scores of generated text.
• Using the top-k extracted features to find a reward modeling failure.

In general, these added results demonstrate more suggestively than the original draft, that the extracted features can be used both as a rough proxy to the quality of the implicitly learned reward model, as well as a means to expose reward modeling features.

On questions posed by Reviewer

• As mentioned earlier in this review, we flesh out at least a sampling of the methodology choices outlined in Section 4, including different ways to select layers and permutations on the autoencoder setup.
• On what we believe to be the most exciting result, we believe it is the (now further examined) correlation between the performance of the trained policy model on the RLHF objective, with the measured utility of the sparse coding feature descriptions extracted via GPT-4 on the activations of the sparse codes.

We thank the reviewer for their thorough and careful responses to our paper, and outline the changes we have made to address these concerns.

Correcting notation

As suggested, English notation has been unified throughout the paper (for example in the case of the token/word confusion, as well as the reference to reward models, for which we provided an acronym for the phrase “implicit reward model” (the object of our study)).

Removing typos

We corrected a large number of typos where they were identified, as well as some formatting issues with the original paper that made the mathematical notation confusing. (See the response to reviewer 1 for more details on the typos removed.)

Adding more results

As suggested by the reviewer, results have been updated to include the Pearson Correlation Coefficient in the appendix, and some additional experiments have been provided to support the conclusion of the paper: * We search for specific features changes caused by fine-tuning that contradict the provided table of utility values for a concrete example of a reward modeling failure. * We correlate the utility of the description of the top-k features with the average utility of generations over a corpus of text, to show the relationship between an increase in the utility of the description of the top-k features and the model’s propensity to generate high utility completions.

Validating GPT-4 descriptions

The process of computing the Pearson Correlation Coefficient are now included in the appendix of the paper, and hopefully constitute validating GPT-4 descriptions. As explained in the updated paper version and in prior work, the Pearson Correlation Coefficient, involves simulating activations if the provided description were true, and then computing the difference between the simulated and true activations over a corpus of text. As such, this is an explicit quantitative measure of the validity of GPT-4 feature descriptions.

Once again, we would like to thank the reviewer for their careful feedback and helpful suggestions.

We continue our response here to Reviewer 1's very thoughtful and helpful feedback.

On typos and unclear bits:

All identified typos (and others found after multiple re-reads) have been corrected in accordance with the reviewers suggestions. In addition we have made all references to tables and figures explicit, rather than positional.

On General Confusion:

By adopting the suggested naming scheme of ‘implicit reward model’ (which we abbreviate to IRM), we hope to have addressed the ambiguity between which reward models are being referred. For example: “Do implicit reward models (IRMs) learned by Large Language Models (LLMs) through Reinforcement Learning from Human Feedback (RLHF) diverge from their intended training objectives?” “As LLMs steered via RLHF scale in capability and deployment, the implications of failures in the IRM amplify.”

As for Reviewer 1's concern with the use of the term ‘reward model’, when the model may be learning some sort of "value function". We clarify that we use the term here not to refer to implicit internal computation of reward, but rather the presence of explicit linear structure within the activations that correspond to the fine-tuning objective.

On statements in the paper.

Parameter divergence was used to discard layers with little divergence to avoid running unnecessary experiments. Training autoencoders on activations from layers with such little change from the base model, that their probability of containing components of IRM was small. This is admittedly a choice also based in convenience and practicality.

On experiment suggestions

We chose two autoencoders, because the cosine similarity metric that has garnered popularity in determining the most effective dictionary features is a pair wise measure, only applicable when two autoencoders are trained. (We hope this is more clear in our rewritten submission.)

This measure could be modified/extended, but it would be a significant and compute intensive perturbation to existing methods, which we were gauging the effectiveness of for reward model interpretation. We agree that it would be interesting to experiment with more autoencoders and alternative similarity metrics in the future.

In general, we thank the reviewer again for their helpful insights, and hope our response demonstrates we have taken the suggestions seriously

We thank the reviewer for their thorough and careful responses to our paper, and outline the changes we have made to address these concerns:

Clarity

• Sections 4, 5 and 6 have been rewritten with clarity in mind, and so that each section is more tightly focused. We now open each section/subsection with why its contents are relevant and finish those sections with a one sentence recap of what the takeaways are from that section.
• Section 4 now gives an overview of the methodology and approach. Section 5 gives all the experiment pipeline details, namely the external reward model definitions, the model hyperparameters for both the RLHF stage as well as for the autoencoder training. Section 6 gives the qualitative analysis of case studies of feature dictionaries, and the quantitative results measuring the summed utility of these via GPT-4 .
• We also re-organize material from sections 5 and 6, so that section 5 focuses on the experiment setup and parameters, and section 6 focuses on the case studies and results, whereas before they were less distinct.

Margins

The issues with margin size have been corrected.

Paper layout

As mentioned prior, we hope to have corrected this via the new structure we adopted for these sections, as well as the reorg of sections 5 and 6.

General confusion

By adopting the suggested naming scheme of ‘implicit reward model’ (which we abbreviate to IRM), we hope to have addressed the ambiguity between the reward models involved. For example: “Do implicit reward models (IRMs) learned by Large Language Models (LLMs) through Reinforcement Learning from Human Feedback (RLHF) diverge from their intended training objectives?” “As LLMs steered via RLHF scale in capability and deployment, the implications of failures in the IRM amplify.” In general we clarify this paper focuses on implicit reward models, and adjust terminology usage accordingly.

As for the concern with the use of the term ‘reward model’ when the model; we use the term not to refer to implicit internal computation of reward, but rather the presence of explicit linear structure within the activations that correspond to the fine-tuning objective. We have added an experiment that shows correlations between these and the actual reward.

Falling short of answering the key question.

We have adjusted the abstract to account for our focus on interpreting the implicit reward model, rather than on correspondence with the desired reward model. We also include additional results that explicitly do illustrate reward model correspondence by correlating the utility of model generations with the utility of the GPT-4 descriptions of the top-k features. We still soften the claim in the abstract, as we can’t yet fully answer this reward model correspondence question, but hope this makes our answer to said question more substantial.

Reproducibility

Source code has since been provided anonymously at https://anonymous.4open.science/r/rlhf-C7C8/. We will continue to update and clean this repository before any camera ready version.

Algorithm presentation

• We adopted the reviewer’s suggestion of additional colors for natural language details in the algorithm in the appendix, and used the suggested convention to convey extracting the dictionary from the autoencoder. We also specify that the activations for the autoencoder were from the train split of the imdb dataset.
• “in lines 10 and 11, shouldn't the autoencoder size be smaller than the Ativation layer dimension?” The power of the sparse autoencoder is not so much in dimensionality reduction, but rather the sparse linear structure of the autoencoder disentangling superposition in the original MLP. In fact, Anthropic’s blog on sparse coding published after our abstract submission scaled up the hidden size dimension by 256x! (See https://transformer-circuits.pub/2023/monosemantic-features/index.html).
• We originally had language in the paper suggesting the power was in the dimensionality reduction. We apologize for this source of confusion, and have since clarified the verbiage.

We respond to the remainder of the review in our next comment.

Response

I thank the authors for their responses to my comments and questions. The reformatting and fixing of typos is great (I only see two typos now: "Seeing appendix", and "Elhage et al" should be \citep).

I'd also like to compliment the authors on their general code style and the presence of docstrings. I'd gently encourage more use of dataclasses, which can reduce clutter. I'd also like to express my appreciation for section 6.1, which recognizes important shortcomings of the approach as is. Finally, I also quite like the new visuals, which add to the clarity of the presentation.

All this said, I still feel like there is more work to be done before this paper is quite at the level I feel comfortable recommending (to give a numerical response, I'd say it's now a 4.5). This is primarily due to an ongoing sense of general confusion (to be clear, it is better than it was before). An illustrative example: a utility table U is mentioned in section 4.2, but as far as I can tell doesn't appear in the paper or appendix. Then in section 5.1, the authors mention giving higher rewards to positive sentiment prefix-completion pairs – what happened to the utility table task?

Misc questions/thoughts

• which checkpoints were used for the Pythia models?
• is it obvious that the Bills et al. approach is well-suited for this setting (models of this size)?
• You say that “good” and “happy” should be more prevalent in the feature descriptions, but it seems like what you should care about is whether the feature descriptions measure positive/negative sentiment? Like the description could be “this sentence contains 'good/bad'” or “this feature senses positivity/negativity in the review” and that would be relevant. But if the description says "this feature recognizes reviews which have good writing" then that would contain the word "good" but not be the kind of feature you're looking for.
• What’s the absolute utility of the top-k most similar feature descriptions, and how do you get it?
• How do you measure similarity between feature descriptions?
• from above: a utility table U is mentioned in section 4.2, but as far as I can tell doesn't appear in the paper or appendix. Then in section 5.1, the authors mention giving higher rewards to positive sentiment prefix-completion pairs – what happened to the utility table task?
• the IRM/ERM distinction is much clearer than before, but there are still some places where "reward model" is used to (I think) mean IRM.
• it would be good to have more justification of why we can assume that IRMs are definitely a thing that's being learned during RLHF.

Conclusion

I hope the authors can find this encouraging vs discouraging, as in just a week, it seems significant progress has been made. I wish the authors the best of luck with this research!