Thank you for the insightful review.

We are pleased that you found our research direction good, and that releasing our SAE infrastructure would be beneficial to the community.

Clarification on SAE Feature Probing

"Unclear why have to probe on top of SAE feature. SAE greatly increase the dimensions of the features, leading to overfitting… Lacking comparison to normal probing."

We want to emphasize several key points:

1. Our SAEs do not modify the dimensions of the activations. The hidden size is the same value as the input size.
2. The accuracy of the probes we report in our paper is the accuracy on an unseen test dataset.
3. Based on your response, we have included a comparison to probing on the raw activations in the PDF attached to the rebuttal visible to all reviewers (Table 2).

Our findings show that probing on the SAE outputs does not meaningfully affect the accuracy of the probes. It also provides the benefits of the inputs being probed being more interpretable (Cunningham et al., Bricken et al.)

Dynamical Perspective on RLHF Training

"Considering the problem from a dynamical perspective can be fruitful… it's also interesting to see the progress of RLHF training, how it warps the features spaces, even the SAE features' relative importances."

We agree that studying the problem from a dynamical perspective would be very interesting. In response:

1. We will mention this as an exciting direction for future work in the camera-ready version of the paper.
2. We aim to include results of our method at various checkpoints throughout fine-tuning in the camera-ready version of our paper.

Analysis of Reward Models

"I wonder if it could be interesting to conduct the same analysis on the reward model… Can we compare the two representation space…?"

We agree this is another interesting direction:

1. We will include this in our discussion of future work.
2. Riggs et al. recently trained some SAEs on preference models with good results, suggesting this approach is feasible.

Challenges and potential solutions:

• Our fine-tuning tasks don't have true reward models (VADER uses lexicon labels, helpful-harmless and toxicity tasks use pairwise preference data).
• If this is a significant concern, we could train classification models for the pairwise preference data and compare the representation space of this classification model with our probes/sparse autoencoders.

The results in the PDF attached to the rebuttal visible to all reviewers will be incorporated into the camera-ready version of our paper.

We thank the reviewer for their thoughtful review and appreciate that you found our paper accessible.

Clarifying the Paper's Objectives and Takeaways

"…the takeaways of this paper are somewhat unclear. … if we finetuned a new model on one of the datasets used in this paper and trained probes in a similar way from its activations, what would that tell us about the the difference between the base and RLHF versions of that model? Alternately, is the goal to discover information about a model where the base and RLHF-tuned versions are available but the data is not, and hence we do not know what factors might have influenced the preference annotations that guided the annotation."

Our method's primary objective is to measure the extent to which a fine-tuned model has learned the fine-tuning feedback. To clarify:

• We do not intend to contrast base and fine-tuned models (although Appendix C briefly investigates this).
• Our goal is not necessarily to find information about fine-tuned models where the fine-tuning data/reward model is unknown.

We hope practitioners will use our method to evaluate and improve their post-training techniques based on how well they cause models to learn the fine-tuning feedback.

Key findings and implications:

1. Our method considers model internals, unlike other metrics for the success of RLHF (e.g., fine-tuning loss or output evaluations).
2. While our probes can accurately recover whether feedback is positive or negative (Table 3), they struggle with more granular feedback used in PPO fine-tuning (Table 4).
3. This could indicate either: a) The model hasn't learned fine-tuning feedback in sufficient detail, or b) Low probe accuracy (which we argue against, see below)

We support the accuracy of our probes through:

• Showing similarity between probe-identified and GPT-4-identified LFP-related features (Table 6)
• New results in our global rebuttal PDF:
  • Probes less correlated with fine-tuning feedback still learn related patterns (Table 1 in the PDF attached to our global rebuttal)
  • Inputs can often be separated by reward through dimensionality reduction (Figure 1 in the PDF attached to our global rebuttal)
  • Probe accuracy in predicting the label of a word in the VADER lexicon is correlated with the frequency that the fine-tuned model generates that word (Figure 2 in the PDF attached to our global rebuttal).

These results suggest that fine-tuned models may not have learned fine-tuning feedback in a detailed manner, rather than indicating probe inaccuracy.

Clarification on Activation Delta Calculation

"I did not fully understand how the activation deltas are calculated… I assume it is not simply an L2 norm of the two A_concat vectors, as then it is unclear what the probe would learn."

The activation deltas are calculated as follows:

For any contrastive triple (positive, negative, neutral), we provide two pairs of input to the linear probe:

a) Input_1 = SAE_repr of (positive)

Output_1 = $+ ||SAE\_{repr}(positive) - SAE\_{repr}(neutral)||$

b) Input_2 = SAE_repr of (negative)

Output_2 = $-1 \cdot ||SAE\_{repr}(negative) - SAE\_{repr}(neutral)||$

The L2 norm here serves as a signed distance measure between either the positive and neutral or negative and neutral pairs.

Although it might be unclear what the probe would learn if trained on a smaller number of examples, the only consistent difference over many examples should be the distance in activation space related to how different the feedback is in fine-tuning. For example, although a given positive and neutral pair of inputs may be separated in activation space in many ways other than just their implicit feedback, the only consistent difference over thousands of positive and neutral pairs should be the difference in their implicit feedback. In the PDF attached to the rebuttal visible to all viewers we show through dimensionality reduction that the separation between positive and negative activation deltas can be seen visually (Figure 1), which we hope shows that activation deltas are sufficient labels for probe training datasets.

Utilization of Probe-Obtained Information

"Assuming that the probes trained in this paper do obtain information about the preferences underlying human preference data, how do we make use of that information?"

If probes are accurate but unable to learn fine-tuning feedback precisely, it suggests the fine-tuned model hasn't fully learned the feedback. Practitioners might then:

1. Consider alternative explanations, such as:
   • The model learning a proxy objective
   • The model failing to find consistent patterns in its activations related to the feedback
2. Take remedial actions, such as further fine-tuning, or adjusting their post-training

Addressing Broader Impact

"The paper has discussed limitations but not broader impact of their method."

We will add a broader impacts section to the camera-ready version of the paper, covering the takeaways stated in this rebuttal and our global rebuttal.

All results from the PDF attached to the global rebuttal will be incorporated into the camera-ready version of our paper.

We thank the reviewer for their insight and time.

We appreciate that you found LFPs gave a new perspective on how LLMs learn from human feedback, and that our use of synthetic data and GPT-4 contributed well to the paper.

Model and Task Selection

"The study primarily focuses on a few specific models… and tasks… More recently released models are of more value"

We'd like to clarify and expand on our model and task selection:

• In addition to the smaller models you mention (Pythia-70m, Pythia-160m and GPT-Neo-125m), we experimented with Gemma-2b, a larger and more recent model released in February 2024 (L152).
• Beyond sentiment generation and toxicity tasks, we included the helpful-harmless task, designed to simulate real-world RLHF (L146-149).
• Results for Gemma-2b and the helpful-harmless task can be found in Table 3 of our paper.

We hope these results help address concerns about the generalizability of our findings to larger models and realistic applications.

We also studied models fine-tuned using different reinforcement learning algorithms (PPO and DPO), demonstrating that our method generalizes across RLHF algorithms.

Addressing Weaker Correlations and Feature Superposition

"…the paper notes weaker correlations for more granular reward predictions… feature superposition… poses a challenge to fully interpreting… features."

We offer the following comments:

1. Lower probe correlation in the controlled sentiment generation task may indicate that the model hasn't learned the granular specification of reward for that task, rather than a failure of the probe.
2. Additional evidence in our global rebuttal shows:
   • Probes learn patterns related to fine-tuning feedback even when they have low accuracy to it (Table 1 in global rebuttal PDF).
   • Inputs can often be separated in terms of their probe classifications through dimensionality reduction (Figure 1 in global rebuttal PDF).
   • Probe accuracy in predicting the label of a word in the VADER lexicon is slightly correlated with the frequency that the fine-tuned model generates that word (Figure 2 in global rebuttal PDF).

Our method will benefit from recent improvements in training sparse autoencoders (e.g., Gao et al. and Rajamanoharan et al.), which should reduce the effects of superposition.

Validation Process and GPT-4 Reliability

"The validation process relies on GPT-4's ability to describe neural features… This could introduce biases or inaccuracies if GPT-4's descriptions are not perfectly reliable."

We acknowledge this concern and note:

• Bills et al. and Cunningham et al. have conducted detailed validations of GPT-4 generated feature descriptions against manual and auomatic analysis.
• Our method for generating feature descriptions with GPT-4 is common in prior literature (e.g., Cunningham et al., Bricken et al., Neo et al., and Templeton et al.).

Practical Applications and Risk Mitigation

"Can you elaborate on how your findings can be practically applied to mitigate risks associated with LLM deployment, such as manipulation of user preferences or harmful behaviors? What strategies would you recommend for developers to monitor and adjust LFPs in deployed models?"

Our method aims to help practitioners identify how well their fine-tuned models have learned the fine-tuning feedback. We recommend:

• If probes indicate divergence between LFPs and fine-tuning feedback, that practitioners consider alternative post-training methods.
• This divergence may suggest the model is learning a proxy for the feedback or failing to find consistencies in high-reward generations.

Method Generalization

"Have you tested your method on other LLM architectures or tasks beyond sentiment analysis and toxicity detection? If so, what were the results? How do you anticipate the effectiveness of your method would vary with different model sizes and types?"

Our method has been tested on:

• Tasks beyond toxicity and sentiment analysis, specifically a helpful-harmless task mimicking real-world RLHF (L146-149).
• Various transformer-based LLMs with architectural differences (Pythia models, GPT-Neo-125m, and Gemma-2b), for example in the attention variant they use.

We anticipate our method would recover more information about fine-tuning feedback from larger models due to their increased parameter count and expected better performance on fine-tuning tasks.

Alternative Validation Methods

"Your validation relies on GPT-4's feature descriptions. Have you explored other methods or models for validating the identified features? How do you ensure the robustness of these validations?"

While we haven't explored alternative models for generating feature descriptions:

• Bricken et al. and Templeton et al. have successfully used Anthropic models to explain features.
• We don't foresee issues with using models other than GPT-4.
• Even if GPT-4 descriptions are sometimes inaccurate, the overlap between probes and GPT-4 descriptions still aids in probe validation.

The results from our global rebuttal PDF will be incorporated into the camera-ready version of our paper.

Thank you for your response. We believe we have addressed the concerns raised in your original review and appreciate the point brought up in your response.

it sounds that many of your justification statements are based on other papers' claims which may not directly verifiable in the submission itself

Most of our assumptions are based on either peer-reviewed publications or seminal interpretability papers that are standard references in mechanistic interpretability. Other assumptions we try to validate in our paper. We acknowledge that our paper could have better highlighted how each assumption has been validated in prior work.

Regarding the sparse autoencoder assumptions:

• Recent work that only studies sparse autoencoders does not justify these assumptions experimentally, and usually defers to past work (e.g., Rajamanoharan et al., Rajamanoharan et al. and Gao et al.)
• In each of these papers sparse autoencoders are a much larger component of the method than in our paper. As a result, we think it would be out of place for us to re-justify these assumptions in our submission.
• The peer-reviewed ICLR print of Cunningham et al. is titled “Sparse Autoencoders Find Highly Interpretable Features in Language Models”, which was a specific assumption mentioned by Reviewer dzK7. These results have been further validated in work such as Bricken et al. and Templeton et al.

For the GPT-4 feature descriptions, prior work cited in our rebuttal has validated the descriptions extensively. We note that we did not intend to use the GPT-4 descriptions as a ground truth, and that the correlation of the GPT-4 classifications and our probes (Table 6 in our submission) shows that both methods identify similar features as being related to LFPs. Even if the feature explanations contain inaccuracies, we believe they still help validate the probes.

We hope that the additional experimental validation of our probes in the PDF attached to our global rebuttal helps support the accuracy of our probes.

We appreciate your time and consideration. We hope that since the reviewer-author discussion period is still ongoing, you will consider our response here and to Reviewer dzk7.

Thank you for your thoughtful review.

We are pleased that you found the question our paper studies interesting, and our explanation for using sparse autoencoders good.

Key Assumptions

"The effectiveness of this probing method seems to rely on many key assumptions being true…"

While we agree with the reviewer that the method relies on many key assumptions, we think that these assumptions are reasonable and are based on prior work:

1. Sparse autoencoder outputs being more interpretable

Several studies support this assumption:

• Cunningham et al., Bricken et al., and Templeton et al. show that features in sparse autoencoder dictionaries are easier for language models to describe than neurons in raw activations.
• Bricken et al. and Templeton et al. found that these features are easier for humans to understand through manual analysis.
• Bricken et al. also show that sparse autoencoders enable finding effectively invisible features in raw activations.
• Additional work (Rajamanoharan et al., Gao et al., and Rajamanoharan et al.) shows that sparse autoencoders that perform better on key metrics (sparsity, reconstruction) are more interpretable.

After papers such as Cunningham et al. and Bricken et al., this assumption is common in interpretability, and motivates the use of sparse autoencoders for interpretability.

2. Sparse autoencoder output representations being faithful to the original representations

• Bricken et al. show that replacing MLP activations with sparse autoencoder outputs incurred only 21% of the loss that would be incurred by zero ablating the MLP.
• Rajamanoharan et al. reduce this to 2% with their improved sparse autoencoder.

We consider these results to show faithfulness of sparse autoencoder outputs to activations.

We argue that this assumption is in line with other interpretability work, and that experimentally supporting it in our paper is out of scope.

3. Probe accuracy

We agree there is more work to be done in validating our probes. In our submission, we validated our probes using GPT-4 feature descriptions, finding a strong overlap between the probes and GPT-4 descriptions (Table 6). We offer additional validation of our probes in the PDF in our global rebuttal:

• We show that the probes are learning patterns related to the fine-tuning feedback (Table 1 in global rebuttal).
• We demonstrate that inputs can be separated by their reward using dimensionality reduction, showing there is structure in the probes' input data tht they can exploit (Figure 1 in global rebuttal).
• Probe accuracy in predicting the label of a word in the VADER lexicon is slightly correlated with the frequency that the fine-tuned model generates that word (Figure 2 in global rebuttal).

We commit to integrating these results into the camera-ready version of our paper to better support our probes.

4. GPT-4 prediction accuracy

Bills et al. and Cunningham et al. have performed detailed validations of GPT-4 generated feature descriptions:

• They prompt another LLM to predict how that feature would activate for a collection of tokens if that description were correct.
• They measure the correlation of these predictions and the true activations.
• Bills et al. also conduct additional validation, using humans to validate the feature descriptions and ablation experiments.

We would like to highlight that our method of generating feature descriptions with GPT-4 is common in prior literature, such as Cunningham et al., Bricken et al., Neo et al. and Templeton et al. We argue that even if the GPT-4 feature descriptions are sometimes inaccurate, the overlap between the GPT-4 descriptions and probes still helps to validate the probes.

Addressing Specific Concerns

"If the correlation is low, we cannot tell whether that is the probe's failure…"

We hope that through some of our additional results, we have increased the rigor of our evaluation of the effectiveness of the probes. We acknowledge that additional validation would be useful, but believe that the rigor of evaluation with our updated results is in line with the standards of interpretability literature. Further improving the validation of our probes is our primary objective for the camera-ready version of our paper.

"as it currently stands, this table's [Table 5] results does not seem to strongly support… that the predicted LFP-related features are relevant to LFPs."

We note that Table 5 was an ablation of the features classified by GPT-4 as related to LFPs and was not intended to support our probes.

"Much of the writing about the methods is unclear, contradictory, or omits many details…"

In an effort to address this concern, we have corrected the issues you highlighted, clarifying that the description on L233-234 is correct, and not L493. The camera-ready version of our paper will include a clearer method section, with more of the writing replaced with equations, in line with your comment.

"In Section 3.4, how is GPT-4 prompted...?"

The camera-ready version of our paper will include an appendix with the full prompts used to generate feature explanations, but they are taken from the public repository of Bills et al.

"In this paper, most of the results were for smaller models (under 1B params)... it is unclear whether this method would be scalable to larger models."

We included results for Gemma-2b in our paper (L152), and note that recent work such as Gao et al. has trained sparse autoencoders on models in the GPT-4 family with success, showing scalability.