Language Models Linearly Represent Sentiment

We greatly appreciate your constructive review and the positive remarks on our manuscript's structure, clarity, and experimental design. We particularly appreciate your interest in the relevance of our findings for mitigating the potential risks from language models.

The statements in the abstract and introduction indicate that this paper investigates the representation of sentiment in LLMs… GPT and Pythia may not be representative of the broader spectrum of LLMs, which reduces the persuasiveness of the conclusions drawn from the study.

We acknowledge your concern about the representativeness of our findings across the broader spectrum of large language models (LLMs). In our latest revision, we have repeated the experiments in Figures 2a and 2b (i.e. demonstrating the consistency and classification accuracy of the sentiment direction) for Pythia-2.8b to compare (in Section A.1.5) with GPT2-small (now in Figure A.6). We are also conducting additional experiments across more model families to demonstrate the summarization motif and the linear representation of sentiment. Our in-progress experiments include a comparison with two StableLM models up to 7B parameters in size which are trained on significantly more data than Pythia, making them closer in capability to today’s state of the art models.

However, we would like to note that in our work so far, our patching experiments are shown for different sizes of Pythia and for GPT2-small in Figure 2e, demonstrating our core claim that across a range of models, there exists a linear and causally relevant sentiment direction. Moreover, in Appendix Figure A.2, we show that for different sizes of GPT2, k-means can classify tokens out of distribution. The largest model studied was 35 times larger than the smallest and the two distinct families of models here are quite different. The families have different training corpuses (Pythia was trained with extensive code and ArXiv papers, unlike GPT-2) and different training and architectural choices (Pythia uses rotary positional encodings, untied embeddings, and was not trained with dropout). These differences make the existence of a similar direction in the residual stream particularly significant.

Also, as noted in the paper, unfortunately some models were too small to accomplish the relevant task, rendering a comparison impossible. On the other end of the spectrum, working with large models presented significant computational challenges.

There is no caption provided for Table 1 in the paper. Furthermore, Figure 2 is predominantly featured in Section 3, but there exists a substantial gap between its initial reference and the section in which it is discussed.

Thank you very much for pointing out these slips in presentation. In the revised version we have ensured that all tables and figures are appropriately captioned and, where possible, placed in proximity to the relevant text for ease of reference and to facilitate a coherent narrative flow.

The proposed ToyMovieReview format, "I thought this movie was ADJECTIVE, I VERBed it. Conclusion: This movie is," appears to be more suitable for sentence-level sentiment analysis… Furthermore, the rationale for emphasizing adjectives and verbs in this specific prompt design remains unclear, as it has not been adequately explained in previous sections of the paper.

Your point about the ToyMovieReview dataset is well-taken. To address this, we provide the additional experiment of replicating the SST all-token ablation experiment on the IMDB reviews dataset and have added the results to Section 3.3. These prompts are significantly longer and thus the results better demonstrate our findings at the document level.

We realize that our choice to focus on adjectives and verbs in the ToyMovieReview dataset was not fully elucidated. This choice was based on their simplicity and their efficacy as a sentiment-laden source of data. To further clarify, we have included a reference to a cosine similarity analysis with a toy adverb dataset in the appendix, which supports our choice.

In Section 4.1, the authors find that the model employs a simple and interpretable algorithm to perform the task. However, this algorithm has not been supported by corresponding experiments…

We understand your reservations regarding the claim in Section 4.1. We provided our experimental evidence for this interpretable algorithm in Appendix A.3, but did not make this prominent in the main body and it may have been missed. We have improved the main text with additional clarifications and a pointer to the relevant experiments in the appendix.

Your feedback has been very helpful, and we hope our revisions adequately address your concerns!

Thank you very much for the reference to Clark et al. which we have added to the introduction in our latest revision. We are also very grateful for your thoughtful comments in general!

Models. The authors claimed that the results were consistent across a range of models… author should either clarify this or lower the claim.

Where possible, we have tried to run the same experiments across different models. In particular, the core experiments of patching the sentiment direction in different layers when evaluating on our real-world dataset (SST) were performed for 4 different sizes of GPT-2 and 4 different sizes of Pythia. We included a representative selection of these in Figure 2e. Some tasks could not be performed by smaller models (such as the mood stories) and other experiments were not performed on larger models due to compute time constraints (such as evaluating Pythia-2.8b on SST with patching at each layer).

To add support to our claim of universality, we have repeated the experiments in Figures 2a and 2b for Pythia-1.4b, in order to provide a comparison with GPT2-small. These show that in both models, the different methods converge on a single direction which can classify sentiment on OpenWebText. In addition, to provide additional support for our “summarization motif” results, we have added activation patching experiments for Pythia 6.9b to Appendix A.4.2, and are currently working on adding SST direction ablation experiments for other models to a future revision.

Linearity. The authors claimed that the sentiment was linearly represented… we could also explain that there was more than one direction or simply nonlinear sentiment space.

Thank you for pointing this out; we agree that this claim could benefit from further clarification and we have added a section to the Appendix (A.2) in the revised version, where we include some text expressing similar limitations to the above for our linearity claim.

Our core contribution lies in identifying a key sentiment direction that is causally influential and consistent across various tasks, as evidenced by our directional patching experiment's ability to leverage a sentiment direction identified from a toy dataset to effectively classify sentiment in more complex datasets like OpenWebText and Stanford Sentiment Treebank. While we acknowledge the potential for additional sentiment representations tailored to specific tasks or non-linear dimensions, our findings suggest the dominance of a primary, shared sentiment direction.

We propose that this primary direction acts as a "first principal component of sentiment," and that there could be further more nuanced, task-specific directions. This concept echoes the "feature splitting" phenomenon described by Bricken et al. in “Towards Monosemanticity.” Although exploring these finer-grained directions is outside our current scope, we believe pinpointing this overarching sentiment direction is nevertheless a meaningful contribution to the literature.

We would also like to draw your attention to Appendix Figure A.6, where we show that allowing DAS to learn a higher dimensional subspace does not improve the loss out of distribution. This supports the linear perspective, showing that it is hard to beat even by increasing the dimension of the DAS subspace when seeking a representation of sentiment across a sufficiently wide range of tasks.

Distance. While the author experimented with a toy dataset and injected irrelevant content, I found the relationship between the "summarization" and the context length interesting. However, this could be made stronger…

Thank you for your interest in our result showing increasing summarization with distance. We would also love to see further study of this result, but we have struggled with a number of confounders in real-world datasets. As this is a very interesting phenomenon but not a central one to our core claims in this paper, we decided to merely flag the finding here and leave these complexities for future researchers.

Inconsistency. In the abstract, the authors wrote 76% and 36%, but these were not consistent with the text in Section 4.3.

Thank you for raising this. In hindsight, we agree that reporting the same result with two different denominators was confusing. We reported the same result in terms of the 76% reduction in accuracy above “chance” (i.e. 50% for a balanced binary dataset) and in the body we reported this in terms of a drop of the 38% drop in raw accuracy (i.e. above 0%), resulting in a factor of two discrepancy. The same was true for the 36% and 18% accuracy drops reported with comma-only ablation. We have since revised our numbers reported in the abstract to only use the more conservative framing

We are grateful for the opportunity to refine our work based on your valuable feedback and look forward to further discussion of these results.

Thank you for your feedback on our paper. We appreciate the opportunity to clarify aspects of our work and refine its presentation.

The writing seems extremely sloppy. For an experimental paper, many of the plots/discussions are not clear, especially for a broad ICLR audience. For example, what are the x- and y-axes of figure 2b?

Regarding the writing clarity, we regret any confusion and have extensively improved our paper’s clarity and readability in the latest revision. We have conducted a thorough review and made many small improvements, shown in red font for your convenience. We have brought key definitions forwards from the glossary into the main body and improved figure explanations, particularly for Figure 2b. We would also like to draw your attention to the helpful glossary in Appendix A.7.

What are "sentiment activations" in GPT2-small, are these related to sentiment directions that are learnt via the 5 methods?

These are indeed highly related. We would like to draw your attention to the definition given in the opening text in Section 3:

We show that the methods discussed above (2.2) all arrive at a similar sentiment direction. We can visualise the feature being represented by this direction by projecting the residual stream at a given token/layer onto it, using some text from the training distribution. We will call this the ‘sentiment activation’.

I feel like the authors are using the term "causal" loosely, do they mean that the sentiment is sensitive to activation addition, and hence they deem it causal? Usually the term causal representations are reserved for representations that generate the data, not the other way around.

By “causally significant,” we do not mean that the sentiment direction is sensitive to activation addition, but rather that causal mediation (i.e. activation patching) experiments demonstrate the causal effect that these directions exert on the models’ outputs in real-world contexts. We would like to draw your attention to methods section 2.3 Causal Interventions, where we outline exactly two causal methods: “Activation Patching” and “Ablations”. Then in results section 3.3 Causal Evaluation, the first sentence reads “We evaluate the sentiment direction using directional patching”. In the figure captions for our causal results, we preface 2d with “Directional patching results” and 2e with “Patching results”. From the opening to Section 4: “Through an iterative process of path patching (see Section 2.3)”. In the reproducibility statement: “Our causal analysis techniques of activation patching, ablation, and directional patching are presented in Section 2.3”.

The activation addition results were not central to our claims and thus were relegated to the Appendix in Figure A.3. We regret that these may have seemed like our basis for using the term “causal”. To avoid any further confusion, in our latest revision we have removed the single reference to activation addition from the main body of the paper. We appreciate your attention to this detail and are open to any further suggestions or questions you might have.

As the authors note, attention heads are used for the analysis and a more thorough analysis should include activations at MLP layers too.

Thank you for raising this point. We note that our core results focus on studying directions in the model’s residual stream (i.e. in the skip connections around layers), which are the sum of the outputs of all prior layers, including both attention and MLP layers, and thus includes the contributions of the MLP layers..

We also provide more detailed circuit analysis that we agree would benefit from a broader examination that includes MLP layers. However, this is not central to our core contribution in this paper, and as such, though we have provided initial findings, we acknowledge this as a limitation and an avenue for future work. In addition, in our path patching experiments we observed that the MLPs were of relatively low importance to the models’ performance on the tasks we analyzed, a finding that parallels that of past circuit analyses such as that in Wang et al. (https://arxiv.org/abs/2211.00593) Since attention heads had a more significant effect on model performance, we focused on them in our circuit-level analyses.

Your feedback is welcome, and we will make all necessary revisions to meet the high standards of the conference. We believe that the core contributions of our work hold significant value and, with your feedback, we aim to present them more clearly. Please let us know if we have addressed your objections, or if there are further changes you would like to see made to the paper.

The consensus amongst the reviewers has been that the two core results of our paper (that is, 1. That we have found a causal, linear direction representing sentiment, and 2. That models aggregate clause-level sentiment information at punctuation, forming a partial information bottleneck) are very interesting, but that we needed to provide further support of their universality. Reviewer 8vUR also found some of our writing unclear. In a timely manner, we have provided novel experimental results in both new models (Pythia-6.9b and some “summarization motif” evidence in Mistral-7b) and a new dataset (IMDB) to more firmly establish our findings. We have also expanded the glossary and in-line definitions to ensure that our work is comprehensible to a diverse ICLR audience.

Based on the feedback of reviewer dUnj, we have provided further evidence of universality and limitations to our linearity claim, improving the rigor and robustness of our paper. We thank them for noting that we were able to address most of their concerns, and for their increase in their indicated rating.

We would also like to thank reviewer 6Ynr for acknowledging our detailed response and that their concerns were resolved.

After reviewer 8vUR provided some criticism of our writing and clarity, we made changes that we believe significantly improved the polish on our paper. Afterwards, the reviewer argued that our changes were too significant to be re-reviewed and justify a re-submission, but we respectfully disagree and invite the area chair to review the revised manuscript and decide if the changes (conveniently shown in red text) are unreasonably large.