Steering Language Models with Activation Engineering

We want to clarify two major concerns you raised.

The primary issue with the paper is that it is outdated. The paper refers to several works published in 2023 as "contemporary," implying that they are based on the presented work. This suggests that the paper may have been rejected in previous conferences and is now being resubmitted to ICLR without any major modifications. However, works from 2023 cannot be referred to as contemporary in a submission to ICLR 2025.

We agree that revisions are important. However, your speculation about no “major modifications” is incorrect. For example, we rewrote the entire paper for ICLR.

Furthermore, works from 2023 would have been submitted to ICLR 2024, which makes comparison in ICLR 2025 more reasonable.

Moreover, the claim that both Liu et al. (2023) and Zou et al. (2023) are based on this work is questionable.

We did not mean to claim that we directly inspired that work (although our ActAdd paper has, in fact, inspired a range of follow-on work). What we said was that those papers "followed" ours, meaning "followed" in a temporal sense. We see how this was unclear and will instead state that those papers "came after" this work.

A quick review of these papers reveals that Liu et al. (2023) merely cites ActAdd as related work, and Zou et al. (2023) actually outperforms ActAdd on one of the tasks. Therefore, I do not believe ActAdd presents any novel idea or result. This undermines the relevance of the method, and I believe this alone is sufficient for rejection. However, if I have misunderstood this point, the authors could clarify their claims.

We made a scientific discovery (in line with some past evidence from e.g. GANs): LLMs are steerable via linear manipulations of their activation space. We want our discovery to be scientifically validated by the peer review process. Zou et al explicitly noted that their approach is a “variant of ActAdd." We also observe that their approach outperformed ActAdd on a different task - TruthfulQA - which is not part of our paper. Those two facts do not invalidate our findings or the scientific contribution of this paper.

Consulting ICLR’s reviewer guidelines:

Q: If a submission does not achieve state-of-the-art results, is that grounds for rejection?

A: No, a lack of state-of-the-art results does not by itself constitute grounds for rejection. Submissions bring value to the ICLR community when they convincingly demonstrate new, relevant, impactful knowledge. Submissions can achieve this without achieving state-of-the-art results.

For another angle on the issue, suppose paper X makes discovery Y. Paper X’ (published substantially later) makes a further discovery Y’. If paper X is not immediately published, and instead is being reviewed one year later for its scientific contributions, should it be considered to “not present any novel idea or result” because people already know Y’ (and also, therefore, a subset of X's discoveries about Y)? We think the answer is “no, the original contribution is still valuable.” If you disagree, we are happy to consult with the area chair to come to a mutual understanding of this issue.

We will address the rest of your helpful feedback and concerns in a follow-up comment. We think many of your concerns are reasonable and fixable.

I thank the authors for their reply. However, I still disagree with the points they make.

Before going into specific points made by the authors, let me reiterate the main argument: The paper includes references to work published over a year ago stating they are contemporary. While these papers use a very similar method as the authors present, a comparison (e.g., in the experiments) is not presented in the paper. This is specifically problematic since one of those papers outperforms the given method for at least one application.

We agree that revisions are important. However, your speculation about no “major modifications” is incorrect. For example, we rewrote the entire paper for ICLR.
Apologies for my mistaken assumption. My assumption was mainly based on the lack of recent citations, the use of outdated models, and the use of outdated baselines. This is in my opinion the most important part to address in an updated paper, and I believe this has not happened to a sufficient degree in this paper.

Furthermore, works from 2023 would have been submitted to ICLR 2024, which makes comparison in ICLR 2025 more reasonable.
No, it does not. The fact remains that these works are well-known, contain many of the same ideas, and are not compared against. One year should have given the authors plenty of time to compare against these baselines and improve upon their ideas. If a paper contained similar ideas to a paper published in ICLR 2024, but did not compare against it, this paper should be rejected.

We did not mean to claim that we directly inspired that work (although our ActAdd paper has, in fact, inspired a range of follow-on work). What we said was that those papers "followed" ours, meaning "followed" in a temporal sense. We see how this was unclear and will instead state that those papers "came after" this work.
"Followed" is a very strange word to use in the temporal sense if you do not compare against the work. We are reviewing your paper against the current state-of-the-art, which now includes these papers.

We also observe that their approach outperformed ActAdd on a different task - TruthfulQA - which is not part of our paper. Those two facts do not invalidate our findings or the scientific contribution of this paper.
But it does! Unless you show that for the tasks on which you evaluate ActAdd, it outperforms their method, I have to assume your method is strictly worse.

We made a scientific discovery (in line with some past evidence from e.g. GANs): LLMs are steerable via linear manipulations of their activation space.
Unfortunately, this scientific discovery has now been made in other papers as well. Please, do not understand me wrong: I do believe this idea is very interesting and worth noting. However, I cannot imagine this be presented at ICLR 2025 as "novel" since it has been already known for a year. In order to be accepted, your paper should now improve upon prior works that use this idea.

Consulting ICLR’s reviewer guidelines:
The guidelines clearly state that papers should contain new knowledge. This is not the case anymore. To cite another part of those guidelines:

Q: Are authors expected to cite and compare with very recent work? What about non peer-reviewed (e.g., ArXiv) papers? (updated on 7 November 2022)
A: We consider papers contemporaneous if they are published within the last four months. That means, since our full paper deadline is October 1, if a paper was published (i.e., at a peer-reviewed venue) on or after July 1, 2024, authors are not required to compare their own work to that paper.

This part of the guidelines clearly states that you should compare against all works before July 1, 2024.

For another angle on the issue, suppose paper X makes discovery Y. ...
However, in this case Paper X' makes the discovery "Y+" since it is outperforming your method and I have not seen evidence to the contrary.

Based on the provided information from the authors, I will almost certainly not change opinions on this topic anymore. If the authors want to consult with the AC to discuss the issue, I would be happy to contribute to that discussion. If, after that discussion, the AC instructs me to ignore this point in my review, I will of course do so. However, please note that the remaining weaknesses would still lead me to reject the paper, although those points are much more addressable in a rebuttal and I therefore expect that the situation might improve there.

We are preparing the following experiments and content which we plan to make available within the discussion period:

• We will add the new Dekoninck et al 2024 method as a more up-to-date baseline.

• Previously we used OPT for compatibility with the reported results of our baselines. We will standardise all our experiments on Llama-3 and run all baselines against them (where possible) with the same hyperparameters.

• We’ll ensure that relevant experiment code is included in the Zenodo, and have removed the playground file.

• PreAdd used davinci-001. We used davinci-002 because the davinci-001 API was shut down, preventing us from matching PreAdd’s environment; we thus picked the model closest to theirs. (And we could not get their method to run in time for submission, thus preventing using davinci-002.) We plan to reimplement PreAdd and then use the same perplexity model (Claude Sonnet 3.5) for all settings and baselines.

• We will cite more-recent work in the area of steering.

In the meantime, some discussion points:

Omission of Fudge: In Lines 378-380, Fudge is omitted, despite performing better on certain aspects and only slightly worse on others. This is a strange misrepresentation of the results.

You are correct, thank you for pointing this out. We did not intend to misrepresent the results. FUDGE is indeed reasonably competitive by these metrics, performing better on some and worse on others. We'll clarify in the camera-ready.

Basic Metrics: Perplexity and cosine similarity are insufficient metrics to fully capture fluency and relevance. Since controlled text generation methods edit the model's internals, they can yield unintuitive results that these metrics may not fully capture. The authors should include human or LLM-based evaluations to assess the outputs in Tables 3 and 4 and compare them with baselines.

Perplexity and cosine similarity are the standard metrics in NLP for measuring fluency and relevance. We needed to include them to enable backwards compatibility with our baselines. Note that the paper suggested by the reviewer also uses perplexity (and omits any relevance metric).

“What is meant by "this section does not have complete statistics" in Line 533?”

This just means that we don’t report all 306 hyperparameter settings “We perform simple grid search, usually between c ∈ [3, 20] and l ∈ [6, 24].”

“How was grid search performed for ActAdd's hyperparameters? Were the results reported for the best set of parameters? If so, was a similar hyperparameter search conducted for the baselines to ensure accurate comparisons?”

See line 533: for the given parameter ranges, simple grid search mixed with qualitative sampling of hyperparameters was performed. We used the reported values from the baselines, and thus depend on the original author’s gridsearch or other optimization. We will try to rerun the baselines ourselves during the response period.

Could you clarify the hyperparameter “a” discussed in Appendix C and explain its function?

This is the sequence alignment parameter: the position the steering vector h_A and the forward pass from the user prompt are aligned at. It is also mentioned in Algorithm 1: a = sequence position to align h_A and h_{p^{∗}} and in Limitations: “So far we have had success with fixing the sequence alignment a = 1.”

For which experiments are the prompts mentioned in Lines 1218-1223 used? Appendix C presents a collection of unrelated details, making it difficult to follow and understand how it fits into the overall context of the paper. Could the authors clarify the connection to the experiments?

These prompts are for Figures 4 and 7. We will improve the pointers throughout the Appendix, thanks.

Thanks for your reply!

Obtaining implementations is essential to provide an accurate comparisons between the new method and baselines. How did you ensure that all prompts (e.g., no extra spaces or newlines), parameters (temperature, ...), or models (e.g., to measure cosine similarity) etc. are the same across all your baselines if you did not have implementations?

Using results reported in other work is common, but we agree that it opens up the possibility of uncontrolled experimental variation. As a result we’re currently re-running all the baselines using identical settings.

Could you add it by the end of the rebuttal period instead?

Yes! Results table forthcoming.

I fiddled around a lot with various algorithms that do internal steering, and found that they sometimes produce nonsense words (i.e., 2% of the words are nonsense while the sentence around it makes sense). This problem is not fully captured by perplexity. I just checked, and the baseline I provided does perform an experiment where an LLM decides which of two completions is the best as an extra experiment.

Interesting! With ActAdd we only see this word-level corruption of completions with very high coefficient values, c~=20 (see Appendix G) or when intervening at the last layer. You can also verify this in the best-of-3 demonstration notebook. We aim to do a quick test of your LLM scorer idea in the remaining time.

“how did you decide that one set of hyperparameters was better than the other? Did you have a separate validation set over which you optimized them based on the numbers you got on that validation set? Or did you directly optimize them on the experiment and numbers your report in the paper? In the latter case, this would be quite problematic, especially if no similar search was done for baselines.”

There are two searches involved:

• Finding a prompt pair $(p_+, p_-)$. We did not iterate over candidate prompt pairs during experiments; instead we manually discovered them and fixed this prompt pair during the gridsearch for the experiments (e.g. for sentiment steering the experiments were all done on (“love” - “hate”)).
• For each experiment, we indeed used a validation set. You can see this in the new Zenodo: act_add_iclr2025.tar/act_add_iclr2025/activation_additions_hf-main/notebooks/sentiment.ipynb and …/toxicity.ipynb.

Thanks for your comments and noting the method’s advantages!

You can see the relative computational cost (the increase in inference time from steering) in the following experiment which we conducted but did not include in this round:

• https://i.imgur.com/fBI32B1.png
• https://i.imgur.com/BhT8KPb.png

If you wish, we will include this in the camera-ready. As for the absolute computational overhead: ActAdd is just $n$ extra forward passes (where for instance our gridsearch over GPT-2 layers was $n=306$).

As to layer-specificity: Figures 3 and 7 show that ActAdd performs well for a relatively wide range of layers (~10 layers), though we agree it’s not layer-agnostic.

Thanks for this! You can see the appendix on overheads as C.1 in the newly revised supplementary information PDF at the top of this page.

We also looked into the works you mentioned:

• [a] Sakarvadia et al 2023 point out that their method relies on the unembedding matrix, which can misrepresent intermediate layers: “may portray attention head behavior inaccurately due to representational drift between model layers..." They also note that their future work will be layer-specific "we aim to address this shortcoming in future work by instead using layer-specific learned projections to transform between hidden states and vocabulary.”

• We think [b] Heimersheim et al 2024 isn't quite the same as activation engineering. They use activation patching to interpret model outputs by sweeping over model components to find locations that, if patched, change performance in the task of interest. (i.e. they replace the component with one from another run). This is different from our case: we add to the activations rather than replacing them. Still, it is clearly somewhat related and we've added a short note.

• We've added [c] Vig et al 2020 as related work.

Thank you for your comments! In response, we are preparing the following experiments:

1. We used OPT for compatibility with the reported results of our baselines. We will standardise all our experiments on Llama-3 and run all baselines against them (where possible) with the same hyperparameters.
2. We used a random n=1000 subset of the benchmarks – as is standard in the area, see Pei et al and Dekoninck et al. Increasing the subset size is thus a discrepancy which would weaken the validity of the baseline. We will however rerun our experiments on 10,000 examples from the RealToxicityPrompts and Sentiment test sets and see if there is any difference from our n=1000 run.
3. We will also run the topic steering experiments represented by Figures 4 and 7 on more prompts (drawn from the IMDb sentiment benchmark) to demonstrate that the steering works across a variety of prompts.
4. We will, if time permits, run the same experiment using Factscore and compare steered and unsteered metrics.

In the meantime, some discussion points:

“benchmarks created by the authors”.

We didn’t create any of the benchmarks used (RealToxicityPrompts, Perspective, Stanford IMDb, ConceptNet, ). Did you mean the topic steering experiment of Figure 4?

Some discrepancies in results are also notable—for instance, the paper draws baselines from this paper ((https://aclanthology.org/2023.findings-acl.636.pdf)), but there are differences in the results for the unsteered OPT (0.152 vs. 0.134 toxicity, 49.9 vs. 8.9 fluency). Such large changes in fluency might suggest a difference in experimental setups, which could potentially affect the interpretation of ActAdd's fluency improvements.

We reported our runs for all the baselines we could reproduce. We agree that the discrepancy is regrettable, but did not find the Pei et al hyperparameters. However, the unsteered OPT is less toxic and more fluent in our run, which makes it a stronger baseline and a better comparator.

We thank the reviewer for suggesting HelpSteer. These two papers introduce training datasets for Attribute-Conditioned finetuned models. The HelpSteer papers do not use the dataset as a benchmark (for helpfulness, correctness, etc) but rather other methods such as MT Bench for helpfulness or TruthfulQA for correctness. Do you mean we should use their validation set as a test set?

“benchmarks created by the authors”.

I'm referring to the experiment in sec 4.2. It's unclear what dataset this is run on. This experiment's conclusion is also unclear and the authors have not addressed my question about the results being different for different topics.

Discrepancies in baselines My point was that the discrepancies could indicate a different experimental setup leading to the results of Pei et al. being unusable for comparison.

I am glad the authors plan to work on making the experimentation more thorough and concise, till they do I will maintain my scores.

“I'm referring to the experiment in sec 4.2. Which dataset serves as the starting point for the prompts? Is the experiment based on a single prompt with 100 generations? If so, using a single prompt might make it difficult to fully verify the claim that "ActAdd can steer the model to talk about a topic."

Yes, that’s right; as noted in Appendix C, “The prompt used for all relevance completions is the neutral one: ‘Did you know that ’”. We think this provides good evidence for ActAdd's topic steering capability, given that the base rate of completing ‘Did you know that ’ with any particular topic is low.

Our new experiment verifies the claim on a range of prompts: “We will also run the topic steering experiments represented by Figures 4 and 7 on more prompts (drawn from the IMDb sentiment benchmark) to demonstrate that the steering works across a variety of prompts”. Sorry for being unclear!

“Why does ActAdd perform well for certain topics but not others (e.g., Art)? Is it effective only for steering toward specific topics? Additionally, it is unclear what accounts for the drop at c=0.5 for weddings? This might indicate some experiments on how reliable ActAdd is.”

We aren’t sure. These artifacts disappear in our new experiment which uses 1000 random prompts, which is encouraging.

Results for the new topic steering experiment (i.e. replicating Figure 4 from the original). Setup:

• n=1000 random Stanford IMDb prompts
• These prompts were filtered out by GPT-4o-mini if they were deemed to be relevant to any of [art, finance, music, politics, science, weddings].
• ActAdd applied at layer 6 (selected beforehand on a validation set)
• A range of coefficients c (values fixed beforehand)
• Prompt pair: "I talk about {topic} constantly" - "I do not talk about {topic} constantly"
• Temperature = 1.0, top-p = 0.3, freq_penalty = 1.0, max_new_tokens = 50 (these sampling parameters are constant across all experiments)
• on completions from GPT-2-XL
• Binary relevance scored by GPT-4o-mini.

Result (absolute % of completions deemed relevant):

relevance_n1000_gpt4omini_gpt2_l6

Note that some topics (like "politics") show non-monotonic response in the steering coefficient. We don't understand what's happening in that particular condition, but the trends look sensible for most of the topics.

Since they were drawn from IMDb, obviously the prompts will be disproportionately about art and music. As well as our GPT-4o-mini filter, we also check that ActAdd nonetheless improves on this base rate by noting ActAdd's change in percentage over the unsteered baseline (that is, the ActAdd % relevant - unsteered completion % relevant):

diff_in_relevance_n1000_gpt4omini_gpt2_l6

We have revised the PDF and supplementary information above to allow the reviewer to see this change in context.

We think this is a much better demonstration of the method's topic steering potential. Thanks for the suggestion!

We thank the reviewer for their extensive comments on ways to improve the write-up, lay-out and clarity of the paper. We will implement the suggestions and show them to the reviewer once complete.

“Can you elaborate on how you search for injection coefficient c and injection layer l? How expensive is this process?”

This is simple gridsearch over 17 * 18 = 306 values. This thus takes only around 1000 forward passes.

“In Figure 2, what is the x axis? Why should we expect perplexity to go down when x axis increases?”

The x-axis is the percentage of words in the tested passage that are wedding related. The graph is intended to show that adding a wedding steering vector improves predictive performance smoothly as the vector’s relevance to the domain increases (i.e. the x-value increases; more words are wedding-related).

In Figure 3, how is “P(steered completion contains wedding related words)” determined? Can you be more explicit about this in the paper?

This is poorly flagged by the y-axis label: “Non-zero wedding word count fraction” and by footnote 4. i.e. it is the fraction of completions that contain at least one of the hand-picked words {wedding, weddings, wed, marry, married, marriage, bride, groom, and honeymoon}. We will add a note in the text explaining this better, thanks.

Can you elaborate on what the p value in table 3 and 4 is? That is, what is the null hypotheses you are testing (and the corresponding alternative hypothesis)?

See Appendix C: “For bolding SOTA, we use a one-sample t-test to calculate p-values for sentiment and toxicity metrics.”

In Figure 5/S4.5, referring to the model’s behavior as “off-target answer probabilities” is rather misleading. That phrase reads as the model’s distribution over the answers for non-target tokens, whereas it seems that the actual probabilities being referred to is the P@K.

By “off-target” we mean that the domain is unrelated to the steering vector. We will clarify this in the text.

How do you determine which example to use to determine the steering vector? Did you do any studies on variance across the effectiveness for vectors derived from different examples?

We discovered the vectors via manual experimentation.

Are there any experiments to support the claim in the intro that activation engineering can enable composition of multiple traits, e.g. speech eloquence and mathematical content? If not, I would remove this to avoid overclaiming.

We said “might” in hopes of flagging that paragraph as speculative. However, there's some preliminary evidence supporting the speculation. [1]'s Appendix C.4 shows the compositionality of two steering vectors in an RL maze-solving setting. Somewhat relatedly, [2] find

the emergence of hidden capabilities, i.e., where latent interventions show the model possesses the capability to manipulate a concept, but these capabilities cannot yet be elicited via naive input prompting.

The notation in Algorithm 1 could use some improved clarity. For example, what is @? In code it can refer to a matmul; even though this seems like an indexing operation the ambiguity is confusing for the reader.

We agree. As the reviewer points out, @ is here the indexing operation. We will clarify this.

[1] Mini, Ulisse, et al. "Understanding and Controlling a Maze-Solving Policy Network." arXiv preprint arXiv:2310.08043 (2023).

[2] Park, Core Francisco, et al. "Emergence of hidden capabilities: Exploring learning dynamics in concept space." arXiv preprint arXiv:2406.19370 (2024).

Thank you for addressing the concerns regarding the experimental setup. The updated results and analysis address some of the original issues with the comparisons in Tables 3 and 4. However, after reviewing the new results, I have the following observations and concerns:

• Even if we consider LMA a "successor" method and exclude it from direct comparisons (though I would argue that it should still be included, given its publication at ICLR 2024), the updated results show that ActAdd does not outperform the PreAdd baseline on any benchmark.
• The performance on the /pol/ benchmark appears to be very poor. ActAdd reduces toxicity by only 2% while significantly increasing perplexity, reaching values close to Gu et al. 2022 (48.0 vs. 54.6). Since Gu et al's perplexity is classified as "too high for practical use" in your work, could you clarify what might be causing this gap in performance for this specific benchmark?
• While negative results can indeed offer valuable insights, the paper in its current form seems primarily structured around proposing ActAdd as a novel method that outperforms baselines. The new results, however, do not support this claim. A substantial revision would be needed to reframe the paper as a study presenting and analyzing a negative result.
• Additionally, the current negative result is somewhat limited in scope. A more compelling negative result might be along the lines of: "internal steering of language models does not outperform methods based on logits for these tasks." The current finding suggests only that a specific approach to internal steering (ActAdd) does not outperform logit-based methods.
• Could you clarify why you dropped the relevance metric in the updated results?
• Finally, the updated results show a significant increase in ActAdd’s perplexity compared to baseline methods. This differs from previous tables, where ActAdd showed lower perplexity. Could you explain what factors contributed to this shift? For example, was this due to differences in temperature settings across experiments?

Overall, the revised paper seem to introduce new weaknesses. The authors describe the new results as not purely positive, but they are actually (at least for these two very important tables) very negative. These concerns lead me to maintain my original evaluation score (reject).