Measuring Effects of Steered Representation in Large Language Models

The work, despite the problem it investigates, does not feel like it is complete. The authors present some interesting results, but the actual contribution here is not very clear and it appears to be quite minimal since the primary difference with prior works is simply using some additional parameters to adjust intermediate hidden activations. Though the method is explained, there's a lack of clarity as to what problem its explicitly attempting to solve and there isn't enough of an exploration of different settings. Furthermore, evaluation is done solely through a lens of a self-defined metric, therefore there may be some direct biasing of the method towards these.

Furthermore, sections 6.3 and 6.4 have been explored quite a bit in the past, such as through needle-in-the-haystack style tasks or simply other long-context problems. The particular link with steered representations here is not presented in a convincing enough way and therefore comes off as nothing more than simply the result of similar observations as those made in prior works with long-context reasoning.

• The experiments are limited to a few tasks and models. Adding more tasks would greatly strengthen the paper. Additionally, although many representation steering methods have been proposed, this paper evaluates only one specific type. While the experiments appear valid, the limited experimental settings raise questions about the generalizability of the findings produced by the proposed framework.
• The motivation for studying steering methods in the context of in-context learning is somewhat unclear. Since steering methods are commonly applied outside few-shot settings, it would be helpful to clarify any specific reasons for focusing on in-context abilities.
• Some evaluation aspects lack clarity. For example, providing examples or deeper discussion on what constitutes "side effects" would enhance the interpretability of the results.

Controllability tends to differ across models of different sizes, but this papers fails to include models above 8b or shows that findings generalize across different model sizes. This is important to understand the degree of generalization of the findings and metrics.

Even within two models from the same lab, of similar sizes Llama3-8B, Llama2-7B – it would be helpful to have further clarification from the authors if the key findings generalize across both. As it appears the trends differ across datasets. A clear understanding of what metrics actually hold in generalization across datasets and models during rebuttals is important.

As is, the major weakness of this paper is that it appears the metrics remain sensitive to model, dataset and the normalization criteria used. An understanding of how sensitive steering success is to the choice of normalization would also be helpful. For example, the statement " The normalization term can be chosen for the condition-specific measures." suggests that one needs to tailor the normalization -- which would require signficant hyperparameter turning and limit utility of the measures as generalizable.

• The novelty of the paper may be limited. The paper conducts an analysis on LM representation steering and presents the observations. Although the comprehensive analysis results can be practically useful to guide the usage of representation steering, the contribution might be slim.

• The paper presentation could become clearer by providing more background information of how activation steering is typically used (e.g. in the beginning sections). The paper studies a particular type of hidden representation modification, and it is better to contextualize the technique along with its variations.

• There are many vague expressions that are not clearly defined and underspecified experimental choices, which affects the overall research clarity. For example, in the end of Introduction, “increase the concept of output” is confusing. For some other examples, see questions below.