To Steer or Not to Steer? Mechanistic Error Reduction with Abstention for Language Models

Thank you for all these important remarks. We’re encouraged that you found our work a timely and relevant contribution to the steering literature and that our focus on steering for error reduction is a differentiator/ strength! We’ve addressed all of your remarks below. Please let us know of any remaining questions.

1) Inconsistent gains. We also note that MMLU-HS is more difficult to steer than the other tasks evaluated (except Llama-IT!). This corroborates [3] + aligns with a broader limitation shared by all additive linear steering methods — assuming that the activations and the target quantity (here, the model error) are linearly related. One of MERA’s key contributions is to explicitly detect and gracefully handle these “unsteerable” instance cases or datasets. Our selective steering mechanism (see Eq. 7) and then also the abstention criterion (see Sec. 3.2), ensure that we intervene only when confident (which is controlled by the parameter $\delta$). Rather than steering on all inputs and datasets, MERA steers conditionally.

On reflection, we see that attributing MMLU's limited steerability only to class cardinality is premature. Other confounding factors such as label ambiguity, semantic class overlap, or general task complexity may be equally or more relevant. It may be that steering toward correct answers might be easier when classes lie along a single semantic axis (e.g., yes/no, positive/negative) than in a general MCQA format. This needs to be empirically verified — characterising steering techniques’ limits wrt distinct dataset properties deserves careful consideration and is an exciting direction for future work (see e.g., [1]). We've now removed the prior formulation citing cardinality as the primary cause for MMLU-HS’s limited steerability. We appreciate your attention to this detail and the opportunity to improve our paper in this regard!

2) MERA may still be expensive. You're right that MERA introduces its own components, such as probe training and calibration. However, these steps are principled and crucially, MERA avoids per-model*, per-layer, per-token and per-multiplier hyperparameter sweeps, which is the common sweep strategy in activation steering**. This creates a huge search space and brings on huge computational costs, typically forcing the practitioner to prioritise one hyperparameter over the other (we refer to Appendix A.1 for a discussion, which we’re also currently expanding). MERA replaces this process with a search of ~10 $\alpha$ values per task (dataset and model combination). This seriously reduces computational costs and as we see it, establishes MERA as a more efficient method in comparison.

*Even with significant compute, the best outcome of steering papers is often model-specific advice like “steer on layer 13 for Llama-7B” which may not generalise well.

**An Anthropic blog post [9] summarises current steering practice well: "For all evaluations, we varied the 'steering factor' between -20 and 20. This decision was arbitrary."

3) Benchmarks. We used the MMLU-HS subset simply because the questions would be consistent in difficulty and thus make per-sample comparisons cleaner. Practically, this subset also matched the sizes of the other datasets in the paper. But we agree that it would be interesting to evaluate MERA on more difficult subsets of MMLU. Consequently, we're currently running the MMLU "professional" subset as well.

4) No steering results in main paper. As steering is a post-training intervention, we found it more important to focus on reporting the relative steering performance in the main manuscript (over the absolute results) but we appreciate that absolute metrics provide complementary information that can be helpful to understand true steering effects! In Table 4 in the Appendix we included all the unsteered accuracies for each model and dataset combination (from which steering gains could be indirectly judged) but to make it simpler for the reader, we’ll complement this table with an additional table containing steered accuracies as well as the raw deltas. We will discuss these results in the main manuscript as well. Also, to understand why a bespoke metric was introduced, see answer (3) of Reviewer u9yC.

5) Code. Please see answer (3) of Reviewer 7dxY.

6) Broken citation. Line 671 is fixed.

7) Difference to CAST. Thank you for sharing this relevant work! We have read the paper (and cite it in Appendix A.1). Our method differs from CAST in several ways (i) conditioning logic: MERA conditions on predicted error and CAST on cosine similarity between “condition vectors” and the model’s activations exceeding $\theta$, (ii) steering direction: MERA learns probe weights per layer and CAST uses PCA on contrastive pairs, with multiple directions, (iii) strength selection: MERA solves for $\lambda^{\star}$ with closed form and CAST uses a grid search.

[8] https://www.anthropic.com/research/evaluating-feature-steering

Thank you for the very detail-oriented and helpful review! We’re excited that you believe our work has benefits for transfer learning and personalisation and that our mathematical framework may become a building block in future work. We have addressed all of your points below:

1) Evidence for over-/under- steering. We appreciate the suggestion — thank you for the suggestion—visualising activation space geometry (as in [4, 6]) can indeed help interpretation. However, instead of using UMAP, we now provide a more direct analysis of over- and under-steering effects by reporting transition counts on a sample-by-sample basis: how many examples move from incorrect to correct (0→1), correct to incorrect (1→0), or remain incorrect (0→0) or correct (1→1). This analysis will be included in the Appendix together with the SPI results and referenced in the main manuscript.

2) Baseline layer sweep. Please see the answer (2) in Reviewer H7hE.

3) Evaluation with SPI. We identified the need to define a new metric (SPI) since we ourselves couldn’t find a suitable metric that not only expresses improvement but critically, also degradation in a bounded, interpretable way. Existing metrics (e.g. from OpenAI [7]) only track positive gains. By normalising relative to performing ceiling (or floor), the reader can get a sense of how good the steering method is compared to how its best or worst case. How much (in percentage %) does steering improve or degrade LM performance? If SPI=1, steering makes the LM a perfect classifier. If SPI=-1, steering made the LM incorrect on all test samples. SPI is easy to interpret and particularly helpful when comparing LMs side-by-side (which often differ vastly in initial unsteered performance!). We’ll make sure to add an expanded, more clear motivation of the metric in the paper.

4) Caveats on scope. Totally fair — our experimental scope includes language models going up to 3B and datasets of multi-choice. Given our compute constraints, we prioritised coverage across model types (base vs. instruction-tuned, and three distinct families), and chose supervised tasks to allow controlled evaluation where error can be perfectly recovered at distinct token positions (and not approximated e.g., using external oracle LMs). We'll emphasise our scope better in Section 7.

5) Clarifying $\delta$. In our method, $\delta$ is user-defined and sets the confidence level for steering. For instance, setting $\delta$ = 0.05 corresponds to a 95% confidence threshold: steering is applied only when we're statistically confident that it will improve performance. To make room for new clarifications from this rebuttal, we’ve decided to remove this section in the revised manuscript! Thank you again!

6) LMs error proneness. Thank you for flagging this — we’ve adjusted the wording to “[can be] frustratingly…” instead of “[are] frustratingly…” to better reflect variability. As shown in Table 4 (Appendix), even capable base/ IT models can sometimes yield low accuracy (e.g., 5.6%) and high error rates such as 0.89) on certain tasks.

7) Clarifying generalisability. While we focus on classification tasks for clarity and control, MERA is a general framework. It only requires a real-valued or binary signal to train the probe — so it can, in principle, be applied to other supervision types like truthfulness, toxicity, or helpfulness! Our optimisation and calibration steps are agnostic to task type and independent of model size. To apply MERA to free-form generation, each output should be paired with a target label. This can be done, for instance, using external oracles or LM-based rating systems. We’ll clarify this in Section 7 to better highlight MERA’s broader applicability.

8) Formatting. We have revised Figure 3 for improved clarity! We have also streamlined the writing and notation in Sections 2 and 3 to make the mathematical content accessible.

9) Questions. a) We have not yet evaluated any larger models than 3 billion parameters, but are planning to do that with access to more compute. b) Yes — in all our experiments, the steering methods are evaluated in two complementary modes: "last" and "exact". (i) In "last", we evaluate the model’s logits at the final prompt token. (ii) In "exact", we check whether the model generates the correct answer in its free-form output… where the second “exact” mode accommodates the open-ended settings! c) Fixed-strength baselines apply the same intervention regardless of the model’s current activation state, making over- or under-steering inevitable by design. Empirically, we observed this in two ways: (i) that baselines often output negative SPIs (i.e., steering inadvertently degrades the model), and (ii) instance-level transitions where steering flips correct predictions to incorrect ones. We’ll now include the transition analysis in the paper!

[6] https://arxiv.org/pdf/2312.01037

[7] https://openai.com/index/weak-to-strong-generalization/

Thank you for all the time taking to provide a very detail-oriented and helpful review! We’re glad to hear that you found our claims straightforward, clear + verifiable, our optimisation framing is a great way to move the fields towards more principled foundations and our experimental designs are clear. Now, we have addressed all of your points below:

1) Regression target for probes. Regarding learning the linear model not directly on the error but on the inverse of the sigmoid (i.e., the logits), we agree that this is a promising direction that could potentially enhance the performance of MERA. We are currently re-running this experiment on all evaluations to assess its impact. Preliminary results steering Llama and Gemma models on SMS spam dataset suggest a slight improvement over the original formulation. We’ll reassess when the complete results are in and if supported, update the methodology accordingly.

2) Describing BASE-$\mu_{k}$. Thank you for pointing this out! We agree that mixing strategies across methods can undermine fair comparisons and we appreciate the opportunity to clarify! In our experiments, we do not use any MERA-specific best-performing layer for the BASE-$\mu_{k}$ baseline. Instead, we choose the layer with the lowest RMSE under a trained linear probe — a shared external signal reflecting where the error is most linearly related to model activations —which we think enables the fairest comparison on balance. As we see it, probe performance is a good heuristic for selecting contrastive steering layers, since it neutrally depends on how linearly our target is related to the activations (and does not favour MERA which steers on all layers). It is worth mentioning that the community hasn’t converged to one single criterion for layer selection for contrastive difference-in-means steering (see distinct approaches in [2-5]) but that many methodologically valid strategies exist. We’ll revise the section to make our approach clearer!

On the choice of $k$ — we agree that results vary significantly between $k = 50$ and $k = 100$. To us, this reflects a general instability in contrastive baselines. To construct these sets, we sorted the error scores of the 3000 training samples and selected the top-k and bottom-k examples. We’ll add these details to the manuscript as well!

3) Theoretical claim derivation. We acknowledge that there is indeed a selection bias in our original derivation, since the same calibration dataset was used both to construct the confidence intervals and to select the best-performing $\alpha$. We have now addressed this issue by adding a Bonferroni correction to the confidence bound. As a result, the reported confidence levels have been adjusted: the results previously shown at 99% confidence are now updated to 90%. Full details are provided in our response to Reviewer 7dxY. We also outline a less conservative alternative and will clarify in the next paper version that our updated method has formal guarantees under the i.i.d. assumption.

4) SPI aggregration. We appreciate this observation. Figure 4 was intended to offer a practitioner-oriented overview (for a quick comparison across settings to help identify overall trends) and Table 1 to provide complete details on model- and dataset-specific effects. As we see it, these provide complementary perspectives which are helpful to the reader in distinct ways. We now emphasise this dual purpose more clearly in the manuscript and explicitly caution readers about over-interpreting the aggregated view.

5) Personalised steering citation. Thank you for pointing us to this interesting read! We find the work highly relevant as a steering reference and have thus cited it in our paper (+ added a discussion in Appendix A.1). This work is different from ours in several ways, most notably, in the specific problem it is targeting — where they don’t directly solve for steering strength ($\lambda$) (but find it via hyperparameter sweeps, see Table 5 on p.8). This contrasts with MERA, which gives the solution in a closed form.

6) Steerability and MMLU. What MMLU can/not teach us is a very fascinating comment! We refer to answer (1) in Reviewer MXKB for a discussion on this topic.

Naturally, linear steering has its limitations. In this paper, we suggest possible extensions in Appendix A (Non-linear Case). A simpler alternative, however, is to use a first-order approximation of the non-linear model. This approach would essentially replace the fixed linear weight used across all instances with the gradient of the non-linear function for each specific input.

7) Improved writing. Thank you for taking the time to write out the example where our writing could be improved! We’re revising Sections 3.2 and 3.3 and also the rest of the paper for clarity.

[2] https://openreview.net/pdf?id=HuNoNfiQqH

[3] https://arxiv.org/pdf/2312.06681

[4] https://arxiv.org/pdf/2310.06824v3

[5] https://arxiv.org/pdf/2402.14433

Thank you for the constructive and informative review! We have addressed your four key points below:

1) Hoeffing inequality — Clarifying Our Selection Procedure.

Our selection procedure ensures we do not choose any $\alpha \in (0,1)$ unless it statistically significantly improves performance. Specifically, we first identify a set of $\alpha$ values with provably positive performance gain (with high probability), then select the empirically best among them.

Let $f(\alpha, X_i) \in [0,1]$ be a performance function and $X_i$ a random input. Given a calibration dataset $D_n = \\{ X_1, \dots, X_n \\}$, the empirical performance is:

$$P(\alpha, D_n) = \frac{1}{n} \sum_{i=1}^n f(\alpha, X_i),$$

with $P(\alpha, D)$ denoting the true (population) performance.

Procedure Overview

1. Discretize $[0,1]$ into $M$ values: $\alpha_{set} = \\{\alpha_1, \ldots, \alpha_M\\}$.

2. Confidence Bands: Apply Hoeffding’s inequality to each $\alpha_j$:

   $$\Pr\left( \left|P(\alpha_j, D_n) - P(\alpha_j, D)\right| \le \delta_n \right) \ge 1 - \frac{\alpha}{M},$$

   where $\delta_n = \sqrt{\frac{\ln(2M/\alpha)}{2n}}$.

3. Union Bound: Ensures all bounds hold simultaneously with high probability:

   $$\left|P(\alpha_j, D_n) - P(\alpha_j, D)\right| \le \delta_n \quad \forall j=1,\ldots,M.$$

This yields a uniform guarantee:

$$\Pr\left(\sup_{\alpha \in \alpha_{set}} \left|P(\alpha, D_n) - P(\alpha, D)\right| \le \varepsilon_n\right) \ge 1 - \alpha.$$

We define the valid set:

$$\alpha_{\text{valid}} = \\{\alpha : P(\alpha, D_n) - \epsilon_n > 0\\},$$

and select:

$$\alpha^* = \arg\max_{\alpha \in \alpha_{\text{valid}}} P(\alpha, D_n).$$

Since the confidence bands hold uniformly with high probability, we guarantee $P(\alpha, D) > 0$ for all $\alpha \in \alpha_{\text{valid}}$. Hence, $\alpha^*$ corresponds to a true performance improvement, or we abstain otherwise.

Note: We will make this guarantee more explicit in the next version of the paper.

---

Alternative Calibration Perspective

Instead of using a Bonferroni-style correction (which can be conservative), one could:

• Split the data into two parts:
  • Use one part to find the empirically optimal $\alpha^\star$.
  • Use the other to estimate a confidence interval for $P(\alpha^\star, D)$.

We accept $\alpha^\star$ only if its confidence interval lower bound exceeds a desired threshold. This avoids overly conservative bounds and allows for tighter, adaptive inference at the cost of splitting the data.

2) Cardinality versus steerability. Thank you for raising this important point. We agree that four classes is not especially high, and in retrospect, attributing the difficulty of steering on MMLU-HS to cardinality alone is premature (dataset charactistics like semantic overlap, label ambiguity, and task complexity could also play a role in a task’s steerability). See further discussion in answer (1) at Reviewer MXKB. As we see it, a dedicated, systematic study (e.g., along the lines of [1]) that carefully varies such characteristics while analysing steering performance would be necessary to understand the limits of additive methods like MERA (+ its baselines). We've revised the formulation in the manuscript.

3) Code availability. Apologies — we missed attaching the code at submission. We had it ready for rebuttal, but then we just learned that ICML guidelines don’t allow updates to the original submission in the discussion phase. If we’re allowed to share an anonymous link (can AC please let us know if this is acceptable) we can upload the source code, notebooks and installation guides directly there!

4) Relation to PaCE. Thank you for pointing out the connection to PaCE! Your analysis is very insightful. While there are mathematical similarities, as you point out (i.e. adaptive intervention strengths based on the inner product of activations and steering directions), the overarching goals of PaCE and MERA are very different. The PaCE model is trained on data demonstrating ‘benign’ and ‘harmful’ concepts from some predefined dictionary, and performs interventions to suppress the harmful ones, while MERA is fundamentally concerned with error mitigation on well-defined prediction tasks. We also can’t see a direct analogue of our calibration step, which uses calibration data to identify the optimal intervention strength for error mitigation on a per-layer basis. That said, we consider the work relevant enough for inclusion in Section 3 and have also added it to the Related Works in Appendix A.1.

[1] https://arxiv.org/pdf/2407.12404