Rethinking The Reliability of Representation Engineering in Large Language Models

Thank you for your detailed feedback. We appreciate your comments, which have helped us better clarify our contributions. Below, we address each of your concerns in detail.

The impact might be weakened as LLMs scale up and improve their instruction-following capabilities.

We appreciate your thoughtful observation. While instruction-following capabilities may improve with scale, but this improvement does not automatically translate to enhanced reliability in representation engineering. Ignoring the inherent limitation of being confounded by spurious correlations could lead to overconfidence in this approach.

To address your concern, we conducted additional experiments on 1.1B and 13B models, comparing them to our original 8B results:

While CARE consistently outperforms PAIR, the performance gap narrows with model scale. Interestingly, manipulation scores is lower for the 13B model, not higher, suggesting that larger models may be more resistant to manipulation. Therefore, it is important to study the reliability of representation engineering as models scale up.

The impact of our work is not weakened. The new results highlight two exiting and promising directions:

• For smaller models: CARE provides a principled and low-cost approach to enhance reliability, making them suitable for safety-sensitive applications with constrained resources.
• For larger models, where manipulation remains challenging, future work could explore techniques in emerging research areas such as red-teaming to further improve safety.

We have added a discussion of these results and their implications in Appendix A.7. If you have further references or suggestions, we would be glad to include and discuss them. Thank you again for your valuable insight.

The RepE paper already uses this setup without making the connection to the naming convention used in control trial design.

We respectfully disagree with this opinion. The RepE paper does not implement matched-pair trial design and is unclear in its setting. For example,

• In the Honesty experiment of the RepE paper (Sec. 4.3), paired stimuli is used but do not consistently elicit opposing behaviors.
• In the Harmfulness experiment of the RepE paper (Sec. 6.2), the stimuli are not guaranteed to be paired at all.

This ambiguity places RepE’s approach in a gray area between our "BASE" and "PAIR" baselines. Importantly, this lack of methodological rigor and understanding of causality is a key reason for the unreliability of representation engineering.

Our work addresses this ambiguity by introducing a rigorous definition and implementation of matched-pair trial design, which is essential for isolating causal effects. This methodological clarity and precision enhance the reproducibility and reliability of representation engineering.

Manipulation and Termination are already studied as evaluations in RepE.

We did not list and will not claim manipulation and termination as our contributions. Instead, our goal was to formalize these metrics and distinguish them from traditional evaluation metrics (e.g., accuracy, precision).

The RepE paper references these concepts but does not provide formal definitions, and no corresponding implementations can be found in its open-source code. Explicitly formulating and explaining these metrics serve to enhance the clarity of the evaluation and help readers better understand our experimental results.

To avoid any misunderstanding about the contributions, we have carefully added a statement in the revised manuscript emphasizing that the RepE paper has considered similar concepts.

L280 in related work section just says: "Our work builds on RepE, focusing on interpreting LLM behaviors using neural activities within the representation space", but this is exactly what RepE does.

This sentence might appear unclear when taken out of context, and we appreciate the opportunity to clarify its intent.

This statement appears at the end of the first paragraph in the related work section, which introduces various interpretability techniques for LLMs. It serves to situate our work within the representation engineering paradigm, highlighting that our research builds on this specific interpretability technique.

In the subsequent paragraph, we delve deeper into recent advancements in representation engineering, which primarily focus on applying this method to different scenarios (e.g., alignment, jailbreaking, hallucinations). We explicitly distinguish our work by emphasizing its focus on the reliability aspect of representation engineering itself, a novel and orthogonal research direction that has not been explored in prior work and is fundamental to its applications.

We understand that the reviewer’s concern may stem from the ambiguity in the RepE paper. To address this, we have strengthened the comparison in the revised manuscript to ensure our focus on reliability is evident. Thanks for the feedback!

It would be useful to provide a concrete example for "inconsistent behaviour" in the intro, before figure 2.

This is a great suggestion! In the revised manuscript, we have included a concrete example (Figure 1) illustrating inconsistent behavior. This will better contextualize the vulnerabilities of representation engineering methods and make Figure 2 more accessible to readers.

Percentage of prompts do LLM fails to follow instructions.

Approximately 50% of prompts exhibit instruction-following failures in our experiment, depending on the dataset being used. This includes cases where the model refused to follow an unsafe instruction, as well as cases where the model behaves unsafely even though it was asked to behave safely.

Why is it infeasible to intervene on neural activities?

Intervening on neural activities is challenging due to their high dimensionality and lack of interpretability. We cannot directly modify individual dimensions in a way that reliably translates to desired changes in model behaviors. Representation engineering achieves intervention by learning a manipulation vector, but it does not reliably translate to desired changes in model behaviors either, as shown by the abundant experimental results.

The linear model setup described is supervised, whereas RepE also focuses on unsupervised methods.

While the RepE paper emphasizes unsupervised methods (e.g., PCA), its actual implementation is supervised, as it requires labels to construct contrastive pairs for PCA.

Interestingly, while the RepE paper observes superior performance with PCA, our experiments show that LR performs comparably or even better when confounding factors are effectively addressed. This finding provides new insights.

L308-310 references are broken.

Thank you for catching this issue. We have corrected the references in the revised manuscript.

If you believe the RepE paper indeed does matched-pair design, we would appreciate it if you could point to the specific sections or experiments that demonstrate this

In the RepE paper [1], see algorithm 1 line 7 and 8 where the experimental template and control template give rise to contrast vectors (line 10) that are used for steering, which is essentially the same as the matched pair design. See Section 4.3 as an application of this methodology to honesty. This discussion makes me wonder if the paper accurately reproduces the LAT method in RepE in their results at all.

the simplicity of a modification does not detract from its significance

I agree with this statement. However, I earlier raised concerns regarding the significance of this contribution, hypothesising that the gain achieved by the proposed methodology change of filtering stimuli would diminish as instruction following capabilities of models improve. This has held true from the followup experiments provided, as quoted by the authors:

the margin of improvement decreases for larger models

I am still willing to change my mind if you show a plot between instruction following capabilities on the X axis, and the gain from your methodology over contrast vectors / LAT in RepE on the Y axis, and if the slope is not negative after adding sufficient models with varying instruction following capabilities.

a foundational change to the methodology

I strongly disagree with this characterisation and think this amounts to over claiming, which is one of the main issues with the paper.

diminishing performance due to improved instruction-following capabilities apply equally to the original RepE methodology

My concern was the relative diminishing gain over RepE with the proposed methodological change in this paper, and not diminished absolute results. Also, it is not clear from the results reported how the original RepE methodology leads to diminishing performance with instruction following to me. I dont think this claim can be made from the results shown in this paper.

[1] Zou, Andy, et al. "Representation engineering: A top-down approach to ai transparency." arXiv preprint arXiv:2310.01405 (2023).

Thank you very much for reviewing our paper and providing constructive feedback. We address each of your concerns below:

Causal Identification/claims.

In CARE, our approach is based on a matched-pair trial design, which approximates a controlled experiment by ensuring that pairs of cases differ only in the type of neural activities (i.e., safe-inducing versus risk-inducing) while all other factors are held constant. By comparing the outcomes between these matched pairs, we can attribute the observed differences in model behaviors $Y$ to the neural activity $X$. This follows the standard definition of causality without making additional assumptions.

More complex causal structures, e.g., those involving multiple confounding factors or mediators, might need sophisticated identification techniques or discussion on causal bounds as seen in causal inference literature. However, this is out of the scope of this paper.

As for your concern on representation distribution shifts under interventions, we would appreciate any clarification on what specific aspects you are referring to. If you have relevant references or literature, we would be more than willing to incorporate and discuss these points in the revised version.

Theoretical guarantees/insights for when input matching implies representation matching in deep networks?

Thank you for raising this thought-provoking point. We understand the concern that, due to the inherent complexity of deep networks, even with content moderation models filtering the stimuli, there may still be unobserved confounding factors that are not explicitly controlled and may introduce subtle challenges in ensuring the matching procedure.

The goal of CARE is to provide a practical and approximate solution to isolate the causal effects of neural activity on model behaviors. By combining matched-pair trial design with content moderation models, we aim to minimize confounding effects to the extent possible within the constraints of the available tools and datasets. While this approach does not guarantee theoretical perfection, it offers a feasible framework to systematically evaluate and improve the reliability of representation engineering.

We believe this broader question of ensuring perfect matching is a significant and ongoing challenge in causality research, one that extends beyond the scope of this paper. If there are existing studies or techniques addressing this specific issue, we would be eager to reference and discuss them in our revised manuscript. We also welcome further suggestions on how to address this limitation in future work.

Evaluation Circularity.

Thank you for raising the concern about evaluation circularity and shared biases when using similar models for filtering and evaluation. This is indeed an interesting perspective, though we believe it is orthogonal to the core focus of this paper. The use of content moderation models (e.g., Llama-Guard) serves as a consistent standard to filter responses and reveal the limitations of representation engineering (RepE). Our objective was to demonstrate the vulnerabilities in RepE methods under a unified setup, not to develop a bias-free approach for evaluating LLMs.

We agree that there is ongoing work, such as using LLMs as a judge, which raises similar concerns about biases in evaluation. This is indeed an interesting direction for future research, and we have revised the manuscript to highlight it in the conclusion.

Gap between the manipulation score and termination score.

These two scores indeed serve different purposes and are not directly comparable in terms of a simple “higher is better” metric. Manipulation measures the ability to change behaviors by enhancing or suppressing neural activities, and termination reflects the necessity of those activities for generating specific behaviors.

To clarify with an analogy: consider that smoking is neither a sufficient nor a necessary cause for developing lung cancer—some smokers never develop cancer, and some cancer patients have never smoked. However, smoking still increases the risk. Similarly, in our study, the identified neural activities are not always necessary for specific behaviors (as indicated by a low termination score), but they are highly effective for reversing model behaviors (as shown by a high manipulation score). These results suggest that manipulation may be a more suitable way for controlling model behaviors.

Thank you very much for taking the time to review our work and for providing valuable feedback.

The proposed modification seems to be incremental and too straightforward.

We respectfully disagree with the characterization of our approach as incremental. As noted by Reviewer 1, the issue we address has not been previously explored. While the proposed solution is straightforward, identifying the vulnerability in representation engineering and designing a principled way to address it is both non-trivial and impactful. Given the widespread use of representation engineering for understanding and improving model behaviors (especially safety), ensuring its reliability is essential for achieving these goals.

The simplicity of our method makes it an ideal first step toward addressing the reliability of representation engineering. The existing body of work focuses on applying RepE in various contexts, but few studies examine its underlying assumptions or its reliability. Our contribution fills this gap with a novel and practically effective approach that lays the groundwork for advancing this research area. Thus, while simple, our method is not incremental but foundational.

The analogy to neuroscience may be unnecessarily complicated.

We appreciate your feedback on this point and have streamlined these discussions in the revised manuscript. The analogy is chosen to highlight an important parallel: much like in early neuroimaging studies, where researchers attempted to infer human behavior (e.g., honesty or criminality) from brain scans, we find that current representation engineering researches face a similar risk. Without establishing a reliable causal relationship between neural activities and behaviors, the interpretations derived from these methods can be misleading or even detrimental if used for safety-relevant applications. By drawing on lessons from cognitive neuroscience, we hope to emphasize the importance of causally grounded analyses in LLM interpretability. Therefore, we streamline some discussions while retaining key references for readers who may find these parallels insightful and wish to draw inspiration from them.

Even without filtering (as in the original RepE method), the activation may still have a causal effect on the behavior ...

We appreciate the opportunity to clarify this point. As you noted, "may still have" means that the identified activation does not consistently cause the intended model behavior. The absence of reliable causal effects reduces the utility of RepE in explaining or controlling behaviors. Our method, built on controlled experiments and supported by extensive empirical results, helps isolate causal effects effectively, thus improving the reliability of representation engineering.

We would like to express our sincere gratitude for your detailed feedback and insightful questions. Below, we address each of your concerns in details.

The link between failures of instruction-following and spurious correlation.

Thank you for raising this point. In the context of this paper, our focus is on cases where inconsistent behavior during neural activity collection leads to spurious correlations. When data includes such inconsistencies, the identified neural activities may not be the cause of model behaviors (e.g., safe and unsafe), thus compromising the reliability of representation engineering.

Safety-relevant biases could also caused by alignment fine-tuning.

To address your concern, we conducted experiments on uncensored models, i.e., models trained on datasets where responses containing alignment / moralizing were removed, thereby the model is not biased towards alignment. The results confirm that the existence of spurious correlations leads to unreliable representation engineering, even in the absence of alignment fine-tuning.

We focused on safety-related topics because it is a critical application of representation engineering. Using existing content moderation models allowed us to efficiently label data without the need for recruiting and training human labelers, making it valuable for the research community to verity the vulnerability of representation engineering with limited computing resources.

Percentage of data with confounding factors for baseline.

About 50% of samples showed behavior inconsistent with instructions for baseline.

Why LR performs worse than mean differences in baseline?

The underperformance of LR aligns with the results reported in the original RepE paper, where LR was the weakest method. The authors of the original RepE paper recommended PCA as the preferred approach but did not provide further explanations for LR's poor performance. Our experiments show that when confounding factors are properly controlled, LR achieves high manipulation scores, suggesting its earlier poor performance was due to sensitivity to confounders rather than inherent limitations of the algorithm.

Typo in Table 3, "Manipuation Score".

Thank you for pointing out this typo. We have corrected it in the revised manuscript.

I wish to disagree with the characterisation of some parts of this summary, specifically focusing on ones where I am quoted.

Additionally, Reviewer frYB noted that our proposed approach is "principled, easy to implement, and useful for practitioners. This feedback highlights the significance of our contributions, not only in proposing a novel method but also in identifying a key research question that worth further investigation.

The "proposed approach" is an incremental change (filtering responses where the model doesn't follow instruction) on the main LAT pipeline in the RepE paper, and hence not novel, significant, or a key research question worth further investigation in my opinion. This opinion on this can differ, but it is important to point out I disagree with this claim made by quoting parts of my review.

We conducted additional experiments using both smaller (1.1B) and larger (13B) LLMs to evaluate scalability. These results, included in Appendix A.7, confirm that our method performs reliably across different model sizes, and point to promising directions for future research.

I had asked for these experiments as I thought the effectiveness of the proposed change to the LAT pipeline would decrease as instruction following capabilities improve. The results in Appendix A.7 only confirm this, and I would not use them to claim the method performs reliably across different model sizes. In fact, I believe if this approach is used on state of the art models, the improvement will be negligible, but this isn't tested in the paper.

and do not control for confounding factors.

Could you please list what these confounding factors are, and quantify the fraction of stimuli each of the confounders affect?