Teaching LLMs to Decode Activations Into Natural Language

Thank you for your thorough review and helpful suggestions. We have implemented them and believe that they will help improve the quality of the paper.

The paper would benefit from an improved pipeline diagram to clarify the process flow.

• We have updated Figure 1 and included a new Figure 2 to clarify the process flow.
• Please let us know if you have additional suggestions to the figures, as we would be excited to make them more legible.

LatentQA heavily relies on extensive data and faces challenges in scaling efficiently.

• We refer the reviewer to the common response part 1 (scaling experiments). We paraphrase the main takeaway of Section 5.3: Latent Interpretation Tuning scales with both increasing dataset size and increasing model size, which demonstrates the power of training-based approaches.

The model’s interpretative ability is limited to the specific personas and activations it was trained on, which restricts its generalizability. For example, it can only interpret personas included in its training dataset. It would be valuable to assess whether LatentQA can perform well in a zero-shot setting

• We note that all of the personas and questions we interpret in the reading and control sections (5.1 and 5.2) are novel, which suggests that our decoder is indeed able to generalize.
• Finally, we refer the reviewer to the common response part 1 (additional control experiments). For convenience, we paraphrase:
  • We are able to control the model to respond differently to user questions depending on the model’s belief of the user’s demographic information, e.g., if the user asks “What car should I purchase?”, the model’s response will differ depending on their income.
  • This suggests that the decoder is able to generalize to novel types of control at test-time.

An ablation study analyzing the impact of different layers’ activations as input would enhance the robustness of the findings.

• We refer the reviewer to the common response part 2 (reading and writing layer ablation). We find that our initial choice of layers (layer $k = 15$ to read from and layer $\ell=0$ to write to) yields the best generalization across our hyperparameter study.
• Intuitively, this result makes sense for two reasons. First, we read from layer $k=15$ because we expect the middle layers of the model to be the most semantically rich [1]. Second, we write to layer $\ell = 0$ because we want to provide the decoder with the most steps to process the activation. Although there is a distribution shift from the 15th to the 0th layer, the decoder is trained, so it learns to handle this distribution shift.

A more detailed description of the dataset construction process, along with an analysis of its diversity, would strengthen the paper's validity.

• We have rewritten Section 3 to more precisely discuss the dataset creation. We have also added Appendix A, which provides the prompts and more thoroughly discusses the process used to create our dataset construction process. We are happy to include more specific information if you have any suggestions.
• Could you clarify what you mean by an analysis of the dataset’s diversity? We are happy to run more detailed analyses.

Will the data used in this study be publicly accessible?

• Yes. We will release the data after the anonymity period.

Dear Reviewer eoci,

Thank you for your detailed and insightful suggestions! We hope that our previous comment and the revised version of the paper (available on OpenReview) have adequately addressed your concerns.

If the revisions meet your expectations, we kindly ask if you might consider updating your review to reflect your revised opinion of the paper.

Thank you for your helpful suggestions. We have implemented them.

I find patching activation from layer 15 to layer 0 weird.

• We refer the reviewer to the common response part 2 (reading and writing layer ablation). We find that our initial choice of layers (layer $k = 15$ to read from and layer $\ell=0$ to write to) yields the best generalization across our hyperparameter study.
• Intuitively, this result makes sense. First, we read from layer $k=15$ because we expect the middle layers to be the most semantically rich. Second, we write to layer $\ell = 0$ because we want the decoder to have the most steps to process the activation. Although there is a distribution shift from the 15th to the 0th layer, the decoder is trained, so it learns to generalize.

so each time a person wants to do this for a different model, they need to collect 1.2 data points from that model?

Yes, although this is not a substantial cost.

• We have improved our dataset (Section 3) and find in practice we are able to achieve similar performance with only 17K data points. Our initial dataset during submission had many duplicated data points, and we find that removing these in fact improves performance.
• At its current size, our dataset is smaller than most instruction-tuning datasets (e.g., Alpaca has 52K instructions) and trains faster.
• Finally, we note that training a new system for each type of LLM backbone is a common paradigm. For example, there is a new copy of LLaVA trained for each LLM.

On 5.1 results: are there other training-based methods you can compare to?

Yes; we refer the reviewer to common response part 3 (benchmarking against trained baselines). For convenience, we paraphrase:

• For the first reading task, we add linear probing, a baseline trained on task-specific data and find that our decoder outperforms linear probing by an absolute 32.3% over 6 tasks.
• For the second reading task, linear probing is not applicable because we do not have a prior knowledge of the system prompts to probe. Unfortunately, there are no other appropriate trained baselines.
• Finally, we compare our decoder to all pre-existing LatentQA systems, neither of which train their decoder. We view part of our contribution as the first method to train a system for LatentQA.

On 5.1: There is a debiasing specific activation steering method... have you tried to compare debiasing results with this method?

The method in the listed paper is customized for position bias (whether the model assigns higher likelihood to the first or last answers in a prompt), whereas the CrowS pairs dataset focuses on stereotype bias (whether the model assigns higher likelihood to sentences with stereotypes). We are not aware of a simple way to convert that method to our application, but would be happy to implement suggestions.

On 5.2: LIT seems to perform the worst on 'Generate Positive'.

LIT performs worse than RepE, however, it still outperforms prompting. We will more carefully remark upon this limitation in the final revision, as it suggests future work may help improve the robustness of the control.

Writing clarifications

Is dialog = control + control response?

• Yes; moreover, we have updated sections 3 and 4 to make their notation more consistent.

What do you mean by repeatedly calling STEER(ACT, "control") and what is STEER(ACT, “control”)?

• We rewrote the explanation of control in Section 4 and in Appendix B.3. Roughly, STEER([Act], control) is the gradient with respect to [Act] of the decoder’s logprob of generating answer given [Act] + question. By repeatedly updating [Act] with these gradients, we can approximately identify the target LLM's activations that maximizes the probability of the control.

Why do you need to sample activations from prompts from he Databricks Dolly?

• When performing the control, we need activations from the target LLM. During training, these activations come from stimulus prompts. During control, we need some set of stimulus prompts. Most sets of stimulus prompts (as long as the prompts are diverse) work for control. We thus need a diverse bank of prompts, e.g., Databricks Dolly dataset.

In my understanding, RepE is a fully training free, representation engineering method. Why did you use LoRRA fine-tuning with RepE?

• The representation engineering (RepE) paper describes two methods for control: a training-free method, which adds steering vectors to guide activations, and a training-based method, which trains a LoRA to approximate the effect of adding steering vectors to activations. The latter training-based method is called LoRRA finetuning.
• Because we want a fair comparison to a trained method, we compare our decoder to RepE using LoRRA finetuning. See [1] for more details.

[1] Zou, Andy, et al. "Representation engineering: A top-down approach to ai transparency." arXiv preprint arXiv:2310.01405 (2023).

Thank you for the rebuttal and clarification. My concerns are mainly addressed. To summarize, i still think the paper has great potential, and the revision made by the authors greatly improve the paper. I still think the results for control experiment is a little shy -- I will raise my score to 5 for now. For future iterations, it would be great if we can see more and better results for control experiments. Perhaps its more suitable for other control tasks -- jailbreak a safety aligned model and safeguard a base model?

Another thing that will strengthen the paper is more extensive evaluation on the jailbreaking control... there is a comprehensive benchmark on jailbreaking like MaliciousInstruct dataset, JailBreakBench dataset.. or you can use a toxicity-based dataset. The evaluation can be performed using win rate %or toxicity score.. rather than the current manual inspection

Thank you for your thorough review and helpful suggestions. We have implemented them and believe that they will help improve the quality of the paper.

Comparing these directly may not be entirely fair, and it may lead to overclaiming the observed performance improvements.

We refer the reviewer to the common response part 3 (benchmarking against trained baselines). For convenience, we paraphrase:

• For the first reading task, we add linear probing, a baseline trained on task-specific data and find that our decoder outperforms linear probing by an absolute 32.3% over 6 tasks.
• For the second reading task, linear probing is not applicable because we do not have a prior knowledge of the system prompts to probe.
• In all of the control tasks, we already compare to a trained baseline, the LoRRA finetuning method from RepE.
• Finally, we compare our decoder to all pre-existing LatentQA systems, neither of which train their decoder. Our method outperforms both.

Another concern is the generalization capability of the model. Specifically, it would be insightful to test whether it’s possible to achieve cross-model understanding, such as using activations from one model (e.g., LLaMA) to interpret the behavior of a different model (e.g., Mistral).

We refer the reviewer to the common response part 1 (additional control experiments). For convenience, we paraphrase:

• We are able to control the model to respond differently to user questions depending on the model’s belief of the user’s demographic information, e.g., if the user asks “What car should I purchase?”, the model’s response will differ depending on their income. Our decoder performs as well as prompting and linear probing.
• This suggests that the decoder is able to generalize to novel types of control at test-time.
• We will post results demonstrating that our LatentQA training applies to other models, such as Mistral’s Ministral-8B-Instruct-2410.
• We agree that your experiment would be interesting to test. However, a more natural setup would be to train a new decoder for each target LLM, similar to visual instruction tuning (e.g., there are multiple versions of LLaVA, one for each LLM).

It’s challenging to see how the studied applications, such as decoding latent information or controlling behaviors, directly contribute to a deeper understanding of the model’s internal workings.

• Thank you for pointing out this inconsistency. Our work focuses more on applications of interpretability, specifically, developing techniques that enable novel applications, such as predicting future model behavior from the activations.-
• We rewrote the introduction to more carefully describe our focus on applications of interpretability, rather than understanding the minutiae of LLM internals (which is more akin to mechanistic interpretability).

What are the practical applications of predicting future model behavior based on latent representations, and why is this an important problem to investigate?

• An important practical application we hope LatentQA addresses is monitoring of harmful model behavior. In particular, models exhibit unexpected harmful behaviors at test time if their goals change, e.g., through a feedback loop [1] or with a prompt injection [2]. A LatentQA system could help interpret the internal state of the target LLM and flag any potentially erratic or harmful behavior, serving as an additional guardrail on model safety.

Since WikiText-103 is used to extract phrases containing the subjects, these phrases may also include objects. Could this lead to potential data leakage?

• Thank you for the insightful observation. The dataset was filtered to remove the objects. We will note this in our dataset construction.

How is the “appropriate layer” determined for replacing activations? Why do you consistently replace activations at layer 0, rather than other layers?

• We refer the reviewer to the common response part 2 (reading and writing layer ablation). We find that our initial choice of layers (layer $k = 15$ to read from and layer $\ell=0$ to write to) yields the best generalization across our hyperparameter study.
• Intuitively, this result makes sense for two reasons. First, we read from layer $k=15$ because we expect the middle layers of the model to be the most semantically rich [3]. Second, we write to layer $\ell = 0$ because we want to provide the decoder with the most steps to process the activation. Although there is a distribution shift from the 15th to the 0th layer, the decoder is trained, so it learns to handle this distribution shift.

[1] Pan, Alexander, et al. "Feedback loops with language models drive in-context reward hacking." arXiv preprint arXiv:2402.06627 (2024).

[2] Wu, Chen Henry, et al. "Adversarial Attacks on Multimodal Agents." arXiv preprint arXiv:2406.12814 (2024).

[3] Ghandeharioun, Asma, et al. "Who's asking? User personas and the mechanics of latent misalignment." arXiv preprint arXiv:2406.12094 (2024).

Dear Reviewer RaEp,

Thank you for your detailed and insightful suggestions! We hope that our previous comment and the revised version of the paper (available on OpenReview) have adequately addressed your concerns.

If the revisions meet your expectations, we kindly ask if you might consider updating your review to reflect your revised opinion of the paper.

Thank you for your review and detailed suggestions. We have implemented them and believe that they will help improve the quality of the paper.

“The term ‘activation’ should be explained clearer at the beginning

• We have changed the description of LatentQA to be more precise. Particularly, from L38-L40: “we consider the task of LatentQA, open-ended question answering (QA) about latents, i.e., model activations, in natural language.”
• In particular, we define latents as model activations. Please let us know if you have further clarity suggestions.

It would be fair and convincing to use the same Llama-3 when comparing to Patchscope

• We reran the Patchscope experiment with Llama-3-8B-Instruct and updated Table 1 with the new numbers. They are reproduced below for your convenience. In particular, we see that our decoder still outperforms Patchscope and linear probing, a trained baseline that requires task-specific data.

I would like to see other types of control in the experiment, e.g., topics

• We refer the reviewer to the common response part 1 (additional control experiments). For convenience, we paraphrase:
• We are able to control the model to respond differently to user questions depending on the model’s belief of the user’s demographic information, e.g., if the user asks “What car should I purchase?”, the model’s response will differ depending on their income. Our decoder performs as well as prompting and linear probing.
• This suggests that the decoder is able to generalize to novel types of control at test-time.

Methods like DExpert and Patchscope are training-free methods, while your LIT needs finetuning LLMs. The comparison seems to be unfair. We refer the reviewer to the common response part 3 (benchmarking against trained baselines). For convenience, we paraphrase:

• For the first reading task, we add linear probing, a baseline trained on task-specific data and find that our decoder outperforms linear probing by an absolute 32.3% over 6 tasks.
• For the second reading task, linear probing is not applicable because we do not have a prior knowledge of the system prompts to probe.
• In all of the control tasks, we already compare to a trained baseline, the LoRRA finetuning method from RepE.
• Finally, we compare our decoder to all pre-existing LatentQA systems, neither of which train their decoder. Our method outperforms both.

I would expect to see more LLMs to be tested.

• We are currently running the reading experiments (Section 5.1) with Mistral’s Ministral-8B-Instruct-2410 and will post the results after they finish.
• We are also happy to run experiments with additional language models for the final revision.

What is LoRRA finetuning?

• We updated the paper to have a more precise definition: We compare to RepE (Zou et al., 2023), which has two methods of control: a training-free method, which adds steering vectors to activations, and a training-based method, which updates weights to approximate adding steering vectors. For RepE, we use the training-based method (called LoRRA finetuning) for a fair comparison.
• Please let us know if you have further questions.

Reviewers RaEp, XCEP, and eoci were interested in how we determined which layer $k$ to read activations from and which layer $\ell$ to write activations to. To robustly determine the best layers to use, we performed a hyperparameter sweep over $k, \ell \in { 0, 7, 15, 22, 30}$ with Llama-3-8B-Instruct. We train on the same dataset obtained in Section 3.

Performance of each checkpoint was calculated using the evaluation set detailed in Section 5.1. We find that our layers $k = 15$ and $\ell = 0$ attains the lowest eval loss, suggesting our original choice of layers has the best LatentQA generalization.

Intuitively, this result is sensible: we read from the middle layers because they contain the most semantically-rich representations [1] and we write to the 0th layer because we want to provide our decoder with as many steps for processing the activation as possible. Furthermore, the decoder is trained, so it learns to handle the distribution shift from layer $k = 15$ to layer $\ell = 0$.

[1] Ghandeharioun, Asma, et al. "Who's asking? User personas and the mechanics of latent misalignment." arXiv preprint arXiv:2406.12094 (2024).