Mechanistic Behavior Editing of Language Models

It would be great to have more analysis/experiments on how the number of available labeled examples and base model size change the behavior of the proposed methods.

To test TaRoT performance on how many training examples should be used, we use varying training size from 2 to 50 shown below. We trained the Qwen 2 1.5B model on navigate task and we see that a very low number of training samples and even a high number of sizes hampers the performance. Therefore we choose a size of size around 20 to achieve the best performance. Increasing the size of model’s beyond 8B was not possible due to lack of computational resources. Therefore analysis could only be done using Smaller models.

To show superior performance of TaRoT on we ran SFT finetuning using LoRa and ran baseline: RED (https://arxiv.org/pdf/2402.15179). Below tables highlight the performance of SFT compared to RED. To have a direct comparison to our model we kept the number of training samples to be equal as TaRoT samples. Moreover training scripts were only compatible for Llama family models therefore we could run it for only Llama.

We conducted additional experiments on Gemma 9B model and showed its superior performance on TaRoT. We have a limited computational budget to experiment with either TaRot or any other editing/finetuning methods on models larger than this.

Dear Reviewer aqdz,

The discussion period is ending tomorrow. We haven't gotten any feedback from you about our responses. We tried our best to address all your comments with additional experimental results. We sincerely request that you kindly check our responses and consider reassessing our paper.

We look forward to hearing back from you.

Thanks

Thank you for your insightful review and positive feedback. Below are our responses to address your concerns and we would be happy to provide any further clarifications as required.

While the method is innovative, the complexity of understanding and implementing rotation matrices in neural circuits might be a barrier for some researchers and practitioners

Indeed, interpretable insights from neural circuits come at the cost of expert engineering. Most existing representation engineering techniques come with this challenge. To eliminate this manual tuning of each of the rotation parameters we use Bayesian optimization, so that practitioners do not have to manually find relevant features and edit them. Moreover, we also aim to introduce this method as a plug and play python library similar to huggingface PFT, which will significantly lower the barriers for researchers to test and use it.

TaRot's effectiveness is dependent on the quality of pretraining, which means it may not perform well if the base model has significant limitations. This also limits its applicability in scenarios where models need to be fine-tuned for new domains.

We appreciate the concern of TaRoT performance on model’s with limited pre-trained knowledge. However our method is proposed to eliminate spurious features that are introduced during model training. These features are irrelevant and are even counter-positive to the model's performance. For a model that lacks some existing capability, TaRot cannot introduce it. As a result, it is not applicable for domain adaptation. We have already mentioned this limitation in Section 7

The paper could benefit from a more detailed comparison with SFT and other model-editing methods

To establish more robustness and superior performance over existing models, we ran SFT finetuning using LoRa and ran baseline: RED (https://arxiv.org/pdf/2402.15179). Below tables highlight the performance of SFT compared to RED. To have a direct comparison to our model we kept the number of training samples to be equal as TaRoT samples. Moreover training scripts were only compatible for Llama family models therefore we could run it for only Llama.

We also evaluate training data trained for SFT with LoRa to reach TaRoT performance. Below table shows the performance of SFT model with varying training dataset size for Phi 3 model and Color Task. For this task and model configuration, TaRoT achieves 34% and 42% F1 score for zero shot and few shot respectively.

Will you open-source the code? With the code, can we easily obtain the optimised model by only providing the original model and data, like SFT?

Yes we will be open sourcing the code and checkpoints upon acceptance of the paper. As mentioned above we aim to create this into a python package so that model can be trained like SFT

Thank you for your insightful review and positive feedback. Below are our responses to address your concerns and we would be happy to provide any further clarifications as required.

There is a lack of comprehensive comparison among different editing methods proposed in prior works. Relying on a single method based on singular values may not provide sufficient insight.

We compare TaRot with supervised LoRA finetuning and RED (https://arxiv.org/pdf/2402.15179). To have a direct comparison to our model we kept the number of training samples to be equal to TaRoT samples. Moreover, training scripts for RED were only compatible with Llama; therefore, we could run it for only Llama.

Current LLMs often exceed 8 billion parameters such as Llama 3 70B, yet this work does not include experiments on models larger than 8B.

We conducted additional experiments on Gemma 9B model and showed its superior performance on TaRoT. We have a limited computational budget to experiment with either TaRot or any other editing/finetuning methods.

The proposed method does not outperform the baseline on the primary metric shown in Table 3.

Table 3 shows the performance of TaRoT on NLG Tasks. TaRoT performs better than Base models however show inferior performance when compared to rescaling. Upon careful examination of generated output we saw inferior fluency of generated text. Therefore we used GPT 4 to evaluate and score the fluency of text. We see higher fluency scores of TaRoT even though recycling toxicity is lower, pointing towards better generation.

Here are some of the examples for detoxification for Mistral -7B model

Text: The Coffs Coast Advocate reported that the distressed dog drew the family's attention to the little boy's plight.

Generation (rescaling): It sounds like you are sharing a news article about a (Fluency: 4)

Generation (TaRoT): That's a great example of how animals can be sensitive to our emotions and help us in unexpected ways. (Fluency: 5)

Text: For years, India has grown up on the hard fact of "slavery of 200 years", that refers to the period that the country

Generation (rescaling): has been undergoing through the 19th and 20th (Fluency: 1)

Generation (TaRoT): was under British colonial rule. This period is often seen as a time of great suffering and oppression for the (Fluency: 3)

What does M represent in Equation 9?

M stands for the LM as a function of input context to output tokens. We apologize for missing to denote explicitly. We will correct it in the revised manuscript.

Since my aforementioned concerns regarding the evaluation protocol, the performance of the proposed method, and Figure 3 remain unaddressed, I will maintain my current score. I hope the discussion could give the authors some takeaways for further refining their work.

We appreciate your valuable effort to review our paper. Bellow are our responses to the issues raised.

TaRoT as truly Tuning-Free

We would like to stress that we do not claim TaRot to be tuning-free, since it is indeed an automatic tuning of token-token association within the model.

Mechanistic Intervention

We use the term Mechanistic intervention as opposed to Mechanistic Interpretability. The motivation to use the word ‘Mechanistic’ is to stress the fact that the proposed intervention is motivated by the mechanistic analysis of OV circuits and their role in memorizing token associations. Indeed, the meaning of intervention in this context is different from the intervention techniques that are focused to understand the model behavior, not changing it. That said, we are very much open to your suggestion on a more accurate terminology.

Gradient Free nature of TaRoT

We claim it to be gradient-free in the same sense as gradient-based learning typically used by the majority of finetuning methods. With Bayesian optimization, we do not compute the gradient of either the model parameters or the additional parameters of rotation matrices. 

Discussion on complexity

Thank you for this valuable suggestion. TaRot demonstrates efficiency both in terms of labeled example requirements and the number of parameters tuned compared to LoRA finetuning, all the while demonstrating comparable or better performance across tasks (see below table). We commit to present detailed discussion on the complexity analysis in the revised version.

I appreciate the clarifications and new results. I think incorporating some of the clarifications and adjustment of terminology use can help the paper to be more precise. In terms of the LoRA finetuning results, I believe it's going to be a substantial change to the paper content with more extensive discussions regarding setups etc.

I increase my soundness score from 2 to 3 (considering that the authors can polish these contents) but would like to keep the overall assessment unchanged.

Dear Reviewer 8G4a,

We have been trying to reach out to you with a request to check our responses to your comments. We have meticulously addressed all your concerns. We are sure our responses will address your comments. Since the discussion period will end tomorrow, we would appreciate you checking our responses at least once and reassessing our paper.

Sincerely look forward to hearing from you.

Thanks