TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMs

We thank the reviewer for their thoughtful comments and constructive feedback. We would like to clarify some points:

Appendix B compares OutputCo and TuCo. But which one is better, and why?

They have different interpretations, and are most appropriate for answering different research questions.

OutputCo tells us how large is the distance between pre-trained and fine-tuned models final hidden states. This could be used in a similar way to e.g. computing distribution distances between the final logprobs of each model.

Meanwhile, TuCo tells us how large is aggregate change in intermediate layer outputs due to fine-tuning. In this sense, it gives a quantitative view of how the computation performed by the model is affected, rather than only the final outcome. We are not aware of comparable metrics in the literature.

I find the experiments in the current version cannot support the claims well. The dataset used in finetuning is very important in measuring the model’s behavioral change. So, ablation studies varying the finetuning data will make the conclusion more solid.

We would like to ask the reviewer if they could point to specific claims they consider are not well-justified, so that we can make the appropriate improvements to the manuscript.

Further, we clarify that, in our experiments, we seek to demonstrate the applicability of TuCo in the wild, i.e. on real-world widely-used open weight models, without relying on bespoke toy datasets.

Rather, in Section 5.1, we make controlled interventions by varying the magnitude of the fine-tuning component $FTC$. We demonstrate this can be used to control model behavior, and even improve its performance on certain MMLU tasks. This validates the relevance of measuring the magnitude of $FTC$ when studying the interactions between prompt content, model behavior and capabilities.

[...] why not directly observe $||x_{FT}-x_{PT}||$

As we argue in Appendix A, an effective metric should be interpretable for practicioners, useful for empirical analysis, and practical to compute.

The fact that TuCo is normalized (i.e. between 0 and 1) allows it to be more intuitively interpreted as a fraction (i.e. "30% contribution of fine-tuning"). An unnormalized metric, such as $||x_{FT}-x_{PT}||$, is potentially subject to significant changes in scale across models and prompts, harming its interpretability and usefulness.

Further, per Section 5.5 and the prior answer, TuCo is qualitatively distinct from simply comparing final hidden states, even if one uses a normalized metric (i.e. OutputCo).

We use TuCo to quantitatively show jailbreaks attenuate the effect of fine-tuning (Section 5.3), that the attenuation is strongest the stronger the jailbreak (Section 5.3, MSJ results), and that successful jailbreaks show stronger attenuation (Section 5.4).

I am not quite sure about how to understand $f_\theta(x, l)$.

As pointed out in Section 3, most commonly-used GPT architectures have residual connections around self-attention and MLP layers. This means that, on a layer $l$ computing a function $f_{\theta_l}$ (where $\theta_l$ are the parameters of the layer), the residual stream is updated as $x_{out} \leftarrow x_{in} + f_{\theta_l}(x_{in})$.

Since there are $L$ layers, we have functions $f_{\theta_1}, \cdots, f_{\theta_L}$. For notational simplicity, instead of writing the function computed by the $l^{th}$ layer as $x \mapsto f_{\theta_l}(x)$, we write it as $x \mapsto f_\theta(x, l)$.

What does the map $(x_1, \cdots, x_n) \mapsto x_n$ look like? Directly select the last column of FTC?

Yes, that is correct: this map picks out only the hidden state for the final token of the prompt.

[...] But on what dataset are these models trained on?

We thank the reviewer for pointing this out as an area of improvement.

Llama 2, Llama 3 and Gemma use a combination of publicly, private and synthetic instruction tuning and preference data, including conversational data and safety data. Mistral and Vicuna are only fine-tuned for instruction following. Zephyr-Gemma is fine-tuned on synthetic chat and preference data. The preference ratings take into honesty into account, but, per Tunstall et al. (2024), the samples are focused on helpfulness rather than harmlessness.

We have added a more detailed overview to the appendix.

if the model is finetuned on web-text data rather than chat-data, will the claim still hold?

In this case, we would expect not to see such a clear separation.

Conclusion

In the above, we hope to have addressed the reviewer's mentioned concerns, with particular regard to providing more details on model data mixes, and on what TuCo contributes over a simple comparison of final hidden states.

Given the above, we would like to ask the reviewer to consider increasing their score. If any concerns remain, we are happy to provide further clarifications and improvements to the manuscript.

We would like to address the queries raised. Some claims dismissing our experiments are incorrect and unjustified.

No direct evidence [...] FTC approximates finetuning

FTC is exactly the difference in layer outputs between the finetuned and pretrained models; if FTC is zero then FT=PT. Therefore it is a rigorous and universal notion of "effect of fine-tuning" for a given prompt.

difference in model weights [instead]?

Comparing model weights would be agnostic to the prompt, which is not our problem setting (Secton 4.1).

TuCo quantifies how much a prompt's $FTC$ contributes to the final hidden state, as an interpretable fraction (0 to 100%). This seems natural.

[In MWE] the 'answer matching behaviour' is always 'Yes'.

This is not true.. The matching answers are balanced, (50% "Yes" and 50% "No"). This is easily seen in https://github.com/anthropics/evals/blob/main/persona/subscribes-to-Christianity.jsonl.

[MWE has] data quality issues identified in [...]

This is an appendix of a blog post, which cannot be taken at face value. Moreover, it does not claim any issues with the Persona section of the MWE dataset, the only one we use in our evaluations. This source is both unreliable and inapplicable.

do not report variance large difference in the magnitude of the steering

We added variance to the plots. But this is redundant: there is no "difference in magnitude" that can skew the mean estimator, since we are averaging booleans (constant magnitude).

unclear what these 'generalized components' are and how they work generalization of earlier work on circuit analysis

For a circuit computing $g: (x_l, l) \mapsto g(x_l, l)$ at layer $l$, when the input hidden state is $x_l$, $g$ is a generalized component (Def. 4.1). Thus, Def. 4.1 applies to circuits in Elhage et al. (2021). We updated the paper to explicitly point this out.

We add that this seemed clear to other readers.

transformer [as] linear sum of circuits?

We do not claim full circuit decompositions exist or are known. We only make this assumption in the thought experiment in Section 4.2, in light of the great diversity of circuits identified in prior work.

only uses model activations, [not] circuit components

This is an important strength of our method: exact circuit decompositions need not exist or be known, but TuCo can nonetheless be computed for any model, because it assumes access to only intermediate model activations and pre-trained and fine-tuned models.

[discuss] model stitching and sparse crosscoder

We appreciate the suggestion, and updated our related work section. But the connection is indirect. These methods do not yield scalar-valued metrics, so their "type signatures" are different.

demonstrate signs of life [...] but do not compare to [...] CAA

We politely ask for a reference to this CAA. We also believe "signs of life" unnecessarily diminishes our results.

flawed evaluation methods [MWE]

As mentioned above, the reviewer's discrediting of MWE is unjustified and based on false claims. We kindly ask the reviewer to either clarify the perceived methodological flaws, or to remove the claim.

[no] technique for preventing the jailbreak

Per Section 5.4 and Appendix F.4, applying a threshold to TuCo detects jailbreaks, and model outputs can then be halted. Note TuCo is an analysis technique not designed to detect jailbreaks, and yet has out-of-the-box predictive power.

[no] insight as to why specific jailbreaks work

As pointed out in line 371, our results in Section 5.4 indicate that the attenuation of the contribution of fine-tuning to the model's final hidden state (which is what TuCo direcly measures) is associated with jailbreak success.

not yet led to interesting insights

Our work yields novel scientific insights on the interplay between LLM jailbreaking and fine-tuning, which are both of widespread interest in the community. In this sense, we consider TuCo to have produced interesting insights.

For example, we quantitativey identify a clear link between jailbreaking and the attenuation of the effects of fine-tuning, which had been merely hypothesized in prior work (Kotha et al. 2023, Wei et al. 2024).

claim "Model developers can use TuCo [...] and adjust accordingly"

This claim is made in the Conclusion and Future Work section -- we leave to future work the application of TuCo to improving fine-tuning methodology and dataset construction.

Conclusion

We hope to have addressed points raised by the reviewer, and pointed out incorrect statements dismissing our experimental results.

In light of these clarifications and corrections, which address the basis for the negative review, we would like to ask the reviewer to consider increasing their score.

We sincerely thank the reviewer for recognizing the original contributions of our work, the comprehensiveness and soundness of our experiments, and the quality of our technical exposition.

In the following, we address the reviewer's points regarding the allocation of space to background and experiments in the manuscript.

The paper spends too much time focussing on "setting the scene" and providing background information as well as deriving TuCo. I believe that it would benefit from moving parts of this into the appendix and instead increase its focus on empirical evaluations in the main manuscript

We thank the reviewer for the suggestions on improving the focus of our paper. We agree that we would like to move some of the figures/tables from the appendix to the main paper, and that, for many readers, an extensive description of transformers is not required. Because the later sections depend on equations introduced in the background, it serves also to introduce necessary notation. However, we will strive to reduce it while keeping essential notation, to make space for some of the appendix's content.

We thank the reviewer for their recognition of our extensive experimental suites and the relevance of our method to interpretability, as well as their thoughtful suggestions on areas of improvement. We would like to address some of the points raised:

(1) Lipschitzness of the layers may be strong in practice in the theoretical bound

In Appendix D.5, we rigorously justify our assumption of Lipschitzness for the commonly-used transformer layer with root-mean-square normalization applied before attention and MLP layers.

Intuitively, normalization ensures the input of attention and MLP layers is always of bounded norm, and such layers are locally Lipschitz (or, for MLP layers, globally Lipschitz). Further, the fact that a numerical $\epsilon$ is used during normalization (i.e. one normalizes $x \mapsto \frac{x}{\sqrt{||x||^2 + \epsilon}}$) ensures the normalization map itself is Lipschitz. Hence, the resulting layers are Lipschitz. The boundedness of PTC hence also follows.

(2) The paper only tests a nice spread of open-source models (various LLaMA 2 and 3 sizes, Vicuna, Mistral, Gemma, etc.) but still only up to 13B parameters, but it is not yet shown how TuCo behaves on 30B, 70B, or even larger-scale systems.

The computation of TuCo requires access to model parameters and a modified forward pass. As such, we would need to host a model ourselves to run our experiments on it. Given GPU and budget constraints, we were unable to evaluate TuCo on models of larger scale. Instead, we sought to evaluate a large suite of models up to 13B parameters to demonstrate the general applicability of our method.

(3) Other model architeictures also need to taken into consideration, like MoE structure.

TuCo is agnostic to the specific architecture of model layers, and applies without modification e.g. to MoE architectures. We will implement and evaluate TuCo for JetMoE-8B (https://huggingface.co/jetmoe), a recent MoE model of tractable size for which pre-trained and fine-tuned checkpoints are freely available. This includes modifying the HuggingFace implementation of the forward pass to support TuCo computation and running our experimental suite, which we were unable to complete in the short rebuttal period. We will include results in a camera-ready version if this work is accepted.

(1) Given the TuCo is amodel-dependent metric (e.g., depending on the model architecture). In real-world practice, how do you suggest the practitioners to use the metric?

We clarify that TuCo places very light assumptions on model architecture (i.e. only that the intermediate hidden states are updated as $x_{l+1} = x_{l} + f_\theta(x_l, l)$). In particular, TuCo does not depend on the use of any particular kind of layer (e.g. self-attention).

As mentioned in the conclusion (Section 9), we suggest practitioners use TuCo to detect inputs where fine-tuning is less effective, allowing them to adjust their datasets and mitigate potential vulnerabilities. This approach not only aids interpretability research by identifying prompts that attenuate finetuning effects but also lays the groundwork for integrating adversarial attack prevention in user-facing applications.

Conclusion

In the above, we hope to have addressed the reviewer's concerns regarding justifications of our theoretical assumptions, the scale and architectures of models considered, and downstream applications for TuCo.

We would like to ask if the reviewer would consider increasing their score in case their points have been addressed. Otherwise, we are happy to provide further clarification.