Independence Tests for Language Models

We thank the reviewer for their feedback.

W1/W3 Experiments mainly use decoder-only architectures, all around 7B parameters, lacking variation in model types and sizes.

We agree with the reviewer’s concerns. We have since evaluated our different tests on various architectures such as Mistral, BERT, StripedHyena, and Llama3. We have also added experiments on models of different parameter sizes such as Llama 3.2 1B and Llama3.2 3B. We emphasize we are able to use our robust statistic even on two models with different architecture or dimensions.

Q1 Whether the proposed framework remains effective for encoder-decoder architectures (e.g., BERT) or encoder-only models?

Yes, the p-values from our statistics \\phi_\\text{CSU} and \\phi_\\text{CSH} are valid by construction for any models trained using a $\\Pi$-invariant learning algorithm, such as SGD or Adam, regardless of architecture (encoder or decoder, etc.). Empirically, our tests seem to achieve very high recall for BERT models. In particular, during the rebuttal period we tried running \\phi_\\text{CSU} on the based model google-bert/bert-base-uncased and the fine-tune thereof dima806/tweets-gender-classifier-distilbert, obtaining a p-value < 1e-308.

Q2 How sensitive is the framework to larger models beyond the 7B parameter range, or to smaller-scale models such as Phi-3?

Our tests work for different architectures and large parameter models as well. For example, we run \\phi_\\text{CSU} on 70B parameter models of the Llama architecture and verify its effectiveness. In the table below, the test correctly identifies Llama2-70B and Miqu-1-70B as fine-tunes, but not Llama2-70B and Llama3.1-70B.

Continuing with the non-robust tests, we also evaluate on Mistral models, and find, for example, layers from StripedHyena-Nous-7B that are not independent of layers from Mistral-7B-v0.1, in the table below

The tests also work on smaller-scale models such as Phi-3. Both \\phi_\\text{CSH} and \\phi_\\text{MATCH} return a statistic < 1e-308 (aggregated with Fisher’s method) on the fine-tuned model pair microsoft/Phi-3.5-mini-instruct and numind/NuExtract-v1.5 (3.8B parameters).

Furthermore, the robust test \\phi_\\text{MATCH} can even be used on model pairs of differing architectures. We run the robust MLP matching statistic \\phi_\\text{match} on (all pairs of blocks of) $\\theta_{8B} =$ Llama 3.1-8B and $\\theta_{3B} =$ Llama 3.1-3B, which have very different model dimensions — the number of layers, embedding dimension, and MLP dimension are all different. (This is possible because the LAP element of \\phi_\\text{match} can match a permutation from hidden activation matrices of different shapes — and returns one of the smaller dimension.) We find that the robust statistic returns a value < 1e-4 and can identify the specific Transformer blocks of Llama 3.1-8B very likely used to initialize Llama 3.1-3B through pruning.

The flexibility of \\phi_\\text{match} for varying size, and running \\phi_\\text{MATCH} on all pairs of MLP blocks, is significant because it prevents adversaries that may take only certain layers of a pre-trained model and inject other layers (i.e. through pruning).

We thank the reviewer for their comments and questions!

W1/Q1: Submission does not evaluate methods against "most relevant prior work" (Zeng et al., 2024)

Thank you for the suggestion. We evaluated the three statistics $M_a$, $M_b$, and $M_c$ proposed by Zeng et al., 2024. In the non-robust setting for non-independent pairs, where the statistics (measure of cosine similarity) are comparable to our findings.

However, we can transform the weights of each model (without changing the output, as we describe in Appendix D), in a way that completely breaks the effectiveness of all of their tests (and our non-robust tests), while our robust test based on \\phi_\\text{MATCH} still yields an astronomically small value.

W2/Q2 Submission does not describe cost of algorithms.

Thank you for the suggestion, we will revise our paper. When running the original permutation test, such as using the L2 statistic and T=99, we must permute the model 99 times and compute the L2 distance between the permuted weights and the other model’s weights each time. On a GPU (Nvidia RTX A6000), this takes around 5 minutes. The runtime of this test scales linearly with T—thus, the cost quickly becomes prohibitive if we want to obtain a small p-value (e.g., in order to obtain a p-value < 1e-10 we would need T > 1e10). This trade-off between cost versus power motivates scipy.stats to estimate p-values (see below) when applicable (i.e., for \\phi_\\text{CSU} and \\phi_\\text{CSH}) instead of running PERMTEST.

We apologize for the confusing usage of “lookup table”, and will change our writing. Our implementation uses the Spearman’s rank correlation statistic using scipy.stats, which indicates the probability of an uncorrelated system producing datasets that have a Spearman correlation at least as extreme as the one computed from these datasets. Note that this p-value is still valid (up to numerical precision issues) under our null hypothesis.

W3 More clarity could be provided related to some of the simplifying steps when computing p-values. Specifically, in section 3.2.2 when stating “instead of running PERMTEST itself, in our experiments we convert the output to an approximate p-value using standard lookup tables for Spearman correlation coefficient”

The reason we can avoid running PERMTEST for the Spearman correlation-based test statistics (e.g., \\phi_\\text{CSW} and \\phi_\\text{CSH}) is because the distributions of these test statistics are the same for any two independent models (contrast this with $\\ell_2$ distance, whose distribution will vary for different pairs of independent models depending on factors such as the amount of weight decay employed during training, etc.). Thus, because the null distribution of the Spearman correlation coefficient is known, we do not have to run PERMTEST to keep simulating these null distributions, and we can simply convert the Spearman correlation coefficient output by both \\phi_\\text{CSU} and \\phi_\\text{CSH} to a p-value using the scipy.stat library.

The fact that we do not have to run PERMTEST means we are not computationally bottlenecked like in the case of $\\ell_2$ distance; in particular, in the case of $\\ell_2$ distance we must permute each model more T times to obtain a p-value less than 1/{T+1}. Thus, we are able to report low p-values that more accurately reflect the true statistical power of our tests. We expect in principle that we could obtain similarly low p-values using $\\ell_2$ distance if we were able to run PERMTEST with T > 1/$\\varepsilon$; of course, doing so is computationally infeasible.

We thank the reviewer for their feedback and questions.

W1 For example, in the introduction, in line 60, the authors state "we do not distinguish whether one is a fine-tune of the other or if the two models share a common ancestor"

Thanks for bringing this to our attention, we apologize for the lack of clarity in our writing and will revise the paper. Indeed, the first line should say: "We consider the following problem of model provenance: can a third party verify whether two language models are trained independently or not?”.

W1 "Independence" is too loosely defined throughout the paper, and sometimes causes confusion.

We define independence formally in Section 3.1. We will update the revision to more explicitly refer to this formal definition at other points in the paper (e.g., the Introduction).

W1 First, because of the problem stated in 1 above; If I get a p-value form the proposed method >0.05, what do I know about the two models I tested?

A p-value of 0.05 means that if the two models were independently trained, then the probability of obtaining a p-value at most this large is equal to 0.05, over the randomness of the training algorithms and initializations used to produce both models (so long as one model satisfies our assumption of being the product of a permutation equivariant learning algorithm initialized from a permutation invariant distribution). In general, low p-values provide quantitative evidence that two models are not in fact independent as defined in Section 3.1

W1 Some sources of co-dependence may be problematic, while others may not, so not knowing where the dependence comes from makes the result difficult to use in practical situations. Secondly, the above leads to think that the method can distinguish if two models are somewhat fine-tuned from each other OR from the same model. Was one fine-tuned initialising from the other? Did they just have identical training procedure with the same data? Was one fine-tuned with the output of the other in a teacher-student manner?

We thank the reviewer for their comments and once again emphasize that we will improve the clarity of our writing. Our tests cannot distinguish whether two models are fine-tuned from each other versus from the same model. On the other hand, even if two models are trained with identical procedures on the exact same data, so long as they are trained independently from independent random initializations our tests will correctly identify them as independent. In the case where one model was fine-tuned using the outputs of another model via teacher-student distillation, our tests will incorrectly identify the models as independent. Crucially, this does not mean our tests are invalid: the definition of validity is that the p-values from our tests are uniformly distributed between 0 and 1 under the null hypothesis, which is still the case. Rather, it means there are cases of dependent models which we will not be able to reliably distinguish. Another example of such a case would be if someone independently trained a model but then copied over a single parameter from another model at the end of training; technically the two models would not be independent of each other, yet none of our tests would be able to determine this correctly.

We state in the paper that we only give a p-value for the hypothesis test of whether two model weights are independent if the test is rejected, we do not provide any causal information (one model is fine-tuned from the other), although we can provide more granular information on which layers of the model are dependent on which other layers. Because the p-values we obtain for dependent model pairs are all astronomically low, essentially any threshold will obtain perfect recall (e.g. a p-value threshold of 1e-10 will result in zero false negative errors—at least based on our experiments— and a false negative error rate of exactly 1e-10—due to the guarantees of our method). We will clarify this point in the revision.

Our work is focused on inferring independence purely from parameters. And, we do not treat data as a random variable, so we infer nothing about if the same data was used (which would be especially tricky as most language models likely overlap in some training data from the web). Hence, we only test for independence as defined above, when one model uses another’s weights.

what do the authors exactly mean by $\perp$ in this context; ~0 mutual information perhaps?

The notation $\perp$ denotes independence of random variables: $X \perp Y$ means that $P(X = x | Y = y) = P(X = x)$ for all $x,y$. Two random variables have zero mutual information if and only if they are independent, so your interpretation of $\perp$ as implying zero mutual information is correct. See [1] for further clarification regarding the $\perp$ notation.

whatever is the definition of $\perp$, from the explanations above, the authors suggest that if $\theta_1^0 \notperp \theta_2^0$ (not independent inizialization) then $A_1(\theta_1^0) \notperp A_2(\theta_2^0)$ (not independent final weights)

There are cases where this implication is not true. For example, both $A_1$ and $A_2$ could ignore their inputs and output a weight vector with i.i.d. Gaussian elements. In this case, we would have $A_1(\theta_1^0) \perp A_2(\theta_2^0)$ regardless of the (joint) distribution of $\theta_1^0$ and $\theta_2^0$. On the other hand, for most learning algorithms people use in practice, we generally expect the implication to be true; we say as much in the paper (Line 137-38) mainly to provide the reader with helpful intuition, but we are happy to remove this language if it is misleading.

$A_1 \notperp A_2$, or even $A_1 = A_2$ (not independent or equal training procedure), then $A_1(\theta_1^0) \perp A_2(\theta_2^0)$, (independent final weights) with no further justification.

In our notation, both $A_1$ and $A_2$ are functions that map an input parameter to an output distribution over parameters. Crucially, we do not consider $A_1$ and $A_2$ random variables, so statements of the form $A_1 \notperp A_2$ are invalid (in our notation) since $\perp$ and $\notperp$ apply only to random variables. Because $A_1$ and $A_2$ are functions, by the post-processing inequality their outputs must be independent if their inputs are independent (i.e., mutual information can only decrease, so if the inputs have zero mutual information then so must the outputs, which in turn implies independence of the outputs). Independence of the output distributions of $A_1$ and $A_2$ immediately implies independence of samples from these distributions.

[1] https://en.wikipedia.org/wiki/Independence_(probability_theory)

The authors may be misunderstanding the post-processing inequality in their setting, which leads to the significant assumption not currently stated or discussed I have concerns with.

The post-processing inequality states that the mutual information through a process between the variable before and after the process can only decrease. In the paper's setting this means that $I(\theta_1^0, \theta_2^0) \geq I(\theta_1^0, A_2(\theta^0_2))$ and $I(\theta_1^0, \theta_2^0) \geq I(A_1(\theta_1^0), \theta_2^0)$. This does not imply that $I(\theta_1^0, \theta_2^0) \geq I(A_1(\theta^0_1), A_2(\theta^0_2))$. In fact, we can make a simple edge case to disproof the latter inequality: Consider the edge case of $A_1()$ and $A_2()$ being both functions that, regardless of the input, return the same set of weights $\theta_1$. Even if the inputs $\theta_1^0 \perp \theta_2^0$, the outputs $A_1(\theta^0_1) = A_2(\theta^0_2) = \theta_1$ are the same, hence the mutual information between them has increased, going from 0 to being maximal.

The unstated assumption in the paper is that the independence of the final weights $\theta_1 \perp \theta_2$ depends only on the independence of their initialisation $\theta_1^0 \perp \theta_2^0$ and not on any other training factor, such as training data, learning parameters, etc., which the authors embed in $A_1$ and $A_2$. This is a very strong statement and, as explained above, it is not a consequence of the post-processing inequality. This would need to be proven in the theory, or strongly demonstrated experimentally to be accepted as an assumption. In the current paper, it is not stated, but implicitly assumed.

In practice, the validity of this assumption is crucial for the proposed test to be taken as viable proof of relationship of initialisation (common initialisation or parents of one-another). I.e., if I get a high p-value, can I conclude that the initialisation of the two models is related, or is this caused by, e.g., the same fine-tuning data. Vice-versa, if I get a low p-value, can I conclude definitively that the initialisations were independent? Or could it be that they still had the same initialisation, but were trained on different data?

We thank the reviewer for the feedback!

W1: The notation of matrices in Algorithm 2 could be improved, as it currently overlaps with the algorithm notation.

Yes, we will change $A_1$ and $A_2$ in Algorithm 2 to be $W_1$ and $W_2$ instead so there is no overlap. Thank you for pointing that out.

W2: It would be helpful to include an evaluation of the L2, CSU, and CSH tests under adversary, the MLP retraining. Yes, we will report those statistics under MLP retraining as well.:

For the table, we retrain the first MLP block according to the setup in 4.3.1, and evaluate each statistic only on the layers of the first MLP. We also find that \\phi_\\text{MATCH} is robust to retraining.

Q1: The models examined are standard, trained or fine-tuned dense LLMs. How might the proposed test perform with other architectures, such as mixture-of-experts models?

Thank you for the question on mixture-of-experts models. We wish to clarify that our model is just as valid to apply to mixture of experts models and that our non-robust tests are valid by construction regardless of model architecture. For the activation tests (including \\phi_\\text{MATCH}), we can choose to only compare the hidden activation matrices for one expert MLP, for example.

Regarding the potential problem of sparse models, we find that there is still signal from activation and weight matching. For example, comparing the sparse (89.32% sparsity) SparseLLM/prosparse-llama-2-7b with Llama 2 7B using \\phi_\\text{CSH} gives a p-value less than $\\varepsilon$ = 1e-308 (for all 32 blocks). And the robust \\phi_\\text{MATCH} gives a p-value less than 1e-05 for at least the first 20 MLPs.