Model Equality Testing: Which Model is this API Serving?

Thank you for your feedback! We refer the reviewer to our general comment, and we address your specific comments below.

Number of Unicode characters

• Thank you for the correction --- you’re completely right. We mistakenly reported the number of possible outputs of Python’s ord function, which is larger than the actual number of Unicode codepoints in use. We will correct this in the manuscript.

How do you compute the kernels if a language model generates a shorter sentence compared to the maximal length?

• We right-pad sequences (Line 181). All string kernels thus are aware of length (the padding token can be compared to other tokens).

The choice of baselines is limited to different choices of string kernels. I was a bit surprised to see Hamming distance to perform the best, and in general felt that discussion and intuition why it works well is not present.

• We thank the reviewer for this writing comment! First, we point out that in addition to string kernels, we also investigate non-MMD tests. For example, in Figure 2, we investigate both the L1 and two-sample chi-squared tests. Additionally, we refer the reviewer to Appendix C.3, where we compare the two-sample tests to a suite of goodness-of-fit baselines.
• Our intuition for why the Hamming kernel is helpful is as follows: the main challenge of the Model Equality Testing problem is the high dimensionality of the space, in particular an exponential dependency on $L$, the completion length. In comparison to the one-hot kernel, for example, the Hamming kernel gives “partial credit” to samples that are more similar, even when they are not exact matches. Other kernels that have this property, e.g. the all-substrings kernel, and something based on the Levenstein distance as you suggested, probably have similar properties. We’ll look into the latter for the final manuscript.

“eq.4, should there z and z' be different?”

• Yes, thank you! We will correct the notation to clarify that $z \neq z’$ in the first two summands.

“162: what does accurate completion means?”

• Here we are referring to prompt distributions / tasks where there’s a clear notion of “correctness” for completions, e.g. through an automated evaluation metric. An example of this is the HumanEval task. The discussion is that for these tasks, it’s possible to define the MMD to use the automated evaluation metric, in which case we’re essentially comparing the task accuracy between samples. We then go on to discuss shortcomings of this approach in lines 163-165, in addition to the issue that many language tasks don’t have clear notions of accuracy / automated evaluation metrics. We’ll clarify the language in the final manuscript.

210: is the only difference in the weights of the models, can generation algorithms also be different?

• In Section 4.1, we only consider quantization and watermarking: quantization is a weight modification, while watermarking is a generation-time algorithm. We also refer the reviewer to our response to Reviewer qBpo, where we show additional experiments that detecting different temperature settings at generation time is very easy (100% power).

326: "significantly different training data": do you use word significantly in statistical meaning here?

• We do not; we’ll correct this language in the final manuscript.

337: semantic caching, quantization: is it possible to put citations here?

• Ah, for some reason, our links in the footnote are not working. We’ll fix the LaTeX issue in the final manuscript.

Figure 4 (right): what are the colours here?

• The gray points indicate pairs where both distributions have accuracy < 10% (line 402). This is to help clarify our statement in lines 413-418: “In several cases, the MMD is high but the accuracy difference is low. These are often when both task accuracies are low: the gray points in the figure highlight pairs where both distributions have accuracy < 10%. In these cases, the MMD captures that there are multiple ways to be wrong for a task.”

We thank the reviewer for your detailed feedback! Please let us know if there are additional things you would like to discuss.

Thank you for your feedback! We appreciate your comments.

Thank you for your feedback! We refer the reviewer to our general comments, and we address reviewer-specific comments below.

One immediate problem I see is that acquiring the reference distribution on a user-defined task requires setting up the reference LM anyways, which was something that is acknowledged to be inconvenient or infeasible in many cases. That would make this method impossible in certain scenarios.

• We agree that this is a limitation. Unlike, e.g., the proof protocols suggested by the cryptography community for this problem (see Section 6), we require sample access to a reference in order to verify an API. Despite this, we still believe that our two-sample setup is substantially more feasible than other possibilities. We provide two examples for discussion:
  • The cryptographic approach using proof protocols requires ongoing cooperation from the API provider; i.e., the API provider must generate a proof for every sample, which significantly increases the time to return each prediction. In contrast, a two-sample setup requires no cooperation from the API provider; we only need to obtain samples, which is the normal mode of operation.
  • Another reasonable approach is the one-sample goodness-of-fit approach (see Appendix C.3). This would require log-probability access to the reference; in contrast, our two-sample setting only requires sample access to the reference. This opens up new applications (e.g., testing if an API has changed over time; we can cache samples from an earlier point in time and compare them to a later point in time).
• Overall, we believe the two-sample setup is quite generalizable. In our work, we highlight several such applications: comparing APIs to locally inferenced weights (Section 5); comparing APIs to each other (Appendix C.8); comparing models before-and-after fine-tuning, even without access to the model weights (Section 4.2). Specifically, even in the case where we could not locally inference a model (Llama-3.1 405B investigations, Figure 4 left), we were still able to compare API implementations to each other (Appendix Figure 26).

The hamming kernel's effectiveness across different tasks raises concerns, particularly for open-ended tasks like creative generation where diverse outputs may be desirable. The significance tests may struggle with high-variance outputs, potentially missing quality differences. Further analysis is needed on how task semantics and natural distribution affect the method's applicability. Additionally, using Wikipedia-based language modeling as the primary test task may underestimate output diversity due to the highly factual nature of the content and potential data contamination. This is partially addressed by the inclusion of HumanEval, but additional comparisons and/or analysis could strengthen the results.

• We discuss this at length in our general comment on prompt distributions.
• Re: data contamination, this is a fascinating point. Surprisingly, we find that different models (presumably all trained on Wikipedia) remain easily distinguishable on Wikipedia prompts (Figure 3 lower left, Appendix Table 8), and we find that quantization still changes the model significantly on Wikipedia prompts (Section 4.1). These results seem to suggest that despite contamination, models still differ in their distributions over completions, even if, e.g., their greedy decoding results might match memorized content. Investigating this further could be an exciting direction for future work.

In Figure 4, you compare absolute task accuracy difference with MMD, but this is slightly less informative than showing non-absolute task difference. You imply on lines 415–418 that task accuracy differences are usually lower for the non-reference models, but this isn’t necessarily obvious that a provider’s interventions are always harmful to accuracy/quality. Could you provide some information about MMD and absolute accuracy?

• We did not mean to imply that the non-fp32 models are worse; in fact, some non-reference optimizations actually perform better than fp32. Below, we provide full results for two models, showing accuracies over 100 samples ($L=250$ per sample), averaged over 20 prompts:

• The direction of performance change (improvement or degradation) depends on the specific optimizations each API makes, including whether they have explicitly optimized for HumanEval when developing their inference stack. Note that when interpreting these results, one needs to account for sampling noise.
• Our main argument in Figure 4 is that regardless of direction, one can somewhat predict the magnitude of change using the MMD. This is important in its own right; for example, it suggests that when the effect size returned by our test is large, users should do a manual examination for quality.
• For HumanEval, this two-step procedure of test-and-then-check-accuracy may seem roundabout, as we have access to an automated evaluation metric. But as emphasized in Sections 1 and 3, our method aims to test even for applications without automated evaluation. Figure 4 shows that the MMD effect size provides a signal for when users should allocate manual resources for performance checks.
• We thank the reviewer for raising this point and will include this discussion in the final manuscript.

I didn’t see any mention of two model customizations that I suspect may be popular: system prompt customization and generation-time safety interventions. It’s not clear to me how the proposed method would behave under these customizations.

• These are very interesting questions to explore, and we leave this to future work. However, we would point the reviewer to our watermarking results, which represent an inference-time safety intervention intended to be more subtle than explicit safety refusals. For example, our qualitative samples in Appendix Box 10 suggest that watermarking is somewhat hard to identify with the naked eye, whereas explicit safety refusals are quite obvious. Despite this subtlety, our results suggest that watermarking is statistically detectable.

We hope we’ve addressed your concerns. If so, would the reviewer consider increasing their score? Please let us know if there are additional topics to discuss.

While I believe there are some limitations to the methods proposed, there is significant novelty and contribution made by your work. I think a good number of my concerns have been addressed and raised my score.

Consider adding an analysis that shows that the method does not detect any changes if the two samples come from the same model.

• Please see the italicized numbers in Appendix Tables 8, 9, 10, and 12.
• In addition, we’ve printed the empirical FPRs on tables in this rebuttal.

Can you extend the analysis in Figure 3 lower left to other models? One thing I’m interested to understand is that the method is able to detect different model sizes and looking at Figure 3 on the right hand side, the models from the same family look very close in terms of distance.

• Please see Appendix Table 8. Even within the Llama family, it’s possible to detect when a larger model of the family (e.g., 70B) has been replaced with a smaller one (e.g., 8B) or vice versa. For example, when Llama-3 70B is replaced with Llama-3 8B, we can detect this with 98% power.

For instance, do I still need to provide 10 samples per prompt if I generate the samples using 0 temperature?

• Since greedy decoding is deterministic, the suggested testing problem is trivial: one only needs to check exact match between one sample from $P_\text{greedy}$ and from $Q_\text{greedy}$ to determine if $P_\text{greedy} = Q_\text{greedy}$. As we discussed in Section 1, this is equivalent to checking if the modes of the unscaled next-token distributions match.
• In contrast, we are interested in whether the overall distributions match. In theory, if the non-temperature-adjusted distributions match, then adjusting both by the same temperature parameters will also result in matching distributions.

It would be great to show the detection power using samples obtained at different temperatures.

• This is a great suggestion; we provide preliminary results below and will include other models in the final manuscript. In general, detecting incorrect temperature settings is very easy, because the distributions are quite different.

The kernel checks for token equivalence but an important piece to capture might be around semantic equivalence. Have the authors considered other kernels that can capture semantic equivalence?

• This is an interesting suggestion! As we discussed in Section 3, checking for task accuracy is a variant of this; we do some examinations about task accuracy in Figure 4 (right).
• However, a more general experiment one might consider could use an embedding model as the feature vector $\phi$. Unfortunately, this would make testing slower and more expensive for the user (requires embedding all samples), which is why we did not explore it. Compare this to the Hamming kernel, which can be done on CPU and takes less than a second to compute in wall clock time.
• Although outside the scope of this work, we think this (or in general, learning $\phi$) could be a fruitful line of future inquiry to develop stronger tests.

We hope we’ve addressed the reviewer’s concerns. Please let us know if there are additional things to discuss.

Thank you for your feedback! Based on your suggestions, we ran 3 additional experiments (including the effect of task creativity and detecting temperature mismatches) and identified an area of improvement in the writing (extended discussion about the effect of $L$). We address your comments below.

I’m not convinced of the significance of this problem. Can this problem be solved through policy if the users ask from providers to disclose such changes? The API providers can even be motivated to charge users differently based on the optimizations done to models.

• This is a great question. Please see our general comment on contributions.

“The authors claim that the proposed method works using an average of 10 samples per prompt across 20-25 prompts. Since the paper relies on empirical analysis, I would love to see more analysis backing this claim/more guidelines around these requirements….What about if my prompts are all from a niche topic versus if I’m testing across various different unrelated topics? Does one still obtain high power using 20-25 prompts?...The evaluations are done using samples from English, German, Spanish, French, and Russian Wikipedia. I’d love to see more diversity in the evaluation tasks. For instance, consider showing the results for coding tasks, which can help with the broader applicability of the proposed method.”

• Please see our general comment on prompt distributions for additional experiments.

“Figure 3 lower left. The power of detecting Llama3.1 8B from Llama3.1 8B (int8) and (watermark) are 0.07 and 0.32, respectively, which are very low. I’d love to hear the author's thoughts on this as one of the premises of the method is to be able to detect such changes in the model.”

• This is a wonderful question. First, we clarify why these numbers are lower than those in Figure 2 or the general comment on prompt distributions: as explained on Line 310, in Figure 3, we have changed the tokenization used for testing from the individual model’s tokenizer ($|\mathcal V| \leq 128256, L=50$) to a Unicode tokenization ($|\mathcal V| = 155063, L=1000$). Recall that the size of the sample space is $m|\mathcal V|^L$; increasing $L$ by 20x makes the testing problem significantly more high dimensional. As one would extrapolate from Figure 2 (middle), a larger $L$ hurts power, although the MMD-based tests are more robust to this harder setting than other baselines.
  • An additional opportunity for improvement is in Section 5, where we suffer an additional loss in power when using the composite null hypothesis setting; see Appendix Table 9 for a discussion.
• These power losses reflect why the Model Equality Testing problem is hard: language is a high-dimensional space! Our contributions are, again, to formalize this problem and evaluate a first set of baselines on this task; we think there’s room for a fruitful line of work on developing more powerful tests for this regime.
• We would like to clarify, however, that these power losses do not affect the soundness of our conclusions. In Section 4.1, our conclusion is that the Hamming kernel outperforms other kernels; in Section 4.2, we provide an “existence result” that the Hamming kernel can detect finetuning; in Section 4.3 we show that even in the reduced power setting, model swaps are easy to detect (and that the MMD provides a way to discuss distances between models); and finally, in Section 5, even in the reduced power setting, 11/31 API endpoints are different than reference weights released by Meta. We do not claim that our results have eliminated false negatives; only that they control false positives.
• Finally, for the sake of discussion, we confirm (as in Figure 2 left) that power losses can be compensated for by collecting more samples. Below we repeat the Unicode-space experiments with $N=50m$ instead of $N=10m$. If we were to use this sample size in Section 5, each audit would still cost < $5.

“I’d think $L$ is affected by the vocabulary size of the tokenizer. Did the authors study the effect of $L$ under different tokenizers?”

• Yes; see discussion above.

“Also $L$ is set to 50 and the sensitivity to $L$ is studied until $L=50$. I think 50 tokens for generations is typically on the lower end when it comes to LLM-based applications. I’d recommend studying it at least until 10 folds.”

• See discussion above; we already study up to $L=1000$ under the Unicode tokenization.
• Additionally, we conduct all HumanEval and UltraChat experiments with $L=250$ in token space.

Thank you for your feedback! We refer the reviewer to our general comment, and we address reviewer-specific comments below.

The study primarily focuses on the LLaMA series models. Although this aligns with the paper’s emphasis, I recommend that the authors verify the generalizability of MMD across more models.

• This is a great suggestion! As the reviewer notes, we focus on the Llama models because they are the primary models served by inference providers in our case study in Section 5.
  • For these, we refer the reviewer to Appendix C.1 for sample complexity results stratified by language model.
• Below, we also provide new results in the style of Section 4.1 using Phi-3 Mini 4K, OLMO 7B, and Gemma-2 9B (all instruct versions). These are tested on Wikipedia at $N = 10m = 250$.
• We observe that there is heterogeneity across models, including within the Llama family. The choice of model affects both the reference distribution $P$ and the effect of interventions like quantization ($Q$). For example, 4-bit quantization of the Llama and OLMO models is consistently noticeable. On the other hand, 8-bit quantization and watermarking have more inconsistent effects.
• On average, however, our results in Section 4.1 still support our claim that the Hamming MMD is more powerful than other kernel choices.

MMD performance declines with small sample sizes, which is likely the case in real-world scenarios. Although the paper discusses sample efficiency, additional experiments with smaller sample sizes could provide further insights into this limitation.

• We respectfully point out that our primary findings are centered on the small-sample regime. Namely, in Section 4.1, we find that at $N = 10m$, i.e., an average of 10 samples per prompt, the Hamming kernel outperforms other kernels in defining a powerful test. For context, 10 samples per prompt, for distributions over 20–25 prompts, results in an audit that costs < 1 dollar in our Section 5 experiments (for most providers, the audit actually costs < $0.02).
• Our results in Sections 4.2, 4.3, and 5 all operate in this small sample regime.
• For further discussion, we refer the reviewer to Appendix C.2, where we investigate smaller sample sizes induced by decreasing $m$ (with $N = 10m$).

The computation cost of MMD appears to be somewhat high.

• This is an insightful point, and we view computational cost as one of the main advantages of the Hamming kernel. The Hamming kernel is fast because its computation is parallelizable via broadcasting; other string kernels, including the all-substrings kernel we study, scale poorly with $N$.
• We discuss computational cost in Section 3.

We hope we’ve addressed your concerns and increased your confidence in the paper. Please let us know if there are additional things to discuss.

Reviewers qBpo and jNji both asked about the dependence of power on the prompt distribution. We agree this is an important question that deserves more discussion. We present new experiments below.

Setup

• Reviewer qBpo asked for power results on coding tasks, and Reviewer jNji asked about power results on creative tasks. The underlying question is whether the “open-endedness” of the task affects power: intuitively, one might expect that more creative tasks lead to completion distributions that are higher entropy, which might make testing harder.
• To answer this question, we evaluate power across several prompt distributions.
  • On the most “constrained” side of the spectrum, we experimented with HumanEval (code); on the most “creative” side, UltraChat (chatbot dialogues), and in the middle, language modeling with Wikipedia. Please see Appendix Boxes 3 and 4 for samples of what these prompts look like.
  • To answer Reviewer qBpo’s question about concentration of topics (“What about if my prompts are all from a niche topic versus if I’m testing across various different unrelated topics?”), we additionally experiment with tightly concentrated prompts (just Wikipedia in English, just Wikipedia in French, etc.) vs. diverse prompt distributions (Wikipedia mixed with HumanEval and UltraChat).
  • We tested power of the Hamming kernel against the local alternatives (nf4, int8, watermark) at $N=10m$ with tests conducted in token space (50-token completions). Standard errors are reported following the procedure in Section 4.1.
  • For comparison, we also provide the results of the one-hot kernel. This is to analyze whether our claims about the relative ordering of which kernels are stronger in the $N=10m$ regime are correct.

Results

In the subsequent comment, the top table gives results for the Hamming kernel, and the bottom for the one-hot kernel. We make these observations:

• Reviewer jNji hypothesized that more creative tasks might be harder to detect: in this case, we should see UltraChat < Wikipedia < HumanEval across all alternatives in power. Instead, the creativity level of tasks doesn't clearly correlate with detection difficulty. int8 quantization is hard to detect across several distributions, while nf4 quantization is only hard to detect on English Wikipedia.
• Reviewer qBpo wondered if tightly clustered topics are harder to detect than mixtures of topics. Instead, topic clustering (single-language vs mixed content) shows no consistent pattern in detection difficulty.
• The prompt distribution mainly affects the difficulty of detecting quantization, not watermarking.
• The Hamming kernel consistently outperforms the one-hot baseline on all prompt distributions, matching our main conclusion of Section 4.1 about the relative strength of kernels.
• As a reminder, power can always be increased by increasing the sample size. For example, collecting $N=50m$ samples instead of $10m$ would still cost < $5 per audit.

Summary: While the prompt distribution $\pi$ does affect power, the effect is specific to the $(P, Q)$ pair and inconsistent. In general, our recommendation of the Hamming MMD as the strongest baseline holds.