Embedding-based statistical inference on generative models

nufo,

Thank you for taking the time to read and review our paper. We have addressed your part of your review below.

$**Summary**$ :

We think your summary is a fair description of our work.

$**Strengths**$:

We agree with your noted strengths of the paper.

$**Weaknesses**$:

The paper claims ... The experiments are narrow ...

We agree that our work is an extension of existing embedding-based analysis. We disagree that it is only a modest extension. To go from analyzing the embeddings of a collection of documents to analyzing the embeddings of generative models requires 1) an aggregation over responses, 2) an aggregation over queries, and 3) a suitable metric on the aggregation over queries. There are a number of options for each of these three steps -- which opens the door for theoretical and empirical explorations of numerous instances of similar methods -- that suggests, to us, that the extension is not trivial.

Our goal with the experiments was to demonstrate the general application of the proposed method. For example, we consider one classification task and three regression tasks and consider three different collections of models across the four tasks -- one collection parameterized by different fixed augmentations, one collection parameterized by different LoRA matrices, and one collection of “off-the-shelf” but related models. On one hand, this demonstrates broad applicability. On the other hand, our analysis was limited to 7B models and quantized versions of 7B models due to very real computational and budget constraints. While we expect our results to hold for collections of larger models, we agree that including such an example would improve our paper.

$**Questions**$:

How does your approach scale ...

Our method is quite general and can be applied to larger models or model collections. With that said, one of the components of our method is querying each model multiple times. We expect that this is unavoidable in the black-box setting that we address -- though we are actively working on ways to get high-quality representations of the models in as few queries as possible. Some promising directions that we are working on include selecting a core-set of queries, developing an “optimal” metric on the aggregation of queries, and ensembling across embedding functions.

The tasks chosen ...

Outside the context of benchmarking, statistical analysis of populations of models is relatively new. As such, there are relatively few standard tasks. We took this as an opportunity to explore what we thought were interesting and relevant tasks.

Given that the treatment of our setting as a classical supervised learning problem is relatively new, the choice of using the “Was RA Fisher great?” as our leading example was mainly due to its simplicity. The covariate y = P[“yes” | augmentation] is easy to understand. Further, the model-level covariate surface should correlate strongly along the dimension in the perspective space that captures difference in the eugenics and statistics augmentations.

We disagree that “detecting [if a model has / had access to] sensitive information” task is narrow and not reflective of more complex model inference challenges. See, e.g., this popular press article https://www.cnbc.com/2024/03/06/gpt-4-researchers-tested-leading-ai-models-for-copyright-infringement.html.

Our sensitive data example is a toy example that is similar-in-kind to detecting if copyrighted information was in the pre-training / finetuning data of a black-box model, if private or personal information is in the pre-training / finetuning data of a black-box model, etc. We’d argue that this is one of the major legal and ethical data security & privacy problems of the current technological era.

You provide results ...

We are not aware of any black-box methods that are capable of addressing our setting. We’d be happy to include comparisons if you could direct us to the methods that you have in mind!

NB: The response to the rest of your review is provided in a follow on comment.

9WTf,

Thank you for taking the time to read and review our paper. We have addressed your review below.

$**Summary**$:

We think your summary is a fair description of our work.

$**Strengths**$:

We agree that it is important to develop new techniques to analyze differences in model behavior -- or, more generally, to understand and study populations of models.

$**Weaknesses**$:

The authors note ...

That is a valid and fair concern. “Tracking perspectives of interacting language models” has since been accepted to EMNLP. The theoretical work that our paper extends is currently under review at a dedicated statistics venue. The study of populations of language models is quite new and, as such, there has been quite a lot of fast-paced progress.

I can't understand ...

$f_{i}$ is one of the n models of the previous sentence. In general, we let $i$ & $i’$ index models and $j$ and $j’$ index queries.

$r$ represents the number of responses from a given model for a particular query. For example, if we fix the model to be GPT4 and the query to be “What is your favorite color?” then $f_{i} (q_{j})1, .., f_{i}(q_{j})r$ could be “red”, “red”, “blue”, .., “red”. This type of repeated querying is typically necessary in black-box settings to characterize the response distribution $F_{ij}$. We recognize that all of this notation can be quite cumbersome, but we believe it to be necessary when first describing the setting and methodology.

The second paragraph ...

After introducing the necessary notation, we wanted to re-orient the reader to our goal and to let them know that we sometimes are loose with what variables represent (either a realization of the random variable X or the random variable X) before proceeding.

Line 102 ...

Yes, it is quite a mouthful -- though we tried to make it precise. Can you suggest an alternative?

Lines 189-192 ...

We wanted to be careful when presenting the theoretical results because consistency results are oftentimes confused with optimality results. In our case, we included the quotation marks because there are different notions of optimality here -- optimality with respect to the class of decision functions, optimality with respect to the distribution $P_{\psi Y}$, and optimality with respect to the distribution $P_{f Y}$.

We plan to improve our discussion of these nuances to allow for discussion of optimality without quotation marks.

In general, from the text after theorems 1 and 2 ...

That is a fair point. Again, we wanted to be careful when presenting the results so as to not oversell them. The results are actually quite informative and impactful -- Theorem 1 says that inference in estimated DKPS is consistent for inference in the true-but-unknown DKPS. Theorem 2 says if a sequence of decision functions (such as k-NN, SVMs, etc.) is known to be consistent for $P_{\psi y}$ when trained on the true-but-unknown $\psi$, then the analogous sequence of classifiers constructed with the estimated DKPS is also consistent for $P_{\psi y}$. Essentially, Theorem 1 comments on the effect of $m$ and Theorem 2 comments on the effect of $n$. We see both of the theoretical results in action in the “Was RA Fisher great?” and “Has a model had access to sensitive-data data” examples: as m gets bigger the performance improves (Theorem 1) and as n gets bigger the performance improves (Theorem 2).

We plan to improve our discussion around the Theorem results, as well, to include more insights as to what they do say, as well as keeping the current discussion around what they do not say.

Line 796 ...

Our goal with appending fruit names to the end of the original 50 augmentations was to perturb the original augmentations slightly so as to get relatively densely populated regions of the DKPS. It is a cheap way to get more “models” for which the behavior in DKPS would be easy to interpret. We thought that this was particularly important given that the treatment of our setting as a classical supervised learning problem is relatively new.

Overall ...

Thank you for your helpful feedback! Please let us know if our responses were helpful in understanding our work.

$**Questions**$:

Please let us know if you have any other questions throughout this discussion period!

vySb,

Thank you for taking the time to read and review our paper. We have addressed your review below.

$**Summary**$:

We think your summary is a fair description of our work.

$**Strengths**$:

We agree that trying to understand various aspects of the ever-growing set of generative models is extremely important. Our goal was to abstract away a lot of the different types of differences in models (such as model weights, augmentation strategies, generation parameters, etc.) in a single, unified framework.

$**Weaknesses**$:

I found the work to be trying to be too general ...

Thank you for this feedback. One of our goals in writing this paper is to establish a general approach to thinking about model-level analysis and inference. Our method -- inference in the data kernel perspective space (DKPS) -- allows for classical statistical inference (classification, regression, etc.) of Euclidean representations of the models. Since we treat the models as black boxes, the method is agnostic to different types of model differences such as model weights, generation settings, or augmentation strategies. While we may sacrifice depth for breadth, we think it is important to make sure that the reader is aware that the DKPS framework can handle a variety of types of model differences.

Our contribution is not intended to be that a particular decision function (1-NN, LDA, etc.) is the best for a particular task. It is to demonstrate that the model representations that are presented in Acharrya et al. can be used in the context of model-level inference. As such, we agree that a logistic regression model trained on the same model-level embeddings (from DKPS) would be effective for the “Was RA Fisher great?” task.

Our theoretical results are very related to our empirical results. In particular, Theorem 1 states that as we get more queries (m) the performance of any classifier gets closer to the performance of the classifier trained on the true representations of the models. Indeed, we see that an increase in m corresponds to an increase in performance across all of the experimental settings. Similarly, Theorem 2 states that if a sequence of decision rules trained on the true-but-unknown representations of the models is consistent then the analogous sequence of decision rules trained on the estimated representations of the models is also consistent. This explains why we see performance improvements as a function of the number of models (n) in both the “Was Fisher great?” and “Has a model seen sensitive data” examples.

We are a bit confused by your comment related to “results concerning actual statistical tests appear to be from other established tests, executed after the initial embedding in this [DKPS]”. As mentioned above, our contribution is exactly that inference in the DKPS is meaningful. We show this in classification settings, regression settings, and hypothesis testing settings. We think that this is an extremely important contribution, as there are currently no existing methods of getting model representations (that we know of, please point us to them if they exist!) that achieve this type of utility for populations of black-box models.

Additionally ...

We plan to edit the background and generic statistical inference framework sections to be more reader friendly. Any suggestions that would help make these sections more readable would be greatly appreciated.

Finally, I have some general concerns about the soundness of the foundations ...

This is a super fair and valid concern.Tracking perspectives of interacting language models has recently been published at EMNLP. Consistent estimation of generative model representations in the data kernel perspective space is currently under review at a dedicated statistics venue. This field is quite new and progress is fast-paced.

With that said, the current work builds upon more established theoretical work related to inference in learned node embedding spaces of random graphs. See, e.g., [1] and [2].

[1] Minh Tang. Daniel L. Sussman. Carey E. Priebe. "Universally consistent vertex classification for latent positions graphs." Ann. Statist. 41 (3) 1406 - 1430, June 2013. https://doi.org/10.1214/13-AOS1112 [2] Avanti Athreya, et al. “Statistical inference on random dot product graphs: a survey” Journal of Machine Learning Research 18 (2018) 1-92.

$**Questions**$:

Thanks again for taking the time to read and review our work. We recognize that we attempt to do a lot in the allotted pages and hope that discussions with you will help improve the presentation of our work.

Please let us know if there is anything else we can clarify or you’d like to discuss.

akqW,

Thank you for taking the time to read and review our paper. We have addressed your review below.

$**Summary**$:

We think your summary is a fair description of our work.

$**Strengths**$:

We agree that it is important to develop black-box methods to address model-level inference in scenarios where the score function is not formalized or unavailable. We think that the level of importance is still growing -- especially with large corporations putting retrieval-enabled language models on user’s phones, tablets, etc. and the ongoing pursuit of "digital twins" of society with the current cohort of language models.

$ Weaknesses:

Distinction ...

The distinction between the current submission and [1] is that we analyze the representations of the models in the estimated DKPS with respect to a subsequent inference task. The results of [1] essentially say that the low-dimensional representations of the models in the DKPS are consistent for a true-but-unknown set of low-dimensional representations of the models. They do not have a notion of model-level covariates. As such, they do not explicitly comment on the utility of the representations or on the convergence of decision rules.

The toxicity and bias ...

Yes, the model’s covariate correlates with the size of the point. & yes, the nature of bias and toxicity is more complicated than the other two examples. One of the goals of the bias/toxicity example was to demonstrate that functional difference (as measured by “closeness” in DKPS) is more important than model lineage (as measured by “closeness” in the model graph), even when the relationship between the DKPS and the covariate is non-linear. Also, while we only highlight a single set of neighbors in the model graph and the two DKPS, the relative error shown in the bottom right of Figure 4 is actually the average relative error over all models in the model tree. We will make sure to make that more clear. We will also emphasize the comparison of “functional” versus “lineage”.

The proposed approach ...

Our goal is to be a practical extension of the more theoretical [1], and so we greatly appreciate your suggestions. Beyond the illustrative example, we tried to pick two interesting experiments demonstrating practicality and attempted to highlight potential impact of important hyperparameters. For example, we explored sensitivity to an important hyperparameter (the distribution of queries G) to show that consistency of the estimated DKPS is not the only important thing, as one might think after reading [1].

We think of the sensitive data classification example as a simple experiment related to predicting if a model has had access to copy-righted or private information, which we think is one of the biggest ethical and legal questions related to training and using generative models. See, for example, this popular press article: https://www.cnbc.com/2024/03/06/gpt-4-researchers-tested-leading-ai-models-for-copyright-infringement.html.

Related to the question of the blue model being more (or less) toxic than the red model, we agree that this is an important question. We expect that one use case for DKPS will be to quickly identify not-passable models so that resources can be better used to evaluate promising models. For example, we could safely remove models from consideration if they are close to the more toxic blue model and only spend evaluation resources on models close to the less toxic red model.

$**Questions**$:

How this work ...

[1] introduces DKPS and focuses entirely on the behavior of the estimate of the DKPS. It shows that the estimated DKPS is consistent for a true-but-unknown geometry of models. [1] does not try to demonstrate the utility of the estimated DKPS with respect to model-level inference. As such, we extend the theoretical results to include inference and (attempt to) show that the DKPS is broadly applicable to many model-level problems.

Did I understand ...

Figure 4 (right, bottom) shows the average performance of the classifiers across all models in the model graph.

Is my understanding ...

There are many reasons why we might now see clear dependence in the DKPS space. One might be because two dimensions is not enough to capture it perfectly. Others, as you pointed out, may be due to the choice in hyperparameters such as the query set, the embedding function, the choice of metric on the models, etc. With that said, the shown DKPS is still better than other simple methods for trying to predict toxicity / bias, such as using the model graph or using the global average. This highlights that while the covariate signal in the DKPS is not necessarily linear, there is still a signal in the shown DKPS geometry.

On line 112 ...

Yes! Great catch, thank you!

Addressing ...

Thank you for your helpful feedback! Please let us know if our responses were helpful in understanding our work or if there are any other questions you have.

Dear authors, thank you for a detailed response! My questions are answered, but as mentioned earlier, more comprehensive empirical study is needed (though I understand that there is no time left to conduct larger scale experiments). For example, testing the method's predictive capacity with model embeddings of higher dimensions in other settings (not bias and toxicity, but more quantifiable ones like standard NLP benchmarks) would help validate the applicability of the method.

After reading the other reviews, I agree that the background literature is very recent and [1] has not been published yet, raising concern. There is another concern on the baselines raised by reviewer nufo, there are several model comparison methods that offer model embedding perspective as well, e.g. [1-3] with KID the closest in spirit to DKPS method.

[1] Bińkowski, Mikołaj, et al. "Demystifying mmd gans." arXiv preprint arXiv:1801.01401 (2018).

[2] Khrulkov, Valentin, and Ivan Oseledets. "Geometry score: A method for comparing generative adversarial networks." International conference on machine learning. PMLR, 2018.

[3] Tsitsulin, Anton, et al. "The shape of data: Intrinsic distance for data distributions." arXiv preprint arXiv:1905.11141 (2019).

SsgE,

Thank you for taking the time to read and review our paper. We have addressed your review below.

$**Summary**$:

We think your summary is a fair description of our work.

$**Strengths**$:

Thank you -- we also think that the method is practical & theoretically grounded and we are excited to continue to push this work in various directions!

$**Weaknesses**$:

We plan to add more discussion on the effects of different hyperparameter choices (such as the embedding function, the metric on the representations of the models, etc.). In experiments not included in the current paper, we have seen that performance can be highly affected by the choice of embedding function. For example, using off-the-shelf token embeddings -- as opposed to purpose-built (though also off-the-shelf) embedding functions such as sentence-transformers/all-MiniLM-L6-v2 or nomic-ai/nomic-embed-text-v1.5 -- can cause a sharp performance drop. The process to go from collections of response embeddings for each model to a collection model embeddings is quite rich in the type and number of impactful hyperparameters.

& thank you for suggesting an abstract re-write to better reach our audience.

$**Questions**$:

Please let us know if any questions come up throughout the discussion period.

We have just updated the paper. We spent the majority of our revision time re-writing the abstract and updating Section 2:

$**1. Updates to the abstract**$

• Per your suggestions, we dropped references of "embedding-based representations of the models" in favor of "vector representations of black-box model". We think that this change helps emphasize that the method we does not rely on access to the model weights, token probabilities, etc. and is broadly applicable.

$**2. Updates to Section 2**$

• We eliminated unnecessary notation and updated the description of the setting to improve readability.
• We updated the presentation of classical statistical inference on generative models to flow better.
• We added paragraphs after both theorems to highlight how they can be interpreted in practice.
• We moved the discussion of what the theorems do not say to a new subsection called "Non-standard considerations". We think that this will help the reader understand the nuances of our setting.

$**3. Misc.**$

• We added a "Contribution" paragraph at the end of the introduction. We hope that this helps readers understand our contribution and how we view the work in the context of recent literature. This is the full paragraph:

Our contribution is the theoretical and empirical validation for using vector representations of black-box generative models for model-level inference tasks. The representations that we study herein are based on the embeddings of their responses. The theoretical results apply to any generic, well-behaved embedding function. Given the representations, any standard vector-based method can be used for inference on the models. We think that this addresses some of your comments that one could use other inference methods for some of the tasks we study.

• We clarified that the model bias and toxicity regression results reported are the average over all models in the model graph.
• We fixed various typos.