Bring Your Own Data! Self-Supervised Evaluation for Large Language Models

Thank you for your time, KAMN!

Entropy

We appreciate the reviewer highlighting the important role that model entropy plays in interpreting sensitivity evaluation results. As noted in the paper, lower entropy impacts effects like requiring more aggressive sampling during text generation. While we provided some preliminary entropy analysis, the reviewer is correct that we do not explore this relationship comprehensively.

Understanding how entropy relates to observed sensitivities is crucial for accurate interpretation, as models with very different entropies may behave differently on the proposed metrics. We believe that marginalizing out entropy completely is likely not feasible or ideal. Rather, analyzing entropy provides another facet to contextualize results as part of the holistic evaluation. We recognize entropy's importance and agree it should be explored beyond the initial results provided. For a more holistic picture of SSE metrics, it is worth looking at these entropies of the model to get a better understanding of the SSE scores.

More cohesive narrative that explains how these methods collectively advance the understanding and evaluation of LLMs.

Thank you for the feedback asking for a more unified narrative and rationale behind the proposed evaluation methods. The transformations we explore - negations, toxicity triggers, word order changes, context/long-range (Appendix), and tokenization error (Appendix) - represent model behaviors that are particularly relevant in the current discourse about responsible LLM deployment.

While we presented a subset of methods, the number of possible of evaluations that could be applied in our framework is large, as any interesting, easily automated, transformation of text that has expected behavior could be considered. For example, if you wanted to measure the sensitivity of the multilingualness of a model, then they could swap English words within a passage for another language, and measure how the JSD of the next sentence changes. The unifying theme is efficiently quantifying model sensitivity in a data-driven way without human annotations. We can certainly do more to connect these ideas in the paper and communicate the broader vision behind this approach.

Simple Transformations

We recognize these straightforward techniques have limitations, however, we believe even simple transformations can provide useful behavioral insights about LLMs in a data-efficient manner. Additionally, we believe that only a subset of transformations of the corpora have to be meaningful (see Appendix A.4) to measure something useful. However, the transformation does indeed need to be meaningful. For example, the methods aim for ease of automated application across diverse text data at scale, traded off with linguistic complexity. Simple metrics have utility despite limitations.

The methods may not be backed by a comprehensive set of experiments to validate their effectiveness across different models, domains, prompts, and languages.

Our main experimental goal was to validate the proposed framework by benchmarking against established metrics, which are supervised. For us, this necessitates the comparison against existing benchmarks using related datasets. For example, we chose Wikipedia as a raw data source because we know that TriviaQA questions are sourced from a similar knowledge base, and many answers are Wikipedia entities. This allows us to look at the effectiveness of the unsupervised metric in isolation from domain shifts between labeled and unlabeled data.

However, the idea is that the framework and the tests (i.e. Knowledge via Negations, Toxicity, etc) can be used on many text corpora. For example, if a lawyer wants to know how well a model understands Bengali Consitutional Law, then the individual can get a textbook on the subject to apply these metrics as a first evaluation. This narrow area would not be possible by just looking at the MMLU performance on law. Yet, without labeled datasets it is hard for us to verify that our approach is reasonable in this exact scenario, requiring us (for the validation within this paper) to go back to datasets that we can test against existing benchmarks.

Memorization

Thank you for raising this excellent point. We acknowledge the limitations that training set memorization could confound observed trends. Quantifying these effects precisely is challenging as there is no standard way of approaching the problem. Ideally, we would evaluate sensitivities on novel texts like news articles where we can guarantee no training set overlap. If sensitivity trends remain consistent, it provides evidence that memorization does not fully explain the results.

Thank you for your time, Reviewer cArL!

Inconsistency of Models in Figures

We use a core set of pretrainined open-source LLMs which GPT-2 models, Pythia models, GPT-Neo models, MPT, and LLaMA-1 for all charts. For Figure 3, we added additional API models mainly for really powerful models like GPT-3 as measurements for knowledge are at per-token log probabilities which were available through the APIs. However, when we measured JSD over the probability distribution for word order, we were unable to get the entire probability distribution from the API, making them not possible models to study for all cases. Additionally, we found running toxicity for API models can often trigger the content filter, and thus nothing can be gained from running the toxicity metrics on API models. Furthermore, for Figures 3 and 4, we add Galactica as a control to see if our metric would be meaningful in clinical vs general settings to verify that scale is not the only factor for increased sensitivity scores. Additionally, your point about adding more pretrained models is well taken, and we will include them in a future version of the paper.

Robustness of SSE and Outliers

The goal of the metric is not to measure the exact same thing as MMLU or TriviaQA, but something different. Thus, we do not expect them to have a perfect correlation. When we find a model that is an outlier, we believe it is a telling tale of the model's underlying behavior compared to other models. Another way of viewing this is as a smoke screen for the models. If the model has a sensitivity that is way too high or way too low, then this gives you insight into the model's behaviors.

Thank you for your time, Reviewer tXds!

Terminological Ambiguity

We'd be happy to describe our reason for this naming scheme: In the SimCLR paper, a self-supervised training procedure is described that requires a human to choose augmentations that the self-supervised learner (SSL) is trained to be invariant against, and the loss function. The augmentations that we describe in this evaluation framework are directly comparable to the transformations in SSL and the SSL loss function is comparable to our measurement procedure (change in ppl, JSD of probability distributions, etc). However, we are open to a discussion on changing the name of the evaluation and would be glad about further feedback.

Metric Soundness

We chose knowledge probing, toxicity, and long-range because we felt like these were particularly relevant in today's literature. For example, one of the community's goals for LLM model deployment is to make the LLM less toxic. Another example is long-range dependency, which is a mainstay question to evaluate the finite context length in transformers. Additionally, we found word order to be compelling given there is no dataset that measures this directly. To validate the soundness of our methods, we compared them with standard benchmarks to find that our methods correlate well with these standard benchmarks. We do think that this is a difficult question because many labeled datasets contain discrepancies themselves (i.e. there was a recent discussion evaluating MMLU https://huggingface.co/blog/evaluating-mmlu-leaderboard). A study that could shed light on this issue in the future would be to ask domain experts to evaluate the model and see how well self-supervised methods align with human domain experts.

The idea is that the framework and the tests (i.e. Knowledge via Negations, Toxicity, etc) can be used on many text corpora. For example, if a lawyer wants to know how well a model understands Bengali Constitutional Law, then the individual can get a textbook on the subject to apply these metrics over as a first evaluation. This narrow area would not be possible by just looking at the MMLU performance on the law.

Questionable Conclusion

We agree that metric is different and that they have different usefulness. We illustrate the usefulness of the metric by grounding it with TriviaQA, but it is independent as shown with the Cohere Command example.

Visualization

Thank you for bringing this to our attn. We will update this in a future version of the paper.

Questions:

Can authors provide some examples of how they perform the transformations to evaluate long-range sensitivity.

Thank you for asking for more details on the transformations used to evaluate long-range/context sensitivity. Let's say the original passage had three sentences, $\{S_3, S_2, S_1\}$, where $S_3$ is the first sentence of the input passage, then the altered passage would be $\{S_X', S_Y', S_1\}$, where $S_X', S_Y'$ are random sentences from another passage in the corpus. We provide a concrete example in Appendix 8, Figure 18.

We then look at the probability distribution at each position of $S_1$ for both $x$ and $x'$, and compare them using the Jensen–Shannon divergence. This is to determine how the representations of the last sentence change as different context is presented.

How do you identify neutral corpus

By neutral corpus, we mean that almost all of the corpus will not contain facts. Thus, when inserting a "not", we are seeing a perplexity change that the model induces naturally by seeing this "not" in a given sentence. Since Bookcorpus contains fiction material, we suspect the different domains of fiction material would not matter as much as these do not necessarily contain facts that the model should be "perplexed" when the sentence is transformed into its counterfactual. We also discuss in Appendix A.4 why we believe that transformations need only be informative on average.

Percent PPL Drops

Thank you for catching this confusing statement about Figure 14. The intention was to reference the analysis in Section 4.2 discussing the need for proper normalization when using raw sensitivity scores based on perplexity changes. Unnormalized scores can follow a square-root type relationship with accuracy benchmarks like TriviaQA. The central evidence for normalization comes from the earlier analysis in 4.2, showing it helps account for overconfidence in certain tuned models.

What data do the metric PPL use? Following this paper’s logic, I think the first step this paper should do is to evaluate the correlation between, for example, normalized PPL and TriviaQA accuracy (and other realistic tasks such as MMLU etc).

We use the PPL metric on the same text as senstivity score was measured over. We use this because PPL is good automatic metric for understanding knowledge in a model. However, it is known to have issues like bias to longer sequence. Thus, we attempt to add a new metric helps understanding a model's knowledge. Looking at the outliers between our metric and ppl, we see that it is a good proxy to measure general knowledge. However, when measuring domain specific knowledge like medical terms, we see that a weak language model's knowledge like Galactica cannot be captured by PPL. This is a core advantage of our proposed metric.

How well do toxicity scores by this paper correlate with those by Perspective API on instruction tuned models? From figure 6 left, it looks like there are no Xs. Same question for word order section and figure 4, 7. Without the results of instruction tuned models, it is hard to see if the scores correlate better on vanilla models than instruction tuned models. And therefore hard to assess the significance of the metrics.

There are very few instruction models that use a different base model. Additionally, it is possible to get scores from the 5 different models but hard to get a meaningful correlation out of them. As new models come out we can get them evaluated and add them in our suite.

How adaptable are these proposed metrics? Would the methodology need alterations to assess LLMs in more dynamic, real-world scenarios?

Thank you for the feedback requesting more clarity on real-world applications of our proposed self-supervised evaluation (SSE) methodology. You raise an excellent point that adaptability and testing in dynamic scenarios is important. Here are a few thoughts on how SSE could be applied:

• The transformations we introduce like negations, toxicity triggers, etc. could be adapted and tailored to test sensitivity on specific types of in-domain text data. For example, a company could construct domain-specific violations to test an LLM's robustness before deployment.
• The metrics could be applied dynamically during live system operation to monitor production model performance over time, catching if behavior deteriorates. New text data could be continually evaluated.
• The approach is model-based rather than relying on external benchmarks, allowing evaluation even in domains lacking standard labeled datasets.
• The methodology is light-weight computationally, allowing scaling to large text corpora for evaluation.
• New transformations could be introduced to test emerging capabilities like logical reasoning, domain knowledge, etc. The framework is extensible.

Our main contribution is the SSE (Self-Supervised/Sensitivity Evaluation) framework. Using this framework, we design many different tests like knowledge probing via negations, toxicity in the presence of F-bombs, word order, context/long-range sensitivity, and tokenization error. We show that in these 5 cases, the SSE framework can give us a meaningful measure. In each case, we show that the sensitivity scores provided can be grounded with labeled data, showing that meaningful insights can be determined from these scores. This validates our the SSE framework as complement to traditional evaluation, giving us more insights into the model's behavior.

Although these transformations are straightforward, we believe that this is a strength of what we have proposed as they can be easily applied to many corpora. Additionally, we use the Perspective API to validate how we measure toxicity which does not rely on an outside model or labelled data. Here, we see that we have a strong correlation with the Perspective API.