ALMANACS: A Simulatability Benchmark for Language Model Explainability

Thank you for your review. If we understand correctly, your main concerns are that our experiments are not comprehensive enough. In particular, you would like to see tests with more model sizes and model families, more pairs of language models used for prediction with language models being explained, and human studies. While we leave human studies as an important direction for future work, we have added experiments with 11 additional language models. These include nine new language models being explained (in Appendix G) and two new language model predictors (in appendix J).

Do these changes address your concerns about the comprehensiveness of our experiments? If not, we would be happy to improve the paper based on further input from you.

Only safety-relevant scenarios are included and this is not general.

Our work contains 12 different scenario topics, and it has a total of 180,000 training examples and 18,000 test examples.

We are confused how this many different scenario topics and dataset examples is not general enough for ICLR. There are other natural language benchmarks published in ICLR that contain fewer topics and dataset examples. For instance:

• “WikiWhy: Answering and Explaining Cause-and-Effect Questions”. ICLR 2023 (notable top 5%). The dataset has 9,406 “why” question-answer-rationale triplets spanning 11 topics.
• “ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning”. ICLR 2020. The dataset has 6,138 questions (and no topic categories).

We hope that our work is judged consistently by the same standards as other papers.

The choice of language models for evaluation versus being explained may affect results.

To test if our choice of GPT-4 for evaluation affected the results, we ran additional experiments with two additional language models for evaluation. The results are in Appendix J. We find that no combination of predictor/explanation, for either model being explained, performs significantly better than using GPT-4 for evaluation with no explanations provided, or than the logistic regression baseline. Consistent with our main results, this indicates that none of these predictors are able to extract helpful information from the explanations beyond what is present in the model’s answers.

Automated evaluation using a language model proxy for humans has limitations vs. human studies.

This is an important point; human studies are a valuable direction for future work. Nevertheless, even if an automated evaluation isn’t totally consistent with a human study, we still think automated evaluation has a valuable role to play. Because automated evaluation is an order of magnitude faster than human evaluation, it can provide quick, coarse-grained feedback throughout the interpretability researcher’s development cycle. A tool like ALMANACS is the difference between waiting weeks or months for the results of a human study, versus getting results of an automated benchmark in just a few hours.

Testing on more model sizes, scaling effects, and model families instead of just focusing on flan-alpaca-gpt4-xl and vicuna-7b-v1.3.

We have added scaling experiments to the paper, in Appendix G. These experiments test models of a variety of families with sizes ranging from 0.77B to 20B parameters on the advanced_ai_risk section of ALMANACS. The benchmark difficulty, as measured by how well logistic regression is able to predict model behavior, is shown in Figure 10. We see a general trend that the behavior of larger models is harder to predict. The trend is more pronounced when we compare benchmark difficulty with model performance on a related task, with more capable models having behavior that is more difficult to predict.

Missing references for explanation distillation:

(1) Li et al. Explanations from Large Language Models Make Small Reasoners Better. 2022.

Thank you. We have added this reference to the paper.

the paper would benefit from a deeper analysis... on the possible reasons [why explanations don't help on unseen data]

Thank you for this excellent suggestion. In Appendix H, we have added additional analysis and discussion of possible reasons why the explanation methods are not improving performance. Specifically, we provide examples of the explanation methods from our dataset, and we give a qualitative analysis of how much information is contained in the explanations. One insight is that to us, the authors of the study, the tokens with high salience scores do not intuitively suggest how the model will behave on new inputs. For example, the tokens are sometimes the question words, such as “would” and “choose”, rather than the details of the scenario. Another observation is that the language model’s rationalizations often discuss the specifics of the input scenario without discussing possible variations of the scenario. This might be limiting how much the rationalizations help the predictor at test-time, when the predictor is presented with scenario variations.

The term "simulatability" appears to be inconsistently defined.... The initial definition of simulatability is "how well the explanations improve behavior prediction on new inputs". Subsequently, it seems to change to a definition centered on distribution distance, KLDiv or TVDist…. Could the authors provide a precise definition of “simulatability”?

It seems our original submission did not clearly present the KLDiv and TVDist metrics. “How well the explanations improve behavior prediction on new inputs” is indeed the definition of “simulatability” that we use throughout the paper. The key idea is that we are interested in distributions of behavior; we define “behavior” as the yes-no probability distribution that the language model gives to the scenario. We also have the predictor give a yes-no probability distribution as its prediction. To measure how good the predictor is, we therefore used the distribution distances KLDiv and TVDist.

Past work has considered binarized model output (taking just the most likely answer, either “yes” or “no”). This allows measuring simulatability with accuracy / F1 score. However, binarizing the model output is throwing away information. We choose to keep the entire distribution to create a more challenging benchmark, where the predictor must distinguish between models which answer “yes” 51% of the time versus models which answer “yes” 99% of the time.

We have added a discussion of why we use distribution distances to Section 2.3: Evaluation Metrics.

In Appendix B, we also added a third quantitative metric, Spearman’s rank correlation coefficient. We calculate the Spearman correlation between the predictor’s probability of “yes” and the language model’s probability of “yes” across all questions in a topic. This provides another way, different from distribution distance, to measure simulatability. In Appendix B, we see that Spearman correlation results are consistent with the KLDiv and TVDist results: no explanation method does substantially better than the explanation-free control.

GPT-4's predictions might be independent of input explanations

It’s important to know if GPT-4 has the ability to understand input explanations and change its predictions accordingly. To test this, in Section 4, we provide GPT-4 with explanations for a synthetic linear model. The explanations are hand-crafted to contain prediction-relevant information. In Figure 4, we see that providing GPT-4 with the “qualitative” and “weights” explanations improves performance compared to the “NoExpl” no-explanation control. Therefore, we have examples where we know that GPT-4’s predictions depend on the input explanations.

Have [you] considered more recent sentence embeddings, such as SimCSE?

We have added experiments with more recent sentence embeddings to Appendix I, including SimCSE and SentenceBERT. With these new embeddings, we find that the baseline methods have improved performance. Therefore, we have updated the main results tables (Tables 1, 2, and 3) to use the more recent embeddings. Our overall pattern of results remains the same: even with new embeddings, no explanation method provides a substantial benefit over the no-explanation control.

PredictAverage outperforms NoExpl in Figure 4 but does not in Table 1. Could this [discrepancy] be clarified?

Thanks for bringing this to our attention.

PredictAverage depends only on the model being explained (predicting the average probability over the dataset). NoExpl, on the other hand, is GPT-4’s prediction when prompted with few-shot examples of the model’s behavior. We interpret this result as indicating that GPT-4 is better at in-context learning of other language models’ behavior than in-context learning of a synthetic linear model. We have added this observation to the paper in the “No-explanation predictions” paragraph of Section 5: Results.

Thank you for your review. If we understand correctly, a primary concern of yours is how KLDiv and TVDist relate to simulatability. You also would like to see a deeper analysis of why explanation methods don't improve performance and if GPT-4's behavior is independent of explanations. Finally, you would like more modern sentence embeddings, such as SimCSE.

We have added additional experiments and analyses to the paper, which we believe addresses all of your concerns. Please see our other comment for a detailed discussion of these experiments and analyses. In summary:

• We added to Appendix H an analysis and discussion of possible reasons why the explanations are not improving performance.
• We have added a discussion of why we use distribution distances to Section 2.3: Evaluation Metrics.
• We added a third quantitative metric, Spearman’s rank correlation coefficient, to Appendix B.
• In Section 4, Figure 4, we see an example where the GPT-4 predictor depends on the input explanations to improve its performance.
• We have added experiments with more recent sentence embeddings to Appendix I, including SimCSE.
• We have updated the main results tables (Tables 1, 2, and 3) to use more recent embeddings.
• We have added an observation about the discrepancy between Figure 4 and Table 1 to the “No-explanation predictions” paragraph of Section 5: Results.

Have we fully addressed your concerns? If not, please let us know. We would be glad to incorporate further suggestions into our paper.

Thank you for your review. If we understand correctly, your concerns are that (i) we only use two evaluation metrics, (ii) we do not do a human study, and (iii) we only consider three explanation methods. While we acknowledge the lack of a human study, we have incorporated all your other suggested changes into the paper. In particular, we have added a third evaluation metric, Spearman's rank correlation coefficient, to the paper. We have also added a fourth explanation method, Integrated Gradients.

Does this fully address the reviewer's concerns? If not, please let us know, and we will be happy to make further changes.

The paper only evaluates the explanation methods based on Kullback-Leibler divergence (KLDIV) and total variation distance (TVDIST).

In Appendix B, we have added a third metric, Spearman’s rank correlation coefficient. We calculate the Spearman correlation between the predictor’s probability of “yes” and the language model’s probability of “yes.” This provides another way, different from distribution distance, to measure simulatability. In Appendix B, we see that Spearman correlation results are consistent with the KLDiv and TVDist results: no explanation method does substantially better than the explanation-free control.

automated evaluation using language models may not be consistent with human evaluation

This is an important point; human studies are a valuable direction for future work. Nevertheless, even if an automated evaluation isn’t totally consistent with a human study, we still think automated evaluation has a valuable role to play. Because automated evaluation is an order of magnitude faster than human evaluation, it can provide quick, coarse-grained feedback throughout the interpretability researcher’s development cycle. A tool like ALMANACS is the difference between waiting weeks or months for the results of a human study, versus getting results of an automated benchmark in just a few hours.

The paper evaluates only three explanation methods

We have added an additional explanation method, Integrated Gradients. Integrated Gradients is a feature attribution method that is axiomatically motivated; it satisfies sensitivity and implementation invariance, and it is the unique path method that is symmetry preserving. (For details, see “Axiomatic Attribution for Deep Networks” by Sundararajan et al.) Integrated gradients was also one of the best-performing methods in Pruthi et al.’s explanation evaluation paper, “Evaluating Explanations: How much do explanations from the teacher aid students?”

In Table 1, we see that the Integrated Gradients results are consistent with the results for other explanation methods: Integrated Gradients does not improve simulatability over the no-explanation control.

With one day left in the discussion period, we are hoping to hear a reply from you. Please let us know if we have addressed your concerns.

Thank you for your comments; our reply is below. If there is anything else we can do to improve the paper, please let us know. We will be happy to incorporate further suggestions into our work.

Although the paper frames ALMANACS as a benchmark, I find it more suitable to call it a dataset—only when paired with a good-enough human approximator like GPT-4 would it become a benchmark.

We agree that the dataset is a primary contribution of ALMANACS. The ALMANACS code we release also provides a complete implementation for using GPT-4 as a predictor; the user needs only to provide an OpenAI API key.

Given the predictor is GPT-4, it seems like the benchmark can be applied to interpretability goals beyond simulatability. Any thoughts on that?

Yes, we think it’s exciting to consider how ALMANACS can be applied to interpretability goals beyond simulatability. Here are a few possibilities:

• Studying minimality, the absence of extraneous information. To do this, one could randomly sample a subset of each explanation. Then, one could see if the GPT-4 predictor has the same level of performance when looking only at explanation subsets. If a subset of an explanation performs as well as the full explanation, then it suggests that the explanation contains extraneous information. (Although one has to be careful, because having redundant information could be desirable; future work might need to distinguish between redundant versus irrelevant information.)
• GPT-4 could automate counterfactual analyses, like Chen et al.’s work “Do Model’s Explain Themselves? Counterfactual Simulatability of Natural Language Explanations.” Future work could try performing similar sorts of counterfactual analyses in ALMANACS.
• In order to scale to larger and larger neural networks, we will need automated interpretability tools. Because ALMANACS allows fully automated analysis, it can help test scalability of interpretability methods.