ALMANACS: A Simulatability Benchmark for Language Model Explainability

1. The results are mostly negative. While this is somewhat expected for a challenging benchmark, it is unclear what the takeaways are from the negative results. For example, when the GPT-4 interpreter fails to predict a model prediction based on the explanations (of other related inputs), is it due to the irrelevance of the training inputs (in which case the embedding model is too weak), or the inability of the GPT-4 model in using the explanation information (in which case the model or the prompting may need to be changed), or the low quality of the explanations? It seems to me that only the last case truly highlights the issue of the explanations, and even so, there is no further insight on identifying the specific aspect of the explanation causing the low quality.

2. I notice that both models studied are quite small, flan-alpaca-gpt4-xl with 3B parameter and vicuna-7b-v1.3 with 7B parameters. They are also released over 1 year ago, which is quite old in the context of recent LLM development. I would recommend trying the newer and larger models, such as llama 3(.1), or the closed source models from OpenAI or Anthropic (without the attention or integrated gradient access).

3. Accuracy or F1 score is not included as the metrics. I tend find them more intuitive, compared to metrics such as KLDiv.

4. There are some missing recent related works on studying the self-explanations (i.e., rationales) of LLMs, such as [1] and [2]. Both work show mixed results, echoing the general conclusions by this paper.

[1]. Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations. Huang et al.

[2]. Are self-explanations from Large Language Models faithful? Madsen et al.

The overarching weakness of the paper is that the author's failed to discover where their method shows discriminatory results in simulatability. I think that until this happens the story of the paper doesn't feel finished to me. I think that some experiments which would really help are those which show that the method can actually convey some actionable information for the LLM, just to show that some insightful results can be gleened from ALMANACS and how exactly to setup tests to do that, so that future researchers using this understand how to use it in their experiments.

The issue is, if I were to literally give random noise to the LLM instead of the explanations it would also likely do nothing, so you have to show some observable effect from your framework to validify it.

1. The goal of developing an efficient tool for automated testing of model simulatability is not adequately achieved. User studies must minimize the influence of prior knowledge to ensure objective and effective testing of explanation quality. However, the proposed method relies heavily on the training data of the large language model (LLM), which may skew results and introduce biases. While the authors acknowledge this concern and caution against over-reliance on ALMANACS, it remains critical. If the reasoning generated by ALMANACS cannot reliably measure the simulatability of explanations, then automation in this context becomes meaningless. How do the authors demonstrate that the prior knowledge of GPT-4 as a predictor does not significantly affect the evaluation of explanations?

2. The focus on binary Yes/No questions, while suitable for preliminary implementation, severely limits the applicability of the tool, especially in scenarios that require outputs of richer forms.

3. The conclusions drawn in the paper are not convincing. The authors designed 15 templates for each topic, with each template containing 15 placeholders, generating the dataset by replacing these placeholders with different texts. Two issues arise here:

   • Could the non-placeholder components of the templates significantly influence the model's predictions? If so, the explanations generated from data produced by the same template are likely to be similar, rendering the replacement of placeholders ineffective. In this case, the hundreds of data points generated are effectively indistinguishable from a single data point.
   • For the average of experimental results to be meaningful, the distribution of the experimental data should be uniform. However, the authors did not verify the uniformity of the datasets across different topics. Instead, they concluded that all explanation methods are insufficient based on average results from various topic datasets, which is unconvincing.

4. Some key details in the paper are unclear. Part of them can be found in the appendix but they should really appear in the main text. Concretely:

   • In Section 2.1, the authors should clarify why choosing GPT-4 as a predictor is crucial, as it is the primary tool for evaluating explanations. How might different kinds of LLMs impact their work?
   • In Section 2.2, it is challenging to understand how to calculate the probability of a Yes answer without reference to the appendix.
   • In Section 2.2, how did the authors compare different embedding methods? What metrics were employed, and why was the Sentence-BERT model all-mpnet-base-v2 chosen?
   • In Section 2.2, the authors conducted a suite of evaluations to assess the models’ capabilities but did not provide any details or results from these evaluations. How can they demonstrate that the model explained has sufficient capability to address the questions posed in their benchmark?

I will consider adjusting my score if my below concerns are addressed.

Weakness 1: Validity of using an LLM. My primary critique of the paper is that from the authors' experimental validation, I remain unconvinced by one of the authors' key claims: that an LLM predictor "can replace humans as [an] automated evaluator of explanations" in this context. To make this point, the authors argue that "the automated GPT-4 predictor is consistent with human evaluations". However, I need more detail to understand how the results in Section 5.2, and Figures 4b-c, support this argument.

• The main piece of evidence that the authors use to justify that the LLM is "consistent with humans" is the wide error bars for all of the treatments in the "all" condition in Figure 4. But, if I understand correctly that the authors defined the "All" category by creating a big pool across all of the different tasks, then of course the error bars are going to be wide here because there's variance across tasks. A better measure of statistical significance wouldn't create a big pool, but instead conduct separate statistical tests for each of the 5 tasks (each "distribution"). I'm skeptical of this measure being used as justification, and am open to hearing other justifications instead.
• As the authors already noted in their paper, I think it is interesting that there are some significant differences across conditions in the human experiments – for example, that humans did much better when given explanations in the hiring decisions context – and these trends are not found by the LLM.
• I would be more amenable to this work if the authors did not claim that the LLM predictor's results are necessarily "consistent with humans", and instead note that there are discrepancies, describe in detail what these discrepancies are (e.g., expand further on some of the findings you already have, like how LLMs tend to be better at the simulation task). I think you can still try to make the argument that there is value in using an LLM to approximate the information different XAI methods give that might be useful for simulation, without necessarily needing to argue that it will be predictive of how a human will perform. But this argument needs to be more fleshed out. (Maybe the LLM is an "upper bound" of the predictive information available in an explanation to aid simulation; but humans may struggle/fail to infer how to use these information to actually complete the task accurately – an argument made by [1]).

Weakness 2: Biases introduced by using models to simulate other models. I am wondering if the reason why using another LLM (e.g., GPT-4) to predict the outputs of another LLM (e.g., flan-alpaca-gpt4-xl), performs well, has something to do with how the two models were developed – for example, flan-alpaca-gpt4-xl was trained to behave as similarly as possible to GPT-4.

• In other words, these models already do a great job at simulating the outputs of other models because of the way that they were trained, in a way that humans cannot.
• This makes me question the appropriateness of using large pretrained models as the "predictor". Past work that has used secondary "predictor" models in simulatability tasks doesn't share this same challenge – to my understanding, the predictor in these past studies was initialized from scratch.
• If the authors believe it is necessary to use a large pretrained model as the predictor, perhaps they can acknowledge how these connections between the predictor vs. the model being explained as one factor that contributes to the predictor's good performance on the task. They might also be able to design additional experiments to explore this – e.g., is it true that a predictor that is the same (or a similar) model as the model being explained, will always perform better than other predictors that are unrelated?

Weakness 3: More discussion of limitations of existing scenarios; or extendibility of the benchmark itself. I would appreciate more discussion in the main text about what exactly the scenario categories are, the limitations of using a template-based approach, and how someone reading your paper could potentially contribute a new category of scenario, or a new template, to your benchmark. More broadly, it is unclear if you intend to support extendibility of your benchmark (i.e., the ability of users to add additional explanation methods, predictor models, or scenario prompts).

Nit: The text in your figures is inaccessible via a screen reader. To fix this, you can include figures as PDFs instead of PNGs.

[1] https://proceedings.neurips.cc/paper_files/paper/2022/file/0b9536e186a77feff516893a5f393f7a-Paper-Conference.pdf