Are Large Language Models Post Hoc Explainers?

We thank the reviewer for acknowledging the reproducibility and experiments of our work. We greatly appreciate your feedback on solidifying the contributions of our work and address your concerns below.

Motivation for using LLMs as a post hoc explainer

We understand your concern about the use of LLMs as a post hoc explainer. Our choice to use Large Language Models (LLMs) for this task was driven by our desire to explore their potential beyond conventional language processing applications. The novelty of our approach lies in its ability to integrate contextual understanding and in-context learning (ICL) capabilities of LLMs to provide richer, more nuanced explanations than what might be achievable through simpler methods. LLMs are useful for the following reasons:

Existing methods often lack the ability to offer detailed, human-like explanations. LLMs can bridge this gap by generating explanations that are more understandable to humans, which is particularly valuable in fields where interpretability is crucial, such as healthcare or finance.

Generating explanations using LLMs is not just about identifying important features but also about understanding the rationale behind these selections in a manner that simpler methods may not provide.

The ability of LLMs to process and explain complex patterns in data can be particularly beneficial when dealing with intricate, high-dimensional datasets where traditional feature selection methods might struggle.

We would like to note that the potential benefits in terms of the depth and quality of explanations justify the exploration of our study, where we intended to be a stepping stone in the direction of using LLMs as possible post hoc explainers.

To maximize the power of LLM, a better way is to post-hoc explain the model's output from the perspective of "natural language" …… Another way is to let LLMs use tools (e.g., use Python to code) to enhance the extra ability of LLMs and further obtain a guaranteed faithful explanation for a black-box model.

Thank you for your valuable feedback on our work. We appreciate your insights on using the LLMs in other explainability frameworks and would like to note that there are inherent challenges in using Large Language Models (LLMs) as explainers. However, we would like to clarify that our work aims to explore the potential of LLMs in a new XAI domain — introducing the first framework to study the effectiveness of LLMs in explaining other predictive models trained on tabular datasets. Regarding the application of LLMs to numerical datasets, we believe this represents an innovative step in understanding the capabilities of LLMs beyond their traditional scope. While LLMs are primarily designed for language tasks, their ability to abstract and generalize can potentially be leveraged in a variety of contexts, including numerical data interpretation. Our paper extends the TabLLM work [1] that studies the application of LLMs to zero-shot and few-shot classification of tabular data.

In fact, the “logical thinking skill”, the ability to process math problems[1], and the instruction-following ability of LLMs are poor or remain unclear; even the order of the input will affect the output of a LLM

We agree with the reviewer that the current set of LLMs suffers from a range of problems, such as processing math problems, lack of instruction-following ability, and inability to process longer inputs. However, we would like to clarify that our exploration of LLMs does not expect them to solve extensive math problems to generate explanations. For instance, we specifically instruct the LLM to explain its reasoning (see the “Instructions” section of the prompt templates on pages 4 and 5) before reaching a decision on which features are most important to the model, and the LLMs follow the rule explicitly stated in the instruction (see Section 3.3). Moreover, recent works like Lightman et al. [2] show that LLMs show enhanced performance on mathematical reasoning tasks, which will improve with the progress of LLM research. Again, our goal is to explore the potential of LLMs as a post hoc explainer, and that like other explanation methods, is not without its limitations.

We are very grateful to the reviewer for all their questions/concerns, as they have helped us improve our paper significantly. We tried to address all the reviewer suggestions and hope the reviewer considers increasing their score.

References

[1] Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X., & Sontag, D. Tabllm: Few-shot classification of tabular data with large language models. In AISTATS, 2023.

[2] Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., ... & Cobbe, K. Let's Verify Step by Step. arXiv, 2023.

Thank you for your rebuttal and clarifications.

Compute and Runtime

Thank you for this clarification in terms of the runtime, this is informative. Do we have a cost comparison too? Please correct me if I'm wrong, but the numbers suggest (to me) that it is i) Slower (albeit expected to get faster) ii) A lot more expensive (due to the API request costs). I think it would be good to acknowledge these limitations and be upfront in the paper (this does not diminish its value, but rather increases it with better transparency).

Another question: Why are these details not in the paper?

Experimental Details

Thank you for these clarifications. I'm not familiar with how OpenXAI picked these specific values -- is there a reason why the authors did not perform a hyperparameter sweep? From the compute time table above, it should not be too costly to quickly run a hyperparameter study, in my opinion.

Overall, I'm still unclear why the authors do not share these experimental details in the main text / at least the appendix. Is there a specific reason? As a reviewer/reader of the paper, I would personally appreciate having these details at least in the appendix (and not in a notebook in the zip file). I don't think these important experimental details are to be found by readers in the lines of a jupyter notebook, neither the rebuttal page is the only place to make these clarifications.

Experimental insights

• Thank you for your clarifications here. I do not see a Figure 13 in the paper, is it perhaps not updated?
• I similarly do not see the discussion in the main text. Could you refer me to concretely where it is?

I will revisit my evaluation after these clarifications.

We thank the reviewer for acknowledging the motivation and presentation of our work. We appreciate your helpful feedback and address all the mentioned questions/concerns below.

Compute and runtime for generating explanations using LLMs

Great point! We would like to clarify that, since submission, the cost of GPT-4 has already reduced 10-fold and inference times have gone down too. In response to the reviewers’ concern, we provide a detailed table below that compares the runtime for generating explanations using LLMs and existing post hoc explanation methods.

From the above table, we show that the time taken by LLMs to generate explanations is greater than the other explanation methods. However, we would like to highlight that the runtime for generating explanations using LLMs by query OpenAI APIs depends on a range of factors, including time of the day, server/requests overload, rate limit imposed by OpenAI, etc. However, the above runtime is expected to go down as the query time for OpenAI APIs improves.

Lack of experimental details

We apologize for the lack of clarity and provide additional experimental details below. PGI/PGU and FA/RA metrics

We have added a detailed definition of the four metrics in Section 6.1 of our revised manuscript.

Hyperparameters for explanation methods

We followed OpenXAI and used the standard hyperparameters for these explanation methods. In response to the reviewers’ feedback, we detail them below for reference and have added them to the revised manuscript.

LIME

kernel_width = 0.75

std_LIME = 0.1

mode = 'tabular'

sample_around_instance = True

n_samples_LIME = 1000 or 16

discretize_continuous = False

grad

absolute_value = True

Smooth grad

n_samples_SG = 100

std_SG = 0.005

Integrated gradients

method = 'gausslegendre'

multiply_by_inputs = False

n_steps = 50

SHAP

n_samples = 500

Reproducibility - what learning rate used to train models etc

All our models are trained in PyTorch using cross entropy and a class weighted term to encourage underrepresented classes to contribute more to the loss than the most popular class.

Optimizer: Adam

LR: 0.001

Epochs: 100

Data normalization: minmax

Batch size: 64

We have shared all these details in the Jupyter notebook shared in the supplementary material (Notebooks/TrainModels.ipynb)

What is our parsing strategy and how many bad replies?

The GPT-4 and GPT-3.5 responses are parseable an average of 96.4% and 85% of the time, respectively. We improve parsing ability by encouraging the LLM’s replies to be nicely formatted. This is done by requesting the ranked features to be on the last line in descending order of importance with no other information present. The exact parsing details can be found in our Supplementary Materials code folder llms > response.py.

In general, we use regex and string manipulation to identify the last line of the reply. We then extract key parts of the last line like “is”, “:”, and “=” which indicate that the ranked features will begin on the right side of the matched string. An example reply is as follows:

“Therefore, the top five most important features, ranked from most important to least important, are: G, A, F, H, I.

Answer: G, A, F, H, I”

This will be parsed by splitting the last line by : into two parts, namely “Answer” and “G, A, F, H, I”. The latter is split by a comma and stripped of any white space to be finally placed into an array for further processing and faithfulness evaluation.

Top k = 1 for Post Hoc Explainers

In response to the reviewers’ feedback, we conducted a new experiment and added Figure 13 to show the top-1 performance of existing XAI methods.

We would like to thank the reviewer for their engagement and for reconsidering their initial score. These discussions have been helpful for us whilst revising our paper.

Choice of hyperparameters

This is a fair point regarding the rigidity we have assumed of the post hoc explainer hyperparameters. While we do believe it best to adhere to OpenXAI for means of comparison and benchmarking, this is an interesting discussion to have. It might well be the case that for even higher dimensional datasets than those used, picking one set of hyperparameters for LIME would not yield optimal results across all settings. We would like to make some quick follow-up points to address this in the context of our work:

• We elected to use LIME with 1000 perturbations to mitigate any inconsistency or sub-optimality in its results.
• More importantly, we also observe in the logistic regression setting that LIME achieves near-perfect FA/RA scores across datasets, as well as, in the neural network setting, performance on par with the other state-of-the-art methods that do not require hyperparameter tuning, i.e. gradients.
• Lastly, the set-up we use evaluates explanations based on the order of top-k features (according to absolute value), and does not penalize based on exact feature importance values. We echo the reviewer's points that hyperparameter choices will affect the exact feature importance values returned by methods such as LIME. However, the overall ranking of the features would be affected to a lesser degree (evidenced by the previous point also) when averaged across the full dataset.

We thank the reviewer again for making this point, as it is an important consideration for follow-up work where explanation metrics could be based on values rather than rankings.

Top-k = 1 experiments (Tables 4 and 5)

Thank you for reporting the misleading statement. We found that GPT-4 exhibits non-trivial performance for most important feature identification, and performs on par with state-of-the-art methods in certain settings like the Credit dataset (Tables 4 and 5). Comparing GPT-3.5 and GPT-4 (Figure 6) additionally demonstrates that LLMs are closing the performance gap to post hoc explainers. We deemed this a new frontier as high performance is achieved through prompting alone, a novel approach in our field.

In light of this, we have updated the sentence in the abstract to be clearer, reading: we observe that LLMs identify the most important feature with 72.19% accuracy, indicating promising avenues for further research into LLM-based explanation frameworks within explainable artificial intelligence (XAI).

We hope these points can help to address the reviewer's remaining concerns, and are happy to provide further clarifications if desired.

Thank you for your insightful questions. We are glad that you recognize the importance of this problem and we appreciate your feedback as they have helped us improve our paper significantly. Below, we address all your questions/concerns.

Our method is uninterpretable

While we agree with the reviewer that the inner workings of LLMs are inherently complex and (currently) uninterpretable by nature, it is not the primary focus of our work and is an important point worth exploring in future work. Further, we would like to clarify that the explanations generated by the LLM using our prompting strategy are not uninterpretable or fundamentally flawed. For example, we specifically instruct the LLM to explain its reasoning (see the “Instructions” section of the prompt templates on pages 4 and 5) before reaching a decision on which features are most important to the model.

We would like to note that identifying key features using LLMs is an interesting future direction as our current exploration shows that they achieve non-trivial performance when compared to some state-of-the-art explanation methods. Further, LLMs provide natural language explanations and a level of textual reasoning that is more accessible/interpretable to end users when explaining a model’s feature space. For instance, LIME performs a specific job that does not adapt to any specific patterns it observes in the perturbations, whereas LLMs are not limited in this respect, and have the capacity to adapt their explanation based on the patterns they see in the local neighborhood, which offers an explanation for why GPT-4 is more sample efficient than LIME with 16 input perturbations.

We thank the reviewer for raising these points as we had not clearly articulated them in the original manuscript.

Comparison to Bills et al. 2023

Thank you for your valuable feedback on our paper. We appreciate your insights and the opportunity to address the concerns you raised regarding the interpretability of our proposed method.

Firstly, we acknowledge your point about the interpretability challenge inherent in using Large Language Models (LLMs) as explainers. The complexity and opaqueness of these models do indeed raise valid concerns about the 'black-box' nature of the explanations they generate. However, our work aims to explore the potential of LLMs in a new domain, specifically as tools for post hoc explanation in XAI (Explainable Artificial Intelligence). Regarding the application of LLMs to numerical datasets, we believe this represents an innovative step in understanding the capabilities of LLMs beyond their traditional scope. While LLMs are primarily designed for language tasks, their ability to abstract and generalize can potentially be leveraged in a variety of contexts, including numerical data interpretation.

With respect to the oversimplification of experimental setup in our approach to interpreting logistic regression coefficients. It was a deliberate design choice for initial experiments as we aimed to establish a baseline understanding of LLMs before moving on to more complex scenarios.

Motivation for using LLM instead of other cheaper alternatives

Thank you for your insightful critique regarding the cost-effectiveness of our approach in selecting the most important features. We understand your concern about there being more straightforward and less resource-intensive methods available for feature selection. Our choice to use Large Language Models (LLMs) for this task was driven by our desire to explore their potential beyond conventional language processing applications. The novelty of our approach lies in its ability to integrate contextual understanding and in-context learning (ICL) capabilities of LLMs to provide richer, more nuanced explanations than what might be achievable through simpler methods.

While we agree that there are numerous simpler and more cost-effective ways to identify important features, LLMs are useful for the following reasons:

1. Existing methods often lack the ability to offer detailed, human-like explanations. LLMs can bridge this gap by generating explanations that are more understandable to humans, which is particularly valuable in fields where interpretability is crucial, such as healthcare or finance.

2. Generating explanations using LLMs is not just about identifying important features but also about understanding the rationale behind these selections in a manner that simpler methods may not provide.

3. The ability of LLMs to process and explain complex patterns in data can be particularly beneficial when dealing with intricate, high-dimensional datasets where traditional feature selection methods might struggle.

Thank you for the response. I'd like to follow up on a few points.

Further, we would like to clarify that the explanations generated by the LLM using our prompting strategy are not uninterpretable or fundamentally flawed. For example, we specifically instruct the LLM to explain its reasoning (see the “Instructions” section of the prompt templates on pages 4 and 5) before reaching a decision on which features are most important to the model.

This response seems to conflate interpretability and explainability -- I stand by my original point that LLM outputs are not interpretable. To the claim that they are explainable because the model outputs its reasoning, I again do not believe this is true because of the challenge of faithfulness. Namely, you do not know that the generated reasoning is reflective of the models' internal processing. Notably, faithfulness can be used in the context of reasoning and explanations, e.g. [1,2], and in the context of interpretability, e.g. [3], with slightly different meanings. Notably, just because your language model's generated explanations are faithful with respect to the explained model, that does not mean that they are faithful with respect to the language model. More concretely, if you had solved the faithfulness challenge in language models, that would be more than worthy of a paper of its own.

Our choice to use Large Language Models (LLMs) for this task was driven by our desire to explore their potential beyond conventional language processing applications... The ability of LLMs to process and explain complex patterns in data can be particularly beneficial when dealing with intricate, high-dimensional datasets where traditional feature selection methods might struggle.

One question that still has not been answered is, why? What about language models makes them particularly well suited to this task? No results in this paper suggest that language models are better at dealing with high-dimensional data than other machine-learning approaches. You described my comment as "Motivation for using LLM instead of other cheaper alternatives," but I think "cheaper" misplaces my main concern: we are talking about a type of model that is fundamentally a black box and is well-known to hallucinate - there's no reason to trust that its explanations are correct, faithful, or generalize.

LLM was able to perfectly match the explanations of the LR model SHAP only matched ‘age’ (in the wrong order) and length of stay in the correct position.

SHAP and LIME impose different priors, but I believe few people in the field would describe LIME as generally better than SHAP. More consistently matching LIME heuristically may suggest that the model's predictions are more local but not necessarily better. I think this additional context is useful, but as I noted in the original review, was not my primary concern.

Taking all of this into account, I stand by my comments and my score.

[1] "Faithful Reasoning Using Large Language Models" Creswell and Shanahan 2022
[2] "Faithfulness Tests for Natural Language Explanations" Atanasova et al. 2023
[3] "A Comparative Study of Faithfulness Metrics for Model Interpretability Methods" Chan et al. 2022