XplainLLM: A QA Explanation Dataset for Understanding LLM Decision-Making

Thank you for taking the time to review our paper and for the constructive feedback. We appreciate the opportunity to address your concerns:

Response to Weaknesses:

1. (W1.1) Clarification on Technical Details:

1. Training of Model M and Model G: Model M is our pre-trained LLM, and Model G is GAT to extract the reason-elements. We use the pre-trained RoBERTa-large as Model M, and the GAT is trained with the Model M (embedding concatenation). They trained with the loss function of cross-entropy loss. The optimization involves fine-tuning both the knowledge representation (via model G) and the predictive components (via model M). The training details can be found in Appendix E.5.1-2. We will revise Section 3.1 to clarify the training details.
2. Evaluation Reason-elements: The reason-elements are the key elements that contribute to the decision-making process of the model. They are extracted from the graph modular, mainly based on the attention weights of the GAT. Therefore, they are evaluated by the accuracy of the decision model. We will revise Section 3.1 to clarify the evaluation of reason-elements.

2. (W1.2) Correction of Typos: We appreciate the reviewer pointing out the typos in our manuscript. We will thoroughly review the document to correct these errors, and ensure that our revised version is free of typos.

3. (W2) Clarification on Model Selection:

The design of Task 2 is primarily focused on exploring the transferability of learned knowledge of one LLM to enhance the performance of another LM. The key points are as follows:

• Purpose of E_why: The E_why serves as a knowledge transfer medium. It encapsulates the reasoning used by the first LLM to arrive at its prediction.

• Transferability via E_why: Feeding E_why to another LLM is not about re-predicting the answer but rather about examining how the explanation influences the understanding and performance of the second LLM. This process tests the effectiveness of the explanation in conveying the reasoning knowledge of the first LLM to another. We investigate the potential of explanations to serve as a knowledge bridge between different models here.

• Impact on Performance: Our experiments demonstrate that by utilizing the explanations generated by one LLM, another LLM can enhance its performance (one example is shown below). With the help of explanations, the second LLM increases its accuracy by 2.7%. The different ratio of predicted answers after transferring is around 2%-4%. The results indicate LLMs can benefit from the reasoning and knowledge encapsulated in the explanations, leading to improved decision-making and understanding.

  Model	Method	Accuracy
  GPT-3.5	Vanilla	72.3%
  GPT-3.5 +explanations	Vanilla	75.0%

4. (W3) Clarification on Dataset Quality: We acknowledge there are limitations in the current design and we have discussed them in the limitation subsection. The quality of KG (which is ConceptNet in our case) and more advanced GNN are important factors that can affect the effectiveness of our method.

Our task is to explain the reasoning process of LLMs. We are different from the typical explanation generation task, which is to generate truthful explanations of certain questions. Such tasks might rely on humans to write explanations for the questions. In our explanation task, we need to extract the reasoning process from the decision model, and "translate" it into human-understandable language. Thus, we use an automatic way to generate explanations.

To ensure the faithfulness of the explanations, we use the key reason-elements extracted from the decision model as the prompt to the generator model. The generator model is more like a "translator" that translates the decision model's reasoning into human-understandable language. The reason-elements are not independent from the decision model. They are extracted based on the attention weights of the GAT. Therefore, they are evaluated by the accuracy of the decision model.

We conducted comprehensive experiments to validate the faithfulness of our explanations. Actually, we have included the reason-elements in the evaluation. We will add more details in the revised version. The evaluation included expert evaluation, human evaluation and automated evaluation to assess the quality of our explanations. Our results show that our explanations are with high quality and accuracy. To reduce the bias, we followed the methodology outlined by Hoffman et al. (2018) [1] for cross-human alignment and evaluation questions design. This approach helps to minimize bias and ensure the faithfulness of our explanations.

[1] Hoffman, Robert R., et al. "Metrics for explainable AI: Challenges and prospects." arXiv preprint arXiv:1812.04608 (2018).

Not yet finished, continue to the next comment...

Dear Reviewer,

As we approach the conclusion of the author-reviewer discussion period, we eagerly anticipate your thoughts on our rebuttal. We have endeavored to comprehensively address the concerns you raised. If there are any remaining issues or queries, we warmly welcome your input.

We appreciate your feedback and look forward to your response.

Best regards,

Authors

Dear Reviewer,

As we approach the conclusion of the author-reviewer discussion period, we eagerly anticipate your thoughts on our rebuttal. We have endeavored to comprehensively address the concerns you raised. If there are any remaining issues or queries, we warmly welcome your input.

We appreciate your feedback and look forward to your response.

Best regards,

Authors

Thank you for taking the time to review our paper and for the constructive feedback. We appreciate the opportunity to address your concerns:

Response to Weaknesses:

1. (W1) Clarification on Model Selection:

a. We selected RoBERTa-Large for several reasons:

• RoBERTa-Large has been widely used in the NLP community and is well-known for its robust performance. Its widespread use in the NLP community makes it a reliable base model for evaluating new methods and approaches. Our explanation quality can be influenced by the performance of the decision model, so we choose a model with good performance as the decision model.
• Our dataset focuses on explaining the reasoning process of LLMs. There is no human-understandable explanation to demystify the reasoning process of LLMs. RoBERTa is a fundamental model in the LLM family, we believe that it is important to start with such a model to establish a baseline for future research.

b. Explanation Generation:

• We chose GPT-3.5 as the generator model because of its advanced language capabilities. While it's true that GPT-3.5-turbo's internal knowledge might contribute to the performance gain, our design tries to minimize its influence. The main advantage of our approach is that it transforms the 'black-box' nature of LLMs into a more transparent and interpretable structure. With our graph modular design, we narrow down the knowledge search space of the LLM, providing a clearer understanding of the model's decision-making process. By using strong constrained instruction and the reason-elements learned from the decision model as our prompt to generator model, we can control the generation content. The generator model is more like a "translator" that translates the model's reasoning into natural, human-understandable language.

As a pioneer study, we believe that our dataset can be further improved and expanded. We will continue to expand our dataset to include a variety of LLMs in our future work. We hope that our work can inspire more research in this area and contribute to the development of XAI, making it more interpretable, trustworthy and human-centered.

2. (W2) Clarification on Our Task: While GPT-3.5 or GPT-4 are powerful in generating human-like text, they are not designed to explain the reasoning process of curtain LLMs. The reviewer mentioned question-answer-explanation triples, such task is different from our task. Our task is to explain the reasoning process of curtain LLMs, which is different from the generation task. Without the knowledge of the decision model, the generator model cannot generate explanations that reflect the reasoning process of the decision model.

Our approach can interpret the reasoning process, and link the reasoning process to a generator model. As mentioned before, the generator model is more like a "translator" that translates the decision model's reasoning into human-understandable language.

Response to Questions:

1. (Q1) Model Selection: First, please refer to the response to W1.1 (We selected RoBERTa-Large for several reasons).

It is important to understand the reasoning process of more advanced LLMs. However, we find there is no human-understandable explanation for LLMs, even for "small"/"not such advanced" LLMs. Our work is a pioneer study in this area. Furthermore, our approach is not limited to RoBERTa-Large. It can be easily adapted to other LLMs. We will keep working on expanding our dataset to include a variety of LLMs in our future work.

2. (Q2) Clarification on Explanation:

First, please refer to the response to W2 (Clarification on Our Task).

Since our task is to explain the reasoning process of curtain LLMs, we need to extract the learning knowledge from the decision model to generate explanations. The use of KGs and GAT for extracting reason-elements ensures that the explanations can reflect the reasoning process of the decision model. KGs provide structured and domain-specific knowledge, which when combined with the GAT, allows us to extract the key reason-elements that contribute to the decision-making process. There are two main advantages of our approach: (1) enhancing the explainability of the decision model's outputs, and (2) reducing the irrelevant knowledge and improving the performance of the decision model.

We want to emphasize that our methodology is universally applicable across a diverse range of LLMs. The foundational elements of our approach, such as the integration of KGs and the interpreting component, are designed with flexibility in mind and are not specific to RoBERTa. These components can be adaptable to the architectures and functionalities of various LLMs.

We are grateful for the opportunity to clarify our work. We hope that our response has addressed your concerns. We look forward to hearing from you.

Dear Reviewer,

As we approach the conclusion of the author-reviewer discussion period, we eagerly anticipate your thoughts on our rebuttal. We have endeavored to comprehensively address the concerns you raised. If there are any remaining issues or queries, we warmly welcome your input.

We appreciate your feedback and look forward to your response.

Best regards,

Authors

Thank you for taking the time to review our paper and for the constructive feedback. We appreciate the opportunity to address your concerns:

Response to Weaknesses:

1. (W1) Clarification on Dataset and Faithfulness:

Our paper primarily introduces a dataset for XAI, and here's how we ensure its relevance and utility:

• Dataset Construction: The core contribution of our work is the development of a novel dataset for XAI. The dataset is designed to introduce a way to dive deep into understanding the decision-making of LLMs, transforming a black box into human-understandable explanations. Moreover, our dataset can be helpful for future research in XAI, for example, debugging a model, evaluating its performance, designing human-AI interactions, etc.
• Clarification on Reason-elements: The dataset is derived from certain outputs of models. The reason-elements generated by the GAT are not used directly as the ground truth for explainability. Instead, it serves as a starting point for generating more refined and human-centered explanations.
• Clarification on Faithfulness:
  • While GATs can be complex, they are not entirely black-box models. They offer a level of interpretability by explicitly modeling the relationships and relative importance between nodes in a graph. This allows us to trace back and understand why certain nodes are given more prominence in the reasoning process.
  • The main advantage of our approach is that it transforms the 'black-box' nature of LLMs into a more transparent and interpretable structure. With our graph modular design, we narrow down the knowledge search space of the LLM, providing a clearer understanding of the model's decision-making process.
  • We conducted comprehensive experiments to validate the faithfulness of our explanations. We included expert evaluation, human evaluation and automated evaluation to assess the quality of our explanations. Our results show that our explanations are with high quality and accuracy. To reduce the bias, we followed the methodology outlined by Hoffman et al. (2018) [1] for cross-human alignment and evaluation questions design. This approach helps to minimize bias and ensure the faithfulness of our explanations.
• Broader Applicability Beyond XAI Model Evaluation: Our dataset is not limited to evaluating XAI models. It has broader applications, such as:
  • Debugging the model: Our dataset can be used to debug a model by identifying the reason-elements that are not aligned with human understanding and expectations.
  • Training and fine-tuning the model: Our dataset can be used to train and fine-tune the model to generate more accurate and interpretable explanations.
  • As a benchmark for future research: Our dataset can be used as a benchmark for future research in XAI. It can be used to evaluate the performance of different XAI models and compare their effectiveness.

As a pioneer study, we believe that our dataset can be further improved and expanded. We hope that our work can inspire more research in this area and contribute to the development of XAI, making it more interpretable, trustworthy and human-centered.

[1] Hoffman, Robert R., et al. "Metrics for explainable AI: Challenges and prospects." arXiv preprint arXiv:1812.04608 (2018).

Not yet finished, continue to the next comment...

Dear Reviewer,

Thank you for your reply. We understand and appreciate your concern regarding the alignment of our dataset with human reasoning and expectations. Your listed task is explanations for potential reasoning of how a correct answer is selected (irrelevant to models). This is not about how the model makes decisions, and is totally different from our task. Our task is to explain the reasoning process of a decision model.

The "explainable AI" is a big topic, and our work here aligns with explainable machine learning, which is not to reflect human expectations, but to interpret and explain machine learning models. Many recent works have been done in this area, such as [1], [2], and [3].

An example of [1] (dataset) from OpenAI, explaining why word "dollars" is generated:

A broad effort is under way to understand what really works in health care, perhaps leading to better value for dollars spent.

Explanation:
Token: dollars
mentions of "American" and related terms

The weakness of [1] is that it can only explain a single word not complete reasoning, and it is still difficult for humans to understand the reasoning process of the LM.

The key idea in this topic is to make the reasoning process of machine models more transparent, interpretable, and human-understandable. Our work is in this direction.

We want to emphasize our novelty: (1) our dataset provides a new way of explaining how LLMs work (via more transparent surrogate components, e.g., reason-elements), and (2) explanations of LLMs' decision-making are more human-understandable compared to other works in this area.

We have conducted a comprehensive human study to evaluate the quality of our explanations (Section 5). The results show that our explanations are human-understandable and can help users understand the reasoning process of the decision model.

We hope this explanation addresses your concerns, and we are grateful for your feedback. Please let us know if you have any further questions or comments, we are happy to address them.

Sincerely,

Authors

[1] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/ paper/index.html, 2023.

[2] Weld, Daniel S., and Gagan Bansal. "The challenge of crafting intelligible intelligence." Communications of the ACM 62, no. 6 (2019): 70-79.

[3] Lai, Vivian, Han Liu, and Chenhao Tan. "" Why is' Chicago'deceptive?" Towards Building Model-Driven Tutorials for Humans." In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1-13. 2020.

Dear Reviewer,

As we approach the conclusion of the author-reviewer discussion period, we eagerly anticipate your thoughts on our rebuttal. We have endeavored to comprehensively address the concerns you raised. If there are any remaining issues or queries, we warmly welcome your input.

We appreciate your feedback and look forward to your response.

Best regards,

Authors

Response to Questions:

1. (Q1) Model Choice:

We selected RoBERTa-Large for several reasons:

• RoBERTa is well-known for its robust performance across a variety of NLP tasks. Its widespread use in the NLP community makes it a reliable base model for evaluating new methods and approaches.
• There are no human-understandable explanations for LLMs. RoBERTa is a fundamental model in the LLM family, we believe that it is important to start with such a model to establish a baseline for future research.

Our current dataset primarily focuses on RoBERTa; however, it's important to emphasize that our methodology is universally applicable across a diverse range of LMs. The foundational elements of our approach, such as the integration of KGs and the interpreting component, are designed with flexibility in mind and are not specific to RoBERTa. These components can be adaptable to the architectures and functionalities of various LMs.

Key aspects of our method, such as the extraction and utilization of knowledge from KGs, as well as the generation and validation of explanations, are inherently model-agnostic. This means they can be readily applied to other LLMs with minimal adjustments. As part of our future work, we plan to extend our dataset to include a variety of LMs.

2. (Q2) Intuition Behind Equation 4:

This equation is used to normalize scores obtained from our human study. In our human study, we aimed to engage participants effectively and obtain accurate evaluation results. To achieve this, we included a variety of questions, each measuring different dimensions of our dataset. Notably, one of these dimensions was 'irrelevance,' where a lower score indicated a better quality. This is opposite to the other dimensions, where a higher score indicates a better quality.

We introduced Equation 4 to address this inconsistency and enhance the readability of our results. The primary goal of this normalization was to align all scores on a common scale where a higher value uniformly indicates better quality.

3. (Q3) Clarification on Reasoning:

Our model's reasoning is very similar to 'why' explanations in the dataset. This is because our model transforms the 'black-box' nature of LLMs into a more transparent graph structure, leading to reasoning in LLMs that is very similar to "why" explanations.

• Transformation to Transparent Structure: The core of our approach lies in converting the opaque reasoning process of LLMs into a more transparent and structured form. This is achieved by integrating the retrieved graph into the LM's reasoning process.
• Narrowing Down Knowledge Search Space: By leveraging KG, we effectively narrow down the knowledge search space for the LM. Our approach helps LM focus on the most relevant and contextually appropriate information, reducing irrelevant or inaccurate reasoning.
• Structured Knowledge for Better Interpretability: The graph provides structured representations of knowledge, which are inherently more interpretable than unstructured data. We utilize this structured knowledge to help LM align its reasoning process more closely with the "why" explanations.
• Guiding the LM's Reasoning Process: The graph not only provides structure but also guides the LM in its reasoning. The LM is constrained to utilize knowledge from the graph, which ensures that its reasoning is grounded in relevant knowledge space.

We are grateful for the opportunity to clarify our work. We hope that our response has addressed your concerns. We look forward to hearing from you.

Thank you for taking the time to review our paper and for the constructive feedback. We appreciate the opportunity to address your concerns:

Response to Weaknesses:

1. (W1) Generalizability of Our Method:

We appreciate the opportunity to clarify the generalizability of the 'why-not choose' of our model.

The 'why-not choose' component of our model is designed to provide explanations for not selecting certain options in a multiple-choice question setting. The intuition behind this design is to provide a more comprehensive explanation for the model's decision-making process. In a multiple-choice question (MCQ) setting, the model's decision is not only based on the correct answer but also on the incorrect options. Therefore, it is important to provide explanations for both the correct and incorrect options.

While it is true that our current implementation focuses on MCQ scenarios, our approach can be adaptable to other formats. One of the key aspects of our methodology is the comparative analysis of different possible answers or outcomes. This can be applied to scenarios beyond MCQs, such as:

• Binary or Open-Ended Questions: By slightly modifying the approach to compare the selected answer with potential alternatives or reasoning paths. For example, in a binary question, the model's decision can be compared with the alternative decision. In an open-ended question, the model's answer can be compared with the counterfactual answer.
• Decision-Making Scenarios: Where the model can explain why certain options were not chosen based on a set of criteria or data points.

We acknowledge that extending our 'why-not choose' approach to scenarios beyond MCQs is an interesting area for future research and development. We are actively exploring ways to adapt and apply this methodology to a broader range of question types and decision-making processes.

It is important to highlight that the primary objective of our dataset is to explain the reasoning behind the LM's decision-making process. Our dataset is designed to enable general users to easily understand this process, which can be helpful for future human-centered applications. We believe that our paper makes a substantial contribution to the field by addressing a critical need for transparency and explainability in AI decision-making.

2. (W2) Clarification on Main Text:

We appreciate your attention to detail and agree that clarity in mathematical expressions is crucial for the understanding of our work. We apologize for the typos in our equations. We will address this issue as follows:

• We will fix the typos in the revision.
• Due to the page limit, we moved the detailed mathematical expressions to the appendix. We will revise the main text to provide a high-level overview of the mathematical expressions and refer the readers to the appendix for the detailed expressions.
• To provide a clear understanding of the mathematical expressions, we will add more details of how the message is computed in the revised version. The message is computed by the node feature embedding, node embedding, and relation embedding, with a linear transformation. It is designed to capture the information of the node and its neighbors. The attention mechanism is used to aggregate the messages from neighbors. We will add the details in the revised version.

Not yet finished, continue to the next comment...

Dear Reviewer,

Thank you for your reply. We appreciate the opportunity to clarify this further.

In our paper, we have a decision model, and a generator model. Our task is to explain the decision-making process of the decision model. I am assuming that you are referring to the generator model when you say "the LLM". The generator model is used to generate explanations based on the key reason-elements extracted from the decision model. The generator model is not used to make decisions, it is like a "translator" that translates the decision model's reasoning process into human-understandable explanations. More specifically, the "why explanation" is reflected how the decision model reasons, it should be "very similar" to the real decision model's reasoning, not the "generator model".

Under our strong constraint, the generator model will NOT reason the question and answer or make decisions. We will add this clarification in the revised version.

Our work aims to make the reasoning process of LLMs (decision model) more transparent and interpretable, acknowledging that the explanations are approximations designed to be as faithful as possible to the LLM's (decision model) outputs.

We hope this explanation addresses your concerns, and we are grateful for your feedback. Please let us know if you have any further questions or comments, we are happy to address them.

Sincerely,

Authors