Enhancing Integrated Gradients Using Emphasis Factors and Attention for Effective Explainability of Large Language Models

Weaknesses

Comparison with Scaled Attention (α∇α):
We acknowledge that including scaled attention (α∇α), which combines attention and gradient information, would provide a stronger benchmark. In future work, we will incorporate α∇α to better assess AIEG’s strengths.

Experimental Scope (Datasets):
Evaluating on SST-2 and IMDb limits the generalizability of our findings, especially given LLMs’ diverse applications. While these are standard benchmarks, we plan to extend future evaluations to broader tasks, including reasoning (e.g., GLUE, BoolQ, HellaSwag), summarization (e.g., XSum), and multimodal datasets to better demonstrate AIEG’s utility.

Model Diversity:
We focused on GPT-2 small, GPT-nano, and LLaMA models due to computational constraints. Including larger models (e.g., GPT-2 medium/large, LLaMA-7B/13B) and diverse architectures (e.g., RNNs or LSTMs) in future studies will ensure a more comprehensive evaluation and validate scalability.

Ablation Studies:
A detailed ablation study isolating contributions of the emphasis factor, attention mechanism, and their combination is needed to clarify their roles. While Section 5.2 provides some insights, a deeper analysis is a priority for future work.

RNN vs. Transformer Examples in Section 2:
We used RNN examples in Section 2 for historical relevance but acknowledge their misalignment with Transformer-based models like GPT and LLaMA. Future revisions will replace RNN examples with Transformer-based ones to better align with evaluated models while demonstrating how AIEG generalizes across architectures.

Impact on Downstream Tasks:
Case studies showing AIEG’s impact on tasks like model debugging, bias detection, or reasoning (e.g., BoolQ, HellaSwag) would strengthen its practical relevance. We plan to explore these applications in future work.

Questions

Intuitive Explanation for Emphasis Factors:
The emphasis factor captures significant changes in model logits, focusing attributions on critical decision points and filtering irrelevant gradients. We will include clearer explanations and examples in future revisions.

Sensitivity to α and Emphasis Factors:
A sensitivity analysis for α and emphasis factors will help evaluate their impact across datasets. We aim to provide guidance on hyperparameter tuning in future work.

Extensibility to Other Gradient-Based Methods:
The emphasis factor can extend to other methods, e.g., refining Saliency Maps or SmoothGrad. We plan to explore these extensions to enhance attribution techniques.

Practical Use Cases:
Examples like bias detection (e.g., analyzing attributions for politically sensitive terms) or debugging misclassifications could highlight AIEG’s utility. These will be explored in future iterations.

Choice of k for Masking:
We used k=20% based on standard practices, with Section 5.2 showing consistent performance across k values. This will be emphasized more clearly in the paper.

Weaknesses:

Human Assessment for Adjudicating the Effect of AIEG:
We acknowledge the absence of human evaluations to assess the qualitative impact of AIEG on token importance. Conducting human studies, such as comparing AIEG attributions with expert judgments, would provide deeper insights into its efficacy. This is a valuable direction for future work.

Statistical Significance Tests:
The omission of statistical significance testing for the reported metrics is noted. We will include appropriate statistical tests, such as paired t-tests or bootstrap analysis, in future experiments to substantiate our findings and ensure their robustness.

Examples Highlighting Emphasis Factor (EF):
While the current paper explains the role of the EF in stabilizing AIEG calculations, we recognize the need to provide illustrative examples showcasing its importance. These examples will be included in subsequent revisions to highlight its contribution to improved attribution stability and effectiveness.

Questions:

Line 081: Sensitivity Axiom:
The sensitivity axiom ensures that attribution methods assign non-zero importance to features that influence the model’s predictions. For instance, if a feature change alters the output, its attribution should reflect this influence. This axiom is adhered to in AIEG by leveraging the emphasis factor, which identifies decision-critical regions for attribution.

Line 474: Use of Quotation Marks:
We will correct the use of opening brackets to `` for consistency and clarity in formatting.

Section 5.2: Quantifying Attribution and Human Evaluation:
Attribution quantification can be assessed using metrics like comprehensiveness, sufficiency, and log-odds masking. In AIEG, the emphasis factor aims to enhance these metrics by improving the fidelity of attributions. While no comprehensive human evaluation has been conducted, we recognize its value and plan to include such assessments in future studies to compare AIEG against baseline methods like IG and scaled attention (α∇α).

We appreciate the insightful feedback and will incorporate these improvements in future iterations of the paper.

Response to Weaknesses and Questions:

Poor Writing and Formatting:
    Citation Formats: We acknowledge the citation errors in lines 30, 35, and 39. These will be corrected in the revised manuscript for consistency with professional standards.
    Figures: Figures will be updated to vector graphics to ensure clarity and readability, even when zoomed.
    Spacing and Quotation Marks: Formatting issues, such as improper spacing (e.g., line 75) and incorrect quotation marks (e.g., Figure 2 caption), will be addressed to enhance presentation quality.

Unclear Motivation and Relation to Recent Work:
    The motivation for this work is to address the limitations of standard Integrated Gradients (IG) for autoregressive models, which remain relevant across RNNs, LSTMs, and Transformers. While the approach builds on a 2023 enhancement, we will expand the introduction to better connect our work to recent advancements and highlight its relevance to ongoing trends in attribution analysis for large language models (LLMs).

Responses to Questions:

Fine-Tuning vs. Zero-Shot Analysis for LLaMA:
Fine-tuning was used to ensure robust evaluation on specific datasets under controlled settings. While zero-shot performance is insightful for assessing model generalization, fine-tuning enables a direct comparison of attributions under comparable training conditions. Incorporating zero-shot analyses for models like LLaMA is a valuable suggestion and will be explored in future work.

Results for Models Other than GPT-2-small:
Due to space constraints, Figures 4 and 5 primarily highlight GPT-2-small results. Results for other models, including GPT-nano and LLaMA, are discussed in Sections 5.1 and 5.2. For clarity, future revisions will include these results in supplementary materials or additional figures to provide a broader perspective.

We appreciate the detailed feedback and will address these points comprehensively in the revised version.

Purpose, Coherence, and Precision: The paper clearly outlines its objective: addressing the limitations of Integrated Gradients (IG) for autoregressive models, including RNNs, LSTMs, and Transformers. Arbitrary choices are minimal and informed by prior work or standard practices.

Section 2 - Gradients and RNNs:
The RNN example demonstrates how standard IG fails for autoregressive models, including Transformers. This foundational insight is applicable across architectures.

Section 3.1 - Theorem:
The theorem provides context-specific adaptation of the chain rule for RNNs in attribution analysis. While the output is a probability distribution, embeddings are intermediate representations crucial for gradient-based methods.

Section 4 - Attention Axiom:
We agree higher attention values do not always imply greater importance. AIEG addresses this by considering gradients alongside attention values, ensuring robustness against such discrepancies.

Section 4 - Eqs 11–13:
PosNorm is not recursive but a straightforward normalization step for AIEG values.

Section 4 - Theorem 1 (Vanishing Gradients):
The current emphasis factor primarily handles exploding gradients. Addressing vanishing gradients is part of planned future work.

Section 4 - Theorem 2 (Attention and Decision Regions):
The theorem applies to any attention head. Decision regions are identified by rapid changes in logits, signaling the model’s decision points, as described in detail (Walker et al., AAAI).

Section 5 - Choice of kk:
We acknowledge the suggestion to explore smaller kk values and plan to include this in future work.

Fine-tuning Autoregressive Models as MLMs:
This step allows adaptation for token-level attribution tasks, ensuring robust evaluation. Details of hyperparameters and fine-tuning procedures will be clarified in revisions.

Testing Procedure Precision:
We will provide explicit details about testing categories and data preparation in the revised version.

Typos and Minor Issues:
Typos, citation errors, and display issues (e.g., Figure 1) will be addressed. The introduction of the attention mechanism will be attributed correctly.

We appreciate the detailed feedback and will incorporate these clarifications and improvements in future iterations.