Noiser: Bounded Input Perturbations for Attributing Large Language Models

Thank you very much for the review and for recognizing the strengths of our work.

I'm not sure the answerability score makes sense. Am I correct that it essentially just measures how diffuse/concentrated the importance scores are (I assume the scores are normalized to sum to 1, is that mentioned somewhere)? If correct, then any of the attribution methods could be transformed to be as concentrated as is desired by raising the scores to a power α and re-normalizing.

You are right. The answerability SCORE evaluates whether the FA puts high attention on the top-k% token or not, only when the top-k% attributed tokens (as identified by the feature attribution method) are semantically sufficient (measured using answerability RATE) for a judge LM to predict the correct output.

In short:

• Answerability Rate: The proportion of cases where the judge model correctly predicts the output using only the top-k% tokens.
• Answerability Score: The aggregated importance scores of these top-k% tokens, reflecting their collective contribution to the prediction.

*** The Answerability SCORE is NOT normalized to sum to 1, as it directly aggregates the raw attribution mass of the selected tokens.

$k$ is used as a variable twice. Once for top-$k$%, and once as the scaling factor for the noise. For clarity, it probably makes sense to change one.

Sorry for the confusion. We will revise the manuscript to use distinct variable names.

Did you try values for k other than 50%? I'd be interested to see if occlusion's performance drops off quicker than Noiser as k approaches 0%.

In Figure 3, we illustrate the average minimum top-k% attribution required for the judge model to maintain the correct prediction across different feature attribution techniques. This should indicate which FA performance drops faster as k approaches 0 (The higher minimum top-k%, the faster the performance drops).

Did you calculate the rate at which the judge-llm can answer the question when you set $k$=1. So, including all the (shuffled) tokens?

We appreciate this excellent suggestion, as it would provide an upper-bound baseline for comparison. Here are the results for Qwen2-0.5B and Qwen2-1.5B, where $k$ is set to 100:

• Qwen2-0.5B (top1-match) → Answerability Score=0.82

• Qwen2-0.5B (top5-match) → Answerability Score=0.88

• Qwen2-1.5B (top1-match) → Answerability Score=0.90

• Qwen2-1.5B (top5-match) → Answerability Score=0.93

We will include the full result in the camera-ready version. We should also point out that the selected tokens are always passed to the judge LM in order.

Thank you very much for the review.

The newly proposed evaluation metric "answerability" purely relies on another LLM, which might reduce the reliance of the evaluation metrics. [...] It would be better to conduct a human evaluation and calculate the correlation between the LLM-based answerability score with the human-based score.

We acknowledge that the choice of judge model and prompts can influence the evaluation results. To mitigate this, we conducted preliminary experiments with multiple judge models (including variations of Llama). While the absolute scores varied slightly, the relative performance rankings of feature attribution methods remained stable. However, our primary goal was to introduce answerability as a complementary metric to faithfulness.

In future work, we will include a more comprehensive analysis of different judge models and prompts to further validate the metric’s robustness. We will strengthen the discussion by comparing it with human evaluations or synthetic benchmarks where feasible.

What is the runtime of this method? For each token, this method requires forward multiple times to find the kmin, which is time-consuming. It would be better to compare the running time between this method and previous baselines.

We monitored the runtime of our method and compared it with other FAs. Attention-based methods are the most efficient in terms of runtime, followed by gradient-based methods that are slightly slower. While perturbation-based techniques require substantially more computational resources. However, our method is significantly more efficient than other perturbation-based methods, such as Value Zeroing and ReAgent, and marginally more expensive than LIME and Occlusion.

Another point worth mentioning is that by setting the number of steps in the binary search algorithm (from 10 in the paper to 5), we can reduce the runtime with a slight sacrifice in faithfulness metrics.

Thank you very much for the review and for recognizing the strengths of our work.

You should provide a discussion on the complexity/computational overhead in computing the attribution scores using your method.

We monitored the runtime of our method and compared it with other FAs. Attention-based methods are the most efficient in terms of runtime, followed by gradient-based methods that are slightly slower. While perturbation-based techniques require substantially more computational resources. However, our method is significantly more efficient than other perturbation-based methods, such as Value Zeroing and ReAgen, and only marginally more expensive than LIME and Occlusion.

Another point worth mentioning is that by setting the number of steps in the binary search algorithm (from 10 in the paper to 5), we can reduce the runtime with a slight sacrifice in faithfulness metrics.

Thank you for your review and for raising important points regarding the clarity of our methodology.

Method (Section 3). [...]si=p(X)−p(Xperturbed|kmin). Where is the dependence on i on the right hand side?

The score $s_i = p(X)−p(X_{perturbed}|k_{min})$ depends on $i$ because $X_{perturbed}|k_{min}$ is the input sequence where only the embeddings of $x_i$ is perturbed, as discussed in line 92. We will revise the text to explicitly highlight this dependence in the camera-ready version.

Method (Section 3) [...] Is each xi a discrete token or is it a continuous input embedding?

Yes, $x_i$ is used in two contexts. We refer to xi as a token or its embedding interchangeably. To resolve this ambiguity, we will explicitly distinguish between tokens and their embeddings in the camera-ready version.

Experiment (Section 4). Section 4.1: In line 10 of the abstract, the authors mention that they use NOISER on 3 separate tasks. But [...]

We refer to the three datasets as distinct tasks due to their linguistic and structural challenges: Known: Focuses on factual completion LongRA: Evaluates long-range dependencies and robustness to distractors WikiBio: Tests open-ended generation

Further, the datasets we used have the following statistics: Known: 1,209 instances, average prompt length ~10 tokens. LongRA: 573 instances, average prompt length ~25 tokens. WikiBio: We followed the ReAgent paper, in which they selected a subset of 22 samples. For each sample, we generate 10 tokens for completion. Therefore, it is equal to having 220 samples.

For Known and LongRA, we filter samples that the target model can correctly predict the gold completion. In Table 7 we showed the number of samples used to evaluate each model given an FA.

Section 4.2: I don't understand how some of these baselines can be applied to transformer LMs. For instance, with the Occlusion method, what does it mean to occlude portions of an input text sequence? [...] Similarly, how is LIME applied in a language modeling setting?

In Occlusion, tokens are masked, and the change in the output probability is considered as an importance score (similarly in computer vision, a portion of the image is blacked out).

LIME is adapted in NLP. It approximates a complex model with a simpler and interpretable surrogate model by training on perturbed samples near a prediction.

As we mentioned in section 4.5, all FAs are implemented using the Inseq library, which is one of the most popular and utilized libraries for feature attribution in the sequence generation research field. We refer the reader to the Inseq library for more details regarding the algorithms.

Section 4.3: [...] What are their minimum and maximum values? What does it mean to "randomly mask parts of the token vector representations" at line 162? [...] In lines 212-214, why report the log ratio against a random baseline instead of just reporting the values for the metrics?

The metrics we used for faithfulness are taken from a recent paper in the field, and therefore, we didn’t describe them in detail and ask readers to refer to the corresponding paper.

These metrics are bounded between 0 and 1.

The masking process perturbs embeddings by randomly zeroing out dimensions proportionally to their attribution scores (Equation 3 in Appendix A). For example, if a token’s importance score is 0.7, each dimension of its embedding has a 70% chance of being retained (30% set to zero).

Using the log-normalization follows the recent paper published in the field (ReAgent paper) to normalize scores across models/tasks and highlight relative performance. Besides, normalizing against the random baseline simply gives a good sense of the effectiveness of each method and indicate better-than-random faithfulness.

Section 4.4: It's not clear to me that answerability as defined by the authors is an appropriate measure of feature importance. For example, [...]

Answerability is designed as a plausibility metric to assess whether rationales are human-interpretable rather than a standalone measure of importance. It complements faithfulness metrics, which directly evaluate how attributions align with model behavior.

Asnwerability SCORE (not to be confused with Answerability Rate) is the aggregation of top-k tokens' scores only if the judge LM can predict the true completion using these top-k tokens (please refer to Figure 1).

In your example, the token “for” is essential for predicting “Lakers” and the FA method must put high importance on it. However, if this is not the case, the judge model is probably unable to predict the true completion and therefore assigns 0 to the answerability score. This is exactly the same for humans, for instance, if we ask a human to guess a completion for [“LeBron”, “James”, “plays”] he/she would be unable to predict “Lakers”. For this reason in Table 6 we show relaxed top-5 candidate evaluation for demonstrating robustness to such edge cases.