Is Forward Gradient an Effective Tool for Explaining Black-box Models?

Regarding the methodology:

1. Could the authors clarify how the gradient estimator described by Equation 2 differs from previous work?
2. The claim in line 91 about gradient estimation having "greater scalability" than true gradients is a bit unclear. Given that estimation variance increases significantly when the input feature space expands, should the claim instead be the opposite? Exact gradient measurement (e.g. through backpropagation) is barely affected by the dimensionality of the feature space.
3. The statements regarding the unbiasedness of the LR estimator seem inconsistent. Lines 65-66 state "allowing for unbiased gradient estimation", whereas lines 251-252 say that the LR estimator "introduces a certain bias".
4. Could the authors elaborate more about the condition in Proof 2? Specifically, why does $\nabla_\zeta \frac{\mu_z(\zeta)}{\mu_z(J\odot\zeta)}=0$ hold? How does this condition facilitate the derivation in Equation 8, particularly the equality between the second and the third lines?

Regarding the writing and experiments:

1. As a main concept defining the problem to be solved, how is the black-box setting defined? What is the relationship between "black-box models" and "black-box settings"?
2. Some statements appear contradictory and confusing, for example:
   • Lines 51-53 state that gradient-based methods can explain "decisions from black-box models", whereas line 161 claims that "… the reliance on these gradients poses a significant challenge in the context of black-box models, …".
   • How many forward passes are needed to derive an explanation? There is the claim in Figure 1 "Our proposed method only needs one forward pass of the studied model", yet line 353 states "…, we leverage 1,000 - 96,000 noisy copies …".
3. How are "the most important features" defined in the deletion & insertion test? What is the deletion score, and how is it computed?
4. What is the target class for the examples shown in Figure 7? Does the coloring have a specific meaning?
5. The visual examples in Figures 7 and 10 are somewhat difficult to comprehend. Without a comparison to explanations by other competitors, it is difficult to determine whether the issue lies with the CLIP model or with the explainer itself.

1. See the Weaknesses section.
2. In the three key findings in the Intro section the paper claims "comparable interpretability with greater scalability". What do you mean by greater scalability with the view of the acknowledged limitation of high computation cost of the method?
3. Why is the method called likelihood-ratio? Where / what is the link to likelihood-ratio?
4. What are the input-output spaces of M and S in equation (1). Please complement the notation.
5. In the beginning of section 3.2 the framework is indicated as "unified". In what sense is it unified?
6. The theoretical proves are derived for $\sigma \to 0$. What is the noise variance used in practice in the experiments? Have you explored different noise levels? What was the effect? It would be good to see this analysis in an ablation study.
7. In section 4.1 in paragraph Visualization you claim that the method highlights only one fish for AlexNet and nearly all fish for ResNet and this is a sign of scalability of the method. What do you mean? What does the number of fish has to do with the method scalability?
8. In the last sentence in page 8 you "argue" that your observation aligns well with human understanding. How can you back this argument? Has it been previously shown anywhere or have you verified it yourself (e.g. through human study)?
9. In section 4.4. you present examples over five selected classes. How have these been selected? Cherry-picked?

It appears that the greatest advantage to the LR method is that it is able to provide gradients, and thus gradient based explanations, without using back-propagation, but only model querying. Major questions affecting the efficacy of the method:

• How does LR method compare in computational time and accuracy other black-box methods. A survey should be done, and comparisons should be made to classic numerical differentiation and any other promising competitors (cost, accuracy, etc.)
• This method is approximating the gradient. What is the computational cost to recover the true gradients/gradient methods with high accuracy over a variety of models?
• Where is the likelihood ratio in this method? The paper could be improved by giving a quick intro of a likelihood ratio and how the method employs the concept.

Regarding the improvements of the method: To list other strengths of the method that I noted: 1) provides interpretability, 2) does not require any knowledge of internals of black box, 3) provides improved scalability, 4) explanations smoothed-out noise.

1. Insofar as you are just approximating interpretability methods in a black-box scenario, this is just fine. However, the experiments make it look like the LR method makes a rough and smoothed approximation of the gradients, that the gradients were not recovered with much accuracy, but then the paper claims that this is an improvement. Do you want to approximation the saliency maps, or make an improvement? Either the original gradients and saliency maps should be recovered with sufficient accuracy, or significantly more justification needs to be given on how a deviation from them is an improvement. Other popular methods (IG, GradCAM) spend significant effort to justify why their methods are an effective interpretability method; if they are not recovered but altered so as to look/perform strikingly different, more work should be put into justifying the change (experimentally and theoretically).
2. Fine and good, the major use case of the method. The paper could be strengthened by a small expositing the practical use of this in the beginning.
3. Not sure what scalability means here. Could benefit from being more precise, having more treatment and specificity, an explanation on why this is beneficial.
4. Are you saying that deterministic methods have noise, while random approximations have less? The paper could improve by being precise about what the noise is, how it is removed. Also, I see in some visualizations that some areas with lots of bright spots for the normal method are not lit up for the LR method. This may be a problem for the claim that LR "smooths out" the original method and needs to be discussed.

Some comments on improving experiments:

• Can you provide theoretical or experimental metrics on computational costs/accuracy of your method vs backpropagation and numerical integration? It would be advantageous if the experiments were varied enough to get a generalized understanding of the differences.

• Can you provide time/accuracy graphs showing the effects of noise SD, number of samples used in the estimation, and also explain further/try the "large number of copies" mentioned?

• You claim that the saliency tests are improved because they are less affected by noise and cover more of the feature. Can the author provide experimental results on how many samples/how small the noise has to be so that the actual gradients and saliency maps are recovered with small error? Runtimes and computing specs should be given.

• Method of gradient estimation (theorem 1) follows other works. Those works should be sited, and any differences or novel additions should be pointed out. Likewise for block method of variance reduction.

Minor Comments: 115 - integrate gradients 195 Fig. fig 224 missing a1/2? Ref for corollary (which theorem is being applied) would be nice) 37 large difference(s) Figure 3: which saliency maps are these? What number of samples are being used for the gradient estimation?

I have no significant question. The outcomes are reasonable.