Feature Accentuation: Explaining 'what' features respond to in natural images

We thank the reviewer for taking the time to read our article and for the valuable feedback.

“My main concern is I'm not sure what the utility of feature accentuation is.”

Feature Accentuation represents a novel tool that, as you rightly pointed out, yields feature visualizations ranging from more to less naturalistic. In recent years, general feature visualizations have significantly impacted the literature, enabling the understanding of internal mechanisms within neural networks (such as specific filters and invariances)[1,2,3], detection of biases [4], enhanced comprehension of complex concepts [6], and visualization of internal states of intricate models[7].

We have specifically chosen to illustrate the utility of our method with a use-case presented in Figure 9. This example demonstrates how Feature Accentuation can provide valuable insights in explaining failure cases. For instance, in the first image, it might seem perplexing that an image of a barber shop is classified as a moving truck. Our approach sheds light on such cases revealing that the model mistakenly associates the white sheet on the wall with the trailer of a truck.

We posit that Feature Accentuation serves as a superset of traditional feature visualizations, offering the potential to accelerate all those use cases simultaneously. By providing a way to explore a diverse set of explanations ranging from a natural image to purely abstract feature visualization, our method contributes to the broader landscape of interpretability tools.

“couldn't I just pull a random other word and visualize that? For example, I could feature accentuate "matchsticks" or "juggler fire torches", and I'm not sure how any of those visualizations would help me diagnose mistakes in the original model.”

This is an excellent point, and something that concerned us as we were developing the method. Indeed, if we can accentuate arbitrary features, it’s possible the technique could mislead the user, ‘revealing’ how features are expressed in an image when in fact there is nothing in the image that excites the feature. We demonstrate this phenomenon empirically in Figure 6. Given this concern, we demonstrate in section 2.4 how feature accentuations can be combined with attribution masks as a means of filtering, revealing exaggerated versions of the original image only in those image regions that excited the feature in the original. In this way, feature accentuation can help intuit the many different ways natural images excite a feature, without suggesting to the user that everything excites it.

“Have you tried the sanity checks?”

Thank you for this suggestion. We have implemented the randomization sanity check from the paper, and our method successfully passes it. We’ve added the results to the appendix (section E).

[1] Cammarata, et al., "Curve Detectors", Distill, 2020. [2] Olah, et al., "Naturally Occurring Equivariance in Neural Networks", Distill, 2020. [3] Schubert, et al., "High-Low Frequency Detectors", Distill, 2021. [4] Singla, et al., “Core Risk Minimization using Salient ImageNet”, ICLR 2022 [5] Fel, et al., “CRAFT: Concept Recursive Activation FacTorization for Explainability”, CVPR 2023 [6] Ghiazi, et al., “What do Vision Transformers Learn? A Visual Exploration”

• The sanity check increases my confidence in the paper; thanks for running those checks.
• Re 2.4: This seems reasonable. Combining feature accentuation with attribution masks is a possibility, and I can certainly see it mitigating accentuation of random features. (But does the need for one mean that feature accentuation is somehow deeply flawed? You could conversely argue that these two are orthogonal visualization methods that are stronger--and only sensible--when combined)
• I'm still concerned about the utility of the method. Whereas I could intuit the utility of feature accentuation, there are more concrete ways of defining and proving interpretability; [1] above was recommended to me before, which accomplishes this -- mainly by quantifying utility through improved human/model performance on certain tasks. It's currently hard to assess the evidence for feature accentuation's utility, besides just a visual "yep looks good"; for example, I can't claim feature accentuation has been shown to improve debug-ability in some quantifiable way.

After reading through the other reviews and their concerns, it seems like most concerns have been addressed; I've bumped my score up.

We are delighted to hear that you found the paper enjoyable, and we extend our sincere gratitude for investing time and effort in providing a thorough review. We address some of your concerns below.

Formatting, references, image/figure quality

Thank you for your feedback; we have incorporated it into the manuscript. As our paper contains a lot of images we were constrained by the file size limit on our submission, and subsequently compressed the PDF too aggressively. Our undated manuscript is higher resolution, and the camera-ready version will feature the highest possible quality. How novel is the frequency domain parameterisation? citations to other literature should be provided. While the frequency domain parameterization is not a novel concept and has been discussed in [1,2], we have, nonetheless, integrated and tested a more recent parametrization (NeurIPS 2023) that proves effective on newer models [3]. Additionally, we have included several missing citations.

"What does it mean to achieve higher correlations in circuit similarity than natural images themselves? In a positive view, this could mean that the feature accentuation technique is settling onto good prototypes that provide a coherent illustration of the core concept of the label, and thereby reduces variance from natural depictions of the concept. A less positive view could be that this means the technique is finding local minima that will fail to generalize."

Our interpretation aligns with your ‘positive view’, which you articulate quite well (perhaps better than we articulate ourselves in the paper)! With regards to generalization, we are confident this result will generalize to new examples given how it was calculated. Each natural image in our experiment can be paired one-to-one with an accentuation of that image towards its class logit. These accentuations are conditioned in no way on any other images in the experiment. Within a given class, we get the average pair-wise correlation for each natural image, and similarly the average correlation of each accentuation to the natural images (excluding the correlations between accentuations and their natural image pair, which would be trivially high). We find the accentuations correlate higher on average, which means accentuating an image towards its class tends to make it follow a circuit closer to other, randomly selected images of the class.

''Figure 8: what are the smooth curves, and how were their hyperparameters chosen? The underlying data are very noisy, so the choice of smoothing and its associated uncertainty should be reported.''

The curves depicted in these figures are a spline interpolation (degree 2), across the underlying data-points (class-wise average correlations) averaged into 10 bins. We took this approach so as to convey the raw data and the general trend simultaneously. A difficulty with plotting this data stems from the fact that there is a large amount of variance in the correlation measure across nearby layers of different architectural type. We can filter out this variance in Figure 8.a and isolate the effect we are interested in by plotting the difference between the correlation measure for accentuations and natural images in each layer. We have added this version of the plot – which requires no smoothing – to appendix A, as well as additional clarification on the figures and design of the path coherence experiment.

“Figure 8 B, C: lambda should probably be 1.0 not 10.0”

We actually meant for this to be 10.0, as we wanted to measure the effects setting lambda to extreme values.

[1] Differentiable Image Parameterizations, Mordvintsev, et al. Distill, 2018. [2] Feature Visualization, Olah, et al. Distill, 2017. [3] Unlocking Feature Visualization for Deeper Networks with Magnitude Constrained Optimization, Fel, et al. Neurips 2023.

Thank you for your review, you raise some important concerns that we address below.

“All results in the main paper are quantitative (assume means qualitative)”

It’s true that a significant portion of our results focuses on qualitative outcomes, as in some sense feature accentuation is an intrinsically qualitative explanation method; it is designed to produce visualizations a user can qualitatively evaluate to gain intuition for what excites a given feature about an image. Attribution methods and feature visualizations, the standard approaches in this area, are also qualitative in this respect. That said, we respectfully disagree with the notion that our paper lacks quantitative depth. We would like to draw attention to an entire section dedicated to quantitative results (Section 3, circuit analysis, Figure 8 a, b, and c) that we feel compelled to highlight. In this specific section, we argue, akin to the findings presented in [1], that feature accentuations for maximizing an activation should do so in a ‘natural’ way, i.e. by a similar path through the network as the natural images that excite the neuron. Quantitatively, we demonstrate that our method – which does not require additional models that enforce a natural-image prior – achieves excellent results when compared to state-of-the-art feature visualizations on this metric (Figure 8.a). Furthermore, we argue this analysis can be extended to address whether feature accentuations are providing veridical, local explanations for their seed image. That is, a feature accentuation is not a good explanation for why a neuron is excited by an image if the accentuation takes a distinct path through the network. Figure 8.c shows that increasing regularization strength keeps the path through the entire network closer to the seed image. The quantitative evidence presented in Section 3, coupled with our method's independence from additional models, make this work an advancement on the original feature visualizations. We believe this unique aspect of our methodology not only adds depth to the interpretability toolkit but also addresses concerns raised by previous studies.

“The framework is sensitive to hyperparameters such as learning rates and regularization weights, as noted in Appendix C-D, which makes it challenging to adapt to different models.”

Indeed, our framework includes a regularization term designed to keep accentuations close to the seed image while performing activation maximization for a target feature. We posit that this hyperparameter doesn't necessarily require tuning, but rather allows the user to control the extent to which the target feature is exaggerated, a desirable property for VCEs [2]. Additionally, we have added a small experiment to the appendix (section F) that further explores the setting of regularization strength in different contexts. We find that while the appropriate level of regularization must be identified across models, the same regularization strength works well across images/features within a model layer. Further, we present a method for finding the appropriate amount of regularization that does not require visual inspection by the user, thereby automating the selection of this hyperparameter. Regarding other hyperparameters, while we do spend a considerable portion of the paper exploring the hyperparameter space, we ultimately prescribe settings that work well in the general case. With regards to the learning rate in particular, our phrasing in appendix C was too strong (it has been updated), a more accurate statement is that the upper limit for the learning rate in feature accentuation is lower than for feature visualization. As long as the learning rate is sufficiently low, it will work well in a variety of contexts. To demonstrate this, we have added Appendix F, which shows feature accentuations for a range of models. The only hyperparameter that is adjusted in this experiment is the regularization strength, which is set once per model, but not adjusted across different images. All other hyperparameters are fixed across all examples. The regularization layer, which of course can not be identical across models, is simply set to the ReLU layer closest to ¼ depth through each model. “how to interpret the results from Figure 12 in the Appendix?” An interpretation of ‘super-natural images’ like those depicted in Figure 12, can be found in Section 3; they are images generated by feature accentuation that take a prototypical path through the network for their corresponding class. We put Figure 12 in the appendix to limit the number of images in the main paper (as there are a lot already). Still, we presumed the reader would be curious to see what these ‘super-natural’ images actually looked like. We have clarified the figure caption.

[1] Don't trust your eyes: on the (un)reliability of feature visualizations, Geirhos et al., 2023 [2] Sparse visual counterfactual explanations in image space, Boreiko et. al. 2022

Unfortunately, the written review pertains to another article; the abstract is different from our paper.