From Feature Visualization to Visual Circuits: Effect of Model Perturbation

1. The proposed method is only compared against Push-Up\Down, which limits the scope of the evaluation. Additionally, results for Push-Up and Push-Down on ResNet-50 are missing in Table 2, leaving it unclear whether the performance improvement comes from greater accuracy degradation in ProxPulse. More comprehensive comparisons with a wider range of related methods are needed.
2. The clarity of the visualizations could be improved. Labels and names in the illustrations are too small and difficult to read, which detracts from the readability of the results.
3. The paper does not provide sufficient insight into why visual circuits are robust to ProxPulse attacks. Since the proposed method falls within the existing paradigm of explanation manipulation, the authors should offer a deeper discussion or analysis to explain this phenomenon, which is crucial to understanding the significance of the findings.
4. Although the authors validate their findings with several results, the lack of experiments comparing their method to related explanation attack methods that manipulate visual circuits raises concerns. Without these comparisons, it is difficult to determine whether the robustness observed is a general property of visual circuits or specific to the ProxPulse attack.

1. The overall writing of the article is not concise. It is hard to understand whether it is explaining DNN or proposing a new method for attack tasks.
2. The limitations of existing methods are not clear in the abstract.
3. The introduction section does not clearly describe the tasks that this method is applied to.
4. There is a lack of description of attack methods in related work.

Major Weaknesses/Concerns:

1. The objective of ProxPulse (Eq 5.), while explained to push up the smallest activations, is not very well motivated or explored, why and how is this objective different/better than the push-up baseline in [1]? The constant C is described as “very large”, how is this hyperparameter tuned and what’s the effect on the resulting model as we change it? Why the inverse logarithmic formulation, why the +1? I can only guess for numerical stability. These should all be discussed.
2. Similar to the ProxPulse objective, the CircuitBreaker objective (Eq. 7) is difficult to decipher. The second term is straightforward and the motivation is clearly stated, but the authors describe the first term as the same component in the ProxPulse objective with the purpose of “maintaining the initial feature visualization”. This raises major questions: firstly, why? Isn’t the objective of this loss term to alter the circuit activations? Two, as described by the authors and what seems from the mathematical form of the ProxPulse term, it simply increases small activations rather than “maintains” them. Overall, it is unclear to me what the connection is between ProxPulse and CircuitBreaker and why the first term in CircuitBreaker is included.
3. The CircuitBreaker objective is based on the Attr function described in Eq. 3, which depends on the partial derivative of the activation f. Therefore, optimizing CircuitBreaker as a loss function using gradient descent methods should require computing the Jacobian of the neural network activations, or at the very least, its products with a vector. This is generally considered to be an arduous task due to the high number of parameters. However, I did not find any mention of this in the paper. Perhaps I’m missing something, but I’d appreciate it if the authors clarify how they went about optimizing this objective.

Minor concerns (not affecting the decision):

1. The authors refer to [2] as they employ a norm-constrained maximization for a first-order approximation. It should be noted that [2] employs first-order Taylor approximation for weight perturbation, which is a different objective than perturbing the input. This might be confusing/misleading to readers whether they are familiar or unfamiliar with that paper. I think talking about a first-order approximation maximizer is sufficient, and [2] can be discussed in related works.
2. Line 370 is referring to a missing figure (Fig ??)
3. Some of the CircuitBreaker-related figures contain red outlines for better readability and some don’t. This is cause for confusion, I would suggest adding the red outlines to all the related figures for better comprehension and consistency.
4. The empirical metrics such as kendal-\tau and clip-\delta are adopted from [1] but not explained in detail with equations in the main text or the supplement.

References:

[1] Nanfack et. al. Adversarial attacks on the interpretation of neuron activation maximization.

[2] Foret et. al. Sharpness-aware minimization for efficiently improving generalization.

• Despite proposing novel methods to highlight the vulnerability of feature visualization and visual circuits, previous research has already demonstrated that both types of feature visualizations are susceptible to model perturbations [1,2]. Consequently, the newly proposed attack lacks a fresh message, and the results do not appear surprising or particularly impactful. This limits the paper’s motivation and relevance.

• Given this point, the first part of the paper, which reiterates prior findings on feature visualization vulnerabilities, lacks strong motivation. The second part of the paper, focusing on visual circuits, could stand as a separate work, potentially more engaging for the audience. This section would benefit from further content supporting its claims, such as extending the analysis to non-vision models (e.g., language models) and alternative architectures beyond CNNs.

• Another major weakness is that the proposed attacks are not tested on more robust models, such as those adversarially trained. This limitation raises questions about the effectiveness of ProxPulse and CircuitBreaker when applied to more resilient architectures.

• The study’s focus on vision-based models also limits the generalizability of ProxPulse and CircuitBreaker to other domains, such as language or multimodal networks. Moreover, feature visualization and visual circuits represent only a subset of interpretability techniques, and the proposed methods may not be applicable to the broader range of interpretability methods in machine learning.

• The experiments are conducted on relatively simple CNN architectures and datasets, leading to questions about whether these findings would hold for more complex, state-of-the-art models.

• The presentation includes numerous visualizations but lacks sufficient summary analysis, which can make it difficult for readers to discern overarching patterns across experiments. Additionally, the presentation could be substantially improved: spacing needs refinement, many figure captions are too small, and several images are so tiny they require zooming to be readable. This hinders the reader’s comprehension and overall experience with the paper.

• Although the paper identifies vulnerabilities in interpretability techniques, it does not offer potential defense mechanisms or mitigation strategies, which would make the work more comprehensive and practical.

[1] Geirhos, Robert, et al. "Don't trust your eyes: on the (un) reliability of feature visualizations." arXiv preprint arXiv:2306.04719 (2023).

[2] Nanfack, Geraldin, et al. "Adversarial attacks on the interpretation of neuron activation maximization." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.