Extracting Robust On-Manifold Interactions Encoded by Neural Networks

Although the problem is interesting, the proposed method is built upon the idea of consistency outputs for different noised versions of input data. This technique has been widely used in deep learning research, for example for images, it has been shown that training with augmented inputs under loss functions tailored to consistency can greatly improve robustness. Many other augmentation techniques, including mixing examples besides adding noise, have also been investigated in this area. Therefore, it is not technically novel or exciting when applying this technique to other optimization objectives for deep neural network including explainable learning.

Other questions:

In Eq. (4), the variable in optimization is $\gamma$ but they are not shown in the equation on the right-hand side of $\min_{\gamma}$. I understand that $\gamma$ determines the values of some variables in the equation, it would be better to indicate how the formula is calculated from $\gamma$ to improve self-consistency.

The motivation is not clearly illustrated.

The prominent contribution is not well-presented. No core contribution is shown in the method.

The utilization of AND and OR interactions is not well constructed.

Clarity

I struggled to understand the key takeaways from the paper. I am not sure what the message is here.

In the introduction, the properties like "(1) Universal matching" are given without proof nor references: I am not sure where in the related work I can find the proof of this fact, and it is not stated like a theorem. For "(2) Sparsity." the paper you cite Ren et al. (2023d) does not seem peer-reviewed.

Fig 2. is unclear, the legend is insufficient.

Link to existing literature

The non-unicity of the AND/OR decomposition is a consequence of the non-unicity of booleans circuits/formula to represent a binary function [%] But the link with this literature is not discussed.
[%] Crama, Y. and Hammer, P.L., 2011. Boolean functions: Theory, algorithms, and applications. Cambridge University Press.

A part of the article is a critique addressed toward the work of Li & Zhang (2023a): "Sparsity does not mean stability." But to the best of my knowledge, the work of Li & Zhang (2023a) has not been peer-reviewed so i am not sure about the impact of criticizing their method.

The approach proposed, which relies on AND and OR relations, is looking at the joint effects / the independent effects of input variables, by relying on masking. This approach is also done in Sobol's indices, but a short literature review is lacking.

• Fel, T., Cadène, R., Chalvidal, M., Cord, M., Vigouroux, D. and Serre, T., 2021. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. Advances in Neural Information Processing Systems, 34, pp.26005-26014.

• Van Stein, B., Raponi, E., Sadeghi, Z., Bouman, N., Van Ham, R.C. and Bäck, T., 2022. A comparison of global sensitivity analysis methods for explainable AI with an application in genomic prediction. IEEE Access, 10, pp.103364-103381.

In order to extract on-manifold interactions, we decompose the network output on each perturbed [...] just like in (Li & Zhang, 2023a),

Once again (Li & Zhang, 2023a) is not peer-reviewed and I am not convinced that their method makes sense.

Claims

Considering that the OR interactions can also be considered as AND interactions from another perspective (please see Appendix A for the proof), we can roughly consider that the OR interactions in a well-trained DNN also tend to be sparse

The hypotheses behind this results are given as a footnote in p1, which make the paper hard to read. The proof in Appendix A does not check the compatibility with the original hypothesis of Ren et al (2023d)

I have the same issue with:

In particular, on-manifold interactions mean that different perturbed inputs have similar sets of interactions, considering it has been proven that each input only has a few salient interactions. Noises on the input may mainly change the effect values of the shared interactions, rather than generate a fully new set of interactions.

Is this a hypothesis of the authors, or a conclusion of the paper? How can we check the result?

Same remark for: "The K different perturbed inputs are supposed to have similar decompositions of output scores". I believe an experiment is needed.

Other references

In the introduction, it is written:

Within the realm of XAI, one of the ultimate questions is whether the inference logic used by a well-trained deep neural network (DNN) can be faithfully explained by a set of symbolic concepts or primi- tives (Zhou et al., 2016; Wu et al., 2018; Kim et al., 2018). This question involves both mathematics and cognitive science. Up to now, people have not reached a consensus on the definition of concepts in DNNs, due to the inherent challenge of formulating concepts in human cognition.

You can add this recent work to the survey:

• A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation T Fel, V Boutin, M Moayeri, R Cadène, L Bethune, M Chalvidal, T Serre, NeurIPS 2023, Advances in Neural Information Processing Systems