ICFI: a Feature Importance Measure For Multi-Class Classification

We would like to thank the reviewer for appreciating the problem in our paper as reasonable.

• Weakness 1) ICFI is a method to explain an already trained machine learning model. After computing ICFI there is no need for further training. In Figure 7 we select the most important k features (for different values of k) for each pair of classes. Each feature selection is then leveraged when training the binary models composing the One vs One evaluation strategy. We want to clarify (here and in the revised version of the paper) that One vs One is only used to evaluate the explanations, but is not part of the ICFI computation as such.
• Weakness 2) ICFI is not a global method in the traditional sense, and indeed outputs feature importance for pairs of classes specifically. As ICFI is the first method that provides such class pair specific explanations, there are no direct competitors. To still be able to assess its benefit, we compare with state-of-the-art global methods and random feature selection as a baseline. For evaluation, we indeed use global feature ranking to choose features for each binary model used in the model retraining experiments. Contrary to ICFI, due to their global nature, PFI, SHAP and LIME (added in the new revised version of the paper) select the same features for each pair of classes (and thus for each binary model). We show that such global method explanations do not effectively capture discriminative features for pairs of classes. This confirms that their global ranking suffers from aggregation bias, and insights at class level granularity are lost due to averaging.
• Weakness 3) LIME’s global explanations aggregate explanations over all instances, yielding a single ranking. As the experiments confirm, this averaging suffers from aggregation bias as features important for discrimination between pairs of classes as identified by ICFI are missed in this global view. Consider a feature highly relevant to discriminate between two classes only, in the global ranking which takes into account every class, its importance may be hidden by features important for a greater number of instances.
• Weakness 4) We have expanded the benchmarking in Section 4.3 by adding comparison with LIME. As for other global methods, the LIME results show that it fails to capture feature importance for pairs of classes, and is clearly outperformed by our ICFI method.
• Weakness 5) Our proposed ICFI method considers single features and computes a feature attribution for each feature individually, as also done by state-of-the-art feature importance methods like LIME. Considering feature interaction is an interesting idea that applies to feature importance measures like PFI and SHAP. Considering different permutation strategies, hinted in the feature works, could also try to tackle the feature interaction aspect.
• Question 1) For the textual data, we use the well known TF-IDF approach to represent for each word its term frequency (how often it appears in a document) weighted by inverse document frequency (how rare the word is across the document collection). TF-IDF values are thus the features' numerical values, and permutation is performed as in the other tabular data sets, i.e., we select one of the N! feature permutations at random.

I'd like to thank the authors for their detailed explanations and response to question 1. Following on your explanations in the second bullet point from your comments, have the authors considered either of these two scenarios?

1- Compare ICFI with global methods for binary classification tasks?

2- Use ICFI under multi-class classification (same way as it is used in the paper) to select top features to train a new one vs. one classifier and then compare the performance of this model vs. a similar binary classifier model that is trained by using top features selected with global methods from an original binary classifier?

Such experiments can shed light into how well ICFI can learn top features for a one vs. one classifier in a multi-class classification setting compared with the ideal scenario that requires more computational power.

To summarize, after reading the authors' comments, I have no further questions related to the first, third, and fourth bullet in my comments under the weakness section. I also have no further questions about my question under the first bullet in the question section. I would also like to change my overall rating from 3 to 5.

We appreciate the reviewer comments, in particular noting that ICFI fills a gap in multi-class explanations.

• Weakness 1 + Question 2) While ICFI uses permutation, as do PFI and SHAP, this is the only similarity (i.e., how we simulate the absence of the feature). Neither of the two methods PFI or global SHAP, for example, merges classes nor takes into account any inter-class relationship. PFI and SHAP average contributions from all the instances to provide global explanations across classes, risking to overlook features discriminative only for a single pair of classes (aggregation bias). SHAP local explanations fail to capture behaviours involving multiple instances. Moreover, SHAP can output explanations only toward one class, meaning for multi-class analysis one needs to resort to a One vs All approach.
• Weakness 2) We thank the reviewer for the detailed comments, which we have fixed in the revised paper version.
• Question 1) The datasets used in Section 4.3 with a relatively low dimensionality have been chosen in order to allow visual analysis, but analyse high dimensional data in Section 4.2, with more than 4000 features. We indeed retrain using a One vs One strategy, increasing the number of features selected by one at each step, more than 4000 features hinder the ease of this visualisation. Analysing more than 4000 features (in section 4.2) shows that ICFI performs on high dimensional data.
• Question 3) PDP does not output a feature importance ranking and is thus not suited for the ranking evaluation in Section 4.3 In Section 4.3 we indeed perform feature selection for every binary model of the One vs One strategy (which are trained only for evaluation purposes, and in a real application are not needed for ICFI), requiring thus a feature ranking to select the most important ones. For completeness, we add retraining using LIME's ranking in the revised version of our paper. As the results show, ICFI clearly outperforms LIME.
• Question 4) We further clarify in the revised version of our paper that the One vs One strategy is purely an evaluation methodology, but not needed to compute ICFI. ICFI is completely model agnostic and can thus be computed on top of any black box machine learning model. The One vs One evaluation strategy is employed in Section 4.3 only to evaluate the quality of ICFI’s feature ranking and to benchmark it against other rankings. This evaluation strategy is the following: train a black box model, compute ICFI for each class combination. Compute global SHAP, PFI, LIME (the latter added in the newest version of the paper), and a ranking based on choosing features at random. ICFI and the random strategy are thus the only two methods outputting a different ranking for every class pair while the other global methods output only one ranking. For evaluation, we then train a binary model for each pair of classes, aggregating predictions using majority voting (One vs One). This is done for the k most important features, for different values of k. Retraining based on ICFI’s ranking thus implies selecting (potentially) different features for every class combination while global method always return the same feature choices (for all class pairs). In the experiments, we show that the most important features globally are not necessarily the most important for discriminating between a particular pair of classes. Global methods show lower performance than ICFI, because averaging causes insights to be lost due to aggregation bias.

We thank the reviewer for the feedback and for appreciating that ICFI is intuitive and interesting.

• Question 1) In order to compute ICFI we do not train any additional model. After the model to explain has been trained, ICFI is computed for each feature and for any selected pair of classes. After feature permutation, ICFI requires only evaluations of the original (black box) model and no further training. Only in Section 4.3, we use a retraining strategy to further evaluate the quality of explanations. Model retraining is used only to evaluate the already produced ICFI explanations here, and is performed when ICFI is already computed, exploring its and the other benchmarked methods’ ranking. We will make this more explicit in a revised version of our paper.
• Question 2) Regarding scalability, ICFI averages the contribution of each instance and thus has a linear dependency on the number of data points. The computation is independent for each feature so a complete ranking on every feature scales linearly also with the number of features (and could be easily parallelized). We would like to point out that this is the same computational complexity as that of PFI. ICFI is computationally much cheaper than SHAP, as we do not need to evaluate every feature coalition but practically just one - for the permutation. To output a global explanation, LIME is computed for each instance and its complexity also depends on the surrogate interpretable model fitted for each single instance. ICFI can also be used on image data, as also done with SHAP; permutation would then be done on more complex features than single pixels, which would likely not affect classification much. E.g. super-pixels are commonly used with SHAP, and ICFI could use them in the same manner. ICFI is computationally much cheaper than SHAP and its application on image data would thus also benefit from a speedier computation compared to SHAP.
• Question 3) ICFI is not a local (instance-based) method and explains the reasoning process of the model regarding two entire classes, finding important features in the task of separating the feature pair. We thank the reviewer for suggesting an instance-level extension of ICFI which could be implemented repeating the permutation we already do for a single instance, with multiple selected permuted samples.