The FIX Benchmark: Extracting Features Interpretable to eXperts

1. Lack of popular feature attribution methods: The paper proposes a benchmark that has the potential to improve explanations based on groups of features. However, the benchmarking of methods is significantly under-developed. The paper claims (e.g. in the abstract) that feature-based explanations fall short in identifying the given "expert features". This claim is not sufficiently covered by the experiments. The authors do barely evaluate feature-based explanations. In the experiments, from current XAI literature only Archipelago is considered (and image-specific variants, such as CRAFT). While in the introduction feature attributions, such as LIME or SHAP are discussed, none of these concepts appear in the empirical evaluation. In contrast, the evaluation in Table 2 centers around pre-processing techniques rather than feature attribution methods. For improvement, to support this claim, I would suggest to include a variety of well-established feature attribution methods, such as SHAP, Integrated Gradients, LIME and their variants for higher-order explanations and evaluate their performance on the FIX benchmark. Moreover, it is unclear to me, why the proposed FIXScores reported in Table 2 are insufficient / unsatisfactory.
2. Unclear modeling choices: Given only the datasets, it is difficult to execute a benchmarking of XAI methods, since a black box model is required. The paper does not address the question on how to model these datasets in the first place. A potential improvement would be to clearly discuss the modeling aspect of the datasets, or incorporate the trained model in the benchmark. As of now, it is unclear if any results obtained by the benchmark is due to a bad modeling choice or a bad interpretability method. Moreover, it is even unclear if the "expert features" and actual model reasoning are aligned, which would be crucial for the benchmark (see Q1-Q3 below).
3. Implications remain unclear: The paper in its current state has limited impact on future research, since it does not provide clear evidence of limitations in current XAI methods (see above), nor where these are rooted in or how they could be mitigated in the future. It would be very helpful to clearly evaluate how these methods fail, why, and to propose improvements or directions for future developments.

1. "We propose FIXScore, a unified expert alignment measure With FIXScore, we find that popular feature-based explanation methods have poor alignment with expert-specified knowledge, highlighting the need for new methods to better identify features interpretable to experts." -- Are the authors proposing a call to the community to develop new feature attribution explanation methods? If yes, how does FixScore or expert features help in this regard? Most classification models are trained on certain downstream objectives which do not necessarily reward the model to learn the task using expert features. Assuming complete faithfulness of explanation methods, aren't we arguing that the underlying training paradigm should change?

2. The authors argue that "the main problem is that features at the individual pixel or token level are often too granular and thus lack clear semantic meaning in relation to the entire input" -- While I agree that individual pixels and tokens do not possess any clear semantic meaning, the model was never trained to learn features that align with domain experts, i.e., they are always trained on a given cross-entropy or some other downstream loss function which may/may not align with domain experts. Hence, this argument is not necessarily a symptom of the feature attribution methods but could be implied from the underlying training of the model itself.

3. The authors propose some specific properties of grouping low-level features to high-level features that are more digestible and the low-level features should align with domain experts’ knowledge of the relevant task -- Wouldn't training a model using concept bottleneck models suffice these properties?

4. The authors argue about the importance of feature attribution that is grounded in domain expert knowledge. However, in sec 4.1, the authors use the purity- and ratio-based metrics that are proxies to ground truth information needed to align attributed features. Since these purity and ratio metrics are essentially based on pixel intensities, isn't this another measure of something like the Intersection of Union (IoU), which we use to evaluate feature attribution maps for real-world images and calculate the IoU between the generated importance map with ground-truth object bounding boxes?

5. For datasets like emotion and politeness, getting expert features may be highly subjective. Are there ways the authors are thinking to address this problem? Further, these subjective notions have different levels to it and are mostly context and scenario-dependent. How do these factors affect the underlying expert features?

6. One way to support the significance of the expert features would have been to train a machine learning model on just the features that are deemed to be aligned with domain experts and report the accuracy of the trained model. If the expert features are indeed grounded in domain knowledge, a model should be able to achieve high predictive performance using only those features.

(Minor points)

1. It would be great for the reader if the authors could include the metric calculation of each of the datasets in the main paper as that is one of the main contributions of the paper.

2. The anonymous link to the codes doesn't work.

• Although the paper uses five datasets for experimentation, it lacks a clear rationale for selecting these specific datasets.
• The structure of the paper could be improved - the results and discussions of the results are very brief in the main paper and lack of interpretability. The paper could consider condense the dataset introduction part and add more discussions on the results to better demonstrate the insights.
• There is an inconsistency in line 478, where the paper mentions “three segmentation methods” for image data but lists four: “Patches,” “Quickshift,” “Watershed,” and “CRAFT.” Additionally, Table 2 includes an extra method, "SAM," which is not cited.

• The formulation of the ExpertAlign metric for various datasets appears somewhat ad hoc, lacking clear justification for the chosen metrics. Why were these particular metrics selected over other potential designs, and how do they relate to Formula 2?
• If the proposed metrics were part of a broader methodological paper as a way to evaluate the primary method, they would likely be suitable. However, as a standalone contribution, they seem insufficient.