CHiQPM: Calibrated Hierarchical Interpretable Image Classification

Thank you for your review, acknowledging several strengths and bringing up important points to discuss. As some arguments build on each other, we would like to answer the questions with answers building on each other.

How does CHiQPM compare with prototype methods like ProtoTree [Nauta et al., 2021]? What are the key advantages of your approach?

The distinction to prototypical methods is a crucial point so we would like to elaborate. CHiQPM offers global interpretability as it has interpretable class representations that are built from shared, general, contrastive and diverse features which empirically align well with human attributes on CUB-2011. Therefore, one can understand the class representations of a CHiQPM globally. For example, the CHiQPM explained in Figure 1 represents a Bronzed Cowbird as:

• A Bird with a red eye
• A Bird with head and belly of black birds
• A Bird with the neck of black birds

In contrast, Prototypical methods represent classes via prototypes which typically (barring PIP-Net) are exact patches from the training data. The crucial difference is that these prototypes are class specific. This originates from training the feature (or prototype) extractor with classification loss, while allowing it to learn at least one prototype per class. Consequently, the class representation of a prototypical method for a Bronzed Cowbird might be:

• A Bronzed Cowbird
• … (They typically have significantly more prototypes assigned per class. Depending on the model, the prediction either depends on just one class-specific feature, or all/most assigned prototypes are class-specific.)

When judging prototype visualizations, humans typically suffer from confirmation bias and think that the model indeed represents the class with the human concept, while it is a class detector. This can be quantified via Class-Independence [35], with many prototypes of PIP-Net being only (> 99%) similar to samples of just one class. Additionally, human studies have shown that humans cannot predict the presence of such prototypes [16,19]. Figure 7 in the ProtoTree paper [31] itself illustrates this very effectively: When the height of the ProtoTree enables one prototype per class (height = 8; $2^8 = 256 > 200$ = number of classes in CUB-2011), the accuracy jumps to a competitive level and does not increase further with increasing height. Notably, even with more allowed prototypes at a height of 9, allowing $2^9 = 512$, ProtoTree still just learns about one per class with 201.6 (see Table 2 in the paper [31]). Finally, ProtoTree learns a deep decision tree, which might seem similar to our hierarchical explanations. However, as its features are not general, the usefulness is very limited.

CHiQPM follows the line of Work of Q-SENN [34] and QPM [35], where classes are represented with shared general features instead of class-specific ones to enable global interpretability. Crucial elements for that are to learn less features than there are classes in the dataset and to ensure that features are equally important for multiple classes. Therefore, typically no more than 50 features are learned, while their weights to classes are binary in QPM and CHiQPM. That way, one feature on CUB has to activate on general, shared concepts of its assigned classes and cannot detect class-specific properties. As shown in [35], this strong restriction on the space of learnable features already causes the features to become class-independent and contrastive.

We thank the reviewer for highlighting this crucial point. We agree that clarifying the distinction to prototypical methods is essential for understanding the significance of our work. We will add this detailed comparison to the appendix and emphasize the core differences in the main paper.

The model heavily relies on learned mid-level features, which are not guaranteed to correspond to human-interpretable concepts. While the authors introduce a contrastive objective and grounding loss, these only encourage sparsity but not semantic alignment. This leaves the interpretation paths inherently fragile and possibly misleading.

How is the interpretability of local features ensured? While contrastiveness is enforced, it does not imply semantic consistency. How do we know the activated features correspond to concepts meaningful to humans?

As we discussed in the limitations Section J, it can indeed be considered a limitation of CHiQPM that it is not enforced to learn human-interpretable concepts. Instead, it by-design learns general and contrastive features that classify the dataset well. However, the feature alignment metric quantifies that CHiQPM empirically does learn features that correspond to concepts meaningful to humans, with an average increased feature value of 3.8 times its mean when the most aligned concept is present (Table 3). It seems that the concepts that humans have annotated for the birds in CUB-2011 are among the most effective general and contrastive concepts, which is why CHiQPM learns to detect them. Notably, this metric also verifies the qualitative impression that the saliency maps of individual classes give. For example, the features 7,8 and 17 of Figure 1 indeed return high alignment values of 2.2, 3.9 and 2.5 for the attributes Red Eye Color, Black Belly Color and Black Throat Color their saliency maps indicate. Please note that the novel Feature Grounding Loss improves the alignment with these attributes, but as you correctly mentioned does not generally prefer the hard-to-define category of human-interpretable concepts, but rather further encourages general contrastive features.

Finally, we would like to point out that restricting the learnt features to relevant concepts that are known to mankind a priori is not necessarily an advantage, as discussed in Point 3 of Section J. The open setting of learning general features that are shared (or the main difference) between classes could lead to scientific discovery, as one can learn where patterns are that are suited for differentiation.

Moreover, relying on a small set of local features per class may hurt performance, especially if global context is important.

We acknowledge that it might be possible that the few features per class become too few in certain conditions. However, previous and this work show that the accuracy on already quite challenging datasets tends to saturate after 5 features per class. For reference, on CUB-2011, 3 features per class already seem sufficient for CHiQPM:

We further want to clarify that we do not enforce local features, but rather bias the model towards learning such features via the Feature Diversity Loss and feature selection bias in the QP. However, if a more global feature is required, it is still learnt, such as Feature 8 in Figure 1. We would also like to point out that CHiQPM performs remarkably well on ImageNet, where the scale of features is drastically varying across images and prototypical models with prototypes restricted to one patch are generally not even converging. We will include this ablation in the appendix and add your point as limitation.

The proposed framework assumes that similar classes will share a few features. However, for more abstract or complex classification tasks, a larger number of features per class may be necessary. As the feature pool expands, feature overlap between classes may decrease, undermining the construction of hierarchical structures and weakening the interpretability claims.

What is the generalization ability of the model structure? If applied to a complex dataset with highly abstract or noisy classes, how would CHiQPM perform? What happens if class pairs do not share enough features to construct a meaningful hierarchy?

We consider ImageNet as already quite complex with very noisy labels [1] and deduce that CHiQPM’s predictive performance generalizes very well to such a complex dataset with just 5 features per class and 50 in total. However, as described in Section J, the interpretability of the features does not scale as well. For a more thorough discussion, please see our first response to Reviewer7M7N.

As discussed in Section 3.1, 4.3 and D, the tree density should ideally be set using the calibration data, as it is hard to know how many class pairs share sufficient concepts. If that analysis determines that $\rho$ should be set to 0, the resulting CHiQPM will lose some of its interpretability, but will still be the most accurate model with global interpretability. Nevertheless, as mentioned in the Limitations (lines 750-753), if classes truly share nothing or are very few in total, CHiQPM is not applicable.

The paper lacks analysis of how the depth and width of the explanation trees vary under different configurations (e.g., feature pool size or class granularity). Such ablations are crucial to understand how robust or shallow the constructed explanation paths truly are.

The depth of the explanation tree generally equals the number of features assigned to each class, as every activating feature is part of the tree. We understand the width of the tree as the number of leaf nodes in the tree and have measured that width of the total explaining tree as in Figure 23 and the tree below Level 1, as e.g. in Figure 2. As one expects, with increasing number of features, the resulting trees become smaller, as fewer classes are assigned to each individual feature. We will add this ablation to the appendix.

We hope these clarifications and new results address your concerns.

[1] Shankar, Vaishaal, et al. "Evaluating machine accuracy on imagenet." International Conference on Machine Learning. PMLR, 2020.

Thanks to the authors for their detailed rebuttal. However, several concerns remain.

1. While I appreciate the clarification regarding the distinction between CHiQPM and prototype-based methods such as ProtoTree, the response revolves the depth and structure of the hierarchical explanations. However, I have found that many prototype networks have already explored hierarchical structures for global interpretability [1, 2, 3, 4]. Many of these approaches also construct semantic trees or hierarchies, and I struggle to see a clear and significant advantage of CHiQPM over these alternatives.

2. I fully agree with the authors that learning human-understandable features should not be seen as an inherent advantage. However, hierarchical structures are precisely valuable because they simplify the explanatory process for humans. Therefore, I find it contradictory that the paper highlights hierarchical interpretability as a major contribution, yet downplays the necessity of human interpretability when discussing feature semantics. If we argue that interpretability for human users is no longer essential, then the motivation for building structured and traversable explanations weakens. One may then question whether interpretability-focused frameworks are justified in the first place, or whether we should instead focus on purely robust and reliable predictive models.

3. I appreciate the additional results and clarifications. However, they seem to raise further concerns. For instance, the fact that prediction accuracy remains stable across different numbers of features suggests that the learned structure behaves more like a general-purpose feature learning module rather than a model with tightly constrained, explanation-driven representations. This undermines the core interpretability guarantee. While I acknowledge the general challenge of scaling such methods to complex datasets (e.g., ImageNet), this should also be explicitly discussed or solved.

Overall, I encourage the authors to pursue this promising line of work further, and would appreciate more efforts that strike a good trade-off between model performance and strengthen interpretability guarantees, as well as more clearly show advantages over existing methods.

[1] Hase, Peter, et al. "Interpretable image recognition with hierarchical prototypes." Proceedings of the AAAI conference on human computation and crowdsourcing. Vol. 7. 2019.

[2] Yang, Peiyu, Zeyi Wen, and Ajmal Mian. "Multi-grained interpre table network for image recognition." 2022 26th International Conference on Pattern Recognition. IEEE, 2022.

[3] Gulshad, Sadaf, Teng Long, and Nanne van Noord. "Hierarchical explanations for video action recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[4] Yu, Zhen, et al. "Hierarchical skin lesion image classification with prototypical decision tree." npj Digital Medicine 8.1 (2025): 26.

Thank you for your positive review, acknowledging several strengths and posing interesting questions. We will of course try to improve readability at the points mentioned and answer each question separately.

Are there other hierarchy construction choices?

Our used hierarchy based on order of the features of the predicted class has significant advantages compared to other choices. Its sample specific nature lends itself to a local explanation, as one can consider the order of the features as how dominant they appear in the image. Additionally, that enables traversing up the hierarchy in a meaningful way, since the least certain features get omitted first, causing accurate and efficient sets at each step. We believe that the main alternative is a class specific fixed order of features, as that would enable a global hierarchical explanation for each class. The comparison between a fixed order of features for each class, computed based on the average order on training data and our prediction up the hierarchy is shown below. Evidently, predicting with such a fixed order causes less accurate inefficient sets compared to our sample specific dynamic hierarchy. For example, our dynamic hierarchy reaches 91.6% accuracy with an average set size of 2.1, while the fixed hierarchy never reaches that accuracy. We believe that this is likely due to the unique feature, e.g. the red eye of the Bronzed Cowbird, being on average quite important to that class and thus causing prediction sets that are too specific to the top-1 prediction. This experiment showcases the superiority of using our sample specific dynamic hierarchy and will therefore be added to the appendix, as it further validates our method. Many thanks for this suggestion which is further strengthening our contribution.

Comparison to fixed order of features per class based on average order of Features:

Sample Specifc Dynamic Feature order as used by CHiQPM:

Static Feature Order based on Average Order of Features on Training Data for each class.

Grounding is measured via concept annotations; however, do humans actually find the hierarchical paths helpful? A small-scale user study (e.g 20 domain experts evaluating 100 samples) could validate the claimed usability of the explanations.

While we agree with the reviewer that such a user study would be very valuable, we are unfortunately unable to accommodate that in this timeframe. We will, however, add it to a newly added future Work section in the appendix.

Are there limitations regarding class imbalance?

Thank you for this intriguing question. We believe that CHiQPM can be more robust to class imbalance than baseline methods for two reasons:

1. The quadratic problem is solved with no weighting based on prevalence of the class in the training data.
2. As underrepresented classes get the same number of feature assignments, the class will be predicted if all its general features are activating.

However, for that to work, the assignment between features and classes has to be good. When some classes are underrepresented, the constants in the QP might not be good enough to obtain an assignment well suited for fine-tuning. Empirically, these effects seem to balance out, indicating no quantifiable limitation regarding class imbalance. We calculated the Pearson correlation coefficient between number of training samples and test accuracy for each class on the slightly imbalanced datasets Stanford Cars and ImageNet. As expected, there is a positive correlation between the number of training samples on both datasets, with a negligible reduction for CHiQPM. We will add this discussion and experiment to the appendix. Additionally, we add robustness to class imbalance as a direction for future work as we believe in the aforementioned potential. Future work might include considering the number of samples per class during the QP as a form of certainty or use some additional form of supervision, e.g. via the class names, to make the constants more robust.

Pearson Correlation Coefficient between number of training samples and test accuracy per class:

What is the sensitivity to $\lambda_{feat}$ and maximum depth of the hierarchy? Please add a grid.

Thank you for that question, as we indeed missed including the ablation. The maximum depth of the hierarchy is the number of features assigned to each class $n_{wc}$. Therefore, we swept $n_{wc}$ and $\lambda_{feat}$ and report accuracy, sparsity and alignment. As generally expected, $\lambda_{feat}$ improves alignment and sparsity across various sparsities. Additionally, features become sparser and more aligned with decreasing $n_{wc}$, as they must activate on fewer samples and can therefore learn less abstract shared concepts. We gladly add these grids for their valuable insights to the appendix.

Accuracy:

Sparsity:

Feature Alignment:

Thank you for your review and appreciating the novelty and soundness of our paper. We also thank you for the valuable feedback regarding clarity. We will revise the paper to make it more self-contained, particularly by providing more background on QPM [1] and how it differs from prototypical methods, as also suggested by Reviewer 7DuL. Regarding the questions:

Please provide some discussions around scalability of the proposed approach.

We gladly elaborate on the scalability of CHiQPM. As shown in Table 1, our model scales exceptionally well to the large-scale ImageNet dataset, achieving a remarkable accuracy of just 0.8 percentage points less than the dense, non-interpretable baseline. This performance is achieved using just 50 features for all 1000 classes. As we described in Section J, scaling to a dataset of this magnitude introduces certain limitations for interpretability. For instance, the learned concepts become more abstract, saliency maps less faithful, and some features exhibit polysemanticity. Nevertheless, our results show that CHiQPM successfully identifies the high-level features needed for accurate classification. Finally, the time it takes to solve the QP scales exponentially with the number of features to select [1]. But remarkably, just 50 features are sufficient for the very complex ImageNet dataset, making this a very manageable limitation in practice. In conclusion, CHiQPM demonstrates robust scalability. While its interpretability does not scale as excellently as its accuracy, it still offers the best global interpretability on ImageNet. We will summarize these limitations in scaling in a dedicated section in Section J.

the evaluation miss some common metrics for local XAI, e.g., delete/insert scores, Pointing Game;

Given the proposed method can do both global and local explanations; if the author consider to compare with some common XAI benchmarks, e.g., OpenXAI.

We would like to point out that we do not introduce a new saliency map method whose faithfulness should certainly be evaluated using delete/insert or similar [2] scores, that alleviate some OOD Problems of the former metrics. Instead, the proposed method mainly offers improved local interpretability as one can provide saliency maps via any of the available methods for each of the individually meaningful, general and diverse features separately, as well as via the novel hierarchical explanations.
However, as visualizations of the activations of individual features are shown, it is a good idea to evaluate their faithfulness. Following your suggestion, we evaluate a form of the Pointing Game, similar to the initial form in [3], using segmentation masks provided for CUB200 [4]. Specifically, we calculate the fraction of the activation of the GradCAM saliency map that is focused on the segmented bird S.

$\phi = \frac{GradCAM}{\sum GradCAM} \odot S$

As the bird can be considered the region of the image responsible for the classification, a higher overlap with the segmentation indicates that the saliency maps localize more faithfully. Importantly, one would not necessarily expect an overlap of 100%, as the edge region is relevant to describing the shape, causing activations both on and off the segmentation. Across the entire test dataset, CHiQPM’s GradCAM focusses to 83.4% on the bird, whereas the Dense baseline only does so with 75.8%. Hence, CHiQPM has activation maps with improved faithfulness on CUB-2011, validating their use for saliency maps of individual interpretable features. We will include this experiment in the appendix.

In practice, can the authors give some examples of how a conformal set of features is better than, say, the use of top-1 or top-k features for decision-making.

We gladly elaborate on why using our approach is preferential to using top-k features for decision-making. We understand “conformal set of features” as the set of features $\hat{F}_i^c$ (Eq. 8), from which the descending classes in the hierarchical explanation are predicted, e.g. {Green, Blue} in Figure 2. There are two main differences to just using top-k features for decision-making.

1. The conformal set of features is determined via conformal prediction which is considered an effective tool for supporting experts [47,48]. The level in the hierarchy is determined dynamically for each test sample via our novel interpretable conformal prediction method to guarantee that the target coverage is met. That way, the conformal set of features also transports certainty via its set size. In contrast, predicting with a fixed k, e.g. top-1, also returns large inefficient sets, when the test sample can be easily classified.
2. Practically, using top-k features is not suitable for building a hierarchy, as those top-k features might not belong to any one class together. This also causes top-k to be slightly less efficient than predicting with a fixed level n in the sample specific hierarchy. To demonstrate this, we compare between predicting with top-k features and at level n.

Note that predicting with top-1 feature will always return the large set of all classes with that feature which is even less accurate than predicting at Level 1 as there simply might be no class that has the remaining dominant features apart from Top-1. A concrete example can illustrate this failure case: A male bird of the species “Scarlet Tanager” is red, while females are olive-green. The feature "red" might be the top activated feature for a male, but CHiQPM may not use "red" to represent the class Scarlet Tanager because it's not a consistent feature. Thus, relying on the top-1 feature "red" would lead to an incorrect prediction. We will add this experiment and discussion to the appendix.

We again thank you for your review and suggestions that strengthen our paper.

[1]: Norrenbrock, Thomas, et al. "QPM: Discrete Optimization for Globally Interpretable Image Classification." The Thirteenth International Conference on Learning Representations. 2025.

[2]: Zheng, Xu, et al. "F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI." The Thirteenth International Conference on Learning Representations. 2025.

[3]: Zhang, Jianming, et al. "Top-down neural attention by excitation backprop." International Journal of Computer Vision 126.10 (2018): 1084-1102.

[4]: Farrell, R. (2022). CUB-200-2011 Segmentations (1.0) [Data set]. CaltechDATA.

Thank you for your review, appreciating several strengths and raising several questions that allow us to clarify our contributions. We also thank you for the detailed suggestions on the text and will of course correct the typos, rename the class-class similarity to an unused symbol, and include the formula for the Feature Diversity Loss in the revised manuscript.

Essential steps lack any formal analysis. There is no approximation bound, convergence argument, or sample-complexity discussion. Theoretical novelty is therefore thin.

Can you provide formal statements for the proposed method as raised in the above?

Our work follows other work in interpretable machine learning, such as PIP-Net, ProtoPool, Q-SENN or QPM, where the primary contribution is a model that offers novel forms of interpretability that are thoroughly empirically validated. As such, the paper’s main contribution is not theoretical. Instead, our main novelty is the introduction of hierarchical explanations and the adaptation of the theoretically well-grounded theory of conformal prediction to predict along them. This enables the first form of built-in interpretable conformal prediction and local explanations that answer an unprecedented range of questions simultaneously while having non-trivial empirical gains (as rightly pointed out):

1. What meaningful features of which classes are found in an image? (Quantified by multiple metrics including Structural Grounding and Feature Alignment in Tables 1, 3, & 16.)
2. How does each feature narrow down the set of potential predictions into increasingly similar classes? (Visualized in our hierarchical explanation and measured by Set Coherence in Figure 6.)
3. Which set shall be predicted to guarantee a configurable average accuracy? (Guaranteed by our conformal prediction framework, with results in Table 2.)
4. Which features would have needed to activate stronger in order to predict a smaller set with sufficient certainty? (Demonstrated by the paths in our hierarchical explanations.) This notion of counterfactual interpretability has recently gotten more attention [1] and is included by-design.

Note that questions 2.-4. are only enabled through our novel contributions, while the answer to 1. is greatly improved compared to what QPM offers. Alongside the improved global interpretability, we consider answering these questions in an explanation a significant step to advance the field of interpretable machine learning. We will include this phrasing and the recent counterfactual paper as we believe that it clarifies our contribution and why it is significant without novel formal statements.

Exchangeability is assumed in the manuscript, but there is no discussion as to when this assumption is satisfied and when it is violated (how realistic an assumption it is). At least there should be a discussion of the assumptions made with respect to the experimental setup reported by the authors.

Can you state whether the reported experimental settings satisfy the assumption? and can you provide several counterexamples?

You correctly pointed out that we assume exchangeability between calibration data and test data for our conformal prediction experiments, which is required for the guarantees of Split Conformal Prediction to hold. We argue that exchangeability is certainly holding for our evaluation, as we

1. Calibrate on the first 10 samples per class of the test data (lines 325-326). This ensures the calibration and test samples are i.i.d., a stronger condition than exchangeability.
2. Very closely match the coverage on the test data for which we calibrate, indicating that the test was exchangeable with the calibration data.

We will explicitly state that we assume exchangeability in the main paper and reference an expanded section on exchangeability in Section J, which contains the surrounding clarification. As our experiments are designed to satisfy exchangeability, there is no counterexample in our experiments. However, there exists work [2, 3, 4] on applying split conformal prediction under conditions where exchangeability does not hold. Investigated scenarios without exchangeability are time series datasets [2, 3] or calibrating on ImageNet and then testing on datasets with distribution shift [4], such as ImageNetR or ImageNetA.

Since the proposed method appears to be somewhat sensitive to a hyper-parameter $\rho$, theoretical discussions on the optimality of hyper-parameter selection strengthen this study. Moreover, the authors set $\lambda_{feat}=3$ in their numerical experiments, but there are no discussion on how to choose this value, and no justification.

Justification of hyper-parameter selection is needed. Can you give more discussion?

We thank the reviewer for the question about selecting $\rho$ and are happy to clarify our reasoning. As described in Sections 3.1, 4.3 and D, $\rho$ should be set using the calibration data and depends on several factors, most notably how many classes are similar in the dataset and if a specific coverage is targeted. Varying coverage goals relate to real-world scenarios, since a bird classifying app can be calibrated for a lower coverage, while a medical application should certainly guarantee smaller error rates. We selected $\rho=0.5$. As shown in our ablation studies (Figures 7 and 8), this value represents a 'sweet spot' for the CUB-2011 dataset. It is high enough to induce a rich semantic hierarchy that improves set prediction performance, yet low enough that it does so without sacrificing point-prediction accuracy. However, we will extend Section D to emphasize that there is no single optimal $\rho$ for all CHiQPMs.

You are right to point out that an ablation on $\lambda_{feat}$ is missing. It was chosen based on the initial experiment on CUB-2011 shown below showcasing Accuracy, Sparsity and Feature Alignment with respect to the weighting. Evidently, alignment and sparsity improve with increasing $\lambda_{feat}$. The hyperparameter was set to the highest value for which the accuracy does not decrease, hence resulted in $\lambda_{feat}=3$. We will include this ablation in the appendix.

We thank you again for pointing out valid points of criticism and believe that the resulting changes will improve the manuscript.

[1] Dominici, Gabriele, et al. "Counterfactual Concept Bottleneck Models." ICLR. 2025.

[2] Barber, Rina Foygel, et al. "Conformal prediction beyond exchangeability." The Annals of Statistics 51.2 (2023): 816-845.

[3] Oliveira, Roberto I., et al. "Split conformal prediction and non-exchangeable data." Journal of Machine Learning Research 25.225 (2024): 1-38.

[4] Alijani, Shadi, and Homayoun Najjaran. "WQLCP: Weighted Adaptive Conformal Prediction for Robust Uncertainty Quantification Under Distribution Shifts." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.