Attention maps. Thank you for the comments. It is worth noting that in our original submissions, Figures 12-17 are indeed hard classes, as we want to ensure that every picked class has misclassified examples (the last row) to match the presentation style of Figure 1. Given the high performance of the Butterfly (BF) dataset, selecting a challenging case (the one pointed out by the reviewer in Fig. 13) was necessary to maintain this pattern.

In the rebuttal, we further present Figures 27-28, which are classes without misclassified cases. These attention maps are much clearer.

Furthermore, it is worth highlighting that the attention heads presented are normalized based on attention scores, which may, in some instances, result in the background being highlighted. However, when we sort them based on the un-normalized scores, we find that only the essential attributes are highlighted, as illustrated in Fig. 4.

We also note that, according to https://www.birds.cornell.edu/home/, habitats are also considered important messages to differentiate species. Thus, we do expect for some species, the attention maps may focus on the background.

About the statement in the limitation. We apologize if we overclaim. We mainly want to mention that this is a usual case. For all the fine-grained datasets we consider, they are within 1000 categories. We will certainly clarify this.

We appreciate that you point out the case of bird species in iNaturalist. Please note that, while the whole iNaturalist contains >10K species, those species span across multiple Kingdoms, Phylums, etc. We thus do not consider inputting all the 10K classes all at once into INTR, but focus on recognizing species of the same Class (e.g., Class Animalia Chordata Actinopterygii) or Order. We will certainly clarify this.

In light of our clarification and additional results, we would like to ask if you are willing to reconsider your score. Thank you.

Lack of interpretability analysis. Thank you for the suggestion and reference. We checked the experiment in Sec 4.2 of [1] and found that it may not be feasible for our INTR. First, INTR does not impose regularizers to segment an object into parts. Second, the experiment will require training a regressor to localize part locations, which needs detailed part annotation.

With that being said, we have conducted additional quantitative studies and discussions. First, the attributes that INTR captures are local patterns (specific shape, color, or texture) useful to characterize a species or differentiate between species. These attributes can be shared across species if the species are visually similar. These can be seen in Figure 23 and Figure 24. In Figure 23, we applied the query of Heermann Gull to the image of Heermann Gull (first row) and Ring Billed Gull (second row). Since these two species are visually similar, several attention heads identify similar attributes from both images. However, at Head 8, the attention maps clearly identify the unique attribute of Heermann Gull that is not shown in Ring Billed Gull. Please see the caption for details.

In Figure 24, we present the cross-attention map activated by the ground-truth queries for two closely related species, the Baltimore Oriole and the Orchard Oriole. Additionally, we manually document some of the attributes by checking whether the attention maps align with those human-annotated attributes in the CUB dataset. We found that INTR can identify shared and discriminative attributes in similar classes.

When the species are visually dissimilar (cf. Figure 9), we see that the attributes specifically to each species (every column) cannot be found in other species (the off-diagonal elements of every row).

We will provide more discussions in the final version and move important messages into the main paper.

Comparison with other interpretable models. Besides Figure 3, we include more comparisons in Figure 10 and Figure 25. As mentioned on page 7 (the paragraph before Sect. 4.2), we do not claim that these compared methods cannot detect attributes. We include them mainly as references to show what our approach can achieve with a simple and extensible model design. One particular advantage of INTR is that it can easily detect attributes of different sizes/shapes. In contrast, ProtoPNet, ProtoTree, and ProtoPFormer often need to pre-define the prototype sizes/shapes.

To quantify the difference between our INTR and ProtoPFormer, we conducted a human study. We randomly selected five images from each of the four species selected randomly from the CUB dataset. To verify the attributes detected by models, we provide the models’ interpretation attention maps and image-level attribute information (from the CUB metadata) to seven individuals who do not have any knowledge about the project. We ask them to list all the attributes that are captured by the attention maps. One attribute is considered detected by a model if more than half of the individuals identify it from the attention maps. Our quantitative analysis reveals that INTR outperforms ProtoPformer, achieving an attribute identification accuracy of 74.7% compared to ProtoPformer's 42.2%.

We will expand this study to contain more individual observers and bird species in the final version.

Image manipulation experiments. We initially provided Figure 5 as an analysis, not an experiment. Thanks to your suggestion, and we expanded the study to be more systematic. We considered images from three species of the CUB dataset namely Red-winged Blackbird, Scarlet Tanager, and Orchard Oriole. We removed the red spot of the Red-winged Blackbird (Figure 5), edited Scarlet Tanager by changing the black wings and tail to red to make it look like Summer Tanager (Figure 26), and replaced the color of Orchard Oriole’s belly from brown to orange to make it look like Baltimore Oriole (Figure 5). All of these manipulations follow the attribute guidance described in https://www.birds.cornell.edu/home/. We randomly selected 10 images from each of these species for manipulation and observed that 29 cases changed their prediction after manipulation (out of 30), which shows a 96.7% success rate. This holds the property that INTR can identify tiny image manipulations that distinguish between classes.

Influence of Encoder. Thank you for the insightful question. The large receptive field of a deep CNN or a plain Transformer indeed indicates that every feature map location has been affected by many pixel locations in the input image. On the one hand, this helps capture the context in the image (to differentiate visually similar pixels or local patterns). On the other hand, it may lose the localizability — each feature map location does not necessarily correspond to its associated pixel/patch locations.

That said, we think several design choices in ResNets and the Transformers used in vision (e.g., ViT or the encoder in DETR) help improve localization. First, both of them have “skip” links. That is, there is a shortcut from the input pixel/patch to the corresponding feature map location. Such a skip link has been shown to improve localization in semantic segmentation (e.g., U-Net) and object detection (e.g., spatial pyramid pooling) as well. In [A], the authors empirically showed that a significant amount of feature information is passed from the previous layers to the later ones through the skip links. Second, in the encoder of DETR, fixed positional encodings are added to all the encoder layers. We surmise that this would further strengthen the location correspondence between the input image and the feature maps. Finally, as evidenced by recent work of ViTs like DINO-v1 and DINO-v2, we do see a fairly accurate correspondence of the object/part boundaries in the feature map and in the image.

We hope that these discussions address your concern about the interpretability of INTR.

[A] Maithra Raghu et al., Do Vision Transformers See Like Convolutional Neural Networks? NeurIPS 2021

Efficiency. Thank you for the insight on the quadratic growth of computation. We have indeed been investigating methods to reduce it. The first method is to perform hierarchical classification following the available taxonomy (e.g., the organism taxonomy). We can learn queries for different taxonomy levels (e.g., for different families, genera, and species). During inference, we then apply these sets of queries in sequence. In other words, we do not need to input all the species-level queries to the decoder at once; only those belonging to the predicted genus are needed. Consider a two-level hierarchy with C^2 classes, where every C of them is grouped. Hierarchical inference takes 2*O(C^2) computation, largely saving it from O(C^4) by the plain inference.

Even without hierarchical training, one may take a pre-trained INTR, perform hierarchical clustering to group the classes (e.g., based on the validation confusion matrix), and select one representative class to represent each cluster. During inference, we then use those representative classes, from the root of the hierarchy down, to perform hierarchical classification. In Fig. 21, we show an example that the top predicted classes by INTR indeed share similar appearances and genera; one can thus use one query to represent the others. In Fig. 6, we also show that, by taking only the most visually similar queries as input, INTR can identify finer-grained attributes.

Pre-training influence. Thanks for carefully checking our paper, especially the footnote on page 6 that discusses our usage of an MSCOCO-pre-trained model.

During the rebuttal period, we tried to pre-train the INTR model from scratch, i.e., randomly initializing the Transformer encoder and decoder in DETR, using the ImageNet data. Unfortunately, we found the hyper-parameter setting of DETR not directly applicable to this pre-training. While we have tried to perform hyper-parameter selection, we were short of time to find a good configuration. We will try to continue the study.

As an alternative, we take the pre-trained DETR model and replace its encoder (i.e., ResNet-50 + Transformer encoder) with a separately pre-trained DeiT-S (using ImageNet-1K) and ViT-H (using ImageNet-21K). Details of this study are in Appendix Section F. We found that using a DeiT-S or ViT-H encoder trained without MSCOCO can sometimes outperform the original ResNet-50 + Transformer encoder in terms of accuracy, as shown in Table 4. However, in terms of the interpretation/visualization, using the pre-trained DETR model is still the best. Figure 22 shows an example. We attribute this to two reasons: 1) the pre-trained DeiT-S and ViT-H are not well-aligned with the pre-trained decoder, and 2) pre-training with detection data does help with focusing on details.

With that being said, we respectfully think that these results should not undermine our main contributions and claims. First, DETR is a model architecture designed initially for detection. Training it from scratch may need some careful investigation. Second, with the accessibility to large-scale detection datasets and pre-trained models, building a model from the most suitable starting point should be allowed and encouraged. In our case, the MSCOCO dataset does not contain fine-grained class labels and part annotations, quite different from our downstream tasks. Besides, what we found interesting is that while existing literature mostly uses classification pre-trained models to facilitate other downstream tasks, we show that the opposite—using detection pre-trained models to facilitate classification—is also worth exploring.

The shared w vector. The “w” vector is a shared binary classifier telling whether a class is seen in the image. During training, it is optimized to align with the outputted vector $z_{\text{out}}^{(c)}$ of the true class (cf. Sect. 3.4). In theory, if the model’s capability is large enough, one could freeze “w” (e.g., 1/len(w), as suggested by the reviewer) and learn the model backbone to generate $z_{\text{out}}^{(c)}$ that aligns with the frozen “w”. As all the classes still share a single “w”, the argument of learning distinct queries per class should still hold (cf. Sect. 3.4).

In practice, we found that freezing “w” makes the training much harder. For example, using the same learning rate and total epochs, the trained model can only obtain a 76.3 classification accuracy on the Pet dataset, compared to 90.4 w/o freezing.

Images with multiple objects. This is an interesting question. As we mainly work on classification datasets, we expect that most images only contain one object (or objects of a single class). We follow your suggestion to stitch pictures of two bird species. We then check the cross-attention maps triggered by each of the two queries. We have the following observations.

1. The two sets of cross-attention maps (triggered by the two queries) can jointly localize both object instances.

2. Each query focuses a bit on the corresponding object instance.

3. Several cross-attention maps distribute their weights to both object instances, especially when the two bird species share some common attributes (e.g., head shapes).

We attribute the observation in point 3 to 1) the mismatch between training and testing and 2) the inner workings of INTR — it aims to find class-specific patterns but may not localize them from the same object instance. In practice, one may use INTR with an object detector, using the object detector to identify object instances (e.g., birds) and then applying INTR to each separately.

Regarding the classification result on the stitched image, we found that INTR tends to rank a third class that contains the attributes of both stitched classes higher than each of them. For example, INTR predicts “Magnolia Warbler” on the stitched images of “Green Violetear” and “Prairie Warbler”; INTR predicts “Painted Bunting” on the stitched images of “Indigo Bunting” and “Summer Tanager”. This result makes sense regarding how INTR was trained: it was trained with the cross-entropy loss to output the most likely class by checking whether its attributes are found in the image.

HxW. Thank you for the suggestion, and we will use grid cells in our final version.

In light of our clarification and additional results, we would like to ask if you are willing to reconsider your score. Thank you.

Dear Reviewer dzqa,

We appreciate your time and effort in reviewing our paper. We are glad to see your positive rating.

However, after a careful reading of your feedback, we found that it may not directly relate to our paper content. May we kindly request that you take a look at this potential problem? Thank you.

Best, Authors

Comparisons to models other than ResNet. Thank you for the comment and suggestion. We consider two variants of INTR. 1) INTR-FC: the same architecture as INTR except for learning a class-specific $w_c$ at the model’s output (cf. Sect. B and Eq. 10). This baseline was included in our original submission, mainly to demonstrate the effectiveness of INTR’s classification rule on interpretability: INTR incorporates class information at the decoder’s input (cf. Eq 2 & 6; the last paragraph of Sect. 1) but not at the model’s output (cf. Eq. 1 & 10). With additional model capacity, INTR-FC achieves a higher accuracy (77.5 and 92.7 on CUB and Pet, respectively) than INTR (71.8 and 90.4). Nevertheless, as shown in Fig. 7, INTR-FC obtains a worse interpretation than INTR, as it does not hold the nice properties of INTR that encourage interpretability (cf. Sect. 3.4 and Sect. B). 2) INTR-Enc: We remove the decoder of INTR and add a fully-connected layer on top of the encoder’s output. (This is to implement the reviewer’s suggestion. We respectfully think there is no need to have the decoder if the class-specific queries are disregarded.) With the fully-connected layer on top, INTR-Enc achieves slightly higher accuracy (73.0 and 90.7 on CUB and Pet, respectively) than INTR but loses the interpretability of INTR.

We hope that the results above support our statements in the main paper (the paragraph before Sect. 4.1 and the first paragraph of Sect. 4.1): achieving a high classification accuracy is not the goal of this paper.; the goal is to demonstrate the interpretability. In Table 2, we compared INTR to ResNet-50 mainly because it is a building block in INTR. Nevertheless, the results are mainly as references, not to claim that INTR outperforms ResNet-50 in accuracy, and we will clarify this.

Quantitative interpretability results. Humans in the loop. Thank you for the suggestion. During the rebuttal, we conducted a human study comparing INTR and ProtoPFormer. Please kindly refer to the second response to reviewer 8cK8. We also provided more systematic results on the manipulation analysis in Figure 5. Please see the third response to reviewer 8cK8. We also offered results using metrics in XAI. Please see the second response to reviewer 4rTr.

We will include the above results in the final version and seek other quantitative metrics to strengthen our paper. To our knowledge, besides showing classification accuracy, prior work on interpretable methods (e.g., ProtoPNet, ProtoPFormer, and ProtoTree) has mostly reported qualitative results. Compared to them, we have conducted more qualitative analyses and provided more visualization results to showcase the inner workings of our approach.

Comparison to typical transformer architectures in terms of interpretability. Thanks for the question. We note that typical transformer architectures for classification (e.g., ViT or Swin-Transformer) follow the classification rule in Eq (1). Thus, a post-hoc explanation method like Grad-CAM is needed to explain why the model predicts a particular class. Even though one may analyze the attention weights within the model, they may fall into the debate mentioned in Sect. 3.4. We also kindly refer the review to Sect. 3.5, where we provide further discussions between INTR and closely related work.

Ethics concerns. Our method aims to interpret the model’s prediction, which can potentially improve the trustworthiness and discover new knowledge for prediction (e.g., new attributes for animal species). We humbly think that our approach does not introduce any additional ethical concern or negative societal impact than other explainable and interpretable work.

In light of our clarification and additional results, we would like to ask if you are willing to reconsider your score. Thank you.

Real utility of the explanations. We appreciate your feedback. To our knowledge, in many scientific or safety-critical domains, such as biodiversity and medical applications, domain experts often care more about whether the model is trustworthy or can discover new knowledge (e.g., traits for animals or biomarkers for diseases) rather than purely achieving higher accuracy. In our current paper, the targeted real utility is indeed automatic attribute/trait identification and discovery for organisms. This is of great impact because the analysis of traits, the integrated products of genes and environment, is critical for biologists to predict the effects of environmental change or genetic manipulation and to understand the significance of patterns in the evolutionary history of life. Specifically for fine-grained species, our approach can aid biologists in rapid identification and differentiation between species, refining and updating taxonomic classifications, and gaining insights into relationships between species.

To demonstrate this capability, we conduct additional analyses focusing on closely related pairs of butterfly species within the Cambridge Butterfly dataset. Our objective was to pinpoint the distinctions discernible by our model and compare them with the differences identified by experts. Specifically, we selected three finer species (Heliconius melpomene and Heliconius elevatus; Heliconius erato and Heliconius melpomene; Heliconius melpomene and Heliconius erato) from the Cambridge Butterfly website and confirmed that our method effectively detects these differences as shown in Figure 18 and 19 (and Figure 6). This implies that our approach can assist experts in accurately discerning (and potentially discovering) the unique characteristics of fine-grained species.

The traits identified by human experts can be found here.

Website: https://www.cliniquevetodax.com/Heliconius/index.html

Figure 6: https://www.cliniquevetodax.com/Heliconius/pages/melpo%20vs%20elevatus.html

Figure 18: https://www.cliniquevetodax.com/Heliconius/pages/melpo%2520vs%2520erato.html

Figure 19: https://www.cliniquevetodax.com/Heliconius/pages/erato%20favorinus.html

Finally, as discussed in the first response to reviewer dzqa, improving the classification accuracy may hurt the model’s interpretability. We thus think it is fine to lose the classification accuracy a bit for trading better interpretability.

Evaluate the explanations using proxy metrics in XAI. Thank you for the suggestion. In our original submission, we indeed included results using proxy metrics in XAI (cf. Sect. F and Table 6 in the Appendix). We apologize if we did not make it clear.

Specifically, we applied the deletion and insertion metrics. Given a post-hoc explanation method that generates a saliency map for the input image (e.g., Grad-CAM, cf. Sect. 2.2 and Sect. A), these metrics first sort the image patches based on the saliency scores (descending order). The deletion metric then iteratively removes patches from the image, inputs the manipulated image to the classifier, and measures the confidence of the correct class. The more sharply the confidence drops (i.e., the lower the AUC is), the better the explanation is, as it highlights the most discriminative patches, which, once removed, would drastically reduce the classifier’s confidence. In contrast, the insertion metric blurs the original image and iteratively discloses the patches. The more sharply the confidence increases (i.e., the higher the AUC is), the better the explanation is, as it highlights the most discriminative patches, which, once inserted, would drastically improve the classifier’s confidence. Please see more details in Sect. F, paragraph: Comparisons to post-hoc explanation methods.

The results are shown in Table 6 of the Appendix (during the rebuttal, we further include the results of RISE). INTR notably outperforms Grad-CAM and RISE, achieving a higher insertion AUC and a lower detection AUC.

That said, XAI metrics are designed mainly for post-hoc explanation methods like Grad-CAM and RISE, which aim to explain a black-boxed classifier like ResNet. For self-interpretable methods like ProtoPNet, ProtoPFormer, ProtoTree, and our INTR, which build specific classifiers, XAI metrics are seldom reported as they may not be fair across different methods.

Performance of INTR. Thank you for the insightful comment. In analyzing INTR’s prediction, we did find that some errors result from intra-class variations. Specifically, for many bird species, different genders and life stages show different appearances and attributes (https://www.allaboutbirds.org/guide/Painted_Bunting). For example, in Fig. 1, the last row (misclassified) is indeed a female Painted Bunting; others, male Painted Bunting. These intra-class variations can be considered sub-categories, and we surmise that learning one query vector per class may only capture the major sub-categories in the dataset but not the entire variations.

During the rebuttal period, we investigated the idea of learning M queries per class. This idea results in M logits per class (cf. Figure 2), and we averagely pool them to represent each class in both training (for the cross-entropy loss) and in testing. (We also investigated max pooling but found it hard to converge.) We see a notable gain on the CUB dataset: using $M=2$ queries per class improves INTR from 71.8 to 75.7. We also include a qualitative result in Fig. 20. The image is an immature male Hooded Oriole, and INTR misclassifies it. Using $M=2$ queries, the model can correctly classify the image and highlight the black throat attribute. We will include a comprehensive evaluation of this multi-query version of INTR in the final version.

In our humble opinion, the improvement by using multiple queries per class does not change our main paper's main contributions and messages. In contrast, it further showcases INTR’s extendability and its potential to discover sub-categories within a class.

The number of heads. Thank you for the comment. In our humble opinion, the number of heads in INTR can be different from the number of concepts per class in ProtoPNet. In ProtoPNet, each class is allowed to learn a separate set of prototypes; in INTR, the model parameters of the cross-attention heads are shared by all classes. About the number of discriminative features, according to the CUB dataset, there are 312 human-annotated attributes across more than 20 part-{pattern, shape, size} combinations. Thus, we respectfully think it is reasonable for INTR to use eight or more heads.

That said, the number of heads is adjustable as it is essentially a hyper-parameter. In our original submission, we mainly followed the DETR paper to use eight heads. We did conduct an ablation study on the CUB dataset in Table 5 and found that eight heads slightly outperform other numbers. During the rebuttal, we further compare different numbers of heads for the Butterfly and Pet datasets and see similar trends. We will include a more comprehensive ablation study in the final version.

In light of our clarification and additional results, we would like to ask if you are willing to reconsider your score. Thank you.