Improving Prototypical Part Networks with Reward Reweighing, Reselection, and Retraining

Thanks for your constructive feedback, and we will try to improve the paper as much as we can based on your suggestions (please check out our updated version, which will be uploaded soon). We will address your main concerns and questions below:

$**The method seems overly hand-crafted:**$ We agree with the reviewer about the beauty of ProtoPNet being able to learn parts from course labels, and we believe that R3-ProtoPNet is actually an extension of that simplicity. As discussed in our work, the original ProtoPNet fails appropriately learn consistent and meaningful prototypes, but does manage to learn from parts of the images. If one wished to resolve these issues without manually collected labels, it would still be necessary for the designer of the system to enforce a bias against spurious and inonsistent prototypes via a penalty in the loss function, or, in the case of the original ProtoPNet paper, designing pruning methods to throw out spurious information.

In our opinion, the beauty of the R3-ProtoPNet is precisely that it does not require such a penalty, but instead relies on the quick collection of human feedback.

$**More diverse evaluation on other datasets:**$ Thanks for pointing this out! We chose the CUB bird dataset because it's the most commonly used dataset in related works and identifying bird species is a task that could elicit very natural human preferences. We are currently working on evaluating our R3 framework on other datasets such as the Stanford Cars dataset. We expect similar performance gain, but the full results on all base architectures might not come out by the end of the discussion period. We will definitely include those results later.

$**Generalization to other models:**$ We have tested our method on ProtoPFormer, which is a transformer-based ProtoPNet (Xue et al. 2022), and we've shown that our R3 framework also leads to better interpretability and improved accuracy. The results will be reported in our modified version of the paper. Furthermore, the high-level notion of our R3 framework could be thought of as a type of RLHF that acts on the interpretability of a given model. We believe even for a model without prototypes, as long as it has an observable intermediate stage that's possible to use a reward model to predict human preferences and thus to evaluate its interpretability, with some modifications our R3 framework could be applicable.

$**The possibility of losing non-obvious but useful such as texture and contrast:**$ Whether this could happen really depends on the learned reward model. In our rating rubric for the bird dataset, the prototype will have a score of at least 4 as long as a substantial part of it is overlapped with the bird body, so it's not likely for a prototype capturing non-obvious features on the body to be updated significantly and lose those features completely - we included plenty of visualizations in the appendix. On the other hand, since the goal of our R3 framework is to make the model more interpretable and align with the preferences of the users, introducing human biases to the classification problem is expected and choosing whether to believe them is more of a philosophical question rather than an issue with our R3 framework.

$**How sensitive is R3-ProtoPNet to the amount and quality of human feedback data for the reward model:**$ Actually we had a recent empirical analysis of the relationship between the amount of human ratings and the predictive performance of R3-ProtoPNet and the reward model. We will include this in our modified paper.

$**Only retraining the classifier layers:**$ Empirically we found that only retraining the classifier layers would lead to slightly worse performance than retraining the entire ProtoPNet. This is because the last layers operate on the similarities between the convolution outputs of the base model and the prototypes: once we update our prototypes (especially after the reward reweighing step), we'd also need to update the base features slightly to allow for better alignment.

$**Computation overhead of R3:**$ The computation cost of R3 is discussed in Appendix A, and we think several hours of extra debugging/training time is acceptable. We'd also like to note that the users of R3-ProtoPNet could heuristically control the proportion of bad prototypes to be updated by changing the reselection and reweighing thresholds (Appendix C) and thus control the runtime.

Thanks for your constructive feedback, and we will try to improve the paper as much as we can based on your suggestions (please check out our updated version, which will be uploaded soon). We will address your main concerns and questions below:

$**No evaluation of the learned reward function:**$ We mentioned in 5.2 that our reward model achieves 90.09% test accuracy on the synthetic dataset (the held-out test set has about 13k pairs of heatmaps), which means it could effectively predict the relative order of prototype quality and thus model the collected human preferences reliably.

$**Why our method empirically improves the performance:**$ In the original ProtoPNet, the prototypes are generated by the model itself only based on supervised predictive signal, and the interpretable reasoning chain is maintained by the "push" operation, which sets the prototypes to be the closest input image patch. However, one issue with this process is that once initialized the prototypes cannot change dramatically, so the bad-initialized (possibly due to some predictive shortcuts) ones will always remain bad - we consider this as the main cause of the spurious features in the ProtoPNet. In our R3-ProtoPNet, with the help of an external reward model, reselection allows the global movement/reinitialization of the prototypes, while reweighing allows local movement of the prototypes to be more aligned with human preferences (e.g. more overlap, more characteristic features, etc.).

$**Reward reweighing seems optional:**$ As stated above, reweighing allows local movement of the prototypes to be more aligned with human preferences (e.g. moving an originally partially overlapped prototype more toward the body), and we did find both interpretability and accruacy gains as the result of reward reweighing. Please refer to 1) our updated performance report tables which will break the R2 step into reselection and reward reweighing as well as 2) an ablation study added in the appendix. Thanks for the suggestion!

$**More illustrative visualizations:**$ Actually We included the closest image patches to each prototype and the prototypes corresponding to each image during different stages in the appendix. That being said, we will consider adding more illustrative visualizations in our modified paper and also revising the captions, and thanks for the feedback.

Thanks for your constructive feedback, and we will try to improve the paper as much as we can based on your suggestions (please check out our updated version, which will be uploaded soon). We will address your main concerns and questions below:

$**Only one single bird dataset is used: **$ Thanks for pointing this out! We chose the CUB bird dataset because it's the most commonly used dataset in related works and it could elicit very natural human preferences. We are currently working on evaluating our R3 framework on other datasets such as the Stanford Cars dataset. We expect similar performance gain, but the full results on all base architectures might not come out by the end of the discussion period. We will definitely include those results later.

$**Why R3-PPNet was not compared to the work of Bontempelli et al.: **$ One major distinction we'd like to point out is that R3-ProtoPNet is an offline debugging framework: it only requires one batch of human feedback to train the reward model, and this happens before we actually debug the ProtoPNet. On the other hand, the ProtoDebug framework proposed by Bontempelli et al. requires iterative online human feedback on the exact models and prototypes to be updated. Furthermore, to the best of our knowledge, ProtoDebug is not designed for large-scale prototype debugging: the required human labor would scale up with the number of prototypes (their collected feedback and evaluation was on 5 out of 200 classes while we evaluated on all 200 classes); by contrast our method only needs a fixed number of human annotations to train the reward model. Given the reasons above, we don't think the two frameworks are quite comparable.

$**Cost-benefit analysis of collecting human labels: **$ While a large and high-quality dataset of human feedback is required for RLHF to improve large language models and other complex generative models, work on preference learning in RL demonstrated that relatively small amounts of human feedback was necessary to learn a useful reward model, as long as the data to be rated is chosen carefully (Christiano et al. 2017). Generally, it is our understanding that the amount of human feedback data needed to learn a particular reward model is dependent on the underlying complexity of the feedback task. And, for learning reward functions relevant to ProtoPNet, the feedback task does not require large amounts of human feedback data.

For our method of learning our reward function, we empirically found that ~500 rated heatmaps would be able to train a reward model with over 90% accuracy (it would converge afterward), and although we collected multiple user ratings for these samples using Mechanical Turk and took the average, we found that it took roughly one hour for a single human rater to provide high-quality ratings for all 500 samples. As stated earlier, since the trained reward model could be reused for different base architectures and prototypes, we don't think the labor cost outweighs the benefits.

$**The reward function becomes obsolete after fine-tuning/debugging the model: **$ Yes, this is largely correct. And if some prototypes are still buggy after the R3 update, it's most likely that it's not detected/properly corrected by the learned reward model, and such cases do happen due to the noise in the entire human-in-the-loop module. Conventional ways to robustify the reward model include comprehensive selection of the heatmaps (so that most typical errors are included) to be rated and increasing the number of raters (which we did). One interesting future direction would be having a ProtoPNet updated by multiple learned reward models sequentially, each capturing one single aspect of human preference (e.g. overlap, body part consistency, etc.), so that each reward model would be less noisy because of the single learning objective.

$**Alpha, Beta, and Gamma: **$ As discussed in section C of the appendix, empirically those hyperparameters are determined by comparing the reward distribution of the prototypes to the visualizations of the heatmaps: first the reselection threshold should separate those completely-off prototypes from the rest, and then the reweighing threshold should separate the prototypes that still need local updates from those already look good. Finally we want to set the acceptance threshold such that it's large enough to improve the quality of the reselected prototypes while not causing too many reselection failures. We found that given a properly annotated synthetic dataset, fixed thresholds would lead to approximately the same performance for the same base architecture, and bad thresholds would influence the number of updated prototypes in the R2 step and make the R3-ProtoPNet not reaching peak performance (but usually no worse than the original model).

$**The paper has limited scope:**$ While we agree that the paper is focused on applying RLHF to ProtoPNet, we respectfully disagree with the reviewer that the paper has limited scope. Many interpretability or explainability methods in deep learning are designed for a particular architecture, data modality, or problem in mind. Part of the difficulty of interpretable deep learning is designing or utilizing interpretable methods for particular problems. This process usually involves intense and time-consuming exploration of the data and understanding of the problem. Combining RLHF with existing interpretability methods, while certainly not eliminating the need for a deep understanding of the problem at hand, has the potential to simplify the modeling process, allowing researchers and practitioners to spend less time designing and building complex methods, and more time interrogating the reasoning and limits of their models. We believe that R3-ProtoPNet is one such example of how RLHF and adjacent ideas can allow for improvements to interpretable deep learning more generally.

An example of this is that ProtoPNet fails to provide consistent prototype labels across images, where a single prototype can still map to multiple parts of a bird across images. Instead of designing a hand-crafted regularization function to promote cross-image consistency, we can now instead consider how to best elicit and model human feedback on cross-image consistency of prototypes, which is something human viewers can do with ease.

We will revise our paper to include a discussion of the general contribution to RLHF and interpretability.

$**Human expert evaluation needed:**$ The CUB-200-2011 dataset is a commonly used benchmark for evaluating ProtoPNets due to the ease with which an amateur can recognize how a method makes classifies a bird, such as focusing on the background or the bird itself, on the bird's wings or head--or some uninterpretable combination of the two. We agree with the reviewer that the utility of such a bird classification method to an actual expert is unclear, given that no expert evaluation is used to justify this method. We hope to provide such an exploration in future work.

$**Prototypes are sometimes soft-edged and blobby:**$ In ProtoPNet, a heatmap is generated by calculating the similarity of the upsampled prototype and each of the 49 patches (each of size 32x32) of an original image, so if the textures of several neighboring patches are similar, then the highly activated region would inevitably be larger/blobby. In fact we would consider this as more of a visualization problem. As we can see in the visualized examples in the appendix, many prototypes are able to clearly focus on wings, tails, etc. We hypothesize that some prototypes don't focus on specific body parts because sometimes more subtle visual information such as characteristic edges and texture may provide better predictive features.

$**A more nuanced discussion of what makes a feature spurious or not:**$ We thank the reviewer for pointing this out. Placing attention on the background is commonly considered 'spurious' information in interpretable deep learning methods for computer vision tasks, with the original ProtoPNet paper doing so in Section 2.6 (Chen et al. 2019). Our understanding is that the reason for this trend is a desire for image classification systems to make decisions not on correlative features but causal ones. While the background of the image being a blue sky raises the probability of the bird in the image being a swallow and not a woodland thrush, the presence of the blue sky does not make the swallow a swallow. We believe this focus on wanting to learn causal and not correlative features comes from the want for interpretable methods like ProtoPNet to generalize across image classification tasks.

$**The application of improving interpretability and high-level understanding about RLHF, ProtoPNets, or general neural networks: **$ As we have discussed, the use of the CUB-200-2011 dataset is largely a demonstration of how one could use ProtoPNet and R3-ProtoPNet to perform image classification according to a particular reasoning framework. A limitation of this work is that we only propose a hypothetical application, but do not provide the reader with a particular application where we work with domain experts to solve a particular problem. The questions of who this is specifically for, and what specific purpose is this for, are left unanswered by our work. We hope to remedy this limitation in future work.

That said, as discussed earlier, we believe our method has general applicability to interpretable deep learning. As we demonstrate, RLHF or related ideas have the potential to simplify the process for designing and training interpretable deep methods, with possible follow-ups extending beyond the architecture of ProtoPNet.

We again thank the reviewer for these helpful comments!