Towards Interpretable Continual Learning Through Controlling Concepts

Thank you for your positive feedback! We are glad to learn that you think our work is interesting and novel. Below we provide a few more details in response to your comments and requests.

#1 Details of CC-CBM

We describe the CC-CBM in Section 4 (p.5 -p.7). Even though it uses a pretrained backbone, we think it is very different from other baselines [2,3,4]. We combine CC with LF-CBM [1] to build CC-CBM. CC is for its continual learning ability. To know where performance gain comes from, we need to investigate CC-CBM in two steps:

1. Finetune standard model v.s. Finetune CBM : To understand the impact of the concept bottleneck.
2. Finetune CBM v.s. CC-CBM : To understand the impact of CC.

Both points have been done in our paper, please see the main results (Section 5.1 from p.7- p.9) and Appendix (p.23 - p.26):

• Overall, Finetune CBM outperforms Finetune standard model by up to 2.3% in $\bar{A}_T$ and up to 16.4% in $\bar{F}_T$.
• CC-CBM makes CBM’s continual learning ability even better, as it outperforms Finetune CBM by up to 1.8% in $\bar{A}_T$ and up to 12.4% in $\bar{F}_T$.

The non-incremental learning setting is studied in LF-CBM, which shows comparable performance to the standard model. In contrast, in this paper we have studied the GPT + concept bottleneck in the continual learning setting, which shows that this direction is promising and is a novel solution to address the catastrophic forgetting as shown in Section 5.1 of the main paper (p.7- p.9).

#2 Comparison with DEN

Following your instructions, we compare CC-CBM (ours) and Finetune-CBM (ours) with DEN in below Table R1. Our methods outperform DEN by 15.07% and 15.81% in $\bar{A}_T$ respectively.

Table R1: Experiment results on CIFAR-100 in a 5-task scenario.

#3 GPT-3 usage and example

CC-CBM is builded upon LF-CBM [1]. It follows the procedure described in LF-CBM to use GPT-3. Example prompts:

• List the most important features for recognizing something as a {class}:
• List the things most commonly seen around a {class}
• Give superclasses for the word {class}:

#4 Summary

To summarize, we have:

1. Discussed in #1 on the details of CC-CBM.
2. Provided in #2 for the comparison with DEN.
3. Discusses in #3 on the GPT-3 usage.

We believe we have addressed all your concerns. Please let us know if you still have any reservations and we would be happy to address them.

Reference:

[1] Label-free Concept Bottleneck Models. ICLR 2023

[2] Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning. CVPR 2022

[3] Training networks in null space of feature covariance for continual learning. CVPR 2021

[4] Gradient episodic memory for continual learning. NeurIPS 2017

I thank the reviewer for their responses. The added results of DEN are good, and it would be good to add DEN to other datasets too. While I think the idea is interesting and results are significant, I also share some of the concerns by other reviewers that the method could suffer from its complexity and the writing could also be clearer to facilitate understanding. I remain my current score rating of the paper.

#4 Further information for additional experiments

Following your suggestion, we performed the zero-shot classification experiment. We found that zero-shot classification on continual learning will lead to bad performance. In the 5-task scenario, we pretrained the backbone only on the first task, and did zero-shot classification for the following tasks. The experiment results of zero-shot classification are much worse than our proposed CC-CBM as below Table R1 shows.

Table R1: Comparison between zero-shot classification and our methods, in the 5-task scenario of CIFAR-100. The experiment results shows that our methods (CC-CBM) outperforms the zero-shot verison in average accuracy $A_T$ and learning accuracy $L_T$ (ability to learn new tasks).

#5 Motivation

We have briefly discussed our motivation in the Introduction section (p.1). Overall, our goal is to design continual learning algorithms which are interpretable. Previous work [6] points out the benefits of human-interpretable concepts for transfer learning.

Meanwhile, we measured the evolution of human-understandable concepts in the models when learning sequences of tasks. The experiment results in Table 3 in main draft (p.9) and Table 9 and Table 10 in Appendix p.19) show that existing continual learning algorithms don’t really preserve human-understand concepts. To prevent this behavior, we think it is reasonable to design an algorithm that aims to mitigate catastrophic forgetting of concepts, which is the main motivation for us to design the proposed Concept Controller. We will add this discussion to the introduction in revision to make the motivation more clear.

#6 Summary

To summarize, we have:

• Clarified in #1 on our contributions when comparing with DER (CVPR 2021).
• Described in #2 on the ablation studies we have done, and why comparing with DER is not a fair ablation study.
• Discussed in #3 about the additional information from GPT-3.
• Provided in #4 for the additional experiments as requested.
• Clarified in #5 on our motivation.

We believe we have addressed all your concerns. Please let us know if you still have any reservations and we would be happy to address them.

Reference:

[1] Post-hoc Concept Bottleneck Models. ICLR 2023

[2] Interpretable Basis Decomposition for Visual Explanation. ECCV 2018

[3] Interpretability beyond classification output: Semantic bottleneck networks. arXiv:1907.10882, 2019

[4] Compare to The Knowledge: Graph Neural Fake News Detection with External Knowledge. ACL 2021

[5] Entity-Aware Dual Co-Attention Network for Fake News Detection. EACL 2023 Findings

[6] Network dissection: Quantifying interpretability of deep visual representations. CVPR 2017.

[7] Label-free Concept Bottleneck Models. ICLR 2023

[8] Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification. CVPR 2023

[9] Learning Concise and Descriptive Attributes for Visual Recognition. ICCV 2023

[10] Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning. CVPR 2022

[11] Dark experience for general continual learning: a strong, simple baseline. NeurIPS 2020

Thank you for your valuable feedback, please see our reply below.

#1 Difference between our methods and DER (CVPR 2021)

We would like to highlight our contributions when comparing with DER (CVPR 2021):

1. We agree that architecture-based methods like DER are explored in other works, as we discussed in Section 2.1 (p.3). However, we are the first one to introduce neuron-level interpretability to continual learning, which is non-trivial. We are happy to hear your comment that our work in this direction is interesting and worth-exploring.
2. As we mentioned in the Introduction session (p.2), interpretability allows us to improve the "method design" in an interpretable manner. This is unexplored in DER and other architectured-based methods.
3. DER requires a different feature extractor for each task to retain learned knowledge. On the other hand, our methods use the same feature extractor to learn all tasks, which is more efficient.

#2 Ablation Study for CC-CBM

We have done ablation studies for CC-CBM in the main results and appendix:

1. When introducing label-free CBM (abbreviated as LF-CBM), we compare the Finetune strategy vs CC strategy in Step 2 (p.6), the experiment results are in the main results (Section 5.1 from p.7- p.9) and Appendix (p.23 - p.26).
2. Ablation studies for freezing $W_C$ and regularizing $W_F$ are in Appendix A.5 (p.17).

These ablation studies have shown the contribution of each part in CC-CBM. Meanwhile, we believe that combining LF-CBM with DER (CVPR 2021) may not be a fair ablation study for our CC-CBM for the following reasons:

1. DER requires a new backbone for each task, while CC-CBM only needs one.
2. DER official code is not complete, since the code to learn a new backbone is missing. The issues have been complained about in their official codebase (https://github.com/Rhyssiyan/DER-ClassIL.pytorch/issues/18).
3. DER is an exemplar-based method, while CC-CBM is an exemplar-free method. We have compared with other exemplar-free methods (EWC, SI, LwF, Adam-NSCL, SSRE) and exemplar-based methods (GEM, MIR, DER-NeurIPS-2020 [11]). Please see Section 5.1 in p.7 for references. Note that the DER-NeurIPS-2020 is a different method published in [11] but happens to have the same name “DER” as the method you suggested (CVPR 2021) and hence we denote [11] as DER-NeurIPS-2020. Overall, our methods outperform other exemplar-free baselines and improve exemplar-based methods when combining them.
4. We have compared with baseline methods that are more recent than DER, e.g. SSRE (CVPR 2022) [10], and show that our results outperforms it in $L_T$ by up to 57.1% in Table 18 (p. 27, appendix A.12). Unfortunately, SSRE did not compare with DER either.

We will add the above discussion into the revision. Thank you for your suggestions!

#3 Additional Information from GPT-3

We would like to highlight several points for the potential issue:

1. Using GPT-3 to general concept sets is firstly proposed from an earlier work LF-CBM [7], which has many concurrent and follow-up works that use language models to create concept bottleneck layers, e.g. see [8, 9]. In fact, introducing additional information is widely exploited when designing CBM [1, 2, 3], and is reasonable task-relevant information that we can leverage.
2. We agree that the potential leaking of CBM is an interesting problem, but we would like to point out that this is not the focus of our work as this is an universal problem in CBM. We will leave it as a future work and add discussion in the revised draft.
3. Using additional information to enhance a model's performance is common in other fields as well [4, 5].
4. Our method is still under the setting of continual learning because during the training phase, the information for each class only appears in one task.

Thus, we believe that our methods are fair in the continual learning setting.

Dear Reviewer Hm1u,

We believe that we have addressed all your concerns and with additional new experiments (see General response and Appendix A. 13, A.14). Please let us know if you still have any reservations and we would be happy to address them. Thank you!

Dear Reviewer Hm1u,

As the rebuttal period is ending within 7 hrs, we would like to ask for the feedback to our rebuttal response.

As we did not hear from you, we are not sure if you have additional questions or comments. We believe that we have addressed all your concerns and with additional new experiments and discussions (see General response and Appendix A. 13, A.14). Please let us know if you still have any reservations and we would be happy to address them.

Thank you!

Thank you for your valuable feedback, please see our reply below.

#1 Scalability

1. Baseline papers [1, 2, 3] mostly focus on the CIFAR dataset. TinyImagenet is the largest dataset in all baseline papers we compared except SSRE [1] which does experiments in ImageNet-100. Due to the rebuttal constraint, we have tried to add additional experiments on large scale dataset (e.g. ImageNet-10), and the results are in Table R1 and R2. The experiment results are still under the same trend: CC outperforms other exemplar-free methods by up to 0.87% in $\bar{A}_T$ and up to 5.77% in $\bar{F}_T$. Meanwhile, can improve exemplar-based methods by combining them by up to 1.83% in $\bar{A}_T$ and up to 2.87% in $\bar{F}_T$.
2. In Table R3 and Table R4, we provide the number of concepts added in the concept bottleneck layer (CBL) for CC-CBM.
3. For large-scale datasets in common continual learning benchmarks, our methods perform well, as shown Table 1 and Table 2 (p.8 -p.9) in the draft and the new ImageNet-10 experiments. Due to the expansion ability of CC-CBM, we believe that we can handle other large-scale datasets as well.

Table R1: Experiment results on ImageNet-10 in a 5-task scenario.

Table R2: Experiment results on ImageNet-10 in a 5-task scenario, the backbones are pretrained on Place365 dataset.

Table R3: Average concepts added per task in a 5-task scenario.

Table R4: Average concepts added per task in a 10-task scenario.

#2 Motivation for the role of concepts and interpretability

We agree that the subnetwork search process can be performed on the classification layer directly, just like DEN [2]. However, this gives us no idea about what knowledge and information is learned and preserved in the model, which lacks interpretability. This is the main motivation of our proposed method. In the detailed motivation (please see #5 in the Author Response (2/2) to Reviewer Hm1u), we saw the catastrophic forgetting of concepts in other baselines, which motivated us to design the proposed Concept Controller. As experiment results in the main results (Section 5.1 from p.7- p.9) and Appendix (p.23 - p.26) show, controlling concepts improves the model’s performance in continual learning. Moreover, our methods provide interpretability, which is different from other continual learning algorithms. Finally, We added a comparison between our method and DEN, please see #2 in the Author Response to Reviewer 6BcB.

#3 Questions regarded to vision-language aligned (VL-aligned) model

1. We build CC-CBM based on LF-CBM [4]. Following LF-CBM’s procedure, we use CLIP to calculate the activation matrix $P$ as described in Section 2.2 (p.3). We can replace CLIP with other VL-aligned models as long as they have a text encoder and an image encoder to calculate matrix $P$. We would like to point out that the choice of the VL-aligned model is related to LF-CBM’s design instead of the continual learning problem. We will add this discussion in the revision and leave it as a future work.
2. As we discussed in #2, controlling concepts provides interpretability and improves performance. Just using a VL-aligned model is not sufficient to understand the concepts inside itself.

#4 Recent baselines in experiments

1. We have compared the two baselines within two years: Adam-NSCL[3] and SSRE [1]. As you requested, we compare our methods with ICICLE, and the experiment results are in Table R5. Our method outperforms ICICLE by 2.02% in $\bar{A}_T$ and 31.68% in $\bar{F}_T$.
2. ICICLE focuses on part-based prototype concepts. Our work focuses on text-based concepts instead, which allows more general interpretability. Meanwhile, it is only suitable for particular model architectures whereas our methods are suitable for any CNN-based models.

Table R5: Experiment results on CUB200 [5] in a 5-task scenario. The backbones are pretrained on Place365 dataset.

#5 Details for freeze concepts and subnetwork

1. In CC, classes in new tasks may reuse old concepts learned in previous classes. CC freezes the subnetwork from the concept units to the bottom, but does not freeze the associations between the classification layer and concept units. Therefore, the new classes can reuse old concepts. We added new experiments on CUB200 [5], which is a bird species classification dataset and has many concepts overlapped between classes. We measured the forward transfer metric [9] (FWT) to understand whether old concepts help new classes. The experiment results are in Table R6, which shows our method can reuse concepts efficiently and improve a model’s performance.
2. We added the subnetwork size of different datasets in appendix A.14 (p.29). Overall, a more complicated dataset has a bigger subnetwork.

Table R6: FWT on comparing our methods with baselines.

#6 Generalization to transformers

We would like to point out that whether concepts can be disentangled in transformers is still under research [6, 7, 8]. We believe this is an interesting future work after transformers’ interpretability has made solid progress.

#7 Presentation

1. Thank you for pointing out the unclear part of our paper. We have revised them in the revision.
2. In Section 3, the subnetwork of a concept unit is frozen. In Section 4, we freeze the concept unit itself. Concept unit is a neuron as we stated in Section 1 (p.1).
3. Due to the page limit, we are not able to put Figures 6 or 7 in the main paper. The discussion for CC’s interpretability is in Section 3 (p.5).

#8 Summary To summarize, we have:

1. Discussed in #1 on our methods’ scalability, and add a new experiment on a large-scale dataset.
2. Discussed in #2 on the role of concepts and interpretability.
3. Discussed in #3 on questions related to vision-language aligned models.
4. Discussed in #4 on the latest baselines we have compared, and add a comparison with ICICLE.
5. Described in #5 on the mechanism and details of subnetwork.
6. Described in #6 on the generalization to transformers.
7. Discussed in #7 on the presentation of our paper.

We believe we have addressed all your concerns. Please let us know if you still have any reservations and we would be happy to address them.

Reference:

[1] Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning. CVPR 2022

[2] Lifelong Learning with Dynamically Expandable Networks. ICLR 2018

[3] Training networks in null space of feature covariance for continual learning. CVPR 2021

[4] Label-free Concept Bottleneck Models. ICLR 2023

[5] The caltech-ucsd birds-200-2011 dataset. 2011.

[6] An interpretability illusion for bert. arXiv:2104.07143 2021

[7] "Toy Models of Superposition", Transformer Circuits Thread, 2022.

[8] "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning", Transformer Circuits Thread, 2023.

[9] Gradient episodic memory for continual learning. NeurIPS 2017

Dear Reviewer zdWA,

As the rebuttal period is ending within 7 hrs, we would like to ask for the feedback to our rebuttal response.

As we did not hear from you, we are not sure if you have additional questions or comments. We believe that we have addressed all your concerns and with additional new experiments and discussions (see General response and Appendix A. 13, A.14). Please let us know if you still have any reservations and we would be happy to address them.

Thank you!