Generate to Discriminate: Expert Routing for Continual Learning

Thank you very much for your time and thoughtful feedback! We have now made the following changes which we believe address all your concerns and strengthen the paper

• Added additional results on ER (50/class) for DomainNet and DermCL (see Tables 1,3)
• Added additional results on CaSSLe for CORe50 and DermCL (see Tables 2,3)
• Added clarification to Table 1 caption on the difference between G2D vs. G2D (Full FT)

In tab 1: where is ER? Whats CaSSLe?...Why are different methods compared for different (vision) datasets

Thank you for bringing this to our attention. As per your suggestion for completeness, we have run ER and CaSSLe numbers for all remaining baselines and have updated Tables 1,2,3 accordingly, which now unifies all baselines assessed for the vision datasets. TLDR: Our method outbeats both baselines across all datasets by a large margin. We have included a summary of the new results in the table below (*denotes results obtained from [1]):

For further clarification: In our earlier version, these two baselines were only evaluated on selected datasets because previous works [1] [2] included ER and CaSSLe results only on selected datasets and we directly obtained those results. CaSSLe [2] is a method for continual self-supervised learning and ER cannot be performed in our setup due to data sharing constraints; and thus, we had not focused too much on evaluation of these two baselines in our earlier draft. As noted, we have now updated our draft to address your concern.

What’s the difference between G2D and G2D (Full FT) –> roughly explained much later in the text?

We apologize for any confusion, due to the description coming much later in the text. We have added this clarification in the caption of Table 1, where G2D (Full FT) is first mentioned. We further elaborate on the difference here: The difference between G2D and G2D (Full FT) is that G2D is using parameter-efficient approaches for fine-tuning, while G2D (Full FT) is a variant of our method that uses full fine-tuning of all parameters. For G2D, our classifier models are fine-tuned using LoRA [3], where low-rank “update matrices” are added to the query and value matrices of the attention blocks of the base model. During fine-tuning, only these matrices are trained, while the original model parameters are kept frozen. At inference time, the update matrices are merged with the original model parameters to produce the final output.

Will the new benchmark dataset be published?

Yes, we plan to publicly release the newly proposed benchmark dataset DermCL along with all code associated with this work.

We hope these answers and additional experiments help clarify your questions. We would be happy to answer any further questions you have. If you do not have any further questions, we hope that you may consider raising your score. Thank you again for your constructive feedback!

References

[1] S-Prompts Learning with Pre-trained Transformers. Wang et. al 2023

[2] Self-Supervised Models are Continual Learners. Fini et. al 2022

[3] Lora: Low-rank adaptation of large language models. Hu et. al 2021

Thank you very much for your time and thoughtful feedback! We have now made the following changes which we believe address all your questions and strengthen the paper

• Added a section on implementation details of Generative Replay in Appendix C.3
• Added a summary of these clarifications to Section 5.2 with a reference to Appendix C.3
• Updated section 4.2 to correct description about ER (see below)

... does not provide enough experimental details to properly assess whether the comparison between G2D and Generative Replay is performed in a fair manner…it is unclear to me whether Generative Replay has been implemented in an optimal manner

We have updated the draft with the following clarifications (see Section 5.2 and Appendix C.3). Further, we plan to release all code regarding our implementation of both G2D and Generative Replay, for the community.

We sought to make the comparison fair by (a) using the same replay buffer (i.e., fixing to the same set of synthetic data) for both methods and (b) using the same fine-tuning approach for classifier models (see below):

How is / are the classifier model(s) finetuned?...Is this approach of fine-tuning the same for the classifier models of G2D and the classifier model of Generative Replay?

To be explicit, we will answer this question for (1) G2D (Full FT) vs. Generative Replay and (2) G2D vs. Generative Replay,

1. Comparison of G2D (Full FT) vs. Generative Replay:

Yes, the approach remains the same. Both approaches (for fine-tuning classifier models of G2D (Full FT) and the classifier for Generative Replay) are done by:

• Sequential fine-tuning from the previous domain checkpoint
• Hyperparameter tuning is done over the same set of hyperparameters (learning rates, batch size, etc.) and chosen based on performance on the held-out validation set of the first domain. Hyperparameters remain fixed throughout the domain sequence, for both approaches (see Appendix C.1 and C.2 for hyperparameter details).

2. Comparison of G2D vs. Generative Replay:

The fine-tuning procedure (above two bullet points) remains the same. There is just one difference: For G2D, we employed parameter-efficient fine-tuning techniques to improve the parameter efficiency aspect of our method:

• G2D uses a parameter-efficient fine-tuning method (i.e., LoRA)
• Generative Replay uses the same full fine-tuning method as G2D (Full FT).

For generative replay, we opted against employing parameter-efficient fine-tuning (LoRA) techniques due to its inferior performance compared to full fine-tuning. The drawback arises from the sequential learning of multiple domains using limited trainable parameters in LoRA, which adversely impacts performance. Therefore, we presented the optimal performance of Generative Replay, achieved through the full fine-tuning procedure, mirroring the approach used in G2D (Full FT). We apologize that this might have not been clear. We have updated the draft to clarify this point (see Section 5.2 and Appendix C.3). Thank you for raising this clarification point!

In section 5.4. it is stated “we fine-tune only 1.04 ~ 2.5% of trainable parameters”. How was this percentage decided? How is it decided which parameters are fine-tuned?

The number of trainable parameters (so-called “update matrices”) and which of them are fine-tuned is controlled by (1) the dimension used by the LoRA update matrices (i.e., r) and (2) the scaling factor (i.e., lora alpha), which gives the flexibility to balance a trade-off between end performance and parameter efficiency [2]. For our vision experiments, we set the default values for both hyperparameters (r = 16, lora alpha = 16). For our text experiments, we set the rank and scaling factor to 32 to match the performance of parameter-efficient finetuning with full finetuning on a single domain. These parameter choices result in 1.04 ~ 2.5% of trainable parameters.

... With Generative Replay, how are the loss on the replayed data and the loss on the data from the current task weighed? Are they simply added? Or are they balanced in such a way as to approximate the joint loss over all domains so far?

Following standard practice [1], we simply add the loss, in an unbiased manner.

Could the authors explain why they took the S-iPrompts results from the Wang et al (2022a) paper, but not the S-liPrompts results?

As noted in Section 4.2 (end of third paragraph), for a fair comparison, we fix the model backbone of the downstream classifiers as ViT-B16 pretrained with ImageNet checkpoint. This is the same across all baselines and our methods. S-liPrompts uses a different pretrained backbone in the CLIP family [3]. Thus, we compare with the S-Prompts method (S-iPrompts), which uses the same pre-trained architecture.

On p5 towards the bottom the authors claim that ER with a limited buffer size is an upper bound for generative replay. This does not seem correct.

Thanks for catching this. We meant to clarify that Experience Replay (ER) with a lot of samples (not “limited buffer size”) is the upper bound. Indeed, Experience Replay (ER) with “limited buffer size” is not the upper bound. We have made this revision in the draft (see Section 4.2).

We hope these responses helped clarify your questions. We would be happy to answer any further questions you have. If you do not have any further questions, we hope that you may consider raising your score. Thank you again for your constructive feedback!

References

[1] Continual Learning with Deep Generative Replay. Shin et. al 2017.

[2] Lora: Low-rank adaptation of large language models. Hu et. al 2021.

[3] S-Prompts Learning with Pre-trained Transformers. Wang et. al 2023.

As seen above, for direct comparison, we compare with all variants (Full FT, LoRA) for both methods (Generative Replay, G2D).

For Generative Replay (LoRA), we use the same LoRA hyperparameters (rank = 16) as we did in G2D (LoRA), resulting in the same number of trainable parameters 1.29% for both DomainNet and CORe50. For comprehensiveness, we also ran experiments where we “match” the number of total trainable parameters, which we term Generative Replay (LoRA - Matched). For instance, for DomainNet, we have 6 domains resulting in 6 expert classifiers each with 1.29% trainable parameters. Thus, in terms of total LoRA parameters, there is a discrepancy between (1) sum of expert classifiers’ LoRA parameters for G2D and (2) single classifier’s LoRA parameters for Generative Replay. To account for this discrepancy, we adjust the rank hyperparameter (16 x num_expert_classifiers) to match the number of total parameters. This results in the exact same number of trainable LoRA parameters across both methods, resulting in using rank=(16x6)=96 for DomainNet and rank=(16x8)=128 for CORe50.

We can clearly observe that even after carefully matching the number of total trainable parameters, we still observe a performance drop when using LoRA for Generative Replay. We hypothesize that using LoRA more often hurts performance for Generative Replay as the classifier here has to perform the more challenging task of classification on all seen domains. On the contrary, for G2D, each classifier expert has a much simpler task of classification for a specific domain. We have reflected these clarifications and additional experiments in the revised draft (see Appendix C.3).

Considering the above issue, I now appreciate that the most direct comparison in the paper between G2D and Generative Replay is between “G2D (Full FT)” and “Generative Replay (ours)”. Based on this comparison, however, the results are substantially less clear / convincing.

As you note, we acknowledge that when comparing G2D (Full FT) vs. Generative Replay for two of the four benchmarks (DomainNet and CORe50), the performance gains are less than G2D vs. Generative Replay. As per your concern, we have reworded clarifications in Appendix C.3 to better reflect this and have run the additional experiments for the two benchmarks for direct comparison between Full FT variants and LoRA variants (Same experiment as above; including Table here too for better readability):

This direct comparison highlights benefits of 0.32 ~ 8.66% improvement for the Full FT setting (see above Table and Tables 1,2,3,4) and 4.9 ~ 8.78% improvement for the LoRA setting (see above Table), which has additional benefits of parameter efficiency. We note that for both settings, our method still outperforms Generative Replay across all benchmarks.

Further, our findings highlight how to better utilize recent parameter efficient fine-tuning techniques in continual learning, in that when learning independent weights, performance is often at least on-par with full fine-tuning. On the other hand, learning one set of parameters for multiple tasks can lead to performance drops due to this restriction of parameters; therefore, not being able to utilize PEFT solutions in an optimal fashion. Understanding this phenomena more in detail is an interesting direction for future work.

Thank you very much for your continuous engagement and useful review! We will try to address your questions below:

...If LoRA helps to improve performance when fine-tuning the classifier of G2D, and it helps to improve performance when continually training the generative model…, it is surprising that it is unable to improve performance when continually training the classifier model for Generative Replay. I would say that this requires more explanation / demonstration.

We have understood your points as follows: (i) if LoRA helps improve performance when fine-tuning the classifiers of G2D and (ii) if LoRA helps improve the performance when fine-tuning the generator (for both G2D and Generative Replay) (iii) then why does it not help for fine-tuning the classifier for Generative Replay.

To address your concern, we have run two additional experiments: Generative Replay (LoRA) and Generative Replay (LoRA - Matched) (see Appendix C.3; also detailed below). First, we will address your first two points (i) and (ii). Then, we will move on to discuss our new results, which answers point (iii).

First, on to point (ii), we apologize for the confusion and want to clarify that LoRA does not help improve performance, when fine-tuning the generator (for both G2D and Generative Replay) (see Appendix C.5). Our main reason to employ the LoRA approach for the generator was for parameter efficiency concerns, as noted in Section 4.3. We used the Classification Accuracy Score (CAS) [1], a widely employed metric to measure conditional generative models' quality, to assess performance of the generator. On DomainNet, full fine-tuning the generator achieves a CAS of 66.28 ± 0.31, while LoRA fine-tuning reaches 64.17 ± 0.20, with only fine-tuning at most 2.5% of trainable parameters (see Appendix C.5). Noting this performance vs parameter efficiency tradeoff, we proceeded with the LoRA-based implementation choice to make the synthetic data generation process parameter efficient,. We have added this clarification in Appendix C.5 and a footnote in Section 4.3 (due to current page limit constraints). Note that this LoRA-based implementation choice remains the same across both methods (Generative Replay and both variants of G2D), since we fix to the same set of synthetic data for fair comparison.

As you note in point (i), for two out of four benchmarks (DomainNet, CORe50), we see that LoRA does not hurt performance for fine-tuning the classifier for G2D, but rather results in slight performance improvement over full fine-tuning. We hypothesize that in the case of G2D, fine-tuning a smaller set of parameters is more tractable and perhaps leads to less overfitting, since we are using distinct sets of weights for simpler tasks (i.e., classification for a specific domain), relative to the more challenging task of training a common classifier for Generative Replay (i.e., classification on all seen domains).

For the aforementioned two benchmarks of concern, we have run the following additional experiments: Generative Replay (LoRA) and Generative Replay (LoRA - Matched). Highest performance is bolded.

For Generative Replay, I am surprised by the authors’ choice to “simply add the loss” of the replayed data to the loss on the current task. This creates an imbalance between G2D and Generative Replay. As described in subsection 3.2, the domain discriminator of G2D is trained on the union of synthetic data from all tasks so far...the domain discriminator is trained in a balanced way (i.e., all tasks so far are given equal weight). However, there is no such balance for Generative Replay. Because with Generative Replay, …, the loss of the classifier is constructed for 50% based on synthetic data from all past tasks and for 50% based on data from the current task…there is an imbalance (i.e., the last task is given higher weight than the previous tasks). I think it is likely this might negatively affect the performance of Generative Replay.

We apologize for the confusion here. The classifier for Generative Replay is also trained on the union of synthetic data from previous tasks and real data from current tasks (i.e., all tasks so far are given equal weight). This is what we meant by “in an unbiased manner” (i.e., correctly balancing the weights of all seen domains); otherwise, this would lead to a biased classifier. More explicitly, if we have seen t-1 tasks so far and currently training on t-th task, then that means we consider all t tasks with equal weight (1/t). Sorry again for the confusion, and we have stated this much more clearly in the revised draft (see Appendix C.3).

Thank you again for your comprehensive and useful review! We hope the above clarifications and additional experiments addresses your concerns. Do you have any further questions? If you have any more suggestions or feedback, please let us know

References

[1] Classification accuracy score for conditional generative models. Ravuri et. al 2019.

[2] CORe50: a New Dataset and Benchmark for Continuous Object Recognition. Lomonaco et. al 2017.

… initial motivation of the setting in the paper, where model weights may be made available, but not the actual data used for training. I have trouble seing healthcare institutions open-sourcing generative models of their data, but not the actual data itself. … authors could point me to such instances.

The setting where model weights may be shared but not the actual training data is a well-known setting in the healthcare domain [1][2][3]. We elaborate on two examples: (1) Kamran et. al 2021 presents a multisite external validation study for early identification of COVID-19 patients at risk of clinical deterioration, which requires sharing the model trained on private EHR data from one US hospital with 12 other US medical centers; (2) Ulloa-Cerna et. al 2022 presents a multisite external validation study for identifying patients at increased risk of undiagnosed structural heart disease, which requires sharing the model trained on private EHR data and patient echocardiography reports from one site with 10 other independent sites. While these are generally examples of discriminative models being shared across facilities as opposed to generative models, this demonstrates the general principle that in such domains, model sharing is often permissible in settings where data sharing is not.

On one hand, as you note, it seems intuitive that healthcare institutions might be queasier about sharing generative models than sharing discriminative models.

On the other hand,

• Healthcare institutions are even queasier about sharing real data — and to this end, there is a large mainstream line of work investigating the use of generative models for direct sharing or for producing synthetic datasets that could be disseminated in lieu of actual patient data [4][5][6].
• From the standpoint of most contractual or regulatory requirements, it is not yet clear if generative models sit in a different category than discriminative models or if they should just follow the same current regulatory requirements.
• How institutional practices develop and the regulatory environment evolve will be informed, to a large degree, by exploratory research that characterizes both (i) the potential benefits and (ii) the potential risks associated with the dissemination of generative models trained on medical data. We see our research as helping to elucidate the potential benefits.

We find this broad ongoing discussion on the utility of generative models in real-world deployment settings might be generally useful for the community. Therefore, we have updated the draft to include this ongoing discourse (see Appendix D).

Thank you for your thoughtful review! We will try to address your concerns below:

...discuss the computational cost of the method. What is the training cost of having to train two additional models (task classifier and generator) compared to expert learning?

Thank you for raising our attention to this. We have added the following paragraph to Appendix C.4, which details the computational cost of our method:

“Our method incurs an additional total of 6-8 hours of computational cost on a single A6000 GPU, due to fine-tuning and sampling from the generator. Training the task discriminator, which takes less than 1-3 hours is done in parallel with training the expert classifier, so given that we permit the use of one more GPU, it does not incur additional compute time.

This increase in compute cost (compared to expert learning), results in a substantial performance improvement - up to an absolute 8.1 point increase (see Table 4). In high-stakes applications such as healthcare, where missing even a single positive case (e.g., a fatal disease) could have critical consequences (e.g., a patient’s death), this performance gain is significant, and thus, we deemed this amount of tradeoff meaningful.”

How is the task discriminator trained? Is it trained from scratch at every new domain, or continually learned?

The task discriminator is trained from the ViT-B16 pre-trained ImageNet checkpoint (vision) or BERT-base pre-trained checkpoint (text) at each domain.

How does this approach scale? My understanding is that it does so poorly if the task discriminator is not trained continually.

As per your concern, we performed an additional experiment (added to Appendix B, Table 5 with a reference in the main text in section 3.2), where we train the task discriminator in a continual manner. Note that this turns the learning of domain discriminator into a class-incremental continual learning problem, introducing a new challenge on top of our original domain incremental learning problem.

We find that the performance impact of continual training of the discriminator is minimal, implying that we can continually train our discriminator model. To assess the continuous training of the discriminator, we selected the dataset with the most number of domains, which is the CORe50 benchmark with a total of 8 different domains. This results in the performance of 88.13 ± 0.01 (averaged over 5 random seeds), while our original implementation results in 89.11 ± 0.30 (see Table 5). Thus. this continual fine-tuning of the domain discriminator results in a performance drop of less than 1 point. Therefore, our conclusions remain the same in that we still outperform existing state-of-the-art baselines by a substantial margin.

... authors don't quite understand the computational efficiency related to PEFT approaches; taking LoRA for example, the computational cost saved from not performing a gradient update step … is quite small compared to … compute forward and backward passes …. The "gains" from peft are really in parameter efficiency …

Our focus in elaborating on the LoRA fine-tuning techniques was to touch on parameter efficiency and not computational efficiency. While we have been careful to describe these methods as “parameter-efficient”, we realize there is a reference to “computational efficiency” in section 4.3 which can potentially confuse the reader. We apologize for this phrasing. To address this, we have updated the draft.