IBCL: Zero-shot Model Generation for Task Trade-offs in Continual Learning

We appreciate your thoughtful comments, and we invite you to read our revised manuscript, which significantly improves readability and compares to two more baselines: A-GEM [1] and L2P [2]. Due to time constraints, we are unable to finish working on the other baselines you suggested, but we will keep working on them.

We indeed believe CLIP [3] is a well-established zero-shot approach. However, it requires natural languages as prompts, which is unavailable in our settings. Therefore, we choose L2P, where prompts are trainable parameters.

Our new experiments show that IBCL is able to obtain high continual learning performance, thanks to its probabilistic Pareto-optimality. Moreover, it is able to maintain a constant training overhead per task, regardless of how many preferences are requested. Compared to rehearsal-based baselines, like GEM and A-GEM, the performance is not worsened while the training overhead is significantly reduced.

Although prompt-based baselines, like L2P, also achieves constant training overhead per task, IBCL has a much higher performance overall. This is due to the fact that there is no standard way of specifying preferences over tasks to prompt-based methods, and we made an attempt to train one prompt per task, and use a preference-weighted sum of the learned prompts to specify a preference. The resulting performance is poor except for on the CelebA benchmark. As a consequence, we point out how specifying task preferences in prompt-based continual learning can be a future research direction.

Again, we appreciate your comments that helped us improve the overall quality of the manuscript, and we sincerely hope you can update the rating accordingly.

[1] Chaudhry, Arslan, et al. "Efficient lifelong learning with a-gem." arXiv preprint arXiv:1812.00420 (2018).

[2] Wang, Zifeng, et al. "Learning to prompt for continual learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

We appreciate your thoughtful feedback, and we invite you to read our revised manuscript, which significantly improves readability and addresses your concerns.

“The paper itself is too dense.”

We have improved readability in our revised manuscript. Please also refer to our comment to everyone for a list of our modifications.

“The proposed setup seems contrived.”

Our setup is simplified and clarified in the revised manuscript. Please read our revised version of the Introduction for a better structured story.

“Poorly performing models can also be sampled from HDRs.”

We are able to completely avoid sampling these models by modifying the hyperparameter alpha. Please refer to our ablation studies in Appendix H.3 of the revised manuscript, where we are able to always sample near Pareto-optimal models with a small alpha (0.01). In fact, avoiding poor models does not depend on validation at all, and we made this correction in our latest manuscript.

“The experimental studies are limited.”

We have added experiments based on the suggestions of reviewer 79Li. As in our revised manuscript, GEM and A-GEM are selected because they are rehearsal-based, and there is an established method to specify task preferences on rehearsal-based continual learning – that is, using preferences as weights on loss functions. The other methods, such as prompt-based continual learning, have no standard way to specify a task preference so far. In our revised experiments, we made an attempt to use a preference-weighted sum of prompts in L2P, but it ended up with poor performance except for the CelebA benchmark.

“Can IBCL be deployed to single-preference benchmarks?”

Yes. We can request IBCL to generate models for any number of preferences (including one preference only), and the total training overhead (measured in number of batch updates) remains constant, regardless of the number of preferences requested.

“What drives the assumption of task similarity?”

Task similarity assumption is needed to mitigate the possible model misspecification, which in turn could lead to catastrophic forgetting even when Bayesian inference is carried out exactly. This is now explicitly mentioned after presenting Assumption 1, and discussed in more depth in Appendix E.

“Does the diameter of F(r) connect to the expected continuous distribution shifts that can be handled? How does the choice of r affect the applicability of IBCL?”

Yes: assuming a bounded diameter implies that the tasks are somehow close to each other. In other words, the distribution shift between the distributions for task i and j, p_i and p_j respectively, must not be too pronounced. If we were to allow arbitrary differences between the tasks the agent needs to accomplish, as a consequence of IBCL being embedded in a Bayesian framework, we may incur catastrophic forgetting. As pointed out in Appendix E, in the future, we will generalize IBCL to ProtoCL instead of Bayesian learning to let go of Assumption 1. Having a bounded diameter does not limit the applicability of IBCL, since in the majority of applications the tasks we want to accomplish are reasonably similar to each other. This is for example the case of classification, where Assumption 1 should always be satisfied, with a reasonably small diameter r.

Again, we appreciate your comments and hope the rating can be adjusted accordingly.

Dear reviewer, thank you for your positive rating and thoughtful comments. We have updated a further revised manuscript that addresses your concerns.

“The HDR concept is not immediately clear.”

We modified Figure 1 as well as the paragraph right after it. In Figure 1, the full names of FGCS and HDR are provided. In the paragraph, we give a more detailed description about what FGCS and HDR are. We also added a reference to Section 2 at the end of the paragraph, to show where formal definitions can be found.

“A more detailed explanation of how IBCL differed from [1] [2] would be beneficial.”

We detailed this explanation in the last two paragraphs before “Contributions” in the Introduction. That is, compared to existing efficient knowledge transfer methods like [1], IBCL enables learning models under specified task trade-off preferences. Compared to existing methods that enable specified preferences like [2], IBCL is not only much more efficient by maintaining a constant training overhead per task, but also guarantees probabilistic Pareto optimality. Moreover, IBCL is a Bayesian CL method, hence it mitigates catastrophic forgetting by design. We prove this and discuss Bayesian CL at length in Appendix A.

“Why not choose a smaller alpha, e.g. 0.001?”

Indeed, a smaller alpha is more preferable. However, when alpha becomes very small, like 0.01, we already have a very high probability of sampling the Pareto-optimal model, and this probability can be hardly improved by using an even smaller alpha. For example, in Figure 2, the shaded blue area that illustrates IBCL’s performance range has already converged to a single curve. This means that the worst sampled model and the best sampled model (Pareto-optimal model) have a negligible difference. Therefore, there is no need to keep shrinking down this difference by using a smaller alpha.

We sincerely hope that you can consider improving the rating based on the changes. Thank you in advance.

[1] Learning the Pareto Front with Hypernetworks.

[2] Controllable Pareto Multi-task Learning.

Thank you for your comments. We believe there are some misunderstandings in our work, and we are happy to address them here. We would also appreciate the reviewer to refer to the comments and our discussions with the other three reviewers. First, for the weaknesses:

1. “In Algorithm-1 paper shows the FGCS Knowledge Base Update, which is based on some distance … There are many similar works based on entropy, loss, and other metrics.”

This distance-based distribution selection is not the main focus of this paper. We use this as a convenient subroutine to exclude distributions that are very similar to what is already cached, so that we have a sublinear buffer growth. Our main contribution is modeling the representation learned in continual learning as an FGCS, which allows zero-shot identification of probabilistically Pareto-optimal models.

2. “The paper is motivated as we have a large number of users, and the model is scalable for the larger number, but the results are shown only for the 5/10 task, which is small and does not align with the motivation.”

The number of tasks and the number of preferences are two different concepts. We can have a small number of tasks, but with each task we have a large number of preferences. These two numbers are distinguished by $n_{pref}$ and $i$ in our Table 1.

3. “The baseline papers are outdated.”

We have explained the baseline selection in Section 5.1, as well as an explanation in the last paragraph before Figure 1 in Section 1 (“However, we identify two major disadvantages …”). That is, so far, people can only specify preferences over tasks on rehearsal-based continual learning algorithms, by using a preference-weighted sum on the losses of the rehearsal memory, making them comparable to IBCL. The "new" and “efficient” algorithms, such as model-based or prompt-based algorithms, do not enable an input of preferences. However, we still made an attempt to input preferences to L2P after the discussion with reviewer 79Li. This is a relatively new prompt-based method for comparison.

4. “The ablation results are insufficient to evaluate the model.”

We would appreciate the reviewer to point out how to make the ablation studies sufficient. We conduct these ablations by using different hyperparameters $d$ and $\alpha$ and the results already show a clear trend of how the performance can be affected. The training procedure is exactly the same as in the main experiments, which is detailed in Section 5.1 and Appendix H, just with different hyperparameters given.

Then, for the justification in weaknesses:

5. “The claim in the motivation is not supported in the experiments.”

Our experiments already show that IBCL can achieve high continual learning performance metrics, supporting the claim of probabilistic Pareto-optimality. We have also analyzed its overhead in Table 1, supporting the claim of zero-shot model generation under preferences. We are not sure why the reviewer thinks the claim is not supported.

6. “ Lack of motivation about Pareto-optimality.”

Pareto-optimality is a common goal in multi-objective optimization, which is the nature of multi-task and continual learning. This motivation is already well supported by the cited previous work. Please refer to Kendall et al. 2018, Sener and Koltun 2018, Lin et al. 2019 and Lin et al. 2020, cited in our introduction section.

7. “The computation of the HDR are not discussed.”

The computation of HDR is an established standard procedure, which we explain right after Theorem 4 and cite Juan et al. 2022 for an R package that implements it. This is also not our focus in this paper. In practice, we use Gaussians to model the parameter distributions, and we select an interval, with the lower and upper bounds symmetrically on the two sides of the mean, such that the enclosed probability density function has an integral = 1 - $\alpha$. The two bounds are computed as the inverse cumulative density function (icdf) of a Gaussian at the quantiles of 0.5 - (1 - $\alpha$) / 2 and 0.5 + (1 - $\alpha$) / 2. A Gaussian’s icdf can be very fast computed by tools like scipy.