IBCL: Zero-shot Model Generation under Stability-Plasticity Trade-offs

We appreciate the reviewer for the positive rating. We address the reviewer’s concerns as follows. If any concerns still persist, we are more than happy to discuss them and edit our paper accordingly. We will be very grateful if the reviewer considers improving the rating.

1. It will be best to have additional ablation studies.

We thank the reviewer for pointing this out. We have added additional ablation studies on different priors and different choices of $beta$ in Appendix I.

2. Does IBCL still have benefits when applying to larger models?

When solving CLuST problems, IBCL does have benefits in efficiency compared to state-of-the-art methods. As current methods require running an optimization of a loss function when generating a model, while IBCL only needs a convex combination. This benefit applies to various scales of models. Still, it will be an interesting future research to identify a use case on large-scale models. This is mentioned in our updated Section 6.

3. For the optimality of $p_{\bar{w}}$: (1) Is the convex combination of true distributions only under cross entropy loss? (2) If so, can it be extended to other loss functions, such as L2 or absolute loss?

We thank the reviewer for these insightful questions. It is indeed true that $p_{\bar{w}}$ minimizes the entropy loss. We did not think of the possibility of other losses because we treated the preference as given by/known to the user. One potential research direction is to generalize IBCL, so that it can derive the preference vector $\bar{w}$ from some inputs. For example, we may learn this preference from additional sequential prompts [1]. In that case, the preference vector itself might be different according to the design, including what loss is used. Once again, we thank the reviewer for this suggestion, and we have edited Section 6 accordingly.

[1] Wu, Yiqing, et al. "Personalized prompt for sequential recommendation." IEEE Transactions on Knowledge and Data Engineering, 2024.

We appreciate the reviewer’s thoughtful comments. Possibly due to the presentation, we believe there are certain results already included in the paper but the reviewer has missed. We have edited the paper accordingly to improve the clarity. If any concern still persists, we are more than willing to discuss with the reviewer and edit our paper. We would be very grateful if the reviewer considers improving the rating.

1. Need to clarify the motivation.

Our motivation is clarified in the second paragraph of Section 1. Specifically, the goal is to solve the CLuST problem efficiently by finding a method that generates every customized model per preference fast. This is because when there are a large number of preferences, if training each customized model is expensive, the overall cost will accumulate to a tremendous amount.

2. How to get preference weights?

In this paper, we assume preference weights are given, which is the same assumption as in [1]. How to obtain preference weights shall be an interesting future research direction. One potential solution is to learn the preferences by sequential prompts [2]. We have added this point in Section 6.

3. What is a setting of infinite preferences?

For infinite preferences, one example would be the movie recommendation system in our Section 1. As there may be a large number of customers, and each customer’s taste in movie genres may vary throughout time, there is potentially an infinite number of preferences. If the company has to train one model per preference, the cost would be huge. Therefore, a more efficient way of generating models to adapt to different preferences is needed, and IBCL is one solution. Generally, when there is a large number of users for a model to customize, IBCL has the advantage of efficient customization. We have edited Section 1 for clarity.

4. Need to clarify the presentation in multiple places.

We thank the reviewer for pointing this out. We have made edits accordingly in the new version of paper. (1) We edited objective 2 in Section 3.2, by defining $\hat{q}_{\bar{w}}$ here. (2) We edited Algorithm 1, and added an explanation of variational inference after Algorithm 1 with a reference, for those who are unfamiliar with this procedure.

5. In Theorem 2, how to find Pareto-optimal parameters? Is there a guarantee?

The Pareto-Optimal (PO) parameters are guaranteed to belong to the Highest Density Region that we build. Our algorithm does not find the PO parameter, but instead the narrowest region that contains it with high probability. In spirit, this result is very similar to what conformal prediction does (for predicted outputs, rather than parameters of interest). In practice, we can sample multiple parameters from the HDR, to estimate the Pareto-optimal parameters, as what we have done in the experiments — we sample 10 models per HDR and average them for estimation.

6. How is the performance in forgetting?

We thank the reviewer for pointing out a potential issue in presentation clarity. We do measure forgetting metrics by backward transfer. These are in the every third subfigure in Figures 3, 4, 5 and 6, and discussed in the paragraph “As illustrated in the figures, IBCL has a slightly negative backward transfer in …” in Section 5.2. To make our presentation more clear, we have edited Section 5.1 on the setups.

7. Are there ablation studies on hyperparameters such as threshold $d$, and why choose equal $\beta$?

In our edited version, we have ablation studies on threshold $d$, significance level $\alpha$, prior std sizes, numbers of priors, and $\beta$ in Appendix I. Equal $\beta$ is a choice of convenience, and the ablation studies show that it has the same performance as randomized $\beta$.

(continue to the next comment)

I thank the authors for the response! Some of my questions have been resolved. However, I am not very satisfied about the answers to 2, 5 and 8. For 5, my question is about if there is any theoretical guarantee on the Pareto-optimality (for any solution or for the Highest Density Region). If yes, could I find any result in the paper. For 8, the assumption is still strong. It is still hard to guarantee that the true data generating processes pertaining to different tasks are not too distant from one another within a radius of $r$. For example, how to ensure and validate this assumption in practice? In many cases, the distributions over different tasks can be quite different from each other. What if $r$ is very large? How does it impact on the final performance?

Best, Reviewer

We appreciate the reviewer for such a positive rating. Here we answer the question.

What are the benefits of IBCL compared to linear weight interpolation?

The major benefit is that using Bayesian models augments the search space. Say we have two models, parameterized by $\theta$ and $\theta’$, linear weight interpolation leads to a search space

$\Theta(\theta, \theta') =$ {$\theta'' = w\theta + (1-w)\theta' \space | \space w \in [0, 1]$}

In contrast, if we adopt Bayesian models $q = \mathcal{N}(\theta, \sigma^2)$ and $q’ = \mathcal{N}(\theta’, \sigma^2)$, where $\sigma$ is some selected std, we have

$Q(q, q') =$ {$w q + (1-w) q' \space | \space w \in [0, 1]$}, and $\Theta_{aug}(\theta, \theta') =$ { $\theta \sim q_w \space | \space q_w \in Q(q, q')$}.

Therefore, we have a chance to sample better models from $\Theta_{aug}$ than simply obtaining them from $\Theta$. This is also illustrated in our Figure 9 in Appendix I, where sampled models may be close to or far away from the Pareto front. Using Bayesian models, we can sample the ones closest to the Pareto front.

We appreciate the reviewer for giving us the opportunity of being clearer. A revised version is uploaded per the concerns. Please let us know if anything still needs further clarification or formalization, and we are more than happy to edit them. We will be very grateful if the reviewer can consider improving the rating.

1. Some definitions are not formal, and wordings should be picked more carefully.

We have edited Definition 1 and 2 in the paper, with a detailed explanation and illustrated example of Definition 2 in Appendix B. We also added Definition 3 to formalize Bayesian continual learning. Moreover, we edited Algorithm 1 and its discussion to make it more formal.

2. What is the definition of $q^j$?

The $q^j$’s are probability distributions. Definition 1 says that, given a finite collection of distributions ${q^j}_{j=1}^m$, $\mathcal{Q}$ is its convex hull, that is, $\mathcal{Q}$ is the collection of all probability distributions that can be written as a convex combination of the $q^j$’s. As the reviewer intuitively points out, if the state space is finite then the $q^j$’s can indeed be seen as probability vectors, whose entries represent the probability mass assigned by distribution $q^j$ to the elements of the state space. We also point out how we used the canonical definition of convex hull (see e.g. [1]); we only slightly generalize it to be the convex hull of distributions instead of elements of a metric space.

3. In Definition 2, what is $\int$ is a minimum?

Requiring that the integral is a minimum corresponds to requiring a minimal cardinality. Formally, this minimal integral generalizes the requirement of minimal cardinality to the case where the underlying set $\Theta$ may be uncountably infinite. In the continuous case, requiring that the integral is a minimum ensures us that the HDR is the set having the least amount of elements, which satisfies the desired condition. We also point out how we are not the first who introduce this notation; it was proposed first in [2]. A detailed definition with illustration can be found in Appendix B.

4. Are $\mathcal{X}$, $\mathcal{Y}$ a subset of Euclidean space?

In a typical classification problem, $\mathcal{X}$ will be a subset of a Euclidean space, and $\mathcal{Y}$ a finite set. In a typical regression problem, $\mathcal{Y}$ will too be a subset of a Euclidean space. In general, we do not limit ourselves to either scenario. As a consequence, we purposefully let the input and the output spaces, $\mathcal{X}$ and $\mathcal{Y}$, respectively, as generic sets.

5. Why say variational inference is computational intensive? Is IBCL feasible in real-world scenarios?

We say variational inference (VI) is computational intensive because it requires optimization of an objective function (usually ELBO loss). This is the major reason why state-of-the-art solutions are computationally expensive, as they must run one optimization per model generation. In contrast, IBCL (1) first runs a small fixed number of optimizations, and (2) then generates models via convex combination, which does not involve any optimization. This design makes IBCL more computationally feasible than state-of-the-art methods.

[1] Phelps, Robert R., ed. "Lectures on Choquet’s theorem". Berlin, Heidelberg: Springer. 2001.

[2] Coolen, Franciscus Petrus Antonius. "Imprecise highest density regions related to intervals of measures." 1992.

Dear reviewers, thank you very much for your patience. We have uploaded our revised paper with additional experiments. Please refer to our comments, and we hope to engage in more discussions with all of you.