Mixture of Experts Meets Prompt-Based Continual Learning

Thank you for your constructive feedback and insightful comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions in final version.

Q1: Comparing NoRGa with other state-of-the-art continual learning methods that do not use prompts would highlight the specific advantages of the proposed method.

A1: Thank you for your valuable suggestion. As our work focuses on the continual learning of pretrained models, we have limited our comparison to pretrained model-based (PTM-based) continual learning methods. Previous works have also demonstrated that utilizing pretrained models has shown great promise and performance for continual learning, surpassing the performance upper bound of non-PTM-based methods. Specifically, we compared our method (NoRGa) against ADAM [1], RanPAC [2], and ESN [3], using ViT-B/16 with pretrained Sup-21K weights. Performance was evaluated using final average accuracy (FA) on Split CIFAR-100 and Split CUB-200. The results can be summarized as follows:

As shown, NoRGa exhibits competitive performance on both datasets. For example, on Split CIFAR-100, NoRGa achieves an FA of 94.48%, surpassing the next best method by over 2%. On Split CUB-200, NoRGa also demonstrates competitive results compared to other baselines. This improvement underscores the effectiveness of our proposed method in mitigating catastrophic forgetting and preserving knowledge across multiple tasks. A more detailed comparison will be included in the final version.

Q2: It may be better to use a additional graph to represent the final method NoRGa of the paper.

A2: Thank you for your valuable suggestion. You can refer to Q1 in General Response.

Q3: It may be necessary to further clarify the differences between the proposed method and other prompt-based methods.

A3: Thank you for your valuable suggestion. We compare the differences between L2P, DualPrompt, HiDe-Prompt, and NoRGa (ours) as follows:

• L2P: Utilizes a shared prompt pool for all tasks. Each prompt is associated with a learnable prompt key. We then employ the query feature $q(\boldsymbol{x})$ to retrieve the top K most similar prompts using cosine distance. Consequently, the most relevant keys and corresponding prompts are explicitly assigned to instances based on the query feature.
• DualPrompt: Enhances L2P by using two complementary prompts during training: a general prompt (G-Prompt) and a task-specific expert prompt (E-Prompt) per task. The set of E-Prompts acts as an expanding pool of task-specific knowledge, similar to the L2P prompt pool, but with the key difference of growing incrementally with each new task. DualPrompt employs the same prompt selection mechanism as L2P for the E-Prompts. In contrast, the G-Prompt is shared among all tasks, requiring no prompt selection.
• HiDe-Prompt: A recent SOTA prompt-based method that employs only task-specific E-Prompts. Prompts for each task are trained with the task's objective and a contrastive regularization that tries to push features of new tasks away from prototypes of old ones. Unlike L2P and DualPrompt, prompt selection is achieved by an additional MLP head placed atop the pre-trained ViT, which employs $q(\boldsymbol{x})$ to determine the suitable prompt.
• NoRGa: Utilizes the same framework as HiDe-Prompt. Specifically, NoRGa only uses task-specific prompts (E-Prompts) with an additional MLP head for prompt selection. Recognizing that MSA layers embody a mixture of experts (MoE) architecture and applying prefix tuning is the process of introducing new experts into these models, NoRGa modifies the gating mechanism of prefix tuning, addresses statistical limitations of prefix tuning, and enhances continual learning performance.

We will add the above discussion in the final version.

Q4: Which dataset is the ViT-B/16 pre-trained on?

A4: Thanks for your question. The ViT-B/16 model is pre-trained on the ImageNet dataset [4] (ImageNet-1K and ImageNet-21K). We utilize several publicly available checkpoints to demonstrate the effectiveness and robustness of our proposed method under varying pretraining settings:

• Sup-21K: Supervised pretraining on ImageNet-21K
• iBOT-21K: Self-supervised pretraining on ImageNet-21K
• iBOT-1K, DINO-1K, MoCo-1K: Self-supervised pretraining on ImageNet-1K.

[1] Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need, arxiv 2023.

[2] Ranpac: Random projections and pre-trained models for continual learning, NeurIPS 2023.

[3] Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference, AAAI 2023.

[4] ImageNet: A large-scale hierarchical image database, CVPR 2009

Thank you for your constructive feedback and insightful comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions in final version.

Q1: The paper could benefit from clearer explanations regarding the core elements of prompt-based methods, particularly in how prompts are defined and utilized within their framework.

A1: Thank you for your valuable suggestion. As detailed in Section 2, within our framework, a prompt is a set of learnable parameters denoted by $\mathbf{P} = [ \mathbf{P}^K, \mathbf{P}^V ] \in \mathbb{R}^{L_p \times d}$. These parameters are utilized to fine-tune the multi-head self-attention (MSA) layer of the pretrained model, where $L_p$ is the prompt length and $d$ is the embedding dimension. Notably, the MSA layer incorporates a specialized architecture comprising multiple mixture of experts (MoE) models. Our approach leverages prompt parameters to refine these models by introducing new prefix experts. Specifically, $\mathbf{P}^V \in \mathbb{R}^{\frac{L_p}{2} \times d}$ encodes parameters for new prefix experts appended to the pretrained MoE models. Correspondingly, $\mathbf{P}^K\in \mathbb{R}^{\frac{L_p}{2} \times d}$ encodes parameters for the associated score functions of these new experts within the MoE models in the MSA layer. We adopt task-specific prompts for each task, ensuring that every task has its own set of experts and score functions. During inference, the task identity is inferred to select the appropriate experts and score functions. We will add the above discussion in the final version.

Q2: Could NoRGa's gating mechanism be seen as a different interpretation or implementation of prompts, albeit operating at a different level or with distinct functional goals?

A2: NoRGa's gating mechanism can be regarded as a distinct implementation of prompts. As demonstrated in Section 3, the MSA layer in pretrained models can be regarded as a specialized architecture comprising multiple MoE models. Prefix tuning finetunes these MoE models by introducing new prefix experts, utilizing prompts to encode the parameters of the new experts' components. The score functions for newly introduced experts via prefix tuning are linear functions of the input, resulting in suboptimal sample efficiency for parameter estimation as detailed in Appendix A. To mitigate this statistical limitation, NoRGa refines the score functions associated with the original prefix tuning's new experts by incorporating non-linear activation and residual connections, substantially enhancing statistical efficiency (polynomial versus exponential) with theoretical guarantees. We will add the above discussion in the final version.

Q3: Are there instances of redundant theoretical explanations or definitions throughout the paper? Streamlining these could improve clarity and focus on the novel contributions without detracting from the foundational concepts

A3: Thanks for your question. Upon careful examination, we found minimal redundancy in the theoretical explanations and definitions presented throughout the paper. Section 2 introduces the foundational concepts of MSA, prefix tuning, and mixture of experts (MoE). Building upon this foundation, Section 3 elucidates the connections among self-attention, prefix tuning, and MoE. Based on these insights, we propose a novel method termed NoRGa to enhance statistical efficiency. Our theoretical analysis considers a regression framework to demonstrate that the non-linear residual gating is more sample efficient than the linear gating in terms of estimating experts. The algebraic independence condition is employed to characterize compatible experts for the non-linear residual gating. Finally, we design a loss function based on Voronoi cells for the convergence analysis of expert estimation in MoE models. Our results indicate that under non-linear residual gating, MoE experts exhibit polynomial-order estimation rates, outperforming the $1/\log^{\tau}(n)$ rate observed in the original prefix tuning design (as detailed in Appendix A).

Q4: Given the complexity of the concepts, would integrating more visual aids (e.g., diagrams illustrating the architecture of NoRGa or the relationship between prompts and gating mechanisms) enhance the paper's accessibility and readability?

A4: Thank you for your valuable suggestion. You can refer to Q1 in General Response.

Thank you for your constructive feedback and insightful comments. Below, we provide a point-to-point response and summarize the corresponding revisions in the final version.

Q1: The placement as a continual learning contribution

A1: Our contributions encompass the introduction of a novel connection between self-attention, prefix tuning, and MoE. This offers a fresh perspective to the design of previous prompt-based continual learning methods. For example, DualPrompt uses two complementary prompts during training: a general prompt (G-Prompt) shared across tasks and a task-specific expert prompt (E-Prompt) per task. As illustrated in our paper, each head in the MSA layer comprises multiple MoE models. We hypothesize that the G-Prompt aims to expand the set of pretrained experts in the MSA layer, which capture generalizable knowledge, with new experts encoded within it. However, unlike pretrained experts, the experts in the G-Prompt are learnable across tasks, potentially leading to catastrophic forgetting. Conversely, E-Prompt encodes task-specific experts capturing each task's knowledge.

Furthermore, we agree that NoRGa can have a broader impact to other domain, such as parameter-efficient model finetuning. However, our work builds on HiDe-Prompt, which uses task-specific prompts, i.e. each task has its own set of experts and score functions. NoRGa modifies the original score functions of prefix tuning, enhance WTP performance with theoretical guarantee. We then use the theory that improved WTP performance is sufficent and necessary condition for improved CL performance. This theory is also discussed in previous continual learning works and the original paper of HiDe-Prompt. This theory is critical and connects our contribution to continual learning. Importantly, NoRGa maintains the same parameter count as HiDe-Prompt, which is crucial in continual learning because of the memory constraint.

Q2: The introduction of 2 hyperparameters $\alpha$ and $\tau$

A2: In our framework, $\alpha$ and $\tau$ are learnable hyperparameters optimized through backpropagation by the objective of the first task, eliminating the need for manual tuning. Moreover, our theory on NoRGa's statistical efficiency holds for any values of $\alpha$ and $\tau$, demonstrating the theoretical robustness. We also experimented with fixed and learnable settings for $\alpha$ and $\tau$. For fixed hyperparameters, we set their values to 1. We report FA on Split CUB-200 and Split CIFAR-100 with Sup-21K weights. The results are summarized below:

Although performance slightly decreased with fixed hyperparameters, it still outperforms HiDe-Prompt, indicating our method's empirical robustness. We will add this discussion in the final version.

Q3: Discussions of recent works on first-task-adaptation and simple PEFT-style tuning for Continual Learning

A3: Thank you for highlighting these excellent related works. We will add the following discussion to the final version:

Previous works have shown that first-task adaptation and simple PEFT-style tuning can achieve competitive performance [1,2,3,4] with prompt-based methods. For instance, [1] demonstrated that appending a nearest class mean (NCM) classifier to a ViT model's feature outputs, can serve as a strong baseline. [2, 3] enhanced this strategy by adapting the pretrained model to the first task using the three PEFT methods for transformer networks [3] and the FiLM method for CNNs [2]. Additionally, [4] improved NCM by incorporating second-order statistics—covariance and Gram matrices. However, these methods, which finetune only the backbone for the initial task, may not always ensure satisfactory separation of new tasks' features. Our work focuses on continually adapting the backbone, utilizing task-specific prompts to consistently capture emerging tasks' characteristics, and proposing a novel method to enhance the CL performance of previous prompting methods.

Q4: Efficiency tests

A4: Thank you for your suggestion. We have added the graph to compare Validation loss of NoRGa and HiDe-Prompt throughout the first task in the attached pdf of General Response.

Q5: Statistical efficiency proofs

A5: Thanks for your questions. We use the HiDe-Prompt framework with task-specific prompts, where each task has its own experts and score functions optimized with only data from that task. Thus, it is reasonable to assume that samples within a task are generated from a single model and are i.i.d.

In Section 3.2, we show that the output attention head with prefix tuning can be expressed as a linear gating prefix MoE model. Recent works [a, b] have shown that the performance of MoE models can be improved by using a more sample efficient gating function in terms of parameter and expert estimation. Motivated by this, we propose using the non-linear residual gating prefix MoE in Section 4 and conduct the convergence analysis to justify its sample efficiency.

Next, Eq.(13) is a common regression framework to study the sample efficiency of the gating function in MoE models [c], rather than an assumption. Finally, the algebraic independence condition characterizes which experts are compatible with the non-linear residual gating. It indicates that under the non-linear residual gating MoE, experts would have estimation rates of polynomial orders rather than of order $1/\log^{\tau}(n)$ when using the linear gating in Appendix A

[a] Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?, ICML 2024

[b] A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts, ICML 2024

[c] On Least Square Estimation in Softmax Gating Mixture of Experts, ICML 2024

Thank you for your constructive feedback and insightful comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions in final version.

Q1: Comparison to Different Parameter-Efficient Fine-Tuning Methods and Theoretical Analysis

A1: Thank you for your valuable suggestion. As the advantages of different PEFT methods remain an open question, we briefly describe them through our revealed connection between self-attention and MoE. Prefix tuning introduces additional parameters at the input of MSA layers to adapt the pretrained model representation, contrasting with adapters, which insert adaptive parameters between layers, often replacing MLP blocks. LoRA approximates weight updates with low-rank matrices and adds them to the backbone weights. Our work shows that the MSA layer in a pretrained model can be seen as a pretrained MoE architecture. Applying LoRA to the MSA layer refines both the pretrained experts and their corresponding score functions for downstream tasks. In contrast, prefix tuning expands the pretrained MoE models by incorporating new experts while preserving the original components, rather than modifying the pretrained experts like LoRA.

For empirical comparison, we used the framework of HiDe-Prompt with different PEFT techniques and Sup-21K weights, evaluating performance using FA on Split CIFAR-100 and Split CUB-200. The results are summarized below:

The table shows that NoRGa consistently outperforms the other PEFT methods on both datasets, suggesting its effectiveness. However, further investigation with LoRA and adapters would be necessary to draw more definitive conclusions. We will add this discussion to the final version.

Q2: Dynamic routing mechanism can be employed for gating-based neural networks (e.g. [c]). To improve the parameter efficiency of the final model, how to integrate this mechanism in the proposed method?

A2: Thank you for pointing out these excellent related works. We will add the following discussion to the final version:

Each head in the MSA layers comprises $N$ MoE models, where $N$ is the length of the input sequence. This allows for a dynamic routing mechanism to enhance parameter efficiency. For instance, [c] proposed a dynamic routing strategy that adaptively adjusts the number of activated experts based on the input. The computation for any MoE model’s gating is directly correlated with the corresponding row in the attention matrix, which encapsulates the MoE model’s score functions. For example, selecting the top $k$ experts via Top-K routing in the $i$-th MoE model is equivalent to identifying the top $k$ largest values in the $i$-th row of the attention matrix. To implement [c], we first sort the elements in the $i$-th row from highest to lowest, then find the smallest set of experts whose cumulative probability exceeds the threshold. Consequently, unselected experts remain inactive, reducing the need to compute all elements of the value matrix within self-attention.

Q3: Important MoE-related continual learning methods are not included in the related works [a]. The difference between the proposed method and other related works should be highlighted

A3: Thank you for providing these excellent related works. We will add the following discussion to the final version:

Recently, the MoE model has been employed to mitigate catastrophic forgetting in continual learning (CL). For example, [a] focused on continual learning in vision-language models by adapting a pretrained vision-language model to new tasks through learning a mixture of specialized adapter modules. [a] introduced an MoE structure onto a frozen CLIP, utilizing a mixture of adapters to modify the MLP block after the MSA layer. In contrast, our work centers on general continual learning with pretrained models, leveraging the inherent MoE architecture of MSA layers. Consequently, our MoE model placement differs from that of [a]. By employing prefix tuning, we demonstrate that it is analogous to introducing new prefix experts to scale and adapt these pretrained MoE models to downstream tasks. Furthermore, while [a] utilizes task-specific routers, our approach employs task-specific prompts that encapsulate both task-specific router and expert parameters.

Q4: Performance improvement on some datasets is limited.

A4: While performance gains on certain metrics may be modest for some datasets, our method consistently outperforms the baseline, HiDe-Prompt, the current state-of-the-art in prompt-based continual learning, in terms of either FA or CA. For example, on the Split Imagenet-R dataset with Sup-21K weights, the improvement in FA is small (75.06%->75.40%), but the CA enhancement is significant (76.60%->79.52%). This trend is consistent across various datasets and pretrained settings, underscoring our method's effectiveness and robustness.

Q5: Running Times and Performance Variance

A5: We utilize a single A100 GPU for all experiments. The training times are summarized below:

Each experiment was conducted 3 times. While NoRGa exhibits slightly longer training times compared to HiDe-Prompt, it consistently achieves significantly better performance as indicated in Table 1. This demonstrates the effectiveness of NoRGa while maintaining competitive training efficiency. We will add the above discussion in the final version. Regarding performance variance, we have already presented the standard deviation of the results in the main results, as displayed in Table 1 of the main text.