Q-Tuning: Continual Queue-based Prompt Tuning for Language Models

Thank you very much for your constructive review and feedback. We are pleased and encouraged to learn that you found our idea promising, our proposed method novel and effective, and our paper well-written and validated. We also thank you for providing all the useful references, as we have added some of them in our related work section. We are addressing each of your comments below:

Q1: Lack of comparison with prompt-based continual learning methods. Only Progprompt can be a straightforward baseline in view of 'prompt-based continual learning', which is insufficient since there are multiple strong prompt-based continual learning approaches [1,2,3].

A1: Thank you for introducing these relevant papers. We didn't include them initially, as we mainly surveyed NLP-related works. We've added them to the resubmitted paper (Lines 79-80, 86-88 and 96-98). We'd like to elaborate on the difference between Q-tuning in the following aspects:

1. The work [1] uses an expanding prompt component set similar to the ProgPromt, which cannot solve an infinite number of tasks. As the new task number becomes larger, there will be a point when the training fails due to out of memory. This limitation actually motivates the design of the Q-prompt and De-Q algorithm of Q-tuning to enable life-long learning.
2. The paper [2] cumulates a prompt pool similar to ProgPrompt, which is considered a baseline approach in Q-tuning. Besides, it cannot tackle extremely long task sequences. In addition, [2] is designed for CLIP-like architectures, whereas Q-tuning is agnostic to model backbones including encoder-only classification models (e.g., BERT) and generative models (e.g., T5 and other advanced language models).
3. A major contribution of DualPrompt [3] is to add the shared prompt and task-specific prompts to different layers of the backbone model. This layer-wise prompt configuration, however, necessitates the prior determination of the total number of tasks, which is orthogonal to our approach, as Q-tuning's focus is on enabling lifelong learning where the total number of tasks is unknown and could be very large.

Based on the above points，the mentioned papers [1,2,3] should not be good baselines in our paper for language models and the experiment settings. Morever, these papers focus on CV tasks and do not provide code for NLP experiments.

Q2: Evaluating the forward knowledge transfer with more diverse continual learning methods, including [4,5].

A2: We did not compare with the methods [4,5] because they are based on different settings. The paper [4] requires a few training tasks equipped with instructions and labelled examples to initialize a model, which is not available in our lifelong learning scenarios. The paper [5] leverages a data buffer from previously observed tasks to enable FKT, therefore it's not rehearsal-free. Q-tuning focuses on the practical situation in lifelong learning where we do not have access to previous data and do not know the total number of incoming tasks.

Q3: Missing ablation studies like efficiency. Recently, some continual learning researchers claim that the training time/real-time computational costs matter in real continual learning scenarios rather than memory occupancy, as the monetary cost for memory (e.g., HDD) is much cheaper than computations (e.g., GPUS) [6]. ....

A3: The proposed Q-tuning aims to solve the computation costs in GPUs rather than the HDD, which can reduce the computation cost regarding GPU memory from $\mathcal{O}(N^2)$ to $\mathcal{O}(1)$ for life-long learning scenarios. For the hardware storage overhead, the proposed Q-tuning assumes it is unlimited because it is very cheap. The proposed Q-tuning maintains a prompt buffer, i.e. Q-prompt of a finite length rather than an infinite length, to make the computation cost for training and inference consistent when the number of tasks grows.

We explain the motivation of Q-tuning in Lines 106-116, the introduction of the De-Q method in Lines 141-163, and the results in Table 3. In Table 3, the experiments demonstrate that the proposed Q-tuning is able to achieve continual prompt tuning across 70 different tasks, while the current SOTA ProgPrompt fails due to the GPU memory problem, i.e. the extremely high computation memory costs. Regarding the inference latency, the proposed Q-tuning can also reduce the computation cost from $\mathcal{O}(N^2)$ to a constant cost determined by the set Q-prompt size $\mathcal{Q}_{size}$, where $N$ stands for the number of tasks.

Q3: It is quite hard to understand that Q-Tuning outperforms ProgPrompt (or Full prompts, MTL) under a long sequence of tasks. Since other methods have a bigger number of parameters (as the prompt size increases), I believe the performance also should be better. Can the author give an explanation or intuition behind these results?

A3: Compared with other approaches with a larger number of parameters, the performance gain of Q-Tuning mainly comes from:

1. The Q-prompt Ensemble module adaptively organizes previous prompts based on their relevance to the current task, instead of directly leveraging previous prompts as ProgPrompt did;
2. Although Q-tuning throws away previous prompts in a long task sequence, the DQ-PCA ensures that only the least relevant parts are discarded. In other words, the DQ-PCA can remove some redundant information in previously trained prompts, hence improving the performance to some extent
3. The shared prompt also bring gains to the overall performance by facilitating the FWT.

We have reported the ablation study results in Tables 5&6. We copy the results here.

We can observe that the ensemble, shared prompt and MR loss contribute to the improvement, especially for the (extremely) long sequence experiments.

Q4:Although I can understand that Q-Tuning can have a better forward transfer than ProgPrompt (due to the Ensemble), it is a little awkward that Q-Tuning with a limited capacity shows better results than Full prompts of Q-tuning.

A4: As for the comparison with the full prompt, we can see the performance is almost identical on average, as in Table 2. We believe the slightly better average performance of DQ-PCA is purely because of numerical fluctuation. E.g, full prompt outperforms DQ-PCA in some columns. In Table 4, we can observe that the Full prompts of Q-tuning outperform the Q-prompt by setting ${Q}_{size}$ to 10 and 5. Indeed, the performance of the trimmed Q-prompt matches that of the full Q-prompt, showcasing the effectiveness of the elaborate De-Q algorithm in preventing information loss when trimming the Q-prompt.

Q5: De-Q rule is indeed reasonable for better forward transfer (as it retains the most relevant prompt for the new task), but it may have a negative effect on the backward transfer (i.e., the forgetting).

A5:

1. Q-tuing does not consider the backward transfer because all the previously trained prompts and the pretrained backbone are fixed. In this case, we cannot update previous statuses to enable the backward transfer. For evaluation, please refer to the added results in the following responses: Q6&A6.
2. Q-tuing solves the catastrophic forgetting problem. This is because we can cheaply store the task-specific prompts and different snapshots of the shared prompts. In this way when we do inference for previous tasks, we can simply use the corresponding task-specific prompts and shared prompts.

Q6: The paper only reports the overall performance in the main table, but dividing the results into i) backward, ii) forward, and iii) overall will be more informative.

A6: Thanks for the suggestions! Following ProgPrompt [5], we provide additional experiments on the long-sequence benchmarks: forward transfer (FWT), backward transfer (BWT), and average accuracy evolution, as defined by Lopez-Paz et al.[6].

Here is a short summary of our results on 15-task experiments with the T5 model:

1. Forward transfer (higher is better) The table below shows the average FWT across the 3 task orders (Order 8, Order 9, Order 10) of long-sequence experiments:

The full FWT results on all tasks across the three orders in few-shot settings (20, 200, 1000 samples per class) are available in Appendix E (Figures 3-11) and the anonymous link: https://drive.google.com/drive/folders/1xtch_kyckijOVBzzQQA6RRO8XyktIDsy?usp=sharing

2. Backward transfer (higher is better, 0 = no forgetting) The table below shows the average BWT across the 3 task orders (Order 8, Order 9, Order 10) of long-sequence experiments:

The full BWT results on all tasks across the three orders in few-shot settings (20, 200, 1000 samples per class) are available in Appendix E (Figures 12-20) and the anonymous link:
https://drive.google.com/drive/folders/1RE0wvRBk9BhRf7MGEnwt1p7hEQScpcPA?usp=sharing

3. Average accuracy Evolution of average accuracies is shown in Appendix E (Figure 21) and the anonymous link: https://drive.google.com/file/d/1ZJYrb83uCKuTJsa_JK8xV1KjniOKI6OQ/view?usp=sharing

All the above results are added to the Appendix E of the updated paper.

Summary of the Results: We observe that the proposed Q-tuning has the best BWT score and the best FWT score in both few-shot and full-shot continual learning setups. Notablely, Q-tuning, following the prompt tuning paradigm, is able to eliminate the catastrophic forgetting problem, because for a new task, all previously tuned prompts are frozen. Moreover, Q-tuning is able to achieve life-long learning ability while achieving FWT in a limited computation resource. These two advantages of Q-tuning make it outperform the baseline [3] mentioned by the reviewer.

Q7: Even though a single prefix prompt well preserves the knowledge by maximizing Eq (5), the optimized prefix will be concatenated with the pruned prompt (i.e., $[\theta^{i-1}_{\mathcal{P}^{\ast}},\mathcal{Q}^{i-1}]$). Therefore, the resulting prompt does not explicitly maximize the mutual information.

A7: For the continual learning task, we assume the knowledge transfer satisfies the Markov process, i.e. the current status of the prompts is only determined by their last status. Hence, in Eq.(5), we maxmize the mutual information between the current shared prompt $\theta^{i}_{\mathcal{P}^{\ast}}$ of the i-th status and the prevous i-1 th status. However, in practice, the ideal FWT does not exist, and the transferred knowledge to the shared prompt $\theta^{i}_{\mathcal{P}^{\ast}}$ will decay along with the coming of new tasks, resulting in the loss of mutual information. This phenomenon aligns with the well-known brain memory management. When we learn new things, the memory will decay over time.

Thank you very much for your constructive and informative review and detailed comments. We are encouraged to know that you found our paper well-written, our proposed method sound and effective and our experiments extensive. We have updated the paper with additional metrics and plots, and we also added some useful references in the related work section based on your recommendation. Below are our answers to all your questions:

Q1: Insufficient related work [1,2,3].

A1: Thank you for your valuable feedback. We have added more related works in the revised version (Lines 77-82). We do not include them in our comparison because:

1. The method [1] needs to maintain previous networks to train the network for a new task, making it not suitable for current large language models. We aim to enable lifelong learning with a single network, which is why we only surveyed papers in that domain.
2. The method [2] dynamically decides which sub-network to train for a sequence of tasks. However, our scope is in the prompt tuning of large language models, where keeping the model frozen is necessary given the size of recent language models. In such cases, even training a sub-network within the language models will become unmanageable. Moreover, [2] is designed based on multi-layer neural networks such as DNN, and its efficacy in transform-like architecture has yet to be tested.
3. Our scope is in prompt tuning. This is because given the large size of modern language models, we'd like to avoid any approaches that involve the training of the whole network or subnetwork. Another reason is, that prompt tuning is naturally immune to catastrophic forgetting (CF), because we can cheaply store task-specific prompts, and retrieve the related prompts for inference when dealing with the previous task. In comparison, the work [3] leverages distillation and knowledge preservation (i.e., EWC). Although it can mitigate CF while not completely eliminating it as prompt tuning does.

Q2: Important baseline is missing [3]. For instance, Progress & compress [3] tackles the same problem ... since it was quite old paper). The overall concept is highly similar, for instance, using past knowledge for better forward transfer and compressing the knowledge into a compact representation. Since, [3] can be used for any model (i.e., model agnostic) and can be easily extended to continual prompting (i.e., progressively expanding prompts), the authors should compare the results. Also, the authors should rigorously discuss the similarities and dissimilarities.

A2: Here, we discuss the similarities and dissimilarities between Q-tuning and the paper [3] from the perspectives of the method paradigm and the application area.

• Similarities: As far as we can understand, Q-tuning is partially conceptually similar to the paper [3] at the framework level, where they both adopt a shared knowledge base and a task-specific knowledge base.

• Dissimilarities:

  (1) The proposed Q-tuning following the prompt tuning paradigm is naturally immune to catastrophic forgetting (CF) because we can cheaply store task-specific prompts, and retrieve the related prompts for inference when dealing with the previous task. But the paper [3] can only mitigate the CF problem while not eliminating it;

  (2) Our scope is in the prompt tuning area. The modern language models often than not involve billions of parameters. It is highly expensive and not smart to tune the whole parameters or subnetworks of the pretrained large-scale backbones for each new task. Prompt tuning paves the way to efficiently tuning large-scale models.

  (3) Based on the prompt tuning technique, the proposed Q-tuning focuses on solving the continual prompt tuning problem. Specifically, continual prompt tuning solves the learning problem defined by Eq. (2) as introduced in Lines 96~116. However, when the prompt list grows asymptotically (i.e. the model is set as a lifelong learner), training the extremely long prompt list becomes intractable due to the finite system resources. The proposed Q-tuning can reduce the computation cost regarding GPU memory and inference latency from $\mathcal{O}(N^2)$ to $\mathcal{O}(1)$ for life-long learning scenarios.

  (4) Moreover, we propose a Memory Retention technique to reduce information loss when trimming the Q-prompt for lifelong learning.

In summary, our Q-tuning is partially conceptually similar to the paper [3]. But there are many differences between them from the perspectives of the technique roadmap and the applications. Last but not least, we cannot find a public codebase for the paper [3]. If there is an accessible codebase, we are happy to compare it with their method.

Thank you very much for your constructive review and feedback! We are encouraged to learn that you found our proposed method well-motivated and intuitive. Here we provide the answers to your questions and comments as follows:

Q1: Fairness with baselines. The proposed Q-tuning uses more trainable parameters by introducing $W^i$ for dynamic prompt ensemble mechanism and $P^{\ast}$ ... depends on the number of trainable parameters, the author should compare the baselines that use the same number of trainable parameters for a fair comparison

A1: The total number of parameters in Q-tuning is actually fewer than ProgPrompt. As mentioned in Appendix C.3 - Table 10, both the length of the prompt per task and the length of the shared prompt are set to 10. In comparison, in ProgPrompt, the length of prompt per task is set to 50 (by copying the configuration of ProgPrompt to run their experiments). Therefore, Q-tuning not only achieves better performance but also requires fewer trainable parameters.

The weight matrix ${\mathcal{W}}^i$ is of rank-one (${\mathcal{W}}^i = \bf{u}_i \otimes \bf{v}_i^{\mathrm{T}}$, where $\bf{u}_i \in \mathbb{R}^{c^i}$, $\bf{v}_i \in \mathbb{R}^{d}$ and $\otimes$ denotes the outer product.) and hence the number of trainable parameters of ${\mathcal{W}}^i$ is $c^i+d$, where $c^i = l\times i$ is the total prompt length of the Q-prompt ${\mathcal{Q}}^i$ and $d$ is the feature dimension of the each prompt. The prompts in the Q-tuning and the baseline ProgPrompt are both parameterized by a 2-hidden-layer MLP. Hence, we can easily know that the number of parameters of ${\mathcal{W}}^i$ i.e. $c^i+d$ << the number of parameters of a 2-hidden-layer MLP for training.

Here, we provide an example to compare the number of parameters for inference. Let us consider the experiments for short sequences (in Table 1 (a)) which include 4 different tasks. In these experiments, Q-tuning sets each prompt length $l=10$ while the ProgPrompt sets $l=50$. Both of them set the feature dimension $d=1024$. For inference, they both use the MLP's output as the prompts. In this context, the total numbers of the parameters are listed as follows:

We can observe that the total number of parameters of ProgPrompt is 204,800 > 52,264 of Q-tuning, while the performance of ProgPrompt is 74.5 < 76.2 of Q-tuning.

Furthermore, in general, we agree with the reviewer that the performance of prompt tuning significantly depends on the number of trainable parameters. However, the experiments in our paper are based on the few-shot continual learning settings, such as 20 samples/class. In this context, we empirically find that the performance is not sensitive to the prompt length. We think this is owing to the MLP-based prompts and the few-shot setting. Even though the input prompt is of a small length, the learning ability of MLP-based prompts is still powerful enough to achieve few-shot learning.

The following table shows the T5 based Q-tuning experiments by setting different lengths for each prompt.

In this exploration of various settings, we can identify marginal differences. For the sake of maintaining consistency across our short, long, and extremely long sequence experiments, we set the prompt length to 10 in our Q-tuning experiments.

Q2: The gain of Q-tuning over ProgPrompt in Table 1

A2: We believe the gain of Q-Tuning mainly comes from the following two aspects:

1. For a new given task, the Q-prompt Ensemble module incorporates previous prompts adaptively based on their relevance to the task;
2. The shared prompt compliments the task-specific prompts in the Q-prompt. For a given new task, the task-specific prompt is responsible for carrying the incremental information that is not captured by previous prompts, whereas the shared prompt carries the global information across all the tasks. Here, we show the ablation study results on orders 1,2, and 3 as reported in Table 5 in the paper.

More results of each order are reported in Appendix Table 11.

Thank you very much for your detailed and constructive review of our paper and all the references you have provided! We are encouraged to know that you found our idea promising and our paper well-written and well-validated. We have revised our paper to address your concerns, and we also added the papers you recommended in our related work section. In order to clarify our core contributions and resolve all your questions, please see our answers below:

Q1: The novelty of the proposed method appears to be relatively limited, as the newly introduced key modules have all been explored by other works.

A1: Thank you for introducing these relevant papers, and we've added them to the related work (Lines 77-78 and 87-88). We'd like to elaborate on the difference between Q-Tuning and the mentioned papers:

1. The paper [1] requires buffer data from previous tasks for the distillation training; hence, it's not rehearsal-free. Q-tuning focuses on the scenario where we no longer have access to historical data;

2. A major contribution of DualPrompt [2] is to add the shared prompt and task-specific prompts to different layers of the backbone model. This layer-wise prompt configuration, however, necessitates the prior determination of the total number of tasks, which is orthogonal to our approach, as Q-Tuning's focus is on enabling lifelong learning where the total number of tasks is unknown and could be very large.

While acknowledging a conceptual similarity with their framework (shared prompt + task-specific prompts), Q-tuning distinguishes itself by excelling in lifelong learning tasks. This success can be attributed to the elaborate memory retention loss for the shared prompt and the application of the Q-prompt with the De-Q rule. Indeed, we have compared Q-tuning with a configuration similar to the DualPrompt which uses a shared prompt and a task-specific prompt for each task. The results are reported in Tables 5 & 6 (extremely long sequence of experiments, including 70 different tasks) in the paper. We copy them here for reference.

The experiments are conducted on an NVIDIA V100 GPU 32GB. Observing the results in the table reveals that training fails unless the proposed Q-prompt with De-Q is adopted. By introducing Ensemble + MR to Q-prompt, we observe a performance increase from 88.1 to 90.8

3. We agree that the ensemble strategy itself is not a novel concept, as evidenced by its widespread use in machine learning (e.g., [3]). We emphasise that the core innovation of our approach lies in designing Q-tuning to facilitate lifelong prompt tuning for an unknown number of incoming tasks. Our current prompt ensemble approach represents one of several reasonable designs for adaptively incorporating previous prompts based on their relevance to the current task.

Q2: Ablation studies focusing on the prompt ensemble, utilizing only a linear combination of prompts instead of incorporating a weighting matrix with numerous parameters, would offer valuable insights.

A2: The weight matrix ${\mathcal{W}}^i$ is of rank-one (${\mathcal{W}}^i = \bf{u}_i \otimes \bf{v}_i^{\mathrm{T}}$, where $\bf{u}_i \in \mathbb{R}^{c^i}$, $\bf{v}_i \in \mathbb{R}^{d}$ and $\otimes$ denotes the outer product.) and hence the number of trainable parameters of ${\mathcal{W}}^i$ is $c^i+d$, where $c^i = l\times i$ is the total prompt length of the Q-prompt ${\mathcal{Q}}^i$ and $d$ is the feature dimension of the each prompt. Therefore, the parameter number of the weighting matrix ${\mathcal{W}}^i$ is very small in our experiments, e.g. $c^i+d = 10+1024$ for the short sequence experiments. In this context, the Q-prompt ensemble using the rank-one weight matrix ${\mathcal{W}}^i$ is a linear combination of the prompts in terms of the parameter complexity. We acknowledge that there could be different ensemble methods. We do not cover all of them, as they are not the focus of our work.

Q5: Could determine whether the algorithm is sensitive to different backbones.

A5: All of our experiments are conducted by following the benchmark empirical settings for fair comparison. The proposed Q-tuning is a model-agnostic method, as verified by the experiments using the T5 and Bert backbones (Table 1). If using the backbone of LLaMA, the current prompt-tuning approaches will also fail in life-learning scenarios, while our Q-tuning can work well. However, due to limited hardware resources, we cannot finish the continual prompt tuning experiments using the 65B-parameter model. In the continual learning setting, four or more prompts should be simultaneously maintained, which are too expensive to afford by our computation resource. Since popular benchmarks in the NLP domain are mainly evaluated on T5 models, we follow the same practice. We'll evaluate Q-tuning in other large models (e.g., LLaMA) in our future work after getting access to instances with larger GPU memory.

Q6: Why the first term of equation (5) does not include $\mathcal{Q}^i$ together with the shared prompt?

A6: The shared prompt $\theta^i_{\mathcal{P}^{\ast}}$ is designed to carry the global knowledge across all previous tasks, so that we use equation (5) to avoid information loss after trimming the Q-prompt $\mathcal{Q}^i$ by the De-Q. For the $i$-th task, the Q-prompt $\mathcal{Q}^i = [\theta^1_{\mathcal{P}},\cdots, \theta^i_{\mathcal{P}}]$, where only the new prompt $\theta^i_{\mathcal{P}}$ is trainable while the rest of prompts $[\theta^1_{\mathcal{P}},\cdots, \theta^{i-1}_{\mathcal{P}}]$ are fixed. In this context, the trainable $\theta^i_{\mathcal{P}}$ in the Q-prompt is appended to learn the new task information rather than memorizing the old information. If we add the $\mathcal{Q}^i$ to the first term of Equation (5), the knowledge information from the new task learned by the trainable prompt $\theta^{i-1}_{\mathcal{P}}$ in $\mathcal{Q}^i$ will be regularized, dropping the performance on the new task.

Q7: Enlarging the queuing size results in marginal performance improvement

A7: We do not enlarge the queuing size to improve performance. The queue $\mathcal{Q}^i$ is used to maintain the trained prompts. The size of an untrimmed queue should be the same as the number of currently observed tasks. If the queuing size exceeds the maximum capability, we will apply a De-Q rule to trim the queue by discarding less informative prompts in the queue.

In Tables 2&3, we conducted experiments on the 15 sequential tasks to evaluate the performance of the queue with two smaller queue sizes. We can observe that, either setting the size to 10 (throwing away 10 prompts) or 5 (throwing away 5 prompts), the performance drops marginally compared to the Full Q-prompt with $\mathcal{Q}_{size}=15$, because the proposed De-Q algorithm can retain the most information in the queue. The marginal performance differences between these settings are expected, which demonstrate the effectiveness of the novel Q-prompt and the De-Q algorithm.

Q8: MTL is expected to serve as an upper bound, but it is puzzling that its performance is even lower than that of the per-task prompt.

A8: MTL is not necessarily an upper bound. Indeed, the performance of MTL varies across different types of language models. BERT adopts an encoder-only structure, which requires training a nonlinear classifier head for each task. T5 is a text-to-text generative model that does not need a nonlinear head for each task. Hence, when conducting the MTL training, for example, for 4 different tasks, the BERT model should have 4 different classifier heads that are optimized respectively. However, the T5 model is trained to learn the average information of the 4 different tasks, to some extent. Therefore, we observe that the MTL's performance is the best in BERT-based experiments but not in T5-based ones. Besides, MTL's performance also depends on the relevance of tasks. Tasks vary a lot in the NLP domain. E.g., our experiments cover topics like yp reviews, banking, emotion, etc. The negative correlation among tasks might hurt the performance of MTL. In comparison, Q-Tuning leverages the Q-prompt Ensemble which adaptively incorporates previous prompts to avoid this issue.

[1] Ermis, Beyza, et al. "Memory efficient continual learning with transformers." Advances in Neural Information Processing Systems 35 (2022): 10629-10642.

[2] Wang, Zifeng, et al. "Dualprompt:Complementary prompting for rehearsal-free continual learning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[3] Smith, James Seale, et al. "CODA-Prompt:COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning."

Q3: A fair comparison with ProgPrompt could be conducted by increasing its prompt length.

A3: We actually made a fair comparison. The total number of parameters in Q-tuning is actually fewer than ProgPrompt. As mentioned in Appendix C.3 - Table 10, both the length of prompt per task and the length of the shared prompt are set to 10. In comparison, in ProgPrompt, the length of prompt per task is set to 50 (by copying the configuration of ProgPrompt to run their experiments). Therefore Q-tuning not only achieves better performance but also requires fewer trainable parameters.

The weight matrix ${\mathcal{W}}^i$ is of rank-one (${\mathcal{W}}^i = \bf{u}_i \otimes \bf{v}_i^{\mathrm{T}}$, where $\bf{u}_i \in \mathbb{R}^{c^i}$, $\bf{v}_i \in \mathbb{R}^{d}$ and $\otimes$ denotes the outer product.) and hence the number of trainable parameters of ${\mathcal{W}}^i$ is $c^i+d$, where $c^i = l\times i$ is the total prompt length of the Q-prompt ${\mathcal{Q}}^i$ and $d$ is the feature dimension of the each prompt. The prompts in the Q-tuning and the baseline ProgPrompt are both parameterized by a 2-hidden-layer MLP. Hence, we can easily know that the number of parameters of ${\mathcal{W}}^i$ i.e. $c^i+d$ << the number of parameters of a 2-hidden-layer MLP.

Here, we provide an example to compare the number of parameters for inference. Let us consider the experiments for short sequences (in Table 1 (a)) which include 4 different tasks. In these experiments, Q-tuning sets each prompt length $l=10$ while the ProgPrompt sets $l=50$. Both of them set the feature dimension $d=1024$. For inference, they both use the MLP's output as the prompts. In this context, the total parameter numbers are listed as follows:

We can observe that the total number of parameters of ProgPrompt is 204,800 > 52,264 of Q-tuning while the performance of ProgPrompt 74.5 < 76.2 of Q-tuning.

Furthermore, in general, we agree with the reviewer that the performance of prompt tuning significantly depends on the number of trainable parameters. However, the experiments in our paper are based on the few-shot continual learning settings, such as 20 samples/class. In this context, we empirically find that the performance is not sensitive to the prompt length. We think this is owing to the MLP-based prompts and the few-shot setting. Even though the input prompt is of a small length, the learning ability of MLP-based prompts is still powerful enough to achieve few-shot learning.

The following table shows the T5 experiments by setting different lengths for each prompt in Q-tuning.

In this exploration of various settings, we can identify marginal differences. For the sake of maintaining consistency across our short, long, and extremely long sequence experiments, we set the prompt length to 10 in our Q-tuning experiments.

Q4: To provide a comprehensive analysis,... by adding a shared prompt to ProgPrompt and evaluating the results in this context.

A4: We have conducted such a comparison in Table 5 (top rows) for the short sequence tasks. In these short sequence experiments, Q-tuning by dropping ensemble and shared prompt is the same as the ProgPromt.

We can observe a 1.7% improvement by adding Ensemble and Shared Prompt to the ProgPrompt. Furthermore, we want to highlight that the ProgPromt cannot achieve lifelong learning as demonstrated by our experiments on the extremely long sequences including Orders 11, 12 and 13 (Table 3).

We observe that training ProgPrompt will fail (after the 15th task on a single V100 GPU (32GB)) due to out of memory caused by the accumulation of prompts. Compared to the per-task prompt tuning, Q-tuning has gained considerable performance benefits (30.4% accuracy improvement on average from 60.4% to 90.8%).

Thank you for your response. As clarified in our earlier rebuttal Q1&A1, the core contributions of the paper are the Q-prompt, MR loss, and the DQ-PCA algorithm, which are not a simple combination of exiting techniques. Our extensive experiments in Tables 2&3 demonstrate the effectiveness of the proposed techniques. We believed that Q3&Q7 had been addressed in our earlier rebuttal. The following response may help us to clarify further.

1. Although we introduce additional parameters in Q-prompt Ensemble and Shared Prompt, our total number of parameters is still less than the baseline (i.e., ProgPrompt). This is because ProgPrompt set the prompt length as 20 for BERT and 50 for T5, whereas we only set it to 10 (check Appendix Table 10 of our paper and the Appendix Table 7 of the ProgPrompt paper). Our experiments show that our Q-tuning outperforms the baseline (74.5 vs. 76.2), therefore we believe our better performance than ProgPrompt is not simply because of adding more parameters. In our previous rebuttal (Q3 & A3), we conducted a detailed analysis of the number of parameters and provided an explanation for the effects of setting different prompt lengths in our experiments.

Moreover, we want to call out that our core contribution that set us apart from existing approaches is the capability of real lifelong learning (i.e., extremely long task sequence such as 70) through DQ-PCA, while avoiding losing previous information through shared prompt and MR. E.g., for the 70-task sequence, ProgPrompt (our baseline) will fail due to the lack of a mechanism to clean up cumulated prompts, as the following table reports:

Experiments on 70 tasks:

2. Marginal performance improvement by enlarging the $Q_{size}$.

This is expected as DQ-PCA ensures that only the least informative previous prompts are thrown away. Table 2 (copied here) demonstrates the efficacy of DQ-PCA compared with other naive approaches of throwing away previous prompts. Compared to the full Q-prompt, randomly evicting prompts in queue causes -2.1% and -1.9% performance drops for $Q_{size} = 5$ and $Q_{size} = 10$, respectively. However, when using the proposed DQ-PCA, the performance of the trimmed Q-prompt matches that of the full Q-prompt, showcasing the effectiveness of the DQ-PCA algorithm in preventing information loss when trimming the Q-prompt (see experiments when setting $Q_{size}=5$). Moreover, since DQ-PCA can remove some redundant information in previously trained prompts, it also serves a data cleaning role on previous prompt, which can improve the performance to some extent. Therefore, the marginal performance improvement resulting from enlarging the $Q_{size}$ of Q-tuning aligns with our expectations.

We hope that we have addressed your concerns, and we would greatly appreciate it if you could adjust your rating of our paper.