DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning

We appreciate the effort and time by the reviewer (XBEs).

soft prompts are already very parameter-efficient; it is not necessary to reduce the #parameters in prompt tuning

We thank the reviewer for the feedback. We agree that prompt tuning is very parameter-efficient. However, this weakness is not applicable to our work, because we do NOT reduce the number of parameters compared to prompt tuning.

Usually, the length of soft prompts is small, e.g., 8, 16, 32; the additional inference cost is minor compared with the current LLM, which can accept thousands of tokens as input

We would like to express our appreciation for the reviewer's feedback. However, we respectfully present a different perspective:

• Previous works [1,2,3,4,5,6,7,8] typically use soft prompts with a default length exceeding 100 virtual tokens. To support this point, we can directly quote previous work [5], which states that "for hard sequence tasks, usually, a longer prompt than 100 would be helpful". Similar claims can be found throughout the literature. To the best of our knowledge, there is no existing work that utilises soft prompts with a length of fewer than 32 virtual tokens while accepting thousands of tokens as the input.
• Let's take a step back and consider the situation more broadly. Assuming that we can only add at most 32 virtual tokens to the soft prompt, our proposed method allows us to achieve a highly competitive or even better performance than using around 100 virtual tokens, with minor operations and costs (i.e., add the multiplication of low-rank matrices element-wisely to frozen word embeddings at the input sequence). We believe that this should be regarded as a notable strength of our work.

though prompt tuning is sensitive to initialization, there are some recent methods to deal with this problem e.g., Effective Structured Prompting by Meta-Learning and Representative Verbalizer, ICML 2023. MetaPrompting: Learning to learn better prompts, COLING 2022.

We thank the reviewer for the feedback. We respectively argue that this weakness is not applicable to our work because:

• Our main goal is NOT to address the initialisation problem of the prompt tuning. Instead, our work primarily addresses the increased computational demands, in terms of training, inference time, and memory costs, which arise from the increased input sequence length due to adding the soft prompt.
• In our work, we have extensively discussed these Parameter-efficient Transfer Learning (PETL) approaches by previous works, which aim to solve the initialisation problem. In Section 3.4 of our paper, we have empirically demonstrated that our proposed method complements PETL approaches in the few-shot learning setting. This substantiates our position that the initialisation challenge, while relevant in other contexts, is not applicable to our method.

We appreciate the reviewer for suggestions on related works. We will make sure to discuss these works in our revised version.

the tasks used in experiments are too simple; better to try more challenging tasks, like GeoQuery (Zelle & Mooney, 1996), NL2Bash (Lin et al., 2018), WebQS (Berant et al., 2013).

We thank the reviewer for this suggestion. However, we respectfully disagree with the reviewer:

• All other reviewers (mz3i,BZux,yLsP) have commented positively on the thoroughness of our experiments, suggesting that they find our current evaluation to be adequate.
• Our work has already included a comprehensive evaluation of 23 datasets. These tasks range from natural language inference and sentiment analysis to open-domain question answering, common-sense reasoning, and even visual-language tasks. We believe that these tasks collectively represent a diverse and challenging set, and we do not consider them to be simple.
• Our experiments adhere to the settings and tasks used in previous works [1,2,3,4,5,6,7,8], strictly following the two most recent works [7,8].
• WebQs is a question-answering task, while GeoQuery and NL2Bash are generation tasks. Both of these task types have already been covered by the existing datasets in our work.

References:

[1] The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP 2021.

[2] On Transferability of Prompt Tuning for Natural Language Processing. NAACL 2022.

[3] SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer. ACL 2022.

[4] PPT: Pre-trained Prompt Tuning for Few-shot Learning. ACL 2022.

[5] P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks. ACL 2022.

[6] PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarization. COLING 2022.

[7] ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts. EMNLP 2022.

[8] Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning. ICLR 2023.

We appreciate the effort and time by the reviewer (mz3i).

The architecture is not well motivated. The architecture appears to be a combination of prompt tuning and LoRA. But, unlike LoRA, DEPT still suffers from prompt length compared to architectures at inference time.

We thank the reviewer for the insightful suggestion. In response, we would like to discuss the comparison between Prompt Tuning (PT) and LoRA, as our work aims to improve the PT, in the following points:

• Relative Performance of LoRA and PT. When adapting language models (LMs) to specialised domains, like mathematical reasoning, which requires much different knowledge than what LLMs have been trained on, LoRA may perform better than Prompt Tuning (PT). However, in case tasks have already been somewhat understood by LMs and the key challenge is just to properly prompt the LMs, PT can be the better option. PT modifies minimal model parameters, focusing instead on improving the input prompt, which has been proven more effective than LoRA in prior studies [1,2].
• Specific Use Cases for PT. PT offers advantages in particular cases. For example, soft prompts can be used to compress few-shot examples in the prompt or long context [3,4]. While the number of trainable parameters is low, LoRA updates the weight matrices across the whole model. In contrast, PT only improves the input of the LM through the soft prompt, which helps the model focus on understanding the task and context better rather than learning new knowledge.
• Parameter Efficiency. Unlike LoRA, which requires trainable parameters at each layer, PT's trainable parameters are more concentrated and less extensive.
• Parameter-efficient transfer learning (PETL) Framework. PETL framework (e.g., [1,2]) can effectively improve the performance of the PT and make it easier to use. In our work, we have demonstrated that our approach is compatible with PEFL framework.

Experiments with only smaller models (<1B model parameters).

We respectfully disagree with the reviewer. While we acknowledge the importance of exploring this avenue, we are currently constrained by limited computing resources. This is a common issue faced by many researchers and practitioners. Therefore, the findings of this paper are relevant not just to us but also to others in a similar situation. However, also in response to this complain, we have conducted additional experiments on T5-3B. We report the average performance on the Glue dataset as well as inference speed, measured in inference samples per second. Our findings indicate that DePT (m=60, r=30) outperforms PT in terms of inference speed by 37%. This suggests the advantage of DePT increases as the model size increases.

(m indicates the length of the soft prompt, r indicates the rank of low rank matrices)

We will do our best to get the necessary resources to investigate this important direction, and we leave this to future work.

Missing experiments. I noticed that the authors do not include results on SuperGLUE for LoRA in Table 1. Since the results for DEPT and LoRA are so close on GLUE, I would be curious to see how close the results are on SuperGLUE.

We thank the reviewer for the insightful suggestion. In response, we have add LoRA and IA3 as baselines and filled the missing number for Table 1. Specifically, we train LoRA and IA3 with 30k steps with a batch size of 16. We use the rank as 35 for LoRA. We perform a learning rate search among 3e-3, 5e-4, 1e-4, and 5e-5, and report the test results of the best model on the development set. Our experiment show that DePT obtains better mean performance on SuperGlue benchmark.

References:

[1] ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts. EMNLP 2022.

[2] Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning. ICLR 2023.

[3] Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models, EMNLP 2022.

[4] Adapting Language Models to Compress Contexts. EMNLP 2023.

Thank you so much for your continued discussion and feedback! We appreciate the opportunity to clarify and discuss:

• Architecture Justification. We understand the reviewer’s question about the architecture's intuition. It's important to highlight that the series of Prompt Tuning works, which form the core of our architecture, has shown distinct advantages in certain tasks over other Parameter-Efficient Fine-Tuning (PEFT) approaches, as we discussed in the previous reply. Therefore, we believe that our work, with the goal of improving the design of prompt tuning, remains a promising avenue for future research.
• Model Performance and Efficiency. Our work focuses not only on improving the model performance. An important contribution of our work is addressing the increased computational demands, in terms of training, inference time, and memory costs, which arise from the increased input sequence length due to adding the soft prompt. Although the improvement in performance may not be dramatic, the efficiency gains are notable. For example, as in our previous reply, when T5-3B is used as the backbone, we find the DePT (m=60, r=30) outperforms PT in terms of inference speed by 37%, underscoring the efficiency enhancements our proposed method has achieved.

We appreciate the effort and time by the reviewer (BZux).

The key contribution of DePT lies in that it is both optimizing a soft context as long as the vocabulary in an efficient manner. The decomposition idea, although novel in its current form, is incremental to current PEFT methods. There is also existing work (e.g. [1]) that explored tuning subsets of vocabularies as a way of PEFT. That being said, DePT still has the advantage of efficient vocabulary tuning.

We thank the reviewer for the suggestion. We appreciate the opportunity to clarify: the meaning of “compose” and the method are fundamentally different between CSP [1] and our work:

• What is the difference? CSP treats the attributes and objects that are composed to define classes as learnable tokens within the vocabulary. In contrast, our proposed method DePT does not train soft prompts associated with any vocabulary token, nor does it add additional tokens to the vocabulary. The main goal of DePT is to improve the efficiency of Prompt Tuning (PT) due to the increased input sequence. DePT does not tune or update anything related to the vocabulary, including tokens and embeddings.
• Which work is closely related to CSP and DePT respectively. CSP [1] aligns more closely with prompt-based fine-tuning [2,3,4] where the soft prompts are trained associated with specific prompts and label words. In contrast, DePT aligns more closely with PT. While soft prompts in DePT and PT could be initialised with word embedding, these prompts are not updated in association with any existing or newly added vocabulary tokens. We will make this clear in our revised version.
• Comments from the reviewer (mz3i). We appreciate that the reviewer (mz3i) has pointed out that while the title of our work is similar to the CSP work [1], the proposed methods are different. We have discussed this in the Appendix of our revised version.

The 20% efficiency advantage also is only revealed with one soft prompt length of 100. It would be appreciated if the authors are to disclose more performance comparison under different prompt length scenarios

We appreciate the reviewer's question. In response, we have performed additional experiments, as shown in the Table below. Specifically, we have increased the size of trainable parameters in both DePT and PT by a factor of two. We use the T5-base as the backbone. We report the average performance on the Glue dataset as well as inference speed, measured in inference samples per second. Our findings indicate that DePT (m=120, r=60) outperforms PT in terms of inference speed by 34%. We believe that this performance advantage can be further enhanced by reducing the value of m, which represents the length of the soft prompt. To provide a concrete example, on the SST-2 dataset, DePT can achieve an inference speed of 77.2 samples per second, while PT can only infer 57.4 samples per second. This suggests the advantage of DePT over PT increases as the model size increases.

(m indicates the length of the soft prompt, r indicates the rank of low-rank matrices.)

I appreciate the ablation experiments on the significance of different learning rates for the soft prompts and low-rank matrices. I also wonder if there are other intuitive explanations.

We thank the reviewer for the insightful suggestion. The intuition of our method is that (1) given the same number of trainable parameters, allowing some updates for word embeddings will improve the performance; and (2) shorter soft prompt will improve the efficiency. To illustrate, the previous study [5] has shown that a soft prompt can interpolate between many token embeddings, enabling the representation of more abstract concepts compared to relying on a single discrete token. However, the soft prompt in the Prompt Tuning is consistently added at the beginning of the frozen word embedding. In contrast, we propose DePT, which decomposes the long soft prompt into a short soft prompt and a pair of low-rank matrices. This approach can (1) reduce the length of the soft prompt for better efficiency; and (2) permit representation updates within the frozen word embedding, thereby increasing the adaptability of input representations that were previously unavailable.

References:

[1] Learning to Compose Soft Prompts for Compositional Zero-Shot Learning. ICLR 2023.

[2] Exploiting cloze-questions for few-shot text classification and natural language inference. EACL 2021.

[3] Differentiable prompt makes pre-trained language models better few-shot learners. ICLR 2022.

[4] Don’t Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner, NeurIPS 2023.

[5] Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models. Findings of EMNLP 2022.

We appreciate the effort and time spent by the reviewer (yLsP).

W1: Some of the important baseline methods like IA3 are missing. See questions below.

We thank the reviewer for the insightful suggestion. In response, we have performed additional experiments as follows:

(1) Firstly, we have added LoRA and IA3 as baselines and filled in the missing numbers for Table 1. Specifically, we train LoRA and IA3 with 30k steps with a batch size of 16. We use the rank as 35 for LoRA. We perform a learning rate search among 3e-3, 5e-4, 1e-4, and 5e-5, and report the test results of the best model on the development set. Our experiment shows that DePT obtains better mean performance on Glue and SuperGlue benchmarks.

(2) We also add LoRA as a baseline in Table 2. While it is worth noting that LoRA performs highly competitively on WinoGrande, YelpPolarity, SciTail, and PAWS datasets, we have encountered challenges when training LoRA on certain MRQA tasks, such as HotpotQA. Our experiment shows that the advantage of DePT persists.

(3) In addition, we have added IA3 as a baseline to Table 4 in the few-shot learning setting (4, 16, or 32 training examples). We report average results across three seeds. Specifically, we find that DePT, along with other PT variants (ATTEMPT and MPT), can outperform IA3, highlighting the effectiveness of DePT with transfer learning approaches.

W2: The idea is interesting, however some more intuition on why this works might strengthen the paper.

We acknowledge the reviewer's inquiry regarding the intuition of our method. The intuition of our method is that (1) given the same number of trainable parameters, allowing some updates for word embeddings will improve the performance; and (2) shorter soft prompt will improve the efficiency. To illustrate, the previous study [1] has shown that a soft prompt can interpolate between many token embeddings, enabling the representation of more abstract concepts compared to relying on a single discrete token. However, the soft prompt in the Prompt Tuning is consistently added at the beginning of the frozen word embedding. In contrast, we propose DePT, which decomposes the long soft prompt into a short soft prompt and a pair of low-rank matrices. This approach can (1) reduce the length of the soft prompt for better efficiency; and (2) permit representation updates within the frozen word embedding, thereby increasing the adaptability of input representations that were previously unavailable.

[1] Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models. Findings of EMNLP 2022.

Q: Most of the improvement in the table-1 glue task seems to come from 1-2 tasks like cola, rte any specific reason for this?

We appreciate the reviewer's question. Our main objective in this study is not solely to improve performance metrics in the table. Instead, we seek to demonstrate that DePT can deliver highly competitive or even better overall performance while substantially enhancing overall efficacy in terms of computational time and memory usage, which is especially useful for large language models.