Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt

[W2,Q1,Q2 - The evaluation is only conducted with PPL: We have in fact evaluated on other tasks]

We kindly direct the reviewer's attention to Appendix A.3 , where we have explored QA (OpenBookQA, PIQA), commonsense NLI (Hellaswag), language understanding (high school European history from MMLU) tasks reported with non-PPL metrics. We highlight that all these reported results are zero-shot. Namely, the prompt is learned on C4, and then we directly stitch the prompt to the compressed model and test the performance on these downstream tasks. We hope the reviewer may find them helpful.

[W3 - The difference between the proposed methods and conventional soft prompt tuning: Transformation of Prompt Tuning from Task-Specific to Transferable]

As we mentioned in our response to W1 above, the key difference between our prompt tuning and conventional soft prompt tuning is that our learned soft prompts are transferable between datasets, tasks, and compression levels. The transferring of prompts saves the prompt tuning time and effort originally required for each specific setups (according to our experience with 4 GPUs, that means 5-ish hours per task per compressed model).

We also already compared our methods with conventional soft prompt tuning in experiment (NOTE: it was originally in Table 4 of Appendix A.3, we move it to the main text in the updated version). For your convenience, we post our results and summarize our observations here:

Perplexity comparison between full model and quantized models with different prompts. Where we report test perplexity on PTB and Wikitext-2 dataset. "w./o. prompt" refers to the quantized model without soft prompts. "w./ direct prompt" means the soft prompts are directly trained on the target dataset. "w./ transferred prompt" means the prompt is trained on C4 dataset and then transferred to the target dataset.

Given direct prompt receives a task-specific loss, our transferred prompt is, as expected, not as competitive as the direct one. However, such transferred prompt may significantly bridge the gap between a compressed and full model — e.g., our 3-bit & 4-bit quantized LLaMA-7B with transferred prompt can deliver on-par or better PPL than the full model on PTB and Wikitext2. We'd say this is an especially worthy contribution in practice, as one may possibly download the open-sourced transferable prompt to help on a compressed model with little effort.

[W1 - The approach is simple and the novelty is not obvious: Our paper uncovers new properties of prompt tuning and they offer new opportunities for model compression.]

While novelty is a multifaceted concept in academic research, we believe it can be roughly viewed from two fronts: empirical novelty, which involves uncovering new properties and behaviors unknown to the wider community (e.g., the emergent ability [1], lottery ticket hypothesis [2]), and technical novelty, which pertains to the development of new solutions or methodologies (e.g., LoRA [3]).

We argue our work is rich in empirical novelty in at least two aspects:

1. Transformation of Prompt Tuning from Task-Specific to Transferable: Prior to our study, only a few works studied the transferability of learned prompts between different tasks [4, 5, 6]. Specifically, [4] finds it is possible to transfer learnable prompts with additional fine-tuning on downstream task. However, as mentioned in the experiment section, all our reported results are zero-shot, i.e., no fine-tuning is needed to obtain transferability. Furthermore, [5] finds that the learned prompt can only be transferred among similar tasks. However, as verified by our experiments, our learned prompts are transferable between datasets, tasks, and compressed models.

2. New avenue to enhance Compressed Model Accuracy: We show that prompt tuning can effectively recover the accuracy drop of compressed models. Traditional model compression methods often demand extensive engineering efforts and intricate designs, as seen in popular model compression papers like GPTQ [7], SPQR [8], and AWQ [8]. However, our method simplifies this process significantly. We leverage prompt tuning to effectively recover the accuracy drop in compressed models, opening a new avenue to optimize the trade-off between accuracy and efficiency.

These findings are not only valid but also generalizable across various model families, datasets, and tasks, underscoring the broad applicability and impact of our work.

Regarding technical contribution, we agree that our method is a direct application of prompt tuning. However, we emphasize that we intensionally kept our method as simple as possible. This simplicity, is in our opinon more of a strength than a weakness, as it just underscores the unknown properties of straightforward prompt tuning methods; where the uncovering of these properties and the new pathways they opened for model compression constitute the true novelty of our paper. Thereby, we argue that we contribute substantially to the field's understanding of prompt tuning and model compression.

[1] Emergent Abilities of Large Language Models

[2] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

[3] LoRA: Low-Rank Adaptation of Large Language Models

[4] On Transferability of Prompt Tuning for Natural Language Processing

[5] SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer

[6] Reducing Retraining by Recycling Parameter-Efficient Prompts

[7] GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

[8] SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression

[9] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

[W2 - Compare against LoRA tuning: Sure! Our methods outperform LoRA in more recent and larger models]

Below, we compare our approach and LoRA tuning on LLaMA-2-13B and BLOOM-7B models with 3-bit quantization using GPTQ.

Results suggest that our soft prompt approach vastly outperforms LoRA. In fact, the 3-bit quantized LLaMA-2-13B and BLOOM-7B equipped with our approach may even exceed their FP16 counterparts.

Given that both LoRA weights and soft prompts are technically transferrable, we also compare the transferability between the two methods by first training them on the C4 dataset, then transferring and testing on the wikitext2 dataset.

Results suggest our soft prompt approach still outperforms LoRA in this transferred setting.

[Q1 - Can the prompt prefix generalize across different models? Depending on whether such models share the same tokenizer and embedding dim]

A learned prompt can be generalized within the models that share the same tokenizer & embedding dim (e.g., LLaMA 1 and 2, or simply any model with different compression levels). While projecting soft prompt into different dimensions is certainly doable (e.g., project LLaMA-7B trained soft prompt with 4096 dim into 8192 to work with LLaMA-65B's), we leave it to future work. Yet, models with different tokenizers will require extra steps to form alignment, as different tokenizers will interpret the same soft prompt differently.

We thank the reviewer for the comments and suggestions to improve our paper. Please see the following clarifications.

[W1 - Novelty is limited: Our paper uncovers new properties of prompt tuning and they offer new opportunities for model compression]

While novelty is a multifaceted concept in academic research, we believe it can be roughly viewed from two fronts: empirical novelty, which involves uncovering new properties and behaviors unknown to the wider community (e.g., the emergent ability [1], lottery ticket hypothesis [2]), and technical novelty, which pertains to the development of new solutions or methodologies (e.g., LoRA [3]).

We argue our work is rich in empirical novelty in at least two aspects:

1. Transformation of Prompt Tuning from Task-Specific to Transferable: Prior to our study, only a few works studied the transferability of learned prompts between different tasks [4, 5, 6]. Specifically, [4] finds it is possible to transfer learnable prompts with additional fine-tuning on downstream task. However, as mentioned in the experiment section, all our reported results are zero-shot, i.e., no fine-tuning is needed to obtain transferability. Furthermore, [5] finds that the learned prompt can only be transferred among similar tasks. However, as verified by our experiments, our learned prompts are transferable between datasets, tasks, and compressed models.

2. New avenue to enhance Compressed Model Accuracy: We show that prompt tuning can effectively recover the accuracy drop of compressed models. Traditional model compression methods often demand extensive engineering efforts and intricate designs, as seen in popular model compression papers like GPTQ [7], SPQR [8], and AWQ [8]. However, our method simplifies this process significantly. We leverage prompt tuning to effectively recover the accuracy drop in compressed models, opening a new avenue to optimize the trade-off between accuracy and efficiency.

These findings are not only valid but also generalizable across various model families, datasets, and tasks, underscoring the broad applicability and impact of our work.

Regarding technical contribution, we agree that our method is a direct application of prompt tuning. However, we emphasize that we intensionally kept our method as simple as possible. This simplicity, is in our opinon more of a strength than a weakness, as it just underscores the unknown properties of straightforward prompt tuning methods; where the uncovering of these properties and the new pathways they opened for model compression constitute the true novelty of our paper. Thereby, we argue that we contribute substantially to the field's understanding of prompt tuning and model compression.

[1] Emergent Abilities of Large Language Models

[2] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

[3] LoRA: Low-Rank Adaptation of Large Language Models

[4] On Transferability of Prompt Tuning for Natural Language Processing

[5] SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer

[6] Reducing Retraining by Recycling Parameter-Efficient Prompts

[7] GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

[8] SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression

[9] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

[W1.3 Extra - Conventional Prompt tuning cannot transfer between domains!]

Here we also emphasize that the conventional prompt tuning trained with a domain-specific loss can no longer be transferred between different datasets. Below we present the results of transferring the soft prompts learned on Wikitext2 ( Featured articles on Wikipedia) to PTB (Wall Street Journal material) and C4 (collection of common web text corpus). The results, as shown in the table below, highlight a significant disparity in performance when using domain-specific prompts across different domains. The prompt trained on Wikitext-2, when applied to PTB and C4, leads to a drastic increase in perplexity, indicating a severe degradation in model performance. In contrast, if the prompt is learned on general domain (C4), then it can be transferred to different domain (PTB & Wikitext) and tasks (QA, Commonsense NLI, language understanding, etc. Please check Appendix A.3)

Perplexity comparison on PTB and C4

[W1.4 - Other Model Family: Sure! Our method is still strong on LLaMA-2 and BLOOM]

We conduct experiments on BLOOM-7B and LLaMA2-13B. Here, we provide the results. We can recover the performance of 3-bit LLaMA2-2-13B and 3-bit BLOOM-7B with even better performance than their fp16 counterparts. The quantization is conducted using GPTQ.

We also test the transferability of our prompt tuning on the wikitext2 dataset. It suggests that our soft prompt can be transferred to other datasets while still outperforming LoRA for the performance recovery of compressed LLMs.

We thank the reviewer for the comments. Below we provide a wider context and evaluation for our proposed method.

[W1.1 - How expensive is prompt tuning? It can be done in hours]

The backward pass of linear layer, represented as $H^{(l+1)}=H^{(l)}W^{(l)}$, contains two parts, namely, calucating the input gradient $\nabla H^{(l)}=\nabla H^{(l+1)} W^{(l)\top}$ and weight gradient $\nabla W^{(l)}=H^{(l)\top} \nabla H^{(l+1)}$. Soft prompt tuning simplifies this process by only requiring the calculation of input gradients. This approach eliminates the need to cache input tensors, as the model weights are already stored in GPU memory. Consequently, this reduces the backward pass time by 50%, since the weight gradient calculation is bypassed. In contrast to methods like LoRA, which requires caching input tensors for computing the LoRA weight gradient, soft prompt tuning allows larger flexible fine-tuning batch sizes, enhancing GPU utilization. In practice, we can recover the performance of a LLaMA2-7B model within 5 hours on 4 RTX 8000 48G GPUs.

[W1.2 - How much time/effort is saved by transferring prompts? It saves specific prompt tuning efforts and makes models 8X smaller]

First, regarding time efficiency, transferring prompts across datasets, tasks, and compression levels saves the prompt tuning time for each specific setup. According to our experience with 4 GPUs, that means 5-ish hours per task per compressed model as well as the engineering efforts).

Second, regarding memory usage, as shown in Table 2, our method can recover the accuracy drop of an 8X compressed LLaMA-7B. In practice, this means we reduce the model memory cost from roughly 13GB (in Float16 format) to 1.7GB without sacrificing performance! This substantial decrease in memory usage opens up new opportunities for deploying LLMs on devices with limited resources or inferencing with enlarged batch sizes.

[W 1.3 - How does accuracy compare to conventional prompt tuning (this is a key missing comparison): We have already compared them in Appendix A.3. In short, transferred prompts are not as good, but they significantly bridge the gap.]

If by "conventional prompt tuning," the reviewer means directly learning the soft prompt on the downstream tasks, then we already have these results in Table 4 in Appendix A.3. For your convenience, we post our results and summarize our observations here.

Perplexity comparison between full model and quantized models with different prompts. Where we report test perplexity on PTB and Wikitext-2 dataset. "w./o. prompt" refers to the quantized model without soft prompts. "w./ direct prompt" means the soft prompts are directly trained on the target dataset. "w./ transferred prompt" means the prompt is trained on C4 dataset and then transferred to the target dataset.

Given direct prompt receives a domain-specific loss, our transferred prompt is, as expected, not as competitive as the direct one. However, such transferred prompt may significantly bridge the gap between a compressed and full model — e.g., our 3-bit & 4-bit quantized LLaMA-7B with transferred prompt can deliver on-par or better PPL than the full model on PTB and Wikitext2. We'd say this is an especially worthy contribution in practice, as one may possibly download the open-sourced transferable prompt to help on a compressed model with little effort.

We thank the reviewer for the strong support and meticulous reading!

[W1 - Including baseline value in Figures: Sure! Thank you for suggesting it]

We have updated the paper by printing out the value for green lines in Figures 2,3,4. Please see the revised submission for a better presentation.