(Dynamic) Prompting might be all you need to repair Compressed LLMs

We thank the reviewer for their in-depth response and criticisms. We will try to respond in order of appearance.

First, in response to the reviewer’s summary of our work, we wish to clarify that IDP is an inference process applied to a set of individually fine-tuned prompts. Contrary to what may have been implied in the summary, we do not fine-tune two prompts simultaneously during training to achieve the performance uplift. The uplift reported in Tables 1, 2, and 3 results from directly using two separately fine-tuned prompts. Therefore, IDP can be scaled up with any combination of prompts without training.

Second, regarding our manuscript's writing and organization, we will implement the suggested revisions. Notably, sections 3.1.1 and 3.1.2 will be renamed to more clearly delineate their content. In line with your recommendation, we will reorder the Table to match the sequence of discussion. For hyper-parameter clarity, we will update the relevant section to state the numbers explicitly. For context, "small" prompts comprise 26 tokens, and "large" prompts comprise 100 tokens. The case of OPT-6.7b and Llama-7b models, the embedding size "e" is 4096. Regarding "m," the number of distinct prompts used, as mentioned in section 3.3.1, is 2, with "n" being 50 and 100. The three prompt sizes employed for the prompting baseline are 25, 50, and 100, consistent across Ptune's three separate settings. LoRA is fine-tuned with hidden dimensions of 2, 3, and 4, and the alpha parameter is set at 36. As suggested, we will also include the baseline uncompressed performance of Llama-7b and OPT-6.7b in table 1 and 2 as well

Third, regarding the novelty of our work, we present two perspectives:

• Similarity to ensembling: While the reviewer is correct to surmise that our method is similar to ensembling, the critical distinction relates to the scalability of IDP. Typically, prompt-ensembling regards each prompt/input as a unique model and would require a number of input replications across the batch dimension and a voting strategy at the end to resolve the different outputs of the same input. This critically impedes scalability. IDP, on the other hand, resolves the selection in a single forward pass.
• Novelty: To the best of our knowledge, the work most akin to our IDP (Inference Dynamic Prompting) approach is ATTEMPT [1], which similarly employs a combined set of soft prompts. A critical distinction, however, lies in ATTEMPT's methodology. ATTEMPT necessitates additional training specific to targeted tasks to generate a set of weights. These weights are then used to perform weighted averages across the prompt set. This approach involves an extra computational layer and requires further training steps, which IDP efficiently bypasses, thereby streamlining the process. We will include ATTEMPT to our previous work section to showcase our novelty better.

Fourth, regarding the significant improvement over baselines, following your suggestion, we have rerun our experimentation using compressed GPTQ Llama-7b for five trials and conducted a T-statistical significant test. The table below marks the T statistical differences between our baselines and IDP and our p-value. Note that since our p-value is below 0.05 and the t-statistic indicates a lower overall sample mean, IDP does produce statistically significant results. In our revision, we will include this ablation study in our result section.

[1] Peng, Xiangyu, et al. "Model Ensemble Instead of Prompt Fusion: A Sample-Specific Knowledge Transfer Method for Few-Shot Prompt Tuning." The Eleventh International Conference on Learning Representations, 2023, https://openreview.net/forum?id=p0yrSRbN5Bu.

Thanks for the response. I mostly agree with what you pointed out. For the performance part, given the Dynamic Prompt method involves non-trivial additional computation, a marginal improvement, even if statistically significant, is not satisfactory.

Thanks for taking my advise regarding the paper writing. Hope the next version of this paper would be more professional.

We appreciate the reviewer's feedback but we believe there is a misunderstanding regarding the cost of operating IDP. We emphasize that IDP’s computational costs are trivial, equivalent to a single prompt. In comparison to LoRA, especially in quantized settings, IDP is 2x more efficient due to its lack of necessity for an additional residual path across linear layers. Regarding parameter size, IDP is at worst 14x smaller than LoRA and 21x smaller than Ptune. To further clarify the effectiveness of IDP and clear any confusion, we highlight several key factors that the reviewer may have overlooked:

• Trivial Decision Cost Because we fully take advantage of the existing attention matrix, IDP’s decision process as described in section 3 is trivial as it's simply the row mean.
• Efficiency Enhancement Consider Figure 7, our IDP outperforms a 100-token single prompt of an average accuracy of 58.42 using what is effectively a single prompt of 38 tokens. This is a 2.6x improvement in computational efficiency.
• KV-Caching Benefits: If allow for caching, due to the input-invariant nature of prompts, computational costs are effectively zero. Because IDP does not allow prompts to intermingle (as described in section 3), there is no need to recompute KV-cache for different combinations of prompts.

In summary, our analysis demonstrates that the operational cost of IDP is trivial. It not only outperforms a naive single prompt approach but also requires fewer tokens. IDP generates performance improvements that are statistically significant when compared to all baselines while costing over one order of magnitude less parameters and saving 2x latency compared to (q)LoRA - this is the whole picture of “gain”, that we conjecture the reviewer has unintentionally missed to see. We will ensure that our paper revision explicitly states these points to prevent any future misunderstandings.

We appreciate the reviewer's support for our paper. In response to their query about why a more significant mean attention might lead to a more suitable prompt for the input, we provide the following explanation. The mean attention metric captures the average token-to-token attention between the input with any given prompt. Our assumption is that prompts eliciting higher levels of activity in response to a given input are likely to be more contextually relevant to that input compared to those that do not. Essentially, what we are measuring is a rough proxy for sequence-to-sequence attention, where higher mean attention values indicate a stronger and potentially more relevant interaction between the prompt and the input sequence.

As the rebuttal period is nearing its conclusion, we invite the reviewer to respond to our latest reply. We hope we have satisfactorily addressed the reviewer's concerns regarding our work. Should any remaining questions or points of clarification are needed, please do not hesitate to reach out.

We thank the reviewer for their support of our paper. As per your advice, we will revise the manuscript to include a section explaining the theory and intuition behind IDP works. But in short, IDP’s mean attention metric captures the average token-to-token attention between the input with any given prompt. We assume that prompts eliciting higher levels of activity in response to a given input are likely to be more contextually relevant to that input compared to those that do not. Essentially, what we are measuring is a rough proxy for sequence-to-sequence attention, where higher mean attention values indicate a stronger and potentially more relevant interaction between the prompt and the input sequence.

First Regarding the foundational theory behind IDP/Prompt’s ability to rechannel inherent knowledge, this is an attribute that we observe and show through deduction. For instance, we reference the example of LoRA, showcased in Figure 6. LoRA modifies the weight representation within the self-attention module, a change that should theoretically result in notable alterations in attention patterns. However, our empirical findings reveal that even when LoRA is trained on the same data source as Prompt and Ptune, it exhibits a remarkably high self-attention similarity. Figure 6 presents the average cosine similarity across various size settings as outlined in Tables 1 and 2 for LoRA, Prompt, and Ptune. This consistency in behavior across different sizes suggests that introducing a new data source does not necessarily necessitate a change in attention patterns. Our conclusion is based on the observation that different attention behaviors exist between the two methods. This indicates an alternative approach to recovering performance and output, one that does not rely solely on restoring self-attention patterns.

As a final point, we use Figure 7 to illustrate that reducing the number of tokens can still effectively recover average performance compared to the baseline. This provides further evidence to suggest the possibility of recovering output without necessarily restoring self-attention and that this phenomenon is not strongly related to the number of parameters or data sources.

Second In addressing whether there are domains where IDP is insufficient, Tables 1 and 2 demonstrate that IDP underperforms in tasks classified under Common Reasoning. These tasks demand the ability to solve problems by comprehending provided information, as opposed to merely recalling information or identifying likely words. The primary challenge here is that the limited number of parameters in the prompts is sufficient only to rechannel attention, thereby recovering information hidden by the compression process. However, these parameters are insufficient to supplement new knowledge that would effectively utilize these facts. This limitation suggests that while IDP can aid in revealing underlying information, its capacity to enhance reasoning or comprehension in more complex task scenarios is constrained.

Third For the final question, given that IDP essentially measures the proxy between input and prompt sequences, it is highly applicable that prompts adapt to different tasks. It can be used together in a mix-domain input for increased efficiency. This is something we are actively exploring.

As the rebuttal period is nearing its conclusion, we invite the reviewer to respond to our latest reply. We hope we have satisfactorily addressed the reviewer's concerns regarding our work. Should any remaining questions or points of clarification are needed, please do not hesitate to reach out.

We thank the reviewers for their feedback. We shall address them in the order of appearance.

First, we would like to address a slight misunderstanding, as noted by the reviewer in their summary of our work. Contrary to their interpretation, we did not set out to argue that the prior work by Xu et al. experiences diminished performance with increased prompt length. As stated in the third paragraph of our introduction, we found this direction "enticing." It is because of our interest in their proposed method which led us to discover, as shown in section 3.1, that there is a flaw in their evaluation, which is based solely on perplexity, as we try to scale up our prompt-tuning. Figure 3 shows a region of optimal prompt length in which perplexity correlates to performance. This realization forms the primary motivation for exploring a method that employs a mixture of shorter prompts, as observed from Figure 3, tailored to individual inputs.

With this misconception cleared, we hope this will also address the first question regarding the motivation and intuition of our proposal.

Second, regarding the ‘modest’ improvement over baselines, following your suggestion, we have rerun our experimentation using compressed GPTQ Llama-7b for five trials and conducted a T-statistical significant test. The table below marks the T statistical differences between our baselines and IDP and our p-value. Note that since our p-value is below 0.05 and the t-statistic indicates a negative overall sample mean differences, IDP does produce statistically significant results. In our revision, we will include this ablation study in our result section.

Third, we apologize for the lack of clarity with our hyper-parameters, which we will make more explicit in the manuscript. Regarding the sizes of our prompts, small refers to prompts with 26 tokens, and large refers to prompts with 100 tokens. For OPT-6.7b and Llama-7b, the embedding size “e” is 4096. As for m, the number of distinct prompts we mentioned in section 3.3.1 is 2, and n is 50 and 100. The three prompt sizes we used for the prompting baseline are 26, 50, and 100. This is also true for Ptune's three separate settings.

We thank the reviewer for their feedback. The connection between Section 3.1/Figure 3 and IDP is straightforward: we demonstrate that scaling prompt-tuning is more effectively achieved with multiple short prompts and dynamic selection, rather than elongating a single prompt. Figure 3 contrasts Perplexity measures with downstream performance, challenging the assumption that better Perplexity always equates to improved task performance, (as assumed in Xu et al). It shows that while longer prompts may improve Perplexity, this does not consistently lead to better performance in downstream tasks. Section 3 explains this by discussing the limitations of longer prompts, particularly their “rigidity” while highlighting the benefit of customizing each input with appropriate shorter prompts. IDP as a result is an efficient way to achieve this scalability while adding trivial compute cost. We will make sure the revised manuscript highlights this point more clearly.

Regarding the “marginal” performance improvement as the reviewer put it, we highlight several key efficiency factors that we keep pointing out shall be taken into the context. Firstly, compared to LoRA, particularly in quantized settings, IDP is 2x more efficient. This efficiency is due to IDP's elimination of the need for an additional residual path across linear layers. Concerning parameter size, IDP is at worst 14x smaller than LoRA and 21x smaller than Ptune. Moreover, IDP demonstrates enhanced efficiency. For instance, as shown in Figure 7, our IDP method outperforms a 100-token single prompt, which has an average accuracy of 58.42, using what is effectively a single prompt of 38 tokens. This represents a 2.6x improvement in computational efficiency, emphasizing the very significant gains IDP offers on efficiency dimension while being better or comparable on task performance dimension.

In summary, as we indicated in the last reply, the reviewer was confused on the impression that our performance gain is “trivial” over other methods but seems to overlook the right context that such performance gain is achieved with 14 times less parameter size and 2 times less latency (compared to the standard LoRA or (q)LoRA). This together shall be treated as the highly nontrivial accuracy-efficiency gain of our method. In order to dive into how much the gain comes up, we dive into the above analysis on several potential factors (e.g., choosing which metric, how long prompt), confirming that a mixture of short prompts is the best option compared to one much longer prompt, and further that such mixture can be cleverly implemented with no overhead at all - that makes the logic line of our paper.

We hope our explanation now has completely clarified any confusion and we’re thankful for your very prompt response!