HiddenKey: Parameter-Efficient FineTuning Meets Dropout under a Unified Framework

1. Considering that GLUE is a mature benchmark where the adoption of various techniques has established very strong baselines and left little space for further improvement, the consistent improvements over such strong baselines on both NLU and NLG tasks are still significant. Especially, HiddenKey improves the BLUE by 1.26 and NIST from 8.615 to 8.765 on the E2E dataset. Besides, we conduct repeated experiments to ensure robust conclusions and insights.
2. We analyze the integration of HiddenKey step by step, of which the process can be regarded as the ablation study. Hence, each component of HiddenKey has been verified to be useful. Besides, we provide the formulation and comparison of various methods to demonstrate the computation.
3. Essentially, HiddenKey provides an option for users. If resources are constrained, HiddenKey can provide better performance with the same iteration number. As you said, if the longer finetuning process is allowed, HiddenKey can improve the performance further. In both cases, HiddenKey provides extra benefits.
4. QA
   1. In Figure 4, DropKey does perform better and more stably than HiddenCut. As shown in Table 1, HiddenKey, combining HiddenCut, DropKey, and KL loss, provides the best performance.
   2. The dashed and solid lines mean the training loss and test acc, respectively. The decreased acc with lower training loss in one trail means the occurrence of overfitting, while lower training loss does not mean better acc for different trails shown in Figure 5.

1. We will refine our introduction part to make it more logical and clear. Our paper focuses on the empirical analysis of the effect of various transformer-specific Dropout methods on the PEFT performance. With this purpose, the main contribution is to create a framework to unify the design of different Dropout methods and compare their effectiveness in the context of PEFT. We conduct comprehensive experiments to evaluate the performance of nine different Dropout patterns and empirically conclude the best combination - HiddenKey, which is a recommended practice to alleviate the overfitting in the context of PEFT. Besides, we derive some inspiring insights and mathematical co-relation (the similarity in gradient flow of DropKey and DropAttn.) between different Dropout methods.
2. For the selection of Dropout Methods, we mainly focus on those specially designed for Transformer architecture. There are so many methods actually. As for the vanilla dropout, element-wise methods in our paper can be regarded as the basic Dropout, which will be explicitly mentioned in our paper.
3. Since the Method section focuses on comparing and unifying different dropout methods, we mathematically analyze the relationship between DropKey and DropAttn. For one iteration, it is hard to gain the clean and noisy gradient simultaneously, disentangle the clean and noisy part due to the complicated back-propagation, and determine the impact on performance. However, we can use detach() to stop the gradient noise to verify its existence.
4. Please refer to the first point for our detailed contribution.

Thanks for the response, I would like to increase my score to 5, which is still below the accept threshold, as I prefer some analyses about the generalization error bound rather than some gradient calculation for a sufficient justification.

1. We modify our introduction to make the logic flow more natural.
2. References of DropKey, DropAttention, and HiddenCut will be added in the introduction section.
3. Yes, Figure 3 is an illustration of HiddenKey. We will correct this typo.
4. Based on our experience, the introduction of KL loss does not significantly increase the training cost. There are two reasons. One is that the second branch does not need backward, which is much slower than forward. The other is that the second branch runs in the torch.no_grad() mode, which does not store the immediate states and thus is much faster than general forward. Besides, two branches can be computed simultaneously, and the increased training time could be ignorable. We will conduct comprehensive research and evaluate it to demonstrate the additional cost given by the KL loss, if any.
5. DropKey_column is the column-wise DropKey. HiddenKey is not included in the Figure 4. The figure is demonstrated for the illustration of the superiority of element-wise HiddenCut and Column-wise DropKey, indicating the rationale behind our proposed HiddenKey → combining the element-wise HiddenCut, the Column-wise DropKey, and KL loss to maximize the performance.
6. For HiddenKey-, two values of superscripts mean the dropout rates of element-wise HiddenCut and column-wise DropKey, respectively. For other variants, one value means the weight of KL loss for the optimal performance, and the dropout rates remain the same with HiddenKey-. As for the reason of 1000 (apparently larger than other values), this is caused by the STS-B dataset, which is a regression dataset, and the scale of KL loss is different from other classification tasks.
7. Sure, we will include it for your reference.
8. Responses to the weaknesses: The training cost of the proposed HiddenKey will be analyzed to illustrate the efficiency of our methods. Meanwhile, we will revise our writing to improve the logic flow and make it more natural and clear. Furthermore, introducing a full fine-tuned result as a baseline will be very helpful for illustrating the upper bound of our proposed method; we will also explore other PEFT methods to generate a more comprehensive framework of the effectiveness of different dropout methods in the context of PEFT.

1. Our paper focuses on the empirical analysis of the effect of various transformer-specific Dropout methods on the PEFT performance. With this purpose, the main contribution is to create a framework to unify the design of different Dropout methods and compare their effectiveness in the context of PEFT. We conduct comprehensive experiments to evaluate the performance of nine different Dropout patterns and empirically conclude the best combination - HiddenKey, which is the analysis result instead of the only contribution. Besides, we derive some inspiring insights and mathematical co-relation (the similarity in gradient flow of DropKey and DropAttn.) among different Dropout methods.

2. The relatively large variance of our results is subjected to the task configuration (i.e. The performance of LoRA is sensitive to the initialization). To remedy this large variance, we repeat each NLU task five times and take the median value to reduce the randomness, given that the median value is more robust to outliers than the average value. In Table 1, we additionally show the variance for the necessity of repeated trials and readers’ reference. Compared with other published works (e.g., LoRA), our variance falls into a reasonable area. Figure 4. provides a visual illustration of our results, and the superiority of element-wise Hiddencut and column-wise DropKey over different task configurations is demonstrated better.

3. The gain given by HiddenKey- is minor before applying KL loss, but it still showcases a positive signal that the average/median performance increases. After applying the KL loss, the superiority of HiddenKey could be confirmed. Another factor contributing to the minor improvements is that the GLUE benchmark is widely adopted, and it is hard to gain large improvement on a strong baseline.

4. Considering the resource constraint, we are still trying to supplement the experiments on NLG tasks with LLaMA.

5. Among all PEFT methods, LoRA is one of the most popular methods widely studied in academia and industry. There is also some work to unify different PEFT methods, like “Towards a Unified View of Parameter-Efficient Transfer Learning.” We hope our analysis is also valid for other specific PEFT methods.