Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Models

We sincerely thank the reviewer for the constructive comments and feedback. We appreciate that our work is considered “pioneering”, “interesting”, and “neat”. We hope our response can address your concerns.

Q1: More intuitive explanation

We thank the reviewer for the valuable comment. Our intuition is that more concentration on prompts would lead to less attention on inputs. Thus, domain shifts (change of the domain that inputs are sampled from) cause less negative effect as the model pays more attention to knowledge-intensive prompts instead of constantly changing inputs.

Moreover, it is also interesting to investigate the reason why prompts optimized by our method are more generalizable from the perspective of calibration. Calibration refers to the model's ability to provide class probabilities that correspond to its likelihood of being true. A well-calibrated model exhibits better domain generalization ability, as claimed in [1]:

In this paper we highlight a novel connection between multi-domain calibration and OOD generalization, arguing that such calibration can be viewed as an invariant representation.

We use a common metric Expected Calibration Error (ECE) [2] for evaluating the calibration:

Here $n$ is the number of samples, $M$ is the number of bins. Each sample has its own label $y_{i}$, predicted label $\widehat{y_i}$, and predicted probability $\widehat{p_i}$. A lower ECE value indicates better calibration. As shown in Table 1, we demonstrate the change in prompt calibration before and after training using our method. We find that our method significantly improves model calibration compared with vanilla method. We infer that good calibration functions as an important factor to avoid significant inductive bias in domain generalization setting, which explains the performance improvement brought by our method.

Table 1. Expected Calibration Error of Vanilla Prefix Tuning and Prefix Tuning with our method. smaller the better.

Q2: Extension to other tasks

We would like to thank the reviewer for the valuable comment. We are glad to share that our method is able to successfully be applied on mainstream Large Language Models and question-answering tasks. We validate the effectiveness of our method on Llama-2-7b-chat, Vicuna-7b-v1.5, and Alpaca-7b-wdiff models for improving domain generalization ability of prompts on question-answering tasks. We evaluate our method on ROC, SCT, and COPA datasets from the TRAM Benchmark [3] (referred as R, S, and C for simplicity), covering multiple-choice questions in reading comprehension and commonsense reasoning. The result is shown in Table 2.

Experimental results show that our method significantly improves the performance of large models on question-answering tasks across multiple domain generalization settings. For instance, for the Llama-7b model, our method improved the average accuracy of soft prompt generalization and hard prompt generalization comparisons by 1.91% and 2.36%, respectively; similar improvements were observed for Vicuna-7b and Alpaca-7b models, ranging from 1.55% to 2.05% and 1.78% to 1.99% respectively.

Table 2. Performance comparison of LLMs on multiple-choice task accuracy under MFDG settings.

In our global response PDF, we illustrate the Concentration Strength Distribution of prompts in the In-Context Demo format for three 7B-sized language models (Llama, Vicuna, Alpaca) across three different tasks (SA, NLI, MCQA). A common observation is that the concentration strength is stronger in deep layers instead of shallow layers. To be more specific, compared to smaller models (Roberta-large), concentration phenomena occur earlier in larger models. Inductively, we get the conclusion that this phenomenon — higher concentration in deep layers, happens independent of model sizes, tasks or prompts.

Q3: In-context Citation Errors

We would like to thank the reviewers for reading our work carefully. We sincerely apologize for the in-context citation errors in our paper. We will check and revise these errors in the next version.

[1] Wald, Yoav, et al. "On calibration and out-of-domain generalization." Advances in neural information processing systems 34 (2021): 2215-2227.

[2] Naeini, Mahdi Pakdaman, Gregory Cooper, and Milos Hauskrecht. "Obtaining well calibrated probabilities using bayesian binning." Proceedings of the AAAI conference on artificial intelligence. Vol. 29. No. 1. 2015.

[3] Wang, Yuqing, and Yun Zhao. "Tram: Benchmarking temporal reasoning for large language models." arXiv preprint arXiv:2310.00835 (2023).

We sincerely appreciate your valuable feedback and insightful discussion! We hope our response has been helpful to you. As the discussion period is drawing to a close, we warmly welcome any further questions from the reviewer. We would be delighted to provide additional clarification!

We sincerely thank the reviewer for the constructive comments and feedback. We appreciate that our work is considered “well organized”, “interesting”, and “generalizable”. We hope our response can address your concerns.

Q1: Extension to other tasks

We would like to thank the reviewer for the valuable comment. We would like to thank the reviewer for drawing our attention to extending our method to broader applications. We are glad to share that our method is able to successfully be applied on mainstream Large Language Models and question-answering tasks. We validate the effectiveness of our method on Llama-2-7b-chat, Vicuna-7b-v1.5, and Alpaca-7b-wdiff models for improving domain generalization ability of prompts on question-answering tasks. We evaluate our method on ROC, SCT, and COPA datasets from the TRAM Benchmark [1] (referred as R, S, and C for simplicity), covering multiple-choice questions in reading comprehension and commonsense reasoning. The result is shown in Table 1.

Experimental results show that our method significantly improves the performance of large models on question-answering tasks across multiple domain generalization settings. For instance, for the Llama-7b model, our method improved the average accuracy of soft prompt generalization and hard prompt generalization comparisons by 1.91% and 2.36%, respectively; similar improvements were observed for Vicuna-7b and Alpaca-7b models, ranging from 1.55% to 2.05% and 1.78% to 1.99% respectively.

Table 1. Performance comparison of LLMs on multiple-choice tasks under MFDG settings. The last column shows the average gap between test performance on vanilla methods and our methods. Results are averages from 3 runs using different random seeds.

Additionally, we would also like to discuss ‘’why our method works well for large generative language models?’’. In our global response PDF, we present the Concentration Strength Distribution of prompts using In-Context Demo across three 7B-sized language models (Llama, Vicuna, Alpaca) on three different tasks (SA, NLI, QA). We observed that all three LLMs exhibit stronger concentration strength in deeper layers compared to shallower layers when confronted with prompts for different tasks. Additionally, we found that this phenomenon occurs earlier in larger models (7B) compared to smaller models like Roberta-large. We speculate that this behavior is related to the alignment stage in pre-training of large models during Supervised Fine Tuning with a large number of prompts.

[1] Wang, Yuqing, and Yun Zhao. "Tram: Benchmarking temporal reasoning for large language models." arXiv preprint arXiv:2310.00835 (2023).

We sincerely appreciate your valuable feedback and insightful discussion! We hope our response has been helpful to you. As the discussion period is drawing to a close, we warmly welcome any further questions from the reviewer. We would be delighted to provide additional clarification!

Q3: Spelling errors and missing hyperparameters:

We sincerely appreciate your careful reading.

• Spelling errors: We apologize for any ambiguities and typographical errors in our paper. We will incorporate your suggestions in the revision to ensure these errors do not recur.
• Missing hyperparameters: We provide specific values of lambda in Appendix B.4 (as shown in Table 5 in our paper). We use a validation set of the same size as the training set to evaluate its selection of different regularizers. We apologize for the lack of this information and will emphasize it in the next version.

Q4: Concentration distribution on larger models:

We thank the reviewer for the valuable comments. In Appendix C.2, we present the Concentration Strength Distribution of different prompts in the Roberta-large model (355M). In our global response PDF, we illustrate the Concentration Strength Distribution of prompts in the In-Context Demo format for three 7B-sized language models (Llama, Vicuna, Alpaca) across three different tasks (SA, NLI, QA). A common observation is that the concentration strength is stronger in deep layers instead of shallow layers. To be more specific, compared to smaller models (Roberta-large), concentration phenomena occurs earlier in larger models and keeps high in deep layers . Inductively, we get the conclusion that this phenomenon — higher concentration in deep layers, happens independent of model sizes, tasks or prompts.

Q5: Evaluation other than fine-tuning

We would like to thank the reviewer for the valuable comment. We would like to clarify that in-context learning is one of our baselines for discrete prompt optimization. In our main experiments (Table 1 in our paper), In-Context Demo method provides examples as prompt, which is a classic version of existing in-context learning methods (Line 497).

We sincerely thank the reviewer for the constructive comments and feedback. We appreciate that our work is considered “important”, “well-written”, and “thorough”. We hope our response can address your concerns.

Q1: Improvement and additional baselines

Improvement: We will try to address the reviewer's concerns about performance improvement in the following points.

• The objective of this work is to enhance domain generalization in current prompt optimization methods, not to design a new framework. Our method acts as a plug-and-play module compatible with mainstream prompt optimization methods. As shown in Table 1, it improves performance in 3 soft and 3 hard prompt optimization methods.
• Our method modifies the training objective or filter-match strategy within the original framework, so performance is limited by the original model structure or initial prompt set. For example, the quality of the candidate prompt set affects the performance of hard prompt optimization (as shown in Table 1 in our paper). Nonetheless, our method consistently enhances domain generalization with prompts from the same candidate set, demonstrating its effectiveness.

Additional Baselines: Following the suggestion of the reviewer, we compare our method with the suggested domain adaptation methods — OPTIMA [1] and GPO [2] and two PEFT methods similar to [3] — IA3 and LoRA. The results are shown in Table 1. Compared with DP2O with both, the best performer in a series of prompt optimization methods piggybacked with our optimization strategy, we find that DP2O with both achieves accuracy improvements of 3.88%, 3.54%, 4.46%, 1.77%, 0.55%, and 3.62% compared to the second-best method in six experimental settings.

Although our method still performs best, we would like to highlight that our work improves model performance in domain generalization setting where no information from target domain is available during training while data from target domain is accessible in domain adaptation methods like OPTIMA and GPO. The Introduction section of our paper (line 35-39) discusses the shortcomings of domain adaptation methods, while the Preliminary section (line 87-95) introduces the basic definition of domain generalization problems. The Related Work section (line 472-480) contrasts the differences between these two types of problems.

Table 1. Performance comparison of text classification task accuracy under MFDG settings. Bold indicates the best result for each column, and underline indicates the second-best result for each column. Results are averages from 3 runs using different random seeds.

Q2: Extension to other architectures, tasks and modalities

We are glad to share that our method is able to successfully be applied on different architectures like large decoder-only models (Llama-2-7b-chat, Vicuna-7b-v1.5, and Alpaca-7b-wdiff) for additional tasks such as question answering. To be specific, we evaluate our method on ROC, SCT, and COPA datasets from the TRAM Benchmark [3] (referred as R, S, and C for simplicity), covering multiple-choice questions in reading comprehension and commonsense reasoning. The result is shown in Table 2.

Experimental results show that our method significantly improves the performance of large decoder-only models on question-answering tasks across multiple domain generalization settings. For instance, for the Llama-7b model, our method improved the average accuracy of soft prompt generalization and hard prompt generalization comparisons by 1.91% and 2.36%, respectively. Similar improvements are observed in Vicuna-7b and Alpaca-7b models, ranging from 1.55% to 2.05% and 1.78% to 1.99% respectively.

We would like to clarify that our method is designed for language model, as we stated in the title of this paper. Incorporating other modalities into our method is beyond the scope of our research. We believe exploring the possibility to generalize our method to tasks in other modalities is interesting and we will do it in the future.

Table 2. Performance comparison of decoder-only LLMs on multiple-choice task accuracy under MFDG settings.

[1] Guo, et al. Improving the sample efficiency of prompt tuning with domain adaptation.

[2] Li, et al. Robust prompt optimization for large language models against distribution shifts.

[3] Tam, et al. Parameter-efficient prompt tuning makes generalized and calibrated neural text retrievers.

[4] Wang, et al. Tram: Benchmarking temporal reasoning for large language models.

We sincerely appreciate your valuable feedback and insightful discussion! We hope our response has been helpful to you. As the discussion period is drawing to a close, we warmly welcome any further questions from the reviewer. We would be delighted to provide additional clarification!

We sincerely thank the reviewer for the constructive comments and feedback. We appreciate that our work is considered “well-written”, “novel”, and “insightful”. We hope our response can address your concerns.

Q1: Applicability to larger models and other tasks:

We would like to thank the reviewer for drawing our attention to extending our method to broader applications. We are glad to share that our method is able to successfully be applied on mainstream Large Language Models and question-answering tasks. We validate the effectiveness of our method on Llama-2-7b-chat, Vicuna-7b-v1.5, and Alpaca-7b-wdiff models for improving domain generalization ability of prompts on question-answering tasks. We evaluate our method on ROC, SCT, and COPA datasets from the TRAM Benchmark [1] (referred as R, S, and C for simplicity), covering multiple-choice questions in reading comprehension and commonsense reasoning. The result is shown in Table 1.

Experimental results show that our method significantly improves the performance of large models on question-answering tasks across multiple domain generalization settings. For instance, for the Llama-7b model, our method improved the average accuracy of soft prompt generalization and hard prompt generalization comparisons by 1.91% and 2.36%, respectively; similar improvements were observed for Vicuna-7b and Alpaca-7b models, ranging from 1.55% to 2.05% and 1.78% to 1.99% respectively.

Our research primarily addresses the analysis and optimization of attention patterns from current decoding tokens to prompt tokens (line. 121-122). Thus, long sequence generation (e.g., open-end generation) is beyond the scope of this research, as we admit in Section 7 (line. 323-325). We will explore the possibility to improve domain generalizability on more generative tasks in the future.

Table 1. Performance comparison of LLMs on multiple-choice tasks under MFDG settings. The last column shows the average gap between test performance on vanilla methods and our methods. Results are averages from 3 runs using different random seeds.

Q2: Baselines and Hyperparameter selection

We thank the reviewer for the valuable comment. For starters, we would like to claim the objective of this work is not to design a brand-new framework, but to improve domain generalization ability of current prompt optimization methods, as stated in lines. 73-75:

With the principle of concentration §3, we propose two algorithms that could piggyback upon popular prompt optimization methods for both hard and soft prompts to improve the domain generalization ability of prompts.

Thus, prompt tuning, as one of the most popular prompt optimization methods, serves as a good baseline to demonstrate that our proposed objective works well for improving the domain generalization ability within the framework of prompt tuning itself. In addition to prompt tuning, we also try to apply our proposed goal to several stronger baselines (such as Prefix Tuning and P-Tuningv2 for soft prompts, In-Context Demo and DP2O for hard prompts), and it consistently shows excellent performance in improving domain generalization ability of prompts on all experimental settings (as shown in Table 1 in our paper).

Additionally, we would like to address the reviewer’s concern in hyperparameter selection. The suggested work [2] (100 prompt tokens and learning rate of 0.3) is based on full training data for the T5 model. In resource-limited scenarios, such settings may lead to severe overfitting issues.

We also conduct experiments with the suggested hyperparameters by using T5-base model. The result is in Table 2. We find more prompt tokens and larger learning rate actually degrades performance of prompt tuning in the few-shot setting. We would like to clarify that our hyperparameter setting is similar with [3], where a learning rate of 1e-5 is used to train XLM-RoBERTa-base model in the few-shot setting with 4 soft prompt tokens.

Table 2. Performance comparison of soft prompt learning on classification task accuracy for T5-base model. Results are averages from 3 runs using different random seeds.

[1] Wang, Yuqing, and Yun Zhao. "Tram: Benchmarking temporal reasoning for large language models." arXiv preprint arXiv:2310.00835 (2023).

[2] Lester, Brian, Rami Al-Rfou, and Noah Constant. "The power of scale for parameter-efficient prompt tuning." arXiv preprint arXiv:2104.08691 (2021).

[3] Zhao, Mengjie, and Hinrich Schütze. "Discrete and soft prompting for multilingual models." arXiv preprint arXiv:2109.03630 (2021).

We sincerely appreciate your valuable feedback and insightful discussion! We hope our response has been helpful to you. As the discussion period is drawing to a close, we warmly welcome any further questions from the reviewer. We would be delighted to provide additional clarification!