Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models

We thank the reviewer for recognizing the contribution and novelty of our work.

Weaknesses 1 & Question 4: “Quality of vocabulary and selection of shared tokens may be important but no analysis is given”. “ How are the anchors chosen? Are they randomly sampled? Will the selection process affect model performance?”

Thanks for pointing out the importance of the shared vocabulary because it provides the ground for prompt transfer. However, we would like to mention that the performance of the transfer is not sensitive to the number of anchors. In Figure 4, we show that even if we use only 512 anchors, there is not much decrease in transfer performance. For the selection of anchors, Appendix A.1 shows that we may simply pick the top-k nearest neighbors based on the prompt embeddings, and we applied this to all the experiments for our method.

Weakness 2: “Inconsistent use of search loss and matching loss in Section 3.4.”

Thanks for pointing out the inconsistency of terminologies. We fixed them in the paper.

Question 1: “Does it mean the vocabularies should be the same or just have some common parts?”

We only require the vocabularies to share a common subset, which is simply the intersection of the BERT, RoBERTa, and ALBERT’s vocabularies in our experiments.

Question 2: “In Equations 5 and 7, what is the optimization method and how do you implement this?”

We used gradient descent. Clarified after Eqn 5. Thanks!

Question 3: “Will this method be applicable to generative models such as GPT?”

Thanks for pointing out this interesting further direction. We didn’t include generative models because we adhered to the framework established in the OptiPrompt paper, utilizing identical models and the factual probing dataset. We’ve now extended OptiPrompt and tried generative models, namely, gpt2. The results are as follows.

We found that our method is indeed applicable to generative models, and transferring the induced prompts from masked language models to generative models is also possible.

We thank the reviewer for the comments.

Weakness 1 “Compare the proposed method with a straightforward baseline: v^t_i=\sum_{l=1}^k cos(v^s_i, a^s_l) a^s_l”:

We believe there is a minor typo in the formula. If we understand correctly, the suggested method is to set “v^t_i=\sum_{l=1}^k cos(v^s_i, a^s_l) a^t_l”. Thanks for the suggested baseline (which we call weighted target anchors below). We conducted a preliminary study by transferring the induced 41 prompts from BERT-base to BERT-large. The result compared with other methods is as follows.

We see that the proposed baseline does not perform very well in our scenario, probably because it weighs target anchors based on the source cosine similarity in a heuristic manner.

Weakness 2: “Evaluate the proposed method on tasks other than factual probing”

Our choice of factual probing was driven by its suitability for effectively testing prompt transfer across various models, considering the range of subtasks and the inherent difficulty of the task.

We did preliminary experiments on text classification using the SST-2 (2-way sentiment classification) and DBpedia (14-way topic classification) datasets. We keep our default settings and tune five prompt embeddings on two smaller source models (bert and roberta base models) and transfer them to the roberta-large model. The results are as follows.

Performance when transferring to roberta-large

We found the transferred prompts' performance is better than manual prompting, which is consistent with the factual probing task.

We hope our response has addressed your concerns, and we believe that our work makes a valuable contribution showing that induced prompt embedding is transferable across different models. Thanks!

We thank the reviewer for recognizing the novelty of our method by saying “The method is novel and elegant”.

Weakness (performance lower than the source model):

Thanks for this insight! We acknowledge that currently our transfer performance is lower than the source model, which may hinder its immediate practice. We added a limitation paragraph in the conclusion section discussing this. Nevertheless, our work has certain practical values, as LLMs tend to unify and it’ll be handy if we may just use one LLM to solve various tasks.

As also recognized by the reviewer, our paper addresses an important research direction, and with increasingly powerful target LLMs and improved transfer techniques, we can expect that our research serves as a stepping stone towards more effective transfer methods.

Question: “How is the optimum of eqn 7 or 5 searched for exactly?”

The search was conducted by gradient descent. In particular, we compute the gradient of the loss (either eqn 5 or eqn 7) wrt to the embeddings of the prompt tokens and update them accordingly. We’ve clarified it in the revision.

It’s also worth mentioning that our code is publicly available (Footnote 1) to support replication.

We thank the reviewer again for the insightful comments and we’ve revised our paper accordingly.

Thank you for your strong support, especially for recognizing the novelty of our work.

Weakness 1: “Why is generative models not included for experiments?” & Question: “Is BERT embedding transferable to a GPT2 model? Or is a GPT2 model embedding transferable to a pythia model?”

Thank you for the suggestion. We are also interested in the prompt transferability across different models other than the masked language models. We didn’t include generative models because we followed the OptiPrompt paper by using the same settings as they did (same models and the same LAMA dataset).

We’ve now experimented with generative models, and obtained some preliminary results:.

In the experiment, we considered the small and medium versions of the gpt2 model. We found that our method is applicable to generative models as well. Moreover, the transfer from masked language models to generative models is also possible.

Weakness2: “the result and the claim by the paper will be much strong if there are more than 1 dataset”

Thanks for the suggestion. Again, we followed previous work and used LAMA as the dataset, which contains 41 sub-tasks and provides a comprehensive evaluation for prompting transferability.

We did a preliminary experiment on two classification datasets: SST-2, a 2-way sentiment classification, and DBpedia, a 14-way topic classification. We tuned five soft prompt tokens on BERT and RoBERTa base models, and transferred them to the RoBERTa large model. The results are as follows.

Performance when transferring to RoBERTa-Large

We see that the transferred performance is much higher than manual prompting, which shows the clue that our method can be adapted to other datasets and tasks.

We thank the reviewer again for the strong support and valuable suggestions. As our paper on soft prompt transfer is the first of its kind, we’re willing to further pursue this direction in our future work.