Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs

We thank the Reviewer for the constructive feedback! We address the main points one by one below:

A more detailed discussion on public dataset selection risks

We investigate the effect of the choice of public data in Table 15 in Appendix (section D.1), where we consider both public datasets from similar task domains as well as those from different task domains compared to the private dataset. Our experimental results show that selecting a public dataset with a similar task to the private dataset is helpful for effective prompt transfer.

To limit the privacy leakage and obtain good performance, we showed that only the target task should be considered when selecting the public dataset. In Table 15 (Appendix D.1), we find that the best transfer performance can be achieved with public datasets from the same task family, even if the datasets are not very similar and have, e.g., different numbers of classes. Our framework assumes that the model provider has knowledge of only the task domain, not the user’s actual data. This is a reasonable assumption, as task metadata is often available in practical settings. For instance, cloud providers and ML platforms (e.g., Google Vertex AI, OpenAI’s API) typically encourage users to specify the task type or provide task demonstrations (i.e., few-shot learning) when fine-tuning models to optimize performance [A]. Similarly, many AI service providers, such as NVIDIA’s NeMo framework, offer task-specific APIs explicitly designed for applications like chatbots, summarization, and generation, which inherently expose task-level information [B].

why soft prompts trained on a distilled model remain transferable？Theoretical insights or at least empirical ablations on feature space alignment between the small and large models

We conducted an ablation study to investigate the impact of feature alignment. We use a model with the same architecture and initial parameters as our distilled 2-layer LLaMA. Instead of applying knowledge distillation on the BookCorpus dataset, we directly fine-tune this model on BookCorpus to achieve comparable performance to the distilled smaller model. This model has no feature space alignment with the large model. We report this model’s transfer performance.

Transfer results for the distilled model are always higher. This indicates that using knowledge distillation to preserve alignment between small model and big model is essential in our designed prompt transfer procedure.

How to select $\alpha$ in the prompt transfer loss (Equation 3)? Have you explored dynamic or task-adaptive weighting schemes for $\alpha$? Could tuning $\alpha$ differently for various LLMs improve performance?

We conducted a thorough ablation study on $\alpha$ (see Table 19), evaluating transfer performance across $\alpha$ values ranging from 0.0 to 1.0. The results indicate that multiple $\alpha$ values lead to good transfer performance, indicating the robustness of our POST to the choice of $\alpha$.

Additionally, we clarify the intuition behind the transfer loss design in Appendix D.5 and propose a heuristic method (Equation 6) for finding an optimal $\alpha$, which supports tuning $\alpha$ differently for different LLMs to improve performance in practice.

How dataset properties (e.g., domain similarity, label space overlap, dataset size) impact transfer success?

We investigated the influence of the domain similarity of the public dataset on the transfer performance (see Appendix D.1). The findings demonstrate that selecting a public dataset with a task domain similar to the private task is helpful for effective prompt transfer.

What if the public dataset is too different from the private task dataset?

Table 15 presents the transfer performance using public datasets from different task families. It is observed that the best transfer performance can be achieved when using same-domain data, while datasets with different tasks can lead to performance degradation.

---

We provide the reference for all the responses.

References:

[A] OpenAI Fine-Tuning Guide – https://platform.openai.com/docs/guides/fine-tuning

[B] NVIDIA NeMo Framework – https://docs.nvidia.com/nemo-framework/user-guide/latest/overview.html

[C] Duan, H., Dziedzic, A., Papernot, N., and Boenisch, F. “Flocks of stochastic parrots: Differentially private prompt learning for large language models.” NeurIPS 2023.

We thank the Reviewer for the positive feedback! We address the main points one by one below:

The quality of the transferred prompt heavily depends on the distilled model.

We fully agree with the Reviewer. The influence of compressed model size (number of layers) on performance and compute time is presented in Table 18 in our paper. And we show the results in the table below for the reviewer’s convenience. Indeed, our findings show that while larger distilled models generally improve performance, they also increase computational costs for the user.

For complex tasks, where the full target LLM can effectively learn a strong prompt, a weakly tuned prompt on the small model will not transfer well, as the distilled model may fail to capture the necessary task structure.

Beyond investigating the trade-offs of different compression ratios, we also designed a heuristic approach to optimize the $\alpha$ in Equation 6, which balances the first and second loss terms in the transfer loss function (Equation 3). Specifically, when the full target LLM has a strong zero-shot performance or the small model has a weak performance, we set a larger $\alpha$, reducing the reliance on the small model’s behavior while primarily incorporating the directional change induced by the prompt. This adjustment enhances the effectiveness of our POST method in scenarios where the small model has weak performance. We perform a wide range of experiments using different $\alpha$ values and compare the best performing $\alpha$ with the one output by our heuristic, as shown in Table 19 in Appendix D.5. For reviewer’s convenience, we show partial results of Table 19, i.e., the performance of Roberta-base, in the following table (transferred accuracy are presented as mean (standard deviation)):

We observe from the results that the heuristic can successfully identify a good range for $\alpha$ and is usually not far off the empirical optimal values.

We appreciate the Reviewer's insightful comments! Below, we address each comment in detail:

More rigorous exploration into why certain public datasets work better for transfer than others; Additional clarification or justification for the choice of public datasets.

We thoroughly analyzed the effect of public dataset selection in Table 15 in the Appendix (section D.1). Our experimental results demonstrate that public datasets from the same task domain as the private dataset yield better transfer performance than others. This aligns with the intuition that task similarity enhances knowledge transfer. Based on these findings, it is suggested to select a public dataset for a similar task as the private data for effective prompt transfer.

A clearer explanation of the hyperparameter choices (especially $\alpha$ in the transfer loss)

We conducted an ablation study for the parameter $\alpha$ in Table 19 in the Appendix (section D.5). We evaluated the transfer performance across various $\alpha$ values, ranging from 0.0 to 1.0. The results indicate that multiple $\alpha$ values lead to good transfer performance. Furthermore, we designed a heuristic to find an optimal $\alpha$ as shown in Equation 6 in the Appendix (section D.5). The intuition of the heuristic is that if the target model performs well, it needs to less mimic the behavior of the smaller model, but only incorporate the directional change induced by the prompt, leading to a larger $\alpha$. It can be observed from Table 19 that the designed heuristic can successfully identify a good range for $\alpha$ and is usually not far off the empirical optimal values. Thus, for each dataset and model, we selected the $\alpha$ according to the heuristic we proposed in the same appendix section (D.5).

Potential limitations in transfer performance to LLMs with significantly different architectures or training paradigms.

In our setup, the LLM provider distills a small model from the LLM with the aim of guaranteeing good transfer performance. The transfer performance between different architectures is orthogonal to our problem setup.

Practical privacy impacts beyond membership inference attacks

Membership inference attacks are widely recognized as a gold standard for evaluating privacy leakage. Notably, in its most stringent form, our POST framework incorporates DP, which, by definition, ensures that the presence or absence of any individual data point remains indistinguishable and provides theoretical protection against every possible attack.

Can the authors clarify how sensitive the method is to the choice of the $\alpha$ parameter in the prompt transfer loss? How was $\alpha$ tuned in practice?

We performed an ablation study on different $\alpha$ values in Table 19, confirming that our approach is robust to a range of $\alpha$-s. Additionally, we designed a heuristic to find an optimal alpha (see Equation 6). The underlying intuition is that if the target model exhibits strong zero-shot performance, it requires less direct imitation of the small model. Instead, it should primarily incorporate the directional changes induced by the prompt, ensuring effective transfer while maintaining the target model’s inherent capabilities.

How would the approach scale to very large LLMs (e.g., 70B+ parameters)?

The LLM providers are assumed to have the computational resources to distill the target models into smaller ones, as they already manage to train such large language models. The overhead on the client end should not depend on the scale of the model used by the LLM provider.

We appreciate the Reviewer's constructive review and address each point as follows:

The problem setting itself is unrealistic to me.

Soft prompts are exposed via public APIs, such as NVIDIA NeMo, as we highlight in the abstract and introduction of our paper. Could the reviewer kindly clarify specific concerns regarding the “realism of the problem setting” in our work?

I don't see clear contributions of the paper.

We enumerated the contributions of our work at the end of the introduction.

distilled model parameters can reveal pretraining details [1].

The problem of protecting pre-training details is orthogonal to our work. However, we do agree that the LLM provider might not want to disclose such details.

To address the reviewer’s comment, we follow the setup from the paper cited by the Reviewer [1] and use mink% to perform Membership Inference Attack on the WikiMIA [1] dataset with the Pythia-2.8B model. We test 2 models distilled from Pythia-2.8B, each with 3, and 4 layers, respectively. Our results are depicted below and show that the AUC and TPR@1%FPR of the distilled models are much smaller than the base model. This shows that the knowledge distillation in our work can even reduce the pretraining data leakage.

The LLM provider shouldn't have knowledge of users' private data.

Our framework assumes that the model provider has knowledge of only the task domain, not the user’s actual data. This is a reasonable assumption, as task metadata is often available in practical settings. For instance, cloud providers and ML platforms (e.g., Google Vertex AI, OpenAI) typically encourage users to specify the task type or provide task demonstrations (i.e., few-shot learning) to fine-tune models [A]. Similarly, many AI service providers, such as NVIDIA’s NeMo framework, offer task-specific APIs explicitly designed for applications like chatbots and summarization, which inherently expose task-level information [B]. Importantly, our definition of privacy leakage follows DP and is measured at the individual datapoint level. Our approach follows the practical setups. We also offer a general statement regarding the privacy-utility trade-off at the end of our response.

Additional baseline 1: Zero-shot performance of instruction-tuned models

We appreciate the suggestion and have included the zero-shot performance of instruction-tuned Llama2-7b-chat in the following table:

The instruction-tuned model has very close zero-shot performance to the non-instruction-tuned model. This is because we convert the problem of classification to a prefix infilling problem for a non-instruction-tuned model and aggregate the results with multiple ground truth text labels, see Table 7 in the Appendix. This enhances the robustness of our evaluation and allows the non-instruction-tuned model to effectively handle classification tasks.

Advantage of prompt tuning

Prompt tuning offers advantages for LLM providers as it enables batch processing, allowing multiple tasks (from different users) to be handled in a single inference pass with prompts. In contrast, LoRA or fine-tuned models require separate inference passes through the model, significantly increasing the computational costs, and even more importantly, separate models per task [C]. The efficiency gain of soft-prompts provides a strong incentive for the LLM providers to adopt our framework.

Additional baseline 2: Full fine-tuning on the compressed model

We have conducted an additional experiment evaluating full fine-tuning on the compressed model:

Full fine-tuning on the compressed model indeed outperforms prompt tuning on the compressed model, however, it still underperforms POST transfer. This is because the compressed model, although it maintains some similarity to the teacher model, is still not powerful enough.

expected similarity between the private and public datasets

Table 15 in Appendix D.1 investigates the impact of public dataset selection. We categorize datasets into three groups based on task similarity and observe that public datasets with tasks similar to the private dataset yield better transfer performance. Thus, it is expected that the public dataset is from the similar task domain as the private dataset in our framework.

Why the DP is “optional”?

Our framework is designed to be modular, allowing private data owners to balance privacy and utility trade-offs based on their specific requirements. DP is an optional component that can be integrated when strong privacy guarantees are needed, but its inclusion depends on the user’s trade-off preferences.

---

References included in answer to Reviewer Qi3Y.