No more hard-prompts: SoftSRV prompting for synthetic data generation

We want to stress that standard soft-prompt methods (e.g. prefix-tuning) are not directly applicable to synthetic data generation. Concretely, consider a standard soft-prompt method trained on a question/answer task, the standard soft-prompt methods will try to generate the answer given the question as opposed to generating a similar question on the same topic. In order to generate a similar question on topic using standard soft-prompt methods, one would, for example, need to create a dataset with pairs of similar questions and then train a soft-prompt method on this new dataset. Building a dataset of paired examples is challenging due to a potential scarcity of similar examples and difficulty in defining a notion of similarity.

Our goal is to develop a synthetic data generation framework and targeting standard soft-prompt setting is beyond the scope of this paper.

• Question: "To build such a data synthetic generation, the proposed method first needs to obtain a lot of samples from the target distributions to train the soft prompt. So, which is advantage to use the proposed method as directly use hard prompt does not require any target samples for training?"

The hard prompt methods do require target samples to populate the templates, that is the templates use example questions that are taken directly from the training set. See Section 3.2 and Appendix B. To train SoftSRV, we use the same amount of training data as hard prompting methods. From mere 100-1000s of examples, we create 10x as many synthetic examples.

• Question: "The generalization of the proposed method is also a concern. It seems if we want to use a different LLM (e.g., LLama-70B) for data generation, we also need to train a corresponding model. Therefore, the proposed method (i.e., the trained soft prompt) cannot be adapted to different LLMs, while hard prompt does not have such a concern."

This is true, but also would be true of any training based approach. Even for hard-prompts, one would potentially need to refine the hard-prompt for each model type to match the idiosyncrasies of each model. This can be due to, for example, different instruction tuning procedures for each model.

• Question: "In authors' setting, training SoftSRV also require some training (e.g., 20K steps). So what happened if our downstream tasks do not have enough data for training?"

We find that 100-1000's of examples is sufficient to train SoftSRV. Note that the hard-prompt baselines also use this data to seed the hard prompts templates.

• Question: "Experiments on more LLMs are required. ... Do you try some other open-domain scenarios, like some datasets which are not in question-answer format (e.g., Chat)?"

We agree that trying open-domain scenarios and testing more LLMs is an important future direction, which we plan on pursuing.

• Question: "The method shows limited novelty.... A model fine-tuned on ~20K samples is naturally better aligned with these domains than zero-shot"

We are not comparing to the zero-shot setting. All methods, including the hard-prompt baselines, are fully fine-tuned on the same number of synthetic data (see Figure 2). The comparison is between fine-tuning on synthetic data generated by hard prompting methods versus by SoftSRV methods.

Perhaps the reviewer’s question is about question generation? At question generation time, both hard prompting methods and SoftSRV use data from the benchmark's training fold. In particular, the hard-prompt seeds the prompt template with examples from the training fold. Please see section 3.3 for more details.

• Question: "Given this, the approach appears straightforward. Rather than fine-tuning a model for data generation, why not simply PEFT fine-tune the model directly on these tasks? As shown in Table 1, the "train" column demonstrates superior performance compared to the "HP" columns."

In Table 1, we are fully fine-tuning Gemma 2B where each column indicates the data we are using (e.g. original training data, synthetic data generated by HP, etc). We find that fine-tuning using the SS_MC generated data performs better than fully fine-tuning on the original training data for all datasets except BOOLQ. See discussion in line 357-363. In general, we expect that PEFT Gemma on the original training data would perform worse compared to fully finetuning Gemma on the original training data. Note also that HP columns are the competitor baseline methods.

Perhaps the reviewer is asking about PEFT the large model to use on the downstream task directly? This is a different experiment and we are imagining a (typical) scenario where we want to use a small downstream model, for example, for serving efficiency.

Perhaps the reviewer is asking about using PEFT for synthetic data generation? Please see discussion with Reviewer 5RCw.

• Question: "Current comparisons are unfair. More baselines that incorporate training-based data generation methods are needed."

The hard prompting baselines do use the training data and we are not aware of other published baselines that are conceptually different than the ones we already tested.

• Question: "Testing with out-of-distribution (OOD) data instead of MBPP, GSM8K, and BoolQ could further validate the method’s robustness. The paper evaluates only one LM; assessing additional LMs could strengthen its claims."

We agree that evaluating on out-of-distribution domains and testing more LLMs is an important future direction, which we plan on pursuing.

• Question: “only [useful] case when you have a lot of human data”

The parametrized soft-prompts have a relatively small number of parameters, compared to the frozen LM, and can be trained with a small amount of data (even just hundreds of examples). Specifically, MBPP only has 384 training examples while GSM8K and BOOLQ have ~7K and ~9K training examples respectively. Once the parameterized soft-prompt is trained, we then greatly expand the fine-tuning set generating tens of thousands of synthetic examples. Thus, we view SoftSRV as very useful in the data scarce setting.

• Question: “application of Lester et al and Li and Liang (who really should be cited)”

Indeed we meant to cite Li and Liang, this was an oversight that is now remedied. However, we do not view it as a straightforward application, given that we introduce the idea of parameterized contextual soft-prompts that allows us to generate a more diverse and representative set of synthetic data. Additionally, unlike Lester et al and Li and Liang, we are not pre-appending our soft-prompts to the input, but are effectively replacing the input with the soft-prompts, see Figure 1. Lastly, the standard soft-prompt methods are not directly applicable to synthetic data generation. See discussion with Reviewer 5RCw.

• Question: “minimum amount of human data”

We find, in our experimental setting, hundreds to a few thousand examples were sufficient, even for the most complex soft prompt family SS_MC. As mentioned above, MBPP only has 384 training examples.

• Question: "Do you see your paper as having novel mathematical contributions, and, if so, what are they?"

Theoretically, it would be interesting to show that generating a sample from the SoftSRV framework is close to drawing a sample from the target distribution. Previous work has shown the soft-prompt guide the LLM towards the target task, but arguing the generated data distribution is close to the target distribution is a more difficult task.

• Question: "Small Training Sample Sensitivity: For datasets with small training sets (like MBPP with only 384 examples), more complex SoftSRV variants perform worse than simpler ones, suggesting the approach may be sensitive to training sample size."

The results in Figure 2 demonstrate that the most expressive SoftSRV variant, SS_MC, consistently exhibits superior finetuning performance, irrespective of the training set size. While Section 3.7 highlights that certain SoftSRV variants achieve improved MAUVE scores compared to SS_MC, this analysis is secondary to our primary objective of enhancing finetuning performance.

The goal of Section 3.7 is that SoftSRV methods result in a synthetic data that is more aligned with the underlying true distributions compared to hard prompted methods (HP, HP_SR) since SoftSRV directly optimizes a data-driven objective guiding the pre-trained model towards the target distribution.

• Question: "No Direct Performance Indicator: The authors note that the MAUVE similarity score they use to measure distribution matching is "not a direct indicator of downstream fine-tuning performance," suggesting a lack of clear metrics to predict effectiveness."

The writing in that paragraph is convoluted and it has misled the reviewer. We will make sure to revise it. Here, we wanted to point out that the MAUVE should only be used as a tool for a secondary analysis since it is not a direct indicator of downstream performance. In the previous sections, we do show that SoftSRV admits direct downstream performance improvements on the test set – please see Figure 2 and Table 1.

• Question: "Problem Setup Limitations: Assumes fixed maximum sequence length m (Section 2, pg 2). Restricts to scenarios where input and output sequences have equal length"

All LLM have some max length in practice. Input and output sequences do not need to be the same and in Section 2, we assume them to have equal length only for notional simplicity, without loss of generality. Note, in our experiments, input and output sequences are not necessarily the same.

• Question: "Comparison Scope: Primarily compares against hard-prompting baselines. Limited comparison with other synthetic data generation approaches. No comparison with other parameter-efficient tuning methods"

All prior synthetic data generation approaches that have shown promising results are based on hard prompting. We would be grateful for the reviewer to point us to other promising approaches in this setting.

This paper focuses on developing a new data-driven framework for generating synthetic data. While SoftSRV could potentially be applied to parameter-efficient tuning methods, exploring that application is outside the scope of our current research. Note that parameter-efficient tuning methods cannot be directly applied to targeted data generation. See discussion with Reviewer 5RCw.

• Question: "Validation Gaps: Initial results focused on only three domains (coding, math, reasoning). No clear guidelines for choosing between different variants (SSNSP, SSMPk, SSMC)."

The SS_MC method outperforms the other SoftSRV methods on all three domains and hence our suggestions is to use this method. We will clarify this in the paper.

• Question: Reviewer asks a series of questions about SoftSRV hyperparameters & empirical setup (prompt length, training steps, learning rate, quality and diversity of the initial sample, minimal sample size, choice of sequence embedder, niche domains, why use MLPs, basis prompts for SSMPk, random seeds, and initializations).

While tuning hyperparameters for SoftSRV (e.g. choice of embedder, SoftSRV parametrizations, etc.) or varying the underlying setting (e.g. training set size, random seeds/initiatilizations, etc) could yield interesting insights and further performance improvements, we of course have limited time and compute for the study and find our methods already outperform hard prompt baselines even when fixing these hyperparameters and experimental setup a priori.

• Question: "Sufficiency of Context Vector: Assumes the context vector derived from an example sequence captures enough information to generate meaningful variations"

We showed empirically that the context vector contains sufficient information to beat hard-prompting baseline methods. It would indeed be interesting to conduct a theoretical analysis to further understand the sufficiency of context vectors.

• Question: "Task Complexity Impact: The approach appears less effective for more complex tasks like BoolQ that require generating longer passages and more diverse content. The authors note this is "perhaps the most difficult task to generate synthetic data for."

Yes, BoolQ is the hardest task among the three, but nevertheless we still find that SS_MC admits the best performance.