TarGEN: Targeted Data Generation with Large Language Models

• R4.1 - Sophisticated Prompt Engineering: We thank the reviewer for the feedback. We employ it to generate samples for a sarcasm identification task:
  • Step 1: Generate a list of domains or settings in which events can take place and where people can interact. Examples: ’A restaurant, ’a museum’, ’a lively rock concert’ Response: “an opera house”. “a classroom”, “a wedding venue” ….
  • Step 2: For all contexts | Prompt: Generate a list of conversational questions you might hear in a conversation in [an opera house]. Response: Did you read the synopsis before coming?
  • Step 3: Label = TRUE | Prompt: Generate a sarcastic response for the following question: Did you read the synopsis before coming?. Example response: Of course not, I love being completely lost for three hours. Label = FALSE | Prompt: Generate a genuine, non-sarcastic response for the following question: Did you read the synopsis before coming? Example response: Yes, I did. It helped me understand the context and enjoy the performance more.

This demonstrates that we can use simple instructions within the TarGEN framework to generate a majority of language tasks.

We would like to highlight that the simple nature of our framework allows for greater flexibility and control, making it well-suited to domain-specific or particularly complex tasks. We would also request the reviewer to go through our response R1.1 addressing the same concern. We are happy to generate any other tasks that the reviewer deems necessary for the verification of this approach.

• R4.2.1 - Lack of comparative analysis: Thanks for the excellent feedback. It has helped us improve our paper. We showed additional comparative analysis to highlight the differences between our work and previous data generation methods. Please check responses R1.2 & R2.2 where we present the qualitative and quantitative results as part of the additional experiments.

• R4.2.2 - Ablation studies: We carry out additional experiments by ablating the "diverse seeds" component as well as the "generative model" in our pipeline. Please check responses R2.3 and R3.1 respectively for the same.

We would be happy to answer additional concerns raised by the reviewer.

• R3.1 - Comparison with other generative models: We appreciate the feedback of the reviewer. We perform the data generation pipeline with Claude Sonnet and Llama 3 70B to create 2 more variants of synthetic SuperGLUE with each model keeping the task set, prompts, and pipeline steps constant. Llama 2 7b was fine-tuned on the above-mentioned datasets and evaluated on the original test set. The results of the experiments are attached below:

We see nearly the same or slightly improved scores when using Llama 3 70B and Claude Sonnet. We would be happy to conduct additional experiments.

• R3.2 - Ensuring consistency: We ensured data consistency by controlling the decoding parameter, temperature, presence penalty, and frequency penalty. Additionally, one can ensure consistency with a more strict constraint of keeping the generated initial seeds frozen to ensure data consistency across five runs.

• R3.3 - Enhancing diversity: The first step of the pipeline allows us to generate multiple semantic contexts within which these instances occur. That in turn helps to generate seed instances with increased semantic diversity. To quantitatively show the effect of seed tasks, we experimented to see the impact if we reduced the number of semantic contexts. The results are mentioned in response R2.3.

• R3.4 - Controlling toxicity: To demonstrate a task-agnostic framework, we rely on guardrails built into the publicly accessible inference interfaces of leading Language models like ChatGPT, Llama 3, and Claude. However, the framework is flexible to the addition of any safeguards that data generators may wish to include for their purposes.

We would be happy to answer any further questions or perform additional experiments as required by the reviewer.

• R2.1 - Qualitative comparison with other methods: Please check the response R1.2.

• R2.2 - Empirical comparison with other methods: To show a quantitative comparison with other data generation methods, we generate synthetic versions of specific tasks from the SuperGLUE dataset using recent data generation frameworks: The tasks we generate are RTE (textual entailment) and AXg (gender disambiguation). We then train a Llama-2 model on these synthetic task variants and evaluate them on original test sets of the datasets. The results are as follows: | Method|RTE|AXg| |-|-|-| |CODA|64.56 |23.46| |ZeroGEN|58.54|19.01| |SuperGEN|77.94|44.67| |SunGEN|75.11|38.16| |ProGEN|78.43|37.94| |Ours (TarGEN)|92.89|51.32|

• R2.3 - Ablation studies: Thank you for your valuable feedback. We have incorporated ablation studies to quantitatively show the effect of each component. For the seed instances, we experimented to see the impact of reducing the number of semantic contexts.

We also showed the effect of self-correction as a part of the ablation study. We are happy to conduct more experiments as per the reviewers' suggestions.

• R2.4 - Regarding \mathcal{V}\-Usable Information: \mathcal{V}\-usable information is calculated as: $I_{V}(X \rightarrow Y) = H_{V}(Y) - H_{V}(Y|X)$, where $H_{V}(Y)$ is the predictive entropy from a null model $LM_{\text{Null}}$ (trained on empty inputs), and $H_{V}(Y|X)$ is the conditional entropy from a model $LM_{X}$ (trained on input samples and labels). Entropies are measured in bits using $\log_2(P)$, where $P$ is the probability distribution of the labels assigned by the specific model. Higher \mathcal{V}\-usable information indicates harder samples, while lower values indicate easier samples for model family \mathcal{V}\. Original datasets have \mathcal{V}\-usable information concentrated in a narrow range of easier samples. Our strategy generates a broader range of \mathcal{V}\-usable information, from easy to hard, leading to better finetuning results.

• R2.5 - ChatGPT version clarification: Yes, Version number is 0613. This information will be added in the camera ready version.

• R1.1 - Generalizability of the framework:The basic framework proposed in this work is task and model-agnostic. As shown in appendix B.4 and B.8, the instructions for (RTE) vs (AXg) are different. While RTE is a relatively straightforward task, AXg is a complex NLU task. Despite the differences in the prompts of these tasks, the underlying TarGEN framework delineating the steps remains common. This proposed framework can be adapted to any task, based on the problem being evaluated. Given the parameters of a task - the label schema, and the problem statement, this framework is easy to adapt for multiple tasks of varying complexities. We would like to highlight that the simple nature of our framework allows for greater flexibility and control, making it well-suited to domain-specific or particularly complex tasks. We are happy to generate any other tasks that the reviewer deems necessary for the verification of this approach.
• R1.2 - Comparison with other methods: We compare our framework with other popular data generation approaches in the table below. We clarify ambiguous column headers:
  • Seedless: Whether the approach requires labeled task samples.
  • Focused diversity: Whether the approach actively injects diversity, incentivizing samples generated across various contexts.
  • Noise mitigation strategy: Whether the approaches actively mitigate noise, either algorithmically or through the inclusion of a module like our Self-Correction as part of their pipeline

• R1.3 - Effect of Instruction: Determining the dynamics of "varying the instructions" and its corresponding impact on downstream tasks are too broad to quantify precisely. We respectfully request the reviewer to elaborate more on this point, following which we would be happy to respond.