Forewarned is Forearmed: Harnessing LLMs for Data Synthesis via Failure-induced Exploration

Thank you for your careful review as well as appreciation of our work.

According to your suggestions, we add:

• An explanation of performance in mathematical reasoning domains in Section 4.4 and Appendix E.1.
• A clarification of the role of iterative learning in the ReverseGen framework in Figure 2.

---

We address the concerns below:

W1: Limited improvements in mathematical reasoning. Nor are many other similar iterative methods which have some sort of synthetic question-answer prioritization.

To investigate the role of challenging samples in target model training, we maintain a consistent setting across the three tasks, i.e., employing the same simple supervised fine-tuning to train the target model on the failure-inducing synthesis data for all tasks.

For math task, by only generating 7.9K samples with 1 iteration, the failure-inducing framework can improve Meta-Llama-3-8B-Instruct from 79.89 (8-shot) to 80.93 (0-shot), compared to WizardMath 80.05 (0-shot). This improvement is non-negligible to Meta-Llama-3-8B-Instruct, which is a strong model in mathematics by pretraining over 10M data with upsample math domain data [1].

We found that the learning efficiency on hard data varies depending on the task: Simple SFT shows notable enhancements for models in non-deterministic tasks like safety defense or honesty expression. However, in the case of complex deterministic tasks involving multi-step logical reasoning, such as math reasoning, simple SFT struggles to fully leverage the benefits of challenging data without substantial scaling.

If we apply a difficulty-aware training method [2], the advantages of ReverseGen would become more apparent:

As suggested by the reviewer, we add an iterative baseline LLM2LLM for comparison, which solely uses hard seed examples as in-context example for generating similar data points. WizardMath generates samples along the knowledge-difficulty level without considering target model. In comparison, ReverseGen balances exploitation and exploration by utilizing failure feedback as a reward to guide the exploration of a learnable generator. We have incorporated these analyses into Section 4.4 and Appendix F.1.

Thank you for your detailed review and for acknowledging our method and comprehensive experiments. We have revised our manuscript to incorporate your valuable suggestions, including:

• An explanation of performance in mathematical reasoning domains in Section 4.4 and Appendix E.1.
• A clarification of the role of iterative learning in the ReverseGen framework in Figure 2.
• An analysis of the importance of failure-inducing exploration in Appendix A.

W1&W2&Q1: The paper lacks clear justification and motivation for why generating failure-guided data would improve model performance. No theoretical framework explaining why failure cases would be more valuable than standard training data

For the motivation of failure-inducing exploration, we would like to clarify three sources of evidence from our paper and previous work:

• Evidence from the target model side:

  • We have observed that for various tasks such as reasoning and safety tasks, the LLM correctly solves simpler or learned examples in the pre-training dataset but may struggle with harder or not yet learned examples. It will be sub-optimal to keep synthesizing data points for which the LLM is already achieving high accuracy on.

  • Furthermore, different models may excel in various areas or distributions, exhibiting distinct strengths and weaknesses across question types. For instance, we found that LLaMA2-7b-chat often struggles with questions related to privacy, while Vicuna-7b-v1.5 is susceptible to attacks involving harmful tool usage. This variability underscores the importance of generating failure-guided data to address the specific shortcomings of each target model.

• Evidence from the synthesis model side: Current synthesized datasets exhibit a severe bias toward easy queries, resulting in low coverage for harder ones [1]. These biases hinder the learning of target models, as challenging examples are often more critical for effective training. This inconsistency results in inferior performance compared to training on human data distributions.

• Evidence from previous research: Recent studies have shown that augmenting solutions for difficult samples [1] or providing additional feedback for failure cases [2] can improve model alignment. Our work transforms the task of addressing weaknesses specific to certain models and tasks into a trainable framework.

In response to the reviewer's concerns, we have elaborated the motivation of failure-inducing exploration in Appendix A.

W3&Q3: Table 6 shows similar result w and wo failure induction in math reasoning task, does this mean failure induction does not benefit math reasoning tasks? Could you analyze why failure induction appears less effective for mathematical reasoning (Table 6) compared to other tasks?

To investigate the role of challenging samples in target model training, we maintain a consistent setting across the three tasks, i.e., employing the same simple supervised fine-tuning to train the target model on the failure-inducing synthesis data for all tasks.

For math task, by only generating 7.9K samples with 1 iteration, the failure-inducing framework can improve Meta-Llama-3-8B-Instruct from 79.89 (8-shot) to 80.93 (0-shot), compared to WizardMath 80.05 (0-shot). This improvement is non-negligible to Meta-Llama-3-8B-Instruct, which is a strong model in mathematics by pretraining over 10M data with upsample math domain data [1].

We found that the learning efficiency on hard data varies depending on the task: Simple SFT shows notable enhancements for models in non-deterministic tasks like safety defense or honesty expression. However, in the case of complex deterministic tasks involving multi-step logical reasoning, such as math reasoning, simple SFT struggles to fully leverage the benefits of challenging data without substantial scaling. Based on this analysis, we adopt a difficulty-aware training method [2], the advantages of ReverseGen would become more apparent:

These findings can further verify the effectiveness of ReverseGen in useful challenging data, which is the important basis for the following task performance enhancement.

Dear Reviewer yaDx,

We sincerely hope you can reconsider your initial assessment (3: reject, not good enough), especially with less than 23 hours remaining until the rebuttal deadline.

In response to your constructive feedback, we have addressed your questions, provided detailed experimental analyses, and revised our manuscript, including:

• An explanation of performance in mathematical reasoning domains in Section 4.4 and Appendix G.1.
• A clarification of the role of iterative learning in the ReverseGen framework in Figure 2 and Appendix E.2.
• An analysis of the importance of failure-inducing exploration in Appendix A.
• A clear experimental configuration including reward designs in Section 4.1 and Appendix C.

We are open to further discussion if needed. Thank you for your support!

Best regards,

The Authors of 8852

Thank you for your feedback and constructive comments. And thanks for your recognition of the novelty and effectiveness of our method.

We have revised our manuscript according to your suggestions, including:

• Rephrasing and reorganizing the confusing sentences in the technical section (Section 3) and move the pseudocode to Appendix B. The main revisions are highlighted in blue.
• Incorporation of a table to clearly map each of the variables in the technical section to the specific choice made for each task in Appendix C.

---

We address your concerns below.

W1.1 Line 126: “... Our work differs from these prior works in that we focus on generating failure-guided data for target models” Can you be more specific in comparison with each of these prior works? It is unclear to me how ReverseGen differs from many of these works mentioned in the related works.

Thank you for your suggestion! We have clarified and expanded these statements for better understanding in Section 2.1.

Most prior research on generating data to address the weaknesses of target models has focused on known instructions, specific tasks, or distilling knowledge from strong LLMs using prompts.

Our work differs from prior research by transforming the feedback process into a trainable framework focused on synthesizing failure-inducing data. We optimize the model by rewarding it for generating instructions that lead to failures in the target model. The incorporation of an iterative preference learning algorithm enables the model to continuously learn from newly generated instructions that challenge the target model.

We hope the current version clarifies the distinctions more clearly.

W1.2 Line 176: The term “solvable” for positive examples is quite unclear. Use a more appropriate name for this.

Thank you for raising this point. We have changed "solvable" to "ineffective", which means these instructions do not cause the target model to fail. We also changed "unsolvable" to "effective", referring to instructions that successfully challenge the target model.

Minor

Line 275: why were these hyperparameters chosen?

We use the default hyperparameters from HALOs (https://github.com/ContextualAI/HALOs), a library that implements preference learning algorithms. We modified the batch size to fit our hardware requirements.

Line 429: “ReverseGen solely relies on synthetic data” - Doesn’t it use MMLU as the initial instruction seed?

Yes, we optimize the target model solely with data generated by ReverseGen, achieving superior performance compared to the target model trained on human data.

Line 284: “Responses from the target model are generated using greedy decoding for deterministic quality measurement, with the temperature parameter set to 0” - Doesn’t greedy already mean temperature doesn’t matter?

To accelerate data generation, we use VLLM (https://docs.vllm.ai/en/stable/dev/sampling_params.html) for inference. In VLLM, temperature parameter set to zero means greedy sampling.

Line 128: Fix citation style

Thanks for your comment. We have revised the citation style.

Line 381: Emplots -> Employs

Thank you for the reminder. We have corrected the typo.

Line 927: “Safety Responses, generated by the target model itself, gpt-4o-mini, through promptings” - I cannot understand this sentence

To prepare the training data for enhancing the safety capabilities of target models, we use crafted prompts to induce them to generate safety responses for harmful instructions generated by ReverseGen (lines 1009-1010). Although we could use GPT-4o-mini to produce these responses, our prior experiments show that self-generated responses provide slightly better defense performance. We will clarify this in the revised version.

---

We really appreciate receiving these valuable and constructive comments from the reviewer HFrR.

We hope the responses above address your follow-up questions. Thank you again for your time and feedback.

[1] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. https://arxiv.org/abs/2404.02151

[2] Zhang-Wei Hong, Idan Shenfeld, Tsun-Hsuan Wang, Yung-Sung Chuang, Aldo Pareja, James Glass, Akash Srivastava, Pulkit Agrawal. Curiosity-driven Red-teaming for Large Language Models. https://arxiv.org/abs/2402.19464

[3] https://ai.meta.com/blog/meta-llama-3/

Line 429: “ReverseGen solely relies on synthetic data” - Doesn’t ReverseGen use MMLU as the initial instruction seed? The rebuttal response does not answer my question. Doesn’t ReverseGen use MMLU as the initial instruction seed and not depend solely on synthetic data?

Thank you for your question, and we apologize for any misunderstanding.

During the initialization phase, we use multiple-choice questions from the MMLU training set as the initial instruction seed, which is essential for adapting the proposer model to the MMLU distribution.

When enhancing the target model, we do not use any MMLU training data. Instead, we rely solely on samples generated by the proposer model for performance calibration, which yields better results than the baseline that uses the MMLU training set.

Q3: Why is Llama3 used in 4.4 but not in the previous experiments? I realize I should have been more explicit in phrasing this question. My main inquiry is: how were the models selected for each experiment? For instance, the "Safety Red-teaming" and "Honesty Calibration" experiments utilized older versions of the Llama models, while the "Mathematical Reasoning" experiment employed the newer Llama-3 model. Why didn’t the authors use the Llama-3 models consistently across all experiments? Additionally, how were the specific proposer models chosen for each task? My primary concern is ensuring that the technique demonstrates robustness across a variety of model choices and does not rely on cherry-picked model and proposer combinations that happen to produce favorable results.

Selection of Target Models: In safety red-teaming, we focused on Llama-2-7b-chat due to its extensive pre-training and post-training efforts related to safety, which enable it to demonstrate strong safety defense performance, even when compared to the later Llama-3-8B-Instruct [1]. To maintain consistency with the baseline Curiosity [2], we selected Vicuna-7b-v1.5 and Llama-2-7b-chat as our target models. We focused on Llama-3 within the math domain due to its significant improvements in reasoning tasks compared to the previous Llama 2 models [3].

• To address the reviewer’s concern, we perform a further evaluation of the harmful instructions generated by OpenLLaMA-3B for attacking Llama-3-8B-Instruct. Similar to ReverseGen with Llama-2-7b-chat, we collected 520 effective harmful instructions that successfully targeted Llama-3-8B-Instruct. With one iteration, we improved the ASR from 1.03% to 2.45%, and with two iterations, we enhanced the ASR to 7.34%. These results demonstrate the generalization and robustness of ReverseGen across different target models.

Selection of Proposer Models: We aim to investigate whether we can depend on tunable open-source models to generate effective instructions, rather than distilling instructions from stronger models or human. To achieve this, there are two key requirements for the proposer models:

1. they must be capable of generating task-related instructions based on few-shot prompts;
2. they should be pretrained on domain-related data to ensure the quality of the generated instructions.

With these criteria in mind, we can flexibly combine proposer models with target models. We will include these analyses in the revised version of the paper.

This comment indicated that one need not say the temperature is set to 0 when greedy decoding is used. Regardless of the generation framework used, the temperature hyperparameter is meaningless in greedy decoding. Authors should fix this line in the next revision of the paper and remove the part that says the temperature is set to 0.

We appreciate the reviewer’s suggestion and will remove this statement in the next revision to improve accuracy.

Thanks for the reply. I'm raising my score to 6.

I had one more question about the additional experiment:

we perform a further evaluation of the harmful instructions generated by OpenLLaMA-3B for attacking Llama-3-8B-Instruct

Why was OpenLLaMA-3B used in this additional experiment? I'd say the experiments will be much more convincing if the authors show an ablation with 7-8 different combinations of target and proposer model in the next revision.

Dear Reviewer HFrR,

Thank you for your valuable comment.

We chose OpenLLaMA-3B as the proposer model for attacking Llama-3-8B-Instruct due to its comparatively weaker general capabilities relative to other proposer models considered in this paper (e.g., Llama-2-7b). Due to time constraints, we investigated whether OpenLLaMA-3B in ReverseGen can identify increasingly effective instructions for the stronger Llama-3-8B-Instruct through iterations. Our goal is to gain insights into the flexibility and generalization of our framework.

We will explore additional combinations of proposer and target models as soon as possible and include those experimental results in the next revision. We greatly appreciate your understanding.

Once again, we would like to express our sincere gratitude to you for these constructive suggestions.

Best regards,

The Authors of Paper 8852

Thank you for your thoughtful review and comment. And thanks for your recognition of the writing and novelty of our method. In response to your suggestion, we have added an analysis on directly distilling from powerful mathematical models in Appendix G.2.

---

We address the concerns below:

W1: To verify the effect of the proposed RLHF approach, there should a baseline finetuning the data generator (proposer LLM) with a collection of failed samples, and generate a dataset.

We do not assume the target model will fail with the initial instruction set. Even when starting with a mixed or unbalanced instruction set to warm up the proposer model, ReverseGen can progressively generate training samples that cause target model failures.

As suggested by the reviewer, we compare two initialization settings before conducting preference optimization: (1) Mix: Training with randomly sampled instructions (a combination of easy and difficult data); (2) Difficulty: Training with randomly sampled hard instructions for the target model.

• For safety red-teaming (target: Vicuna-7b-v1.5):

  Proposer	ASR	Diversity
  OpenLLaMA-3B (Mix)	6.68	0.571
  OpenLLaMA-3B (Difficulty)	12.62	0.313
  ReveseGen (t=1)	19.76	0.596
  ReveseGen (t=2)	32.83	0.620
  ReverseGen (t=3)	56.73	0.465

  In safety red-teaming, preparing sufficient harmful instructions in advance is difficult. Therefore, we prepare the training data for the Difficulty setting by using the same in-context demonstrations as in the Mix setting to induce the proposer model to generate instructions and collect harmful instructions for warm-up.

• For mathematical reasoning:

  Proposer	GSM8K	GSMPlus
  Llama-3-8B (Mix)	77.96	58.29
  Llama-3-8B (Difficulty)	81.44	59.00
  ReverseGen	80.93	59.31

  Training begins with a high-quality set of challenging instructions, which provides a strong initial policy for further preference learning. We use this as the initialized policy for ReverseGen when sufficient training data is available (e.g., in the math domain). However, currently the main bottleneck for math optimization lies in developing an efficient difficulty-aware algorithm, which we analyze in detail in Appendix G.1.

Summary:

1. Initializing with challenging samples can help us establish a strong initial policy if we have sufficient failure cases for training.

2. However, collecting numerous failure cases is often impractical in many real-world domains. Despite this, ReverseGen remains effective even when the initial seeds are not necessarily challenging instructions.

W2: A strong LLM (i.e., gpt-4o) plays an important role in the proposed method when obtaining the oracle label, so there should another baseline directly prompting the gpt-4o multiple times to generate a synthetic dataset.

Owing to restricted API resources, we have chosen the robust open-source model Qwen2.5-Math-7B for the reviewer's reference. Qwen2.5-Math-7B achieves an 8-shot score of 91.6 on GSM8K, a score comparable to GPT-4o's performance of 92.9 on the identical task.

Mix: Prompting with randomly sampled instructions.

Difficulty: Prompting with randomly sampled hard instructions for the target model.

Relying on prompting is insufficient for effectively exploring challenging instructions without preference learning from target model feedback, resulting in subpar performance on the standard benchmark GSM8K and the adversarial benchmark GSMPlus. We will include prompting for GPT-4o-mini in later versions.

Follow the demonstrations and ask the fourth question. Do not generate any verbose information. 

Ask math questions:
Q1. A drink vendor has 80 liters of Mazza, 144 liters of Pepsi, and 368 liters of Sprite. He wants to pack them in cans, so that each can contains the same number of liters of a drink, and doesn't want to mix any two drinks in a can. What is the least number of cans required?

Q2. A sum fetched a total simple interest of Rs. 4016.25 at the rate of 5% p.a. in 5 years. What is the sum?

Q3. There is food for 760 men for 22 days. How many more men should join after two days so that the same food may last for 19 days more?

Q4. 

Dear Reviewer 29jz,

Thank you for your constructive comment and for acknowledging our method's clear presentation and novelty. We sincerely hope you can reconsider updating your assessment, especially with less than 12 hours remaining until the rebuttal deadline.

We have revised our manuscript and clarified your questions in response to your comments, including

• adding an analysis on directly distilling from powerful mathematical models in Appendix G.2.
• comparing the difference by initializing on difficulty-prioritized instructions or random task instructions.
• explaining the performance of prompting GPT-4o-mini or Qwen2.5-Math-7B for data generation.

We are open to further discussion if needed. Thank you in advance for your recognition!

Best regards,

The Authors of 8852