SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models

Dear Reviewer AshF,

We are deeply thankful for your positive feedback and for acknowledging the novelty and significance of our contributions. Your recognition of our method's scalability, the motivation behind leveraging test-time scaling for training, the promising experimental results, and the clarity of our writing are genuinely encouraging. We are especially honored by your clear and strong support!

1. On Higher Costs

Weakness 1: Inference-time scaling is well-motivated for model improvement but will introduce higher costs.

Thank you for your insightful comment! As the ultimate goal is to enhance the model's performance in instruction following, inference-time scaling is actually more efficient than traditional pretraining solutions or creating high-quality data with human annotators.

We have calculated the average number of expanded tree nodes in our framework and included these details in Appendix C. Specifically, with the LLaMA3-8B model, the average number of expanded tree nodes is around 3.8, which is within an acceptable range of added computation.

Moreover, to effectively understand the costs and gains, we have conducted an additional series of experiments controlling the maximum number of response generations (inference times) during the training data construction process. We perform these experiments on two models, LLaMA3-8B and Mistral-7B, and report the average performance on IFEval. Here are the results:

The results indicate that SPaR surpasses the Self-rewarding method when additional computational resources are allocated for inference. This highlights the advantage of our method in scaling inference-time costs to achieve superior performance.

2. Clarification of Writing Issue

Weakness 2: Typo in line 239

Thank you for your careful and detailed reading! We have corrected this issue and thoroughly reviewed the entire paper to address any other errors.

Dear Reviewer 7aGY,

Thanks for your comprehensive and detailed suggestions for our work! We really value your comments on the training overheads and clarification of experimental details. We hope our responses below could address your concerns:

1. On Training Overheads

Weakness 1: SPaR introduces additianal training overheads.

Thank you for your insightful comment! As the ultimate goal is to enhance the model's performance in instruction following, inference-time scaling is actually more efficient than traditional pretraining solutions or creating high-quality data with human annotators.

We have calculated the average number of expanded tree nodes in our framework and included these details in Appendix C. Specifically, with the LLaMA3-8B model, the average number of expanded tree nodes is around 3.8, which is within an acceptable range of added computation.

Moreover, to effectively understand the costs and gains, we have conducted an additional series of experiments controlling the maximum number of response generations (inference times) during the training data construction process. We perform these experiments on two models, LLaMA3-8B and Mistral-7B, and report the average performance on IFEval. Here are the results:

The results indicate that SPaR surpasses the Self-rewarding method when additional computational resources are allocated for inference. This highlights the advantage of our method in scaling inference-time costs to achieve superior performance.

2. Clarification of Experimental Details

Weakness 2: Some experimental details are not shown in paper.

Thank you for your valuable suggestions! We have included the average number of search nodes in our experiments in Appendix C. For LLaMA3-8B, the number of average expanded nodes is 4.3, 3.7, and 3.4 across different iterations, demonstrating a decreasing trend as the model becomes better. The performance of LLaMA3-70B at each iteration, previously omitted due to space constraints, has now been added to Appendix D.1 Table 9.

Question 1: line 527, GPT-4-Turbo or GPT-4o-mini?

The model is GPT-4-Turbo. SPaR-trained LLaMA3-8B-Instruct outperforms the GPT-4-Turbo on the IFEval benchmark. The performance of GPT-4-Turbo is derived from the original benchmark [1].

[1] Zhou, Jeffrey, et al. "Instruction-following evaluation for large language models." arXiv preprint arXiv:2311.07911 (2023).

Question2: Can you compare the training cost of SPAR, Self-Rewarding and Meta-Rewarding?

We would like to clarify that the training costs for all three methods are nearly identical, as we have controlled the number of training samples to ensure fairness.

The inference times required for data construction vary among these methods. For instance, in the case of LLaMA3-8B, the average number of responses generated by Self-Rewarding and Meta-Rewarding methods is 5, whereas for SPaR, it is approximately 8.8. This increase is within an acceptable range. As mentioned in our response to Weakness 1, we have conducted experiments to compare the costs and gains.

3. On Iterative Improvement

Question3: Does more iteration bring higher performance?

Thank you for your insightful comment! Additional iterations can generally improve the model's performance but with diminishing returns. For instance, in our experiments with LLaMA3-8B, extending to a fourth iteration showed smaller improvements of an average of 0.3. This can be caused by model capacity limitations or challenges in iterative DPO training.

Furthermore, most self-training methods, such as Self-rewarding and SELF, typically use three iterations. We follow this practice to make it easier to compare our results with these methods.

Dear Reviewer 2gma,

Thanks for your comprehensive and detailed suggestions for our work! We really value your comment on experiments with additional models, compute resources and baseline details. We hope our detailed response could address your concerns:

1. Experiments with Additional Models

Weakness 1: Limited validation across models. The effectiveness of the method was validated on only three models.

Thank you for your valuable suggestion! We have added additional experiments using GLM-4-9b, a popularly used open-source LLM in the community. The results are shown below (Cf. Table 9 for full results):

SPaR significantly improves the instruction-following capabilities of GLM-4-9B. We believe this further validates the effectiveness of our method.

2. On Computational Resources

The framework's iterative training, including tree-search refinement and multiple model roles, may require significant computational resources.

Thank you for your insightful comment! As the ultimate goal is to enhance the model's performance in instruction following, inference-time scaling is actually more efficient than traditional pretraining solutions or creating high-quality data with human annotators.

We have calculated the average number of expanded tree nodes in our framework and included these details in Appendix C. Specifically, with the LLaMA3-8B model, the average number of expanded tree nodes is around 3.8, which is within an acceptable range of added computation.

Moreover, to effectively understand the costs and gains, we have conducted an additional series of experiments controlling the maximum number of response generations (inference times) during the training data construction process. We perform these experiments on two models, LLaMA3-8B and Mistral-7B, and report the average performance on IFEval. Here are the results:

The results indicate that SPaR surpasses the Self-rewarding method when additional computational resources are allocated for inference. This highlights the advantage of our method in scaling inference-time costs to achieve superior performance.

3. Clarification of Comparative Details

Weakness 3: Lack of comparative details. The paper lacks sufficient details in its comparisons with other methods, such as how each baseline initializes the model.

Thank you for your valuable suggestion! We have made every effort to ensure that the comparisons between baselines are fair, including the model initialization. We have expanded detailed information in Appendix C.

4. On Iterative Improvement

Question 1: The paper only lists results from the first three iterations, and the data indicate that the model's performance still has room for improvement.

Thank you for your insightful comment! Additional iterations can generally improve the model's performance but with diminishing returns. For instance, in our experiments with LLaMA3-8B, extending to a fourth iteration showed smaller improvements of an average of 0.3. This can be caused by model capacity limitations or challenges in iterative DPO training.

Furthermore, most self-training methods, such as Self-rewarding and SELF, typically use three iterations. We follow this practice to make it easier to compare our results with these methods.

5. On Base Model Capability

Question 2: Does the framework heavily depend on the model's initial performance? Can it be directly applied to the raw models provided officially?

Our method can be applied directly to capable open-source models like LLaMA3-70-Instruct, as shown in the experiments.

Moreover, for weaker models, like Mistral-7B-Instruct, we can use a small-sized curated dataset to bootstrap their capabilities, after which iterative training can be effectively applied.

6. Clarification of Experimental Details

Question 3: It is suggested to directly specify the strong LLM used in Section 2.2.2

Thank you for your suggestion! We have mentioned this in Appendix C Implementation Details.

Dear Reviewer pYV7,

Thank you for your thoughtful and constructive feedback and comments! We deeply appreciate your suggestions and spare no effort during this response stage to make improvements accordingly. Below are our responses:

1. On Model Capability for Self-improvement

Weakness 1: The applicability of the method may be limited. It might be suitable primarily for further improvement of models that already possess strong instruction-following capabilities.

Thank you for your insightful comment! Self-improvement methods are typically designed to enhance models that already possess reasonable capabilities. Competitive baselines, including Meta-rewarding and AutoIF, rely on strong base models. Meta-rewarding uses LLaMA3-8B-Instruct, while AutoIF uses LLaMA3-70B-Instruct.

However, our method also demonstrates its applicability to less powerful models, such as Mistral-7B-Instruct, as shown in Table 9. For these models, we provide a dataset to bootstrap their capabilities before our iterative training. To ensure fair comparisons, we also used our constructed dataset to initialize the baseline methods.

2. New Bseline

Weakness 2: Missing comparison with a key baseline: Self-Alignment with Instruction Backtranslation.

Thank you for your valuable suggestion! Initially, the lack of this method's official implementation and dataset deterred us from including it. We have since reproduced this method based on the original paper and available unofficial resources (https://github.com/Spico197/Humback). Here are the results:

Our method has demonstrated stronger performance compared to this baseline.

3. On Task Combination

Question 1: Why are judgment and refinement performed by the same model? What would happen if they were separated, or combined with the actor model, using a single model for all tasks?

We have conducted additional experiments using LLaMA3-8B to explore the impact of separating judgment and refinement tasks.

The results showed that the performance of combining these tasks sightly outperforms that of separating them. Additionally, since judgment and refinement naturally constitute a two-turn task, it is efficient to handle them using a single model.

Our preliminary experiments indicate that combining all tasks into a single model slightly reduces performance on instruction-following benchmarks (also observed in Figure 3, as mentioned in line 377), likely due to the judgment and refinement tasks differing notably from the instruction-following task.

4. Clarification of Baseline Details

Question 2: I haven't closely checked the details of the baselines. Do they use a strong LLM, or do they rely solely on the model being evolved?

All baselines used in our experiments require bootstrapping or seed datasets, which can be generated either through a strong LLM or curated by human experts. We have maintained uniform settings across all baselines and our method to ensure a fair comparison.