Many thanks for the constructive reviews!

---

Q1: The criticism (optimization instability) of SPIN seems only empirical. Do you believe the performance degradation in Figure 2 for SPIN is due to optimization instability or generalization issues?

A1: We note that there are several studies also pointing out the training instability of SPIN [1, 2], which appears to be a widely-recognized issue in the literature. In our work, we have provided an in-depth discussion showing that gap-based methods (including SPIN) are susceptible to performance collapse when extended to self-play fine-tuning (Lines 183–195). This phenomenon is also empirically validated through extensive experiments (Table 1). The extreme cases presented in our analysis are intended to facilitate understanding. In practice, during the iterations, the loss of SPIN may become very small (even a constant), but the learned distribution may not align with the target distribution; instead, it could be any distribution, leading to performance collapse. We will provide a more detailed clarification in the revised version.

---

Q2: Lack of related work discussion, such as GAN and GAIL.

A2: Thanks for the helpful suggestion. GAN [3] is proposed to learn generative models via an adversarial process, and GAIL [4] extends this adversarial paradigm to imitation learning. Overall, both [3, 4] and SPACE adopt two-player adversarial frameworks. However, SPACE is specifically designed for self fine-tuning of LLMs, where the two players are instances of the same LLM with different parameters. We will include a detailed discussion of this connection in the revised version.

---

Q3: Results on Mistral and Zephyr may not generalize to modern ones.

A3: Thanks for the valuable suggestion. Here, we conduct additional empirical studies on Qwen2.5-7B-base model. The results are presented in the table below, where base model refers to the average performance of Qwen2.5 over multiple tasks (the same task set as shown in Table 5; only average scores are shown due to character limitations). From the results, we observe that (i) SPIN continues to suffer from unstable iterative optimization, while SPACE demonstrates consistent performance improvements; and (ii) the performance gains achieved by SPACE on Qwen2.5 are less pronounced compared to those on Mistral and Zephyr. This may be because Qwen2.5 has already been well optimized during pretraining, leaving less room for further improvement when only 50K annotated examples are used.

---

Q4: Empirical comparisons with SFT are not rigorous.

A4: Thanks for the thoughtful consideration. Figure 5 is intended to highlight that SPACE, using a small amount of annotated data, outperforms SFT trained on a significantly larger number of annotated examples. In Figure 5, we present the performance of SPACE trained over two iterations. For example, the reported average performance 51.48 is measure on the model trained with a total of 150K samples, including 50K annotated data and 100K synthetic data. To ensure a fair comparison under the same computational budget, we run SFT for multiple rounds with more annotated samples, and present the results in the following table. The results show that, even when using more annotated data, SFT still underperforms compared to SPACE (51.48 with a total of 150K samples).

---

Q5: Advantages of self-playing algorithms.

A5: We appreciate your valuable question! We summarize the advantages of self-play methods and SPACE as follows:

• Better Performance with Fewer Annotations. SFT typically requires a large amount of annotated data, which is often impractical in real-world scenarios, and thus motivates the development of self-play fine-tuning methods. In this paper, we theoretically prove that with limited annotated samples, SPACE is able to capture the target data distribution (Theorem 2). Moreover, empirical results demonstrate that SPACE achieves superior performance with a small amount of annotated data, compared to SFT trained on significantly larger datasets (Figure 4).
• Stable Performance Improvement. Compared to the existing self-play method SPIN, the primary advantage of SPACE lies in its stable iterative optimization. To support this, we first provide a theoretical analysis demonstrating that SPACE is able to maintain the optimal distribution (Theorem 3). We then conduct extensive empirical studies to validate the stable convergence of SPACE over iterations (Table 1).

Additionally, we would like to emphasize that while SFT, SPIN, and SPACE can theoretically capture the underlying data distribution, they differ in their convergence rates (i.e., the amount of annotated data required) and stability during optimization. We will include a detailed discussion in the revised version.

---

Q6: Test more challenging tasks like MATH and other datasets.

A6: Thanks for the helpful suggestion! We have conducted additional experiments on MATH [5], openbookqa [6], and MT-Bench [7], which evaluate mathematical reasoning, common knowledge, and conversational ability in open-ended scenarios, respectively. We present the results in the following table, where columns 3 to 7 correspond to iterations 0 to 4 of SPACE. From the table, we observe that SPACE can progressively improve performance while maintaining stability.

---

Q7: Have the authors observed the overfitting of self-play methods?

A7: Thanks for the insightful question. We acknowledge that this concern is well-founded, since self-play methods are designed to handle the scenarios with scarce annotated data, where fine-tuning LLMs while maintaining diversity over multiple benchmarks is inherently challenging. In our experiments, we observe that SPACE indeed demonstrates relatively marginal performance improvements on certain benchmarks, such as Winogrande. However, it is important to note that this is a common issue with self-play methods, not just specific to our SPACE. Nevertheless, we would like to emphasize that SPACE is proposed to address the instability issue of SPIN, and we consider the investigation of diversity for self-play methods a meaningful direction and leave it as the future work.

---

Reference.

[1] Investigating regularization of self-play language models. 2024.

[2] Dynamic noise preference optimization for LLM self-Improvement via synthetic data. 2025.

[3] Generative adversarial nets. 2014.

[4] Generative adversarial imitation learning. 2016.

[5] Measuring mathematical problem solving with the math dataset. 2021.

[6] Can a suit of armor conduct electricity? a new dataset for open book question answering. 2018.

[7] Judging llm-as-a-judge with mt-bench and chatbot arena. 2023.

---

We will carefully revise the paper according to your suggestions. If our responses have properly addressed your concerns, we would appreciate it so much if you could re-evaluate our work. We are also happy to provide further clarifications during the reviewer-author discussions if needed.

Many thanks for the constructive reviews!

---

Q1: Is the sync data a noise?

A1: Thanks for the insightful question. In fact, the sync data generated by the previous iteration's LLM is not conventional uninformative noise in the traditional NCE. Instead, we leverage its progressively improving nature as a self-generated supervision to guide the iterative optimization. The sync data $y'\sim p_{\theta_{t-1}}$ can be viewed as hard negatives for $p_{\theta_{t}}$, playing a critical role in the two-player framework [1, 2]. We note that, [3] demonstrates the quality of self-generated samples can potentially exceed that of annotated samples in self-play fine-tuning methods. This is also possible in SPACE. However, it is important to notice that when the sync data $y'$ aligns well with $p_{\text{data}}$, i.e., $p_{\text{data}}(y'|x) = p_{\theta_t}(y'|x)$, our SPACE can provably maintain the optimality on $y'$, as shown in Theorem 3.

---

Q2: Limited datasets evaluated.

A2: To ensure the consistency with SPIN, we follow their experimental setup and conduct our experiments on the Ultrachat200k dataset. In addition, we have already validated the effectiveness of SPACE on many standard benchmarks, including ARC, GSM8K, and MMLU. Moreover, we have conducted additional experiments on MATH [1], openbookqa [2], and MT-Bench [3], which evaluate mathematical reasoning, common knowledge, and conversational ability in open-ended scenarios, respectively. We present the results in the following table, where columns 3 to 7 correspond to iterations 0 to 4 of SPACE. From the table, we observe that SPACE can progressively improve performance while maintaining stability.

---

Q3: Figure 5 may be an unfair comparison on SFT and SPACE.

A3: Thanks for the insightful comment. Figure 5 is intended to highlight that SPACE, using a small amount of annotated data, outperforms SFT trained on a significantly larger number of annotated examples. In Figure 5, we present the performance of SPACE trained over two iterations. For example, the reported average performance 51.48 is measure on the model trained with a total of 150K samples, including 50K annotated data and 100K synthetic data. To ensure a fair comparison under the same computational budget, we run SFT for multiple rounds with annotated samples, and present the results in the following table. The results show that, even when using more annotated data, SFT still underperforms compared to SPACE (51.48 with a total of 150K samples).

---

Q4: Missing recent methods

A4: Thanks for the helpful reminder. DNPO and SPACE share a similar motivation, both of which aim to address the model update stagnation issue in SPIN, but they adopt different strategies. DNPO proposes to selectively filter generated samples for continual updates, whereas SPACE introduces a novel optimization objective that ensures provably stable convergence (shown in Theorem 3). We will include a more detailed discussion in the revised version.

---

Q5: What happens if synthetic data quality degrades?

A5: Thanks for the insightful question. The experiments in Figure 4 demonstrate that using synthetic samples generated by a weaker model leads to limited performance improvement, even with more training epochs. For this reason, SPACE employs an iterative strategy that updates synthetic samples using the most recent model for progressive evolution. We will include a more detailed discussion in the revised version.

---

Q6: Could the iterative process gradually degrade capabilities in non-targeted areas?

A6: Thanks for the valuable question. We acknowledge that this concern is well-founded, since self-play methods are designed to handle the scenarios with scarce annotated data, where fine-tuning LLMs while maintaining diversity over multiple benchmarks is inherently challenging. In our experiments, we observe that SPACE indeed demonstrates relatively marginal performance improvements on certain benchmarks, such as Winogrande. Notably, this is a common issue for self-play methods, not just specific to our SPACE. One possible explanation is that self-play methods gradually reinforce their attention to annotated data during iterations, which thereby reduces the diversity of outputs. Nevertheless, we would like to emphasize that SPACE is proposed to address the instability issue of SPIN, and we consider the investigation of diversity for self-play methods a meaningful direction and will explore it in the future.

---

Referece.

[1] Generative adversarial nets. 2014.

[2] Mastering the game of go without human knowledge. 2017.

[3] Dynamic Noise Preference Optimization for LLM Self-Improvement via Synthetic Data. 2025.

[4] Measuring mathematical problem solving with the math dataset. 2021.

[5] Can a suit of armor conduct electricity? a new dataset for open book question answering. 2018.

[6] Judging llm-as-a-judge with mt-bench and chatbot arena. 2023.

---

We hope that our responses can address your concerns, and we are also happy to provide further clarifications during the reviewer-author discussions if needed.

Many thanks for the constructive reviews!

---

Q1: Connection to off-policy RL.

A1: Thanks for the valuable suggestion! In the standard policy gradient formulation (e.g., REINFORCE [1]), the gradient is formulated as $\nabla_\theta J(\theta) = \mathbb{E}\_{u\sim p_\theta}[R_u \cdot \nabla_\theta \ln p_{\theta}(u|x)]$, where $R_u$ denotes the reward for a response $u$. Intuitively, (9) shares a similar structure to this formulation, if we set $R_u=\sigma_{\mu^{-1}}(-r(x,u))(p_{data}(u|x) - p_{\theta_t}(u|x))$. However, we would like to emphasize that SPACE is based on the self-play fine-tuning framework, where the goal is to iteratively improve model performance through two-player competition. This fundamentally differs from the exploration-exploitation trade-off in RL. Additionally, SPACE optimizes (8) in a supervised-learning manner, without involving explicit rewarding or rollout phrases, which are essential components of standard RL methods. We will include a more detailed discussion in the revised version.

---

Q2: Why does SPACE do so much better than SPIN on GSM8K and IFEval?

A2: Thanks for the insightful question. GSM8K and IFEval evaluates the abilities of mathematical problem solving and instruction following, respectively. Both benchmarks require strong logical reasoning and accurate responses. Based on the experimental results, we consider that for self-play fine-tuning methods, stable evolution during iterations is the key to ensuring rigorous reasoning and precise responses. Moreover, we present response comparisons on GSM8K below. From the responses, we observe that Mistral exhibits strong fluency but lacks understanding of the problem. SPIN demonstrates partial understanding but often fails to solve the problem correctly, likely due to unstable optimization. In contrast, SPACE, with its stable iterative evolution, is able to effectively solve practical problems.

Prompt: Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?

Mistral: Guidance\n\nIn the last concept, we learned how to solve proportions involving rates. In this concept, we will be solving proportions as well. The difference is that rather than solving for a rate which compares two rates, we will be solving for an unknown quantity in a proportion. The formula to find any unknown amount is to cross multiply.\n\n#### Example A\n\nIf 24 is to 88 as 5 is to 12, how much is the unknown value 5?\n\nSolution: To find the value of an unknown amount in a proportion, we cross multiply the amount given in the proportion and the amount we are solving for... \n\nYou must check all answers, so check the answer of 12. ... The answer of 12 checks.\n\n#### Example B\n\n ... #### Concept Problem Revisited\n\nWeng earns 12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?\n\nThis is a proportional situation.\n

SPIN: If the word problem asks you to find the total number of dollars earned (the solution), then first, you need to translate the words into a math problem. When you read the problem, ask yourself, “What is the unknown?” In this case, it is the number of dollars earned.\n\nThe money earning rate is given.\n\nWeng earns 12 an hour for babysitting.\n\nThe time taken is known.\n\nYesterday, she just did 50 minutes of babysitting.\n\nWhat is the multiplication formula for calculating the total sum from rate and time?\n\nrate × time = total\n\n12 × \underline{\underline{\phantom{\rule{1em}{0ex}}}} = total\n\nWrite the question word to fill the missing part of the equation.\n\nMultiply the rate and time.\n\nHere, the unknown is total, which is also unknown to find.\n\nYou can find the answer to the problem by using the multiplication formula and the information given in the word problem.\n\nNote: 1 hour = 60 minutes\n\nRate × time = total\n\n12\⨯50=1,200\n\nShe earned 1,200.

SPACE: ## The math Rate = (\12) per hour Number of hours \= (50 \\div 60) \= (0.833333) h Earned \= Rate x Number of hours \= (\\12 \times 0.833333) = (\10) Therefore, Weng earned (\10) for babysitting.

---

Q3: The method only looks at one model, so it's not clear if it works for other models.

A3: Thanks for the helpful suggestion. In fact, we have included the experiments on Zephyr in the Supplementary Material, and the results also demonstrate the superiority of SPACE over SPIN. Additionally, we also conduct empirical studies on Qwen2.5-7B-base model. The results are presented in the table below, where base model refers to the average performance of Qwen2.5 over multiple tasks (the same task set as shown in Table 5; only average scores are shown due to character limitations). From the results, we observe that (i) SPIN continues to suffer from unstable iterative optimization, while SPACE demonstrates consistent performance improvements; and (ii) the performance gains achieved by SPACE on Qwen2.5 are less pronounced compared to those on Mistral and Zephyr. This may be because Qwen2.5 has already been well optimized during pretraining, leaving less room for further improvement when only 50K annotated examples are used.

---

Q4: There's no proof for the theorems.

A4: Thanks for the kind reminder. The proofs of all theorems are included in the Supplementary Material, and we will highlight this in the revised version.

---

Reference.

[1] Simple statistical gradient following algorithms for connectionist reinforcement learning. 1992.

---

We hope that our responses can address your concerns, and we are also happy to provide further clarifications during the reviewer-author discussions if needed.

Many thanks for the constructive reviews!

---

Q1: Assumes Frozen Baseline Quality.

A1: Thanks for the insightful comment. The quality of the reference model is indeed a crucial factor in self-play fine-tuning for LLM, and SPACE is designed with this in mind. In particular, we consider two scenarios: (i) where the reference model is weak, and (ii) where it is strong. For the first case, Figure 4 shows that using synthetic samples generated by a weaker model leads to suboptimal performance, even with more training epochs. For this reason, SPACE employs an iterative strategy that updates synthetic samples using the most recent model for progressive evolution. In the second case, a strong reference model may lead to unstable optimization (as observed in SPIN), and our SPACE is specifically designed to deal with this issue. Moreover, Theorem 3 shows that SPACE can preserve optimality even with strong reference model. We will include a detailed discussion in the revised version.

---

Q2: Diversity and exploration of SPACE.

A2: We appreciate the thoughtful question. We acknowledge that this concern is well-founded, since self-play methods are designed to handle the scenarios with scarce annotated data, where fine-tuning LLMs while maintaining diversity over multiple benchmarks is inherently challenging. In our experiments, we observe that SPACE indeed demonstrates relatively marginal performance improvements on certain benchmarks, such as Winogrande. Notably, this is a common issue for self-play methods, not just specific to our SPACE. One possible explanation is that self-play methods gradually reinforce their attention to annotated data during iterations, which thereby reduces the diversity of outputs. Nevertheless, we would like to emphasize that SPACE is proposed to address the instability issue of SPIN, and we consider the investigation of diversity for self-play methods a meaningful direction and will explore it in the future.

---

Q3: The behavior of SPACE in open-ended settings like dialogue or summarization is unexplored.

A3: Thanks for the helpful suggestion! We have conducted additional experiments on MATH [1], openbookqa [2], and MT-Bench [3], which evaluate mathematical reasoning, common knowledge, and conversational ability in open-ended scenarios, respectively. We present the results in the following table, where columns 3 to 7 correspond to iterations 0 to 4 of SPACE. From the table, we observe that SPACE can progressively improve performance while maintaining stability.

---

Reference.

[1] Measuring mathematical problem solving with the math dataset. 2021.

[2] Can a suit of armor conduct electricity? a new dataset for open book question answering. 2018.

[3] Judging llm-as-a-judge with mt-bench and chatbot arena. 2023.

---

We hope that our responses can address your concerns, and we are also happy to provide further clarifications during the reviewer-author discussions if needed.