Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs

We sincerely appreciate the valuable feedback! We provide our responses as follows, and hope the reviewer could reassess our work. We are looking forward to addressing any further question in the reviewer-author discussion period!

Q1: It was on older model, and do not compare to preference-pair-based methods.
A1: Thanks for the helpful suggestion. We present our responses as follows.

• LLM choice. We conduct experiments on Zephyr-7B and Mistral-7B to remain consistent with the original settings in SPIN. Furthermore, we have conducted additional experiments on Qwen2.5-7B-base, and evaluated performance on ten benchmarks (in Lines 218-223). We report the average score in the following table:

  	Qwen 2.5-7B	iter0	iter1	iter2	iter3	iter4
  SPIN	53.49	53.84	53.92	53.71	53.89	53.98
  T-SPIN	53.49	53.68	54.17	55.04	54.97	55.03

  From the results, we observe that: First, both methods yield less performance enhancements on Qwen2.5-7B compared to Zephyr‑7B and Mistral‑7B. This may be attributed to: (i) Qwen2.5 is already a strong LLM, making it harder for further improvements via fine-tuning; and (ii) our experiments are conducted with limited annotated samples, where achieving performance enhancements on strong LLMs is inherently challenging. Second, SPIN still suffers from unstable optimization during iterations, which prevents further performance improvements. In contrast, T-SPIN introduces historical advantages to guide the optimization, resulting in stable performance enhancements.

• Comparison to preference-pair methods. We would like to emphasize that our study focuses on the problem of self-play fine-tuning, which is fundamentally different from pairwise preference optimization. Specifically, preference optimization typically relies on data in the format of (prompt, chosen response, rejected response), while our T-SPIN is designed for scenarios where only SFT-style data, i.e., (prompt, high-quality annotated response) is available. Our setting is more common in practice, as SFT-style data is more widely available than preference pairs. Therefore, a direct comparison between self-play fine-tuning and preference optimization may not be appropriate. Nevertheless, we sincerely appreciate the reviewer's helpful suggestion, and we will include a comparison with pairwise preference optimization methods in the revised version.

Q2: Isn't the reference still playing a role in the loss function?

A2: Thanks for the valuable question. We feel that there are some misunderstandings regarding the term 'reference' . In our work, reference specifically refers to the reference policy. SPIN is considered reference-dependent because its objective (3) explicitly relies on a reference policy $\pi_{\theta_t}$. In contrast, T-SPIN is reference-free, as its loss function (7) depends solely on the current policy $\pi_\theta$ and does not involve any other policies during optimization. We will include a detailed clarification in the revised version.

Q3: The reward function in T-SPIN.

A3: In SPIN, the reward function is defined as the log-ratio $c(x, y)=\lambda \log \frac{\pi\_{\theta}(y|x)}{\pi_{\theta_t}(y|x)}$ between the current model $\pi_\theta$ and a reference model $\pi_{\theta_t}$. According to (3), SPIN aims to maximize the reward for annotated samples while minimizing that for synthetic ones. However, in SPIN, the reward function used for training is not aligned with the metric $\log p_{\theta}(y|x)$ used by LLMs for generation. In other words, a response with a high reward does not necessarily correspond to a high generation probability, which has been verified in the contingency table of Figure 3(a) . In contrast, T-SPIN adopts the reward function $c(x, y)=\alpha \log \pi_{\theta}(y|x)$ for training, matching that used for generation. As a result, samples with higher rewards are also more likely to be generated, as shown in the contingency table of Figure 3(b) . We sincerely thank the reviewer for raising the question and will include a clear explanation in the revised version.

Q4: An experiment comparing the variance of the training process.

A4: Thank you for the constructive suggestion. In our paper, stability refers to the consistent improvement of model performance over iterations, without suffering from performance collapse. In contrast, the reviewer seems to refer to the stability of the training process itself, which differs from our intended focus. We will offer a detailed clarification in the revised version. Moreover, additional experiments about training process variance will also be included.

We sincerely appreciate the valuable feedback! We provide our responses below, and are looking forward to addressing any further question in the reviewer-author discussion period!

Q1: The proto-synthetic responses $y_0$ may become stale over successive iterations.

A1: Thank you for the insightful comment. We understand your concern that proto-synthetic response $y_0$ may become stale over iterations. As shown in (7), $y_0$ is incorporated to compute the historical advantages, which can be viewed as a regularization to prevent performance collapse when current advantages diminish during iterations. In fact, its static property is a key factor in providing stable optimization. Detailed discussions can also be found in Lines 136-144.

Q2: The isolated effect of changing the reward formulation.

A2: Thank your for the constructive suggestion. We currently employ the reward function $c(x, y)=\lambda \log \frac{\pi_{\theta}(y|x)}{\pi_{\theta_t}(y|x)}$ , and substitute it into (4) to obtain a new loss function, as shown below:

Then, we conduct an ablation study on Zephyr-7B, comparing the performance of models optimized with (7) (i.e., T-SPIN) to those using the above loss function (namely, T-SPIN_ref). The average score over ten tasks is presented in the following table. From the results, we observe that T-SPIN demonstrates superior performance compared to T-SPIN_ref. This can be attributed to the fact that using $c(x, y)=\alpha \log \pi_{\theta}(y|x)$ as the reward function preserves the alignment between training and generation, which in turn facilitates performance improvement.

Q3: A more thorough discussion of the historical advantage.

A3: Thank you for the helpful suggestion. To clarify the underlying intuition, we start by examining the fundamental issue that causes the performance collapse observed in SPIN during iterative optimization.

• Vanishing current advantages. The core idea of SPIN is to progressively improve performance by minimizing the relative advantage of the current policy $\pi_{data}$ over the target policy $\pi_{\theta_t}$ during iterations. Specifically, as shown in (3), SPIN aims to reduce the confidence gap between high-quality annotated samples and self-generated responses, i.e., minimizing $\ell(c(x,y)-c(x,y'))$ where $y \sim \pi_{data}$ , $y'\sim \pi_{\theta_t}$ and $c(x,y)=\lambda \log \frac{\pi_\theta(y|x)}{\pi_{\theta_t}(y|x)}$. As the model continues to evolve, the gap between $\pi_{data}$ and $\pi_{\theta_t}$ gradually narrows. Once this gap vanishes, the objective in (3) degenerates into a constant independent on $\pi_\theta$, making any policy a trivial solution, leading to unstable optimization and performance collapse ultimately.

To stabilize the iterative optimization, it is thus necessary to introduce a regularization term that prevents this degeneration. A natural choice is to incorporate the historical advantage​, i.e., the relative advantage of the current policy $\pi_{\theta_t}$ over the initial policy $\pi_{\theta_0}$, which remains meaningful even when the current advantage vanishes, thereby mitigating objective degeneration and maintaining training stability. We will include a more in-depth discussion of the underlying intuition and mechanisms of T-SPIN in the revised version.

Q4: Have the authors considered using responses from an intermediate past policy.

A4: We sincerely appreciate this insightful suggestion! To evaluate its effectiveness, we conduct additional experiments on Zephyr-7B comparing our T-SPIN with the variant (namely T-SPIN_mid) that employs proto-synthetic responses $y_{\lfloor t/2 \rfloor}$ from the intermediate past policy $\pi_{\lfloor t/2 \rfloor}$ to optimize (7). Notably, at iterations 0 and 1, both T-SPIN and T-SPIN_mid use proto-synthetic responses generated from the same initial policy $\pi_0$, so the comparison starts from iteration 2. We present the experimental results as follows:

From the table, we observe that T-SPIN_mid suffers from the performance decline across iterations. We hypothesize that this degradation may stem from the vanishing of the historical advantage at iteration 2 (since both $y'$ and $y_0$ are sampled from $\pi_{\theta_1}$), which may potentially destabilize the optimization in subsequent iterations. Due to limited computational resources, we are unable to further validate this idea over other models during the rebuttal period. This is indeed a very interesting direction, and we plan to explore it in future work. Once again, we sincerely thank the reviewer for this valuable idea.

We sincerely appreciate the valuable feedback! We provide our responses below, and are looking forward to addressing any further question in the reviewer-author discussion period!

Q1: This paper lacks deeper theoretical guarantees about convergence. The stability claims are primarily based on empirical studies rather than theoretically proven.

A1: Thank you for the valuable comment. We agree that establishing theorems about convergence and stability is important for T-SPIN. However, due to the non-convex property of LLMs and the arbitrariness of the monotonically decreasing function $\ell(\cdot)$ in (7), it is highly challenging to provide rigorous convergence and stability guarantees for T-SPIN. As an alternative, we provide theoretical analysis for gradients in Theorem 1 with corresponding discussions in Lines 196–200. We recognize the importance of convergence guarantee and plan to investigate it in future work.

Q2: Lack of discussion on memory and computational requirements.

A2: Thank you for this practical concern. The additional cost of T-SPIN compared to SPIN comes from the proto-synthetic response $y_0$ generated by the initial policy $\pi_0$, including

• One-time computational cost: At the beginning of the training, we conduct a single forward pass to generate $y_0$ with the initial policy $\pi_0$ (see Algorithm 1). The computational cost is equivalent to generating synthetic response $y'$ at one iteration, but is not repeated during the training process. In practice, the generation of proto-synthetic responses only takes 0.8 hours, which is significantly lower than the 3.5 hours for training.
• Increased memory cost: In T-SPIN, the training data for each iteration expands from pairs ($y, y'$) to triplets ($y, y', y_0$). We argue that this is a modest and acceptable trade-off for the significant improvements in final performance and optimization stability, as shown in Table 1 and Figure 2. Moreover, compared to SPIN, although T-SPIN utilizes additional samples for training, it eliminates the need of reference model. As a result, the additional memory cost in experiments is minimal.

We will include the discussion about computational and memory costs in the final version.

Q3: Performance of larger models.

A3: Thanks for the helpful suggestion. While our T-SPIN is model-agnostic, we are currently unable to provide results on larger LLMs due to limited computational resources and the tight rebuttal timeline. Nevertheless, we plan to include expanded results in the final version.

Q4: All experiments use Ultrachat200k. How does T-SPIN perform on datasets from different domains.

A4: Thank you for the insightful question. Following the experimental setting in SPIN, we select a subset of Ultrachat200k as the training set. In our paper, we have evaluated the performances over a wide range of domains, including Math and Logic (GSM8K, MATH, MUSR), Multi-Domain Knowledge (MMLU, MMLU-Pro, GPQA), Commonsense Reasoning (HellaSwag, Winogrande, BBH), and Instruction Following (IFEval), and present the results in Table 1. Furthermore, we also evaluate the performance of T-SPIN on Code Generation (HumanEval [1]) and Scientific Question Answering (OpenBookQA [2]). The results presented in the following table demonstrate that T-SPIN consistently yields stable performance improvements across these diverse benchmarks.

Reference

[1] Chen et al. Evaluating Large Language Models Trained on Code. Arxiv, 2021.

[2] Mihaylov et al. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. EMNLP, 2018.

We sincerely appreciate the valuable feedback, and provide our responses as follows. We hope our responses help strengthen your confidence in this work, and we are looking forward to addressing any further questions during the reviewer-author discussion.

Q1: It is unclear if this method also holds for stronger baselines.

A1: Thank your for the helpful suggestion. We have conducted additional experiments on Qwen2.5-7B-base, and evaluated performance on ten benchmarks (in Lines 218-223). We report the average score in the following table:

From the results, we have the following two observations:

• First, both methods yield less performance enhancements on Qwen2.5-7B compared to Zephyr‑7B and Mistral‑7B. This may be attributed to: (i) Qwen2.5 is already a strong LLM, making it harder for further improvements via self-play fine-tuning; and (ii) our experiments are conducted with limited annotated samples, where achieving performance enhancements on strong LLMs is inherently challenging;

• Second, SPIN still suffers from unstable optimization during iterations, which prevents further performance improvements. In contrast, T-SPIN introduces historical advantages to guide the optimization, resulting in stable performance enhancements.

Q2: The extra computational overhead of generating proto-synthetic responses is not measured.

A2: Thanks for the thorough consideration. We report the runtime, FLOPs, and memory cost for generating proto-synthetic samples below:

Additionally, we highlight that the cost of generating proto-synthetic samples $y_0$ is affordable for self-play fine-tuning because: (i) the generation for $y_0$ is performed offline and only once over the multi-iteration training process (see Algorithm 1); (ii) in each iteration, the training process takes approximately 3.5 hours, which is significantly longer than generation, and takes up most of the total computation time.

Q3: The improvement over SPIN is not significant on some datasets.

A3: We would like to emphasize that: (i) we use only a small amount of high-quality annotated data from Ultrachat200K for training, which makes it inherently difficult to achieve significant improvements across all benchmarks; (ii) we mainly focus on overall performance over multiple benchmarks, which provides a more comprehensive view of T-SPIN’s advantages than results on individual datasets. In our experiments, T-SPIN has demonstrated higher average scores than baselines (see Table 1).

Q4: Not compared with other improved methods based on SPIN.

A4: Thank you for the kind reminder. Recently, we have found several studies that propose improved variants of SPIN [1, 2], and we will include comparisons with these advances in the revised version.

Q5: How many iterations do you observe diminishing returns?

A5: In experiments, we run a total of five iterations. On Zephyr, SPIN reaches its performance peak at iteration 2 and begins to decline thereafter, as shown in Figure 2(c). On Mistral, SPIN exhibited performance fluctuations starting from iteration 1 and continuing until iteration 4, as shown in Figure 5(c).

Reference

[1] Alami et al. Investigating Regularization of Self-Play Language Models. Arxiv, 2024.

[2] Yang et al. Dynamic Noise Preference Optimization for LLM Self-Improvement via Synthetic Data. Arxiv, 2025.