RePO: Understanding Preference Learning Through ReLU-Based Optimization

To Reviewer uBRY:

We sincerely thank you for your detailed and constructive review. Your insightful questions have helped us identify key areas for clarification and further strengthening of our work. Below, we address each of your points.

Q1: As DPO is one of the core methods in alignment tuning, although the authors focus on SimPO, extension to DPO would make the contribution stronger. As the author stated that RePO++ can be integrated to DPO, it would be great if they can show it through additional small experiment.

Response: Thank you for this excellent suggestion. To demonstrate the generalizability of our proposed binary filtering mechanism, we integrated it into IPO [20], a canonical DPO-like method that directly addresses DPO's overfitting issues. The resulting loss function, which we term IPO-RePO, applies ReLU filtering to IPO's core objective:

$L_{\text{IPO-RePO}} = \mathbb{I}\left(M_{\theta, \text{ref}} < \gamma\right) \cdot \left( M_{\theta, \text{ref}} - \frac{1}{2\tau} \right)^2 \quad \text{where} \quad M_{\theta, \text{ref}} = \log \frac{\pi_{\theta}(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi_{\theta}(y_l|x)}{\pi_{\text{ref}}(y_l|x)}$

We conducted experiments on the Llama3-Instruct-v0.2 (8B) model, with results on the AlpacaEval and Arena-Hard benchmarks as follows:

These results show that incorporating RePO's binary filtering mechanism provides consistent and significant performance improvements over the original IPO framework. This further substantiates our core claim that selectively applying gradients to challenging samples (i.e., "which examples to learn from") can be more critical for effective preference optimization than the specific weighting scheme of the loss function itself.

Q2: Although over-optimization is a core topic discussed in this paper, the authors do not fully replicate the experimental setups used in prior work (e.g., Gao et al.). Including KL divergence in controlled experiments (even once) would strengthen the empirical claim.

Response: Thank you for this valuable suggestion. We agree that a direct comparison with KL divergence provides important context. We have indeed conducted such controlled experiments on the IMDB dataset, and the results fully align with the conclusions presented in the paper—specifically, that the mean implicit reward margin ($m_{\text{batch}}$) serves as a reliable and effective proxy for the KL divergence trend.

Our rationale for focusing on $m_{\text{batch}}$ in the main draft was to emphasize its utility as a computationally efficient proxy that eliminates the need to calculate KL divergence, which is a key practical advantage of our method.

Given the rebuttal's constraints on figures and anonymous links, we are unable to present the detailed plots here. However, we will add this comparative analysis, including the KL divergence plot, to the appendix in the final version to strengthen the empirical evidence as you suggested.

Q3: The evaluation of SimPO under varying $\beta$ values didn't include the high-$\beta$ regime (e.g., $10^2, 10^3$), which the authors reference in the motivation. The authors could include or discuss results in this range to better support the claim of RePO being equivalent to high-$\beta$ SimPO.

Response: We thank the reviewer for pushing for greater empirical rigor on this key claim. We have run additional experiments on Llama3-Instruct-v0.2 (8B) with high $\beta$ values, keeping $\gamma=0.3$ fixed as per our experimental setup. The results are presented below, alongside RePO's performance for direct comparison:

The trend clearly shows that as $\beta$ increases into the high regime, SimPO's performance converges towards that of RePO. This provides strong empirical support for our theoretical claim that RePO is the limiting case of SimPO as $\beta \to \infty$ (Lemma 3.1).

We add two important clarifications:

1. This trend is predicated on an appropriately chosen $\gamma$, a point we also highlight in the paper (Observation 1, Line 148).
2. The small performance gap remaining even at $\beta=1000$ may be attributed to numerical instability in the log-sigmoid function when its input becomes extremely large. This observation inadvertently highlights another advantage of RePO: its ReLU-based formulation is numerically more stable and directly achieves the desired binary filtering without being susceptible to such floating-point limitations.

Q4: Although the authors compare RePO with SLiC-HF, DPO, and SimPO, it would be useful to explore whether RePO’s binary filtering mechanism could enhance other alignment paradigms such as RLHF (KL-regularized PPO variants) pipelines.

Response: This is an excellent point and a very promising direction for future research. Exploring whether RePO's efficient binary filtering mechanism can enhance the sample efficiency and stability of online RL paradigms like PPO is a compelling next step. The paradigm of focusing on "which examples to learn from" could potentially offer significant benefits in an online setting where data is generated iteratively.

We agree this is a valuable avenue to explore. In line with this suggestion, we have incorporated this discussion into our limitations and future work section (Section 6) to guide and inspire subsequent research in this area. We thank the reviewer for this insightful suggestion.

To Reviewer M1uf:

We thank you for your critical feedback. We appreciate this opportunity to clarify our core contributions, which we believe address the perceived limitations in novelty and resolve the apparent contradiction noted with RePO++.

Q1: The primary weakness is the paper's limited novelty.

Response:

We respectfully argue that viewing our work as a "design swap" overlooks its primary conceptual contribution: we demonstrate that why a mechanism works is as important as what the mechanism is. Our primary contribution is the conceptual reframing of preference optimization, supported by new theoretical and empirical evidence. The novelty lies in the following key areas:

1. Identifying a Core Principle: Filtering is More Fundamental than Weighting. Our key insight, and the central thesis of the paper, is that effective preference learning hinges more on which challenging examples are selected for updates than on how their gradients are continuously weighted. While prior methods like DPO/SimPO entangle these two functions within the log-sigmoid activation and the $\beta$ hyperparameter, our work explicitly decouples them. RePO serves as a "computational lens" to isolate and study the filtering mechanism in its purest form, revealing its primary importance.

2. Providing Theoretical Grounding for this Principle. Our work is the first to establish a formal justification for this binary filtering approach. By demonstrating that the ReLU-based loss is the convex envelope of the ideal 0-1 loss (Theorem 4.2), we provide strong theoretical evidence for its optimality as a surrogate loss. This crucial theoretical contribution explains why this simple filtering mechanism is so effective and fundamentally distinguishes our work from prior heuristic uses of similar functions, such as the hinge loss in SLiC-HF, which lacked this formal justification.

3. Simplifying and Deepening the Understanding of Over-Optimization. By showing that a simple, theoretically-grounded filtering mechanism can match or exceed the performance of more complex methods (Table 1), we challenge the conventional wisdom that intricate components like sigmoid-based gradient constraints are essential for alignment. Our analysis of the hyperparameter $\gamma$ in creating a natural learning curriculum (Observation 4, Figure 4) and directly controlling over-optimization (Figure 6) offers a new, simplified lens through which to understand and manage the alignment process.

As stated in our introduction (Lines 65-67), our primary aim is to motivate researchers to reconsider the fundamental mechanisms behind preference optimization. We believe our work provides a significant conceptual shift and a valuable, simplified baseline to facilitate this deeper understanding.

Q2: How do the authors reconcile RePO++'s performance with the claim that sigmoid-based gradient constraints are an unnecessary component?

Response: This is a critical question, and we are grateful for the chance to clarify what we see as a hierarchy of importance in preference optimization mechanisms.

Our central claim, to be more precise, is not that sigmoid weighting is ineffective, but rather that the mechanism of which samples to filter is more fundamental to preventing over-optimization than the secondary mechanism of how to weight the remaining samples.

RePO and RePO++ were designed to demonstrate this hierarchy:

• RePO Establishes the Foundation: RePO's strong performance (Table 1) proves that binary filtering alone is a sufficient and powerful mechanism for effective alignment. It successfully prevents over-optimization without needing any continuous gradient weighting, thus supporting our claim that such weighting is not an essential component for this primary purpose.

• RePO++ Reveals the Hierarchy: RePO++ was designed as an investigative tool to explore what happens when a secondary weighting mechanism is layered on top of the essential filtering mechanism. The results demonstrate that a hybrid approach can yield further gains (e.g., Table 2, +11.7% Arena-Hard gain for DPO++). This does not contradict our claim; it refines it by showing:

  1. Primary Mechanism (Essential): First, you must effectively filter out easy examples to prevent over-optimization. This is the most critical step.
  2. Secondary Mechanism (Optional Enhancement): Then, for the remaining hard examples, applying a fine-grained weighting can provide an additional, complementary benefit.

In summary, prior methods conflated these primary and secondary roles. Our work decouples them, identifies filtering as the most critical and previously under-appreciated mechanism, and shows that sigmoid weighting, while not essential, can be a useful "add-on" once the fundamental filtering problem is solved.

Thank you for your rebuttal!

Our key insight, and the central thesis of the paper, is that effective preference learning hinges more on which challenging examples are selected for updates than on how their gradients are continuously weighted. While prior methods like DPO/SimPO entangle these two functions within the log-sigmoid activation and the $\beta$ hyperparameter, our work explicitly decouples them.

The wording "selected" in "effective preference learning hinges more on which challenging examples are selected for updates" seems to hint that this is a binary operation. However, I believe that RePO also uses continuously weighted gradients via the positive half of ReLU. Furthermore, the strength of the weighting can also be adjusted by selecting $\gamma$ in RePO, as it shift the positive half of ReLU. So I don't agree with the "decouple" argument that the authors mentioned.

Fundamentally, I believe ReLU works not because it removes continuous weighting (the positive side of ReLU still weights), but because, similar to logsigmoid (as in standard DPO), ReLU is an increasing function that is differentiable almost everywhere.

We thank you for the insightful follow-up. You are correct that the final gradient is continuous for active samples; the crux of our argument lies not in the final gradient's magnitude, but in the form of the weighting function that the loss applies to the gradient direction. Here is a more precise breakdown:

1. The Decoupling Happens in the Gradient's Weighting Function, not the Loss Function Itself.

You correctly observed that the loss function, $\mathcal{L}\_{RePO} = \mathbb{E}[ReLU(-(M\_{\theta}-\gamma))]$, is continuous and differentiable for $M\_\theta < \gamma$. However, the key to our "decoupling" claim becomes evident when examining the gradient expression itself, as presented in our paper.

• For RePO, the gradient (Equation 8) is: $\nabla\_{\theta}\mathcal{L}\_{RePO}(\pi\_{\theta})=-\mathbb{E}\_{\mathcal{D}}[\mathbb{I}(M\_{\theta}<\gamma)\cdot(\nabla\_{\theta,y\_{w}}-\nabla\_{\theta,y\_{l}})]$ Here, the term weighting the gradient direction $(\nabla\_{\theta,y\_{w}}-\nabla\_{\theta,y\_{l}})$ is the indicator function $\mathbb{I}(M\_{\theta}<\gamma)$. This function is fundamentally binary (0 or 1). It cleanly separates the training data into two sets: those that receive a gradient (weight 1) and those that do not (weight 0). This is the filtering mechanism. The magnitude of the update then depends only on the gradient direction term, which is a function of the model's current state.

• For SimPO, the gradient (Equation 7) is: $\nabla\_{\theta}\mathcal{L}\_{SimPO}(\pi\_{\theta})=-\beta\mathbb{E}\_{\mathcal{D}}[s\_{\theta}\cdot(\nabla\_{\theta,y\_{w}}-\nabla\_{\theta,y\_{1}})] \quad \text{where } s\_{\theta}=\sigma(\beta(-M\_{\theta}+\gamma))$ In contrast, SimPO's weighting term, $s\_{\theta}$, is a continuous function of the margin $M\_\theta$. It simultaneously performs filtering (as $s\_\theta \to 0$ for large margins) and re-weighting (the slope of the sigmoid) for all samples. The hyperparameters $\beta$ and $\gamma$ are entangled in controlling this continuous landscape.

This is the explicit decoupling we claim: RePO isolates the binary decision to update (the filter, $\mathbb{I}$) from the subsequent gradient calculation. SimPO/DPO conflate the two.

2. The Role of $\gamma$ is Different in RePO vs. SimPO.

You suggested that $\gamma$ in RePO also adjusts the "strength of the weighting." We respectfully clarify that this is not the case.

• In RePO, changing $\gamma$ only shifts the boundary of the binary filter. It changes which samples are included (the domain of the filter), but for any included sample, the weighting function $\mathbb{I}(M\_{\theta}<\gamma)$ is still just 1. The "strength" of the weight does not change.
• In SimPO, changing $\gamma$ shifts the entire sigmoid curve, which alters the continuous weight $s\_\theta$ for every single sample in a non-uniform way, thus directly adjusting the weighting strength across the board.

3. Why ReLU is not "Just Another Increasing Function" for this Task.

Your final hypothesis is that ReLU works simply because it's a good, mostly differentiable surrogate loss, similar to log-sigmoid. Our theoretical and empirical results challenge this view.

• Theoretical Optimality (Theorem 4.2): Our key theoretical contribution is showing that the ReLU-based loss is the convex envelope of the 0-1 loss. This is a much stronger claim than just being a "good surrogate." It means ReLU provides the tightest possible convex approximation to the ideal (but intractable) classification loss. As per Corollary 4.4, the logistic loss used in DPO/SimPO does not share this property. This provides a fundamental theoretical justification for why this specific functional form is preferable.

• Empirical Disproof (Appendix Table 7): We empirically tested the hypothesis that any similar function would work. As shown in Table 7, simply replacing log-sigmoid with ReLU in the DPO formulation (i.e., without an explicit, tunable margin $\gamma$) leads to a catastrophic performance collapse (e.g., -44% on AlpacaEval LC). This directly demonstrates that it is not the ReLU function in isolation, but the combination of the ReLU form with an explicit margin threshold $\gamma$—which enables the theoretically-grounded binary filtering—that is responsible for the success.

Summary:

In short, our novelty is in identifying and theoretically grounding (as the convex envelope of the 0-1 loss) the principle that explicit binary filtering is more fundamental to preventing over-optimization than continuous gradient weighting. We hope this clarification resolves the issue and are grateful for your rigorous feedback.

To Reviewer 1juA:

We are sincerely grateful for your positive, thorough, and insightful review. We are especially pleased that you recognized the core contributions of our work regarding conceptual simplicity, theoretical backing, and the principle of dynamic data filtering. Your understanding of our paper is spot on.

Q1: What would be an effective way of determining an optimal schedule for dynamically adapting the value of $\gamma$ during the training process?

Response: This is an excellent question that points directly to a key area for future research. Based on our current findings presented in Section 5.5 (Figure 7), our empirically validated recommendation is a "large-to-small" scheduling strategy.

The rationale for this strategy is that it creates a natural and effective learning curriculum. As we note in the paper (Lines 318-320), this works because:

"early in training when the model is underfitting, a larger $\gamma$ permits more aggressive updates across more examples. As training progresses, the decreasing $\gamma$ naturally focuses learning on increasingly challenging examples, effectively preventing over-optimization."

Looking forward to a more general and potentially optimal schedule, we believe an adaptive strategy holds the most promise. Instead of a pre-fixed decay schedule, $\gamma$ could be dynamically adjusted based on the live statistics of the training process itself. For example, one could set the value of $\gamma$ at each training step to match a specific percentile (e.g., the 40th or 50th) of the implicit reward margin ($M_\theta$) distribution observed in the current or a recent batch. This would ensure the model consistently trains on a fixed proportion of what it currently considers to be "hard examples," adapting gracefully to changes in model capability and data difficulty.

We agree that exploring such automated and adaptive scheduling methods is a valuable and exciting direction for future work. We have incorporated this discussion into our limitations section to inspire further research.

Formatting Concerns:

We also thank you for the helpful note regarding acronyms. We will perform a thorough pass on the final manuscript to ensure that all acronyms are clearly spelled out upon their first use.

Thank you once again for your supportive and constructive review.

To Reviewer HXdp:

We sincerely thank you for your expert and detailed critique. Your deep knowledge of the field has pushed us to more clearly articulate the precise scope and contribution of our work. We appreciate the opportunity to address your specific concerns.

1. On Primary Contribution and Novelty (Addressed Weakness1, Weakness2, Question3)

We agree that the mathematical principle of hard thresholding is known in classical theory. Our contribution is not its invention, but its application, analysis, and the new understanding it provides specifically for modern LLM preference alignment, a field where complex methods have become the default. Our novelty lies in:

• Identifying a Core Principle: Filtering is More Fundamental than Weighting. Our central thesis is that effective preference learning hinges more on which challenging examples are selected for updates than on how their gradients are continuously weighted. Our work explicitly decouples these two functions, which are entangled in methods like DPO/SimPO, and reveals the primary importance of the filtering mechanism.
• Providing Theoretical Grounding for this Principle in LLMs. Our work is the first to formally justify this mechanism in this context by demonstrating the ReLU-based loss is the convex envelope of the ideal 0-1 loss (Theorem 4.2). This explains why this simple approach is so effective.
• A Strong, Simple, and Practical Baseline. By showing RePO's effectiveness, we challenge the notion that complexity is required for alignment. The primary use case for RePO is as a strong, simple, and theoretically-grounded baseline to encourage the community to "rethink the fundamental mechanisms" of preference learning, as stated in our introduction.

2. On Theoretical Soundness and Assumptions (Addressed Weakness3)

Thank you for pushing for more clarity on our theoretical claims. You are correct that our theory relies on assumptions.

Our claim of "optimality" is made in the context of finding the tightest convex surrogate for the ideal (but intractable) 0-1 loss in binary classification. This framing, where preference learning is reformulated as binary classification, is consistent with recent foundational work in the field, such as Tang et al., ICML 2024. Our proof for the convex envelope (Theorem 4.2) holds for a domain D = [-a, b], which assumes the implicit reward margin can be both positive and negative—a condition that holds true in practice.

We acknowledge this theoretical optimality does not guarantee convergence to a global minimum in the non-convex landscape of LLM training. However, the strength of our paper lies in demonstrating empirically how well this theoretically "optimal surrogate" performs in practice. We will make these assumptions and context clearer in the main text.

3. On Experimental Scope and Practical Advantages (Addressed Weakness4, Question1)

We interpret your concern about "significant practical advantages" in two ways, and we will address both:

3a. Direct Performance Comparison against State-of-the-Art Baselines We respectfully note that our paper provides extensive comparisons demonstrating RePO's practical advantages. Table 1 shows RePO consistently outperforming DPO, ORPO, R-DPO and SimPO. Furthermore, Appendix Table 5 presents a large-scale comparison on the Open LLM Leaderboard against a wide range of methods, including ORPO, R-DPO, and SLIC-HF, where RePO remains a top performer, validating its effectiveness.

3b. Generalizability of the Filtering Principle to Other Methods To show our core insight—the primacy of data filtering—is a generalizable principle, we integrated RePO's filtering mechanism into IPO, a method with a distinct squared-error loss paradigm. The resulting loss function, which we term IPO-RePO, applies ReLU filtering to IPO's core objective: $L_{\text{IPO-RePO}} = \mathbb{I}\left(M_{\theta, \text{ref}} < \gamma\right) \cdot \left( M_{\theta, \text{ref}} - \frac{1}{2\tau} \right)^2 \quad \text{where} \quad M_{\theta, \text{ref}} = \log \frac{\pi_{\theta}(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi_{\theta}(y_l|x)}{\pi_{\text{ref}}(y_l|x)}$

We conducted experiments on the Llama3-Instruct-v0.2 (8B) model, with results on the AlpacaEval and Arena-Hard benchmarks as follows:

This significant lift demonstrates that our filtering principle is a general mechanism for improving preference alignment, not merely a modification of SimPO.

3c. On Benchmark Selection and MT-Bench We conducted the MT-Bench experiment on Llama3-Instruct-v0.2 (8B) as requested and confirmed RePO's effectiveness:

However, we chose not to emphasize this metric in our paper for reasons of scientific rigor, following public guidance from the SimPO authors themselves, who stated:

"MT-Bench scores often show minimal differentiation between models... and exhibit high variance... For these reasons, we don't consider MT-Bench scores to be a reliable metric." (Source: Official SimPO GitHub repository, Issue #61)

Therefore, our focus on AlpacaEval2 and Arena-Hard was a deliberate choice to use more reliable and discerning benchmarks. We will add the MT-Bench results and this justification to the appendix.

4. On the RePO-SimPO Limit and Stability (Addressed Question2)

Regarding the question of a setting where RePO demonstrates clear advantages over SimPO, our new high-$\beta$ experiments offer several insights. First, we respectfully argue that while the mathematical limit of the sigmoid function is known, its formalization within a preference optimization loss (our Lemma 3.1) is a novel contribution of this work.

Our experiments on Llama3-Instruct-v0.2 (8B) show RePO's clear advantages:

• Stability and Generalization: The table shows that even at an extremely high $\beta=1000$, SimPO's performance saturates and fails to close the gap with RePO. This suggests the log-sigmoid formulation encounters practical optimization limits or numerical instability at such extremes, preventing it from fully realizing the benefits of hard thresholding. In contrast, RePO's ReLU formulation is perfectly stable and robustly achieves this goal, leading to better overall performance.
• Hyperparameter Efficiency: RePO eliminates the need to tune the sensitive $\beta$ hyperparameter, offering a simpler and more robust optimization process.

5. On Writing and Tone (Addressed Weakness5)

Finally, we appreciate the feedback on the manuscript's tone. We will thoroughly revise the paper to ensure our claims are presented with appropriate academic neutrality and that the connection between our theoretical motivation and empirical results is made more explicit and seamless.

Thank you again for your time and expertise. Your critique has given us clear directions for substantially improving our paper.

To Reviewer tpNX:

We sincerely thank you for your thoughtful and detailed review. Your questions have helped us identify several areas where providing more evidence can significantly strengthen our paper. We address each of your points below.

Q1: How does RePO perform across models of different sizes, such as smaller (e.g., 1 - 3B) or larger (e.g., 30 - 70B) architectures?

Response: Thank you for this important question regarding model scalability. To address this, we conducted additional experiments on smaller open-source models: Pythia-1.4B and Pythia-2.8B. The results on AlpacaEval are as follows:

The results show that RePO consistently outperforms both DPO and SimPO on these smaller architectures, maintaining the performance advantage seen in the 7-9B range.

Due to time and resource constraints during the rebuttal period, we were unable to conduct experiments on 30-70B scale models. However, the consistent success of RePO across the 1.4B, 2.8B, 7B, 8B, and 9B model scales provides strong evidence for its effectiveness across a range of small to medium architectures. This consistent outperformance is a promising indicator of its potential to scale favorably, which we propose as a key direction for future work. We will add these new results to the appendix in the final version.

Q2: How well does RePO generalize to other alignment datasets beyond AlpacaEval and Arena-Hard?

Response: Thank you for asking about broader generalization. As you suggest, we evaluated RePO on a wider set of benchmarks from the Huggingface Open LLM Leaderboard. These results are detailed in Appendix D.2, Table 5 of our submission.

The evaluation covers six diverse downstream tasks: MMLU, ARC, HellaSwag, TruthfulQA, Winograd, and GSM8K. In these experiments, RePO demonstrates strong, competitive performance, achieving one of the highest average scores across all tasks for both the Llama3-Instruct and Llama3-Instruct v0.2 models. This robust performance across varied domains further supports our central claim: that the principle of gradient sparsity (filtering trivial pairs) is a more critical driver of successful alignment than the specific weighting scheme of the loss function.

Q3: In practice, how do the gradient directions produced by RePO differ from those of DPO or SimPO? A 2D toy example or visualization illustrating this difference and linking it to the convex-envelope theory would be helpful.

Response: This is an excellent question that gets to the heart of the optimization dynamics. The key difference is not in the gradient direction but in its application and magnitude.

For a given data pair $(x, y_w, y_l)$, the fundamental gradient update direction, $\nabla_{\theta,y_{w}}-\nabla_{\theta,y_{l}}$, is actually the same for RePO, SimPO, and DPO. The distinction lies in how this gradient is scaled:

• DPO/SimPO apply a continuous, smooth weighting to the gradient. The magnitude is scaled by a sigmoid function, meaning all pairs receive a non-zero gradient, though it becomes vanishingly small for well-separated pairs.
• RePO applies a binary, all-or-nothing filter. The gradient is either applied (with a uniform effective magnitude, as adaptive optimizers like Adam normalize it) or it is set to exactly zero if the margin $\gamma$ is met.

To provide the 2D intuition you requested: imagine a parameter space where, for any given data point, the gradient vector points in the same direction for all three methods. With DPO/SimPO, the landscape of gradient magnitudes would be smooth, like rolling hills, gradually approaching zero. In contrast, for RePO, this landscape is a plateau of uniform height on one side of the $\gamma$ boundary and completely flat (zero) on the other. This sharp 'cliff' is precisely what Figure 2 in our paper illustrates in one dimension.

This piecewise-constant behavior is a direct consequence of RePO's loss being the convex envelope of the 0-1 loss, which theoretically favors such sharp, efficient boundaries over the smooth curves of logistic-based losses. It provides a hard, efficient threshold for updates, linking our theory directly to the practical gradient behavior you asked about.

Q4: Can you provide quantitative evidence for the “emergent curriculum” effect—for example, plots showing how the margin distribution shifts toward harder pairs over the course of training?

Response: Thank you for this prompt. Figure 4 in our paper provides exactly this quantitative evidence. The green-shaded areas in the plots represent the proportion of training samples whose reward margin $M_\theta$ is less than the threshold $\gamma$, meaning they are still actively receiving gradient updates.

We can summarize the percentage of samples being updated at each training snapshot:

This data clearly shows the emergent curriculum:

• Early in training (Step 0), nearly all samples are considered "hard" as the model has not learned to distinguish preferences, so 99.5% of pairs are updated.
• As training progresses, the model improves, and a growing fraction of the data is "filtered out" (gray area).
• Late in training (Step 400), only 42% of the samples receive updates. These remaining samples are the most "challenging" or "confusing" pairs for the mature model.

This dynamic filtering, where the training automatically focuses on progressively harder examples, is what we define as the "emergent curriculum" effect driven by the simple $\gamma$ threshold.

Q5: Shallow ablation coverage: Other hyper-parameters (except for $\gamma$) and data regimes remain untested.

Response: Thank you for this point. We agree that exploring a wider range of hyperparameters could potentially tune overall performance further. However, for this study, our primary scientific goal was to isolate and understand the fundamental impact of the gradient filtering mechanism itself, which is RePO's core conceptual contribution.

To achieve this and ensure a controlled and fair comparison, all other training hyperparameters (e.g., learning rate, optimizer settings) were kept consistent across all methods and aligned with the established SimPO baseline, as detailed in our Appendix D.1 (Implementation Details). We believe this focus is crucial for a clear scientific conclusion, and our extensive ablation study on γ (presented in Figures 5, 7, and 8) provides a thorough and insightful analysis of this central component.

Q6: Limited theoretical support: The convex-envelope proof shows RePO is an optimal surrogate loss, but it offers no guarantees that training will converge to optimality (which is rare in practice). Also, the advertised benefits --- such as mitigating over-training and filtering data --- remain theoretically unsupported.

Response: Thank you for this precise and important clarification regarding the scope of our theoretical support. You are entirely correct, and we appreciate the opportunity to be precise about what our theory does and does not claim.

• On Optimal Surrogate Loss vs. Training Convergence: Our theoretical contribution is to prove that RePO's loss function is the optimal surrogate loss—specifically, the tightest possible convex approximation to the ideal 0-1 loss within a binary classification framework (a setup consistent with foundational work like Tang et al., ICML 2024). This provides a strong theoretical reason for why one should prefer RePO's formulation over, for example, a log-sigmoid loss from a surrogate loss perspective. We agree that this does not provide a guarantee that the training process itself will converge to a global optimum in the highly non-convex landscape of LLM optimization, as such guarantees are exceptionally rare for deep learning in practice.

• On Empirical vs. Theoretical Support for Benefits: Similarly, you are correct that the advertised benefits of mitigating over-optimization and creating an emergent data filtering curriculum are currently supported by our extensive empirical analysis (e.g., Sections 3.3, 5.5; Figures 4, 6) rather than a formal theoretical proof. We believe that bridging this gap—developing a formal theory for these emergent, dynamic properties of the training process—is a challenging but very exciting direction for future research.

We will revise the manuscript to make these distinctions clearer. Thank you for pushing for this level of precision.

Q7: Minor: The link in citation [66] is broken.

Response: Thank you for catching this typo. We have corrected the link in the manuscript.