We thank you for your thoughtful and constructive review. We are encouraged by your positive assessment and appreciate the insightful questions regarding the practical assumptions and deployment of our method. Below are our detailed responses.

Q1: Weakness 1 & 3: Strong assumptions on noise (strict tail contamination) and limited noise type analysis (symmetric label flips).

A1: Our assumption is weaker and more practical than classical approaches, and it handles asymmetric noise.

1. On the Tail Contamination Assumption vs. Infinitesimal Contamination

You are correct that our theory relies on this assumption (tail contamination). However, we sincerely argue it is a reasonable and practical abstraction of how a model learns from noisy data, under the setting where we have no access to clean validation dataset. A model trained on a majority of clean preferences should learn to assign very low likelihood to gross errors like flipped labels. Our empirical results across multiple datasets (Figs. 3, 4, 5) provide compelling evidence that this assumption holds sufficiently well in practice to deliver state-of-the-art robust performance, even if real-world data does not perfectly match this theoretical ideal.

Furthermore, this assumption is more realistic for large-scale datasets than the other assumptions often used in classical robust statistic, such as infinitesimal contamination ($\epsilon \approx 0$). In the LLM alignment context, where datasets can have substantial noise (e.g., >25% in Anthropic HH), an infinitesimal contamination model is simply not applicable. Our tail contamination assumption, which allows for heavy contamination, is a more practical abstraction. Furthermore, as we note in our paper (lines 95-96), this type of assumption—that outliers can be identified in the tail of a distribution—is a well-established principle implicitly or explicitly used in many robust learning and outlier detection algorithms.

2. On the Symmetric Label Flip Model and the True Limitation

We would like to clarify that our method's core theoretical assumption is not strictly limited to symmetric flips, but rather the more general condition of tail contamination. This means our method is effective against any type of noise—be it symmetric, asymmetric, or even some systematic annotator biases—as long as the noisy preference pair is correctly identified by the learned model as a low-likelihood event (an outlier).

The true limitation lies with noise that is too systematic to be identified as an outlier by the model. It is plausible to hypothesize that the method's performance would degrade when noise is structured in a specific way—for example, when all preference labels for a single, specific prompt or a narrow topic are consistently flipped.

The reasoning is that these structured errors would no longer form a clear "tail" in the overall data distribution. Instead, they would create a consistent but incorrect signal for that specific sub-task, which the model would likely struggle to distinguish from a genuine, albeit niche, preference. Exploring such complex, systematic noise structures is a vital direction for future research.

Q2: Weakness 2: Sensitivity to Hyperparameter $\gamma$

A2: It is shared challenge in robust statistics community, including prior work, not specific to our work.

The choice of $\gamma$ is indeed a key practical consideration, as is common for many robust statistical methods. We would like to emphasize that this kind of challenge is shared by the other robust methods like R-DPO and Dr. DPO. These methods also depend on a critical hyperparameter (the estimated noise ratio $c$), which is often unknown a priori and requires tuning.

Our key finding is that a single default value ($\gamma=2.0$) works consistently well across our diverse experiments. This suggests that while $\gamma$ is an important dial for tuning the robustness-efficiency tradeoff, our method is not prohibitively sensitive and can be effectively used by practitioners without exhaustive hyperparameter sweeps.

Q3: Role of Hölder-DPO.

A3: Preprocessing if expensive two-step training is permissible given budge, otherwise final training.

This is an excellent and insightful question that touches upon the practical deployment of our method. We believe Hölder-DPO is a versatile tool, and its optimal role depends on the scale of the dataset and the specific application scenario.

• For Medium-Scale or High-Stakes Datasets: In this scenario, a two-stage approach is likely optimal.

  • Stage 1: Train a model with Hölder-DPO to leverage its robustness and data valuation capabilities.
  • Stage 2: Use the trained model to identify and filter out the noisy data points.
  • Stage 3: Train a final model (using standard DPO or another method) on the newly cleaned, high-quality dataset. This approach, validated by our Figure 5c, maximizes final performance by combining robustness for cleaning and efficiency for final alignment.

• For Very Large-Scale Datasets: When dealing with massive datasets where a two-stage process might be computationally prohibitive, a one-stage approach is more practical.

  • Train the final model directly with Hölder-DPO. This provides a robustly aligned model in a single, efficient training run.
  • The data valuation scores generated during this process can then be used to create a cleaned dataset that is held in reserve for the next training cycle, continuously improving data quality over time.

In summary, Hölder-DPO can be used as both a direct training objective and a preprocessing tool, and practitioners can choose the best strategy based on their specific constraints and goals. We will add a new appendix section to explicitly discuss these practical, data-dependent deployment scenarios, highlighting the dual-role capability of our work.

Q4: Limitation 1: Complexity of Real-World Noise.

A4. Fully agreed. We added the explanation in the Limitation section.

To summarize our position: we clarified that our method's core assumption of "tail contamination" is more general than just symmetric flips and can handle various noise types. The true limitation, as you astutely point out, lies with complex, systematic noise that does not form a clear "tail" in the data distribution—such as the excellent example you provided of "easy to agree, hard to differentiate" noise. Investigating such complex noise types is indeed an important and significant direction for our future work.

We will ensure this nuanced perspective is reflected in the final paper's Limitations section.

Q5: Limitation 2: Scalability and Filtering Strategy in data-scarce domains.

A5: We understand the sentiment, but in data-scarce settings, flipped data points matter more—so our method performs better.

We argue that our method is particularly valuable in such scenarios, guided by the principle of "less is more for alignment."

As our own Figure 5c demonstrates, removing noisy data points leads to better alignment quality, even if it reduces the dataset size. We believe that carefully cleaning a small dataset is often more effective than training on a larger, noisy one.

Crucially, our method offers the flexibility needed for these high-stakes domains and is not just an aggressive, automated filter. Instead, it empowers domain experts by providing a ranked list of all data points from most to least likely to be noisy. An expert in a data-scarce setting does not need to blindly trust the model's estimated noise ratio $\hat{\epsilon}$. Instead, they can use our method's ranking to make a more informed, conservative decision:

• For efficient manual review: The expert can start inspecting data from the top of the ranked list (the most suspicious items). This allows them to focus their valuable time on the most problematic $\hat{\epsilon}$ fraction of the data, rather than having to review the entire scarce dataset.
• For conservative filtering: The expert can choose a much smaller, more conservative threshold for removal (e.g., only removing the top 1-5% of most suspicious items) to preserve as much data as possible while still eliminating the most harmful outliers.

Therefore, rather than being a risky tool in data-scarce settings, Hölder-DPO serves as an intelligent assistant that makes the process of cleaning valuable datasets more efficient and controllable.

We will add this discussion to our Limitations section (Section 6), highlighting how the ranked output of our method enables flexible, human-in-the-loop filtering strategies suitable for data-scarce domains.

Thank you again for the helpful suggestions and being constructive. We have incorporated them into our revision and will continue to strengthen the experimental section in the final version. We hope our rebuttal clarifies our points.

Many thanks,

All the authors

Thank you for the valuable feedback. Below are our detailed responses.

Q1: The definition of the IF function is syntactically problematic and confusing.

R1: We appreciate this observation and respond in two parts:

Q1.1: The function $\mathrm{IF}(x, \theta, p_D)$'s inputs do not appear on the right-hand side of Eq. (4).

R1.1: This is a shorthand; the full form is provided in Appendix E.1.

We apologize for the confusing notation. The expression in Eq. (4) follows a common shorthand in robust statistics literature, omitting explicit dependence for readability. However, we agree this may not be accessible to all readers and should have been explained more clearly.

The complete derivation is provided in Appendix E.1, where the influence function is given as:

$$\text{IF}(x, \theta, p_D) = -H_1^{-1} F_1,$$

where

$H_1 = \mathbb{E}_{p_D}[ H_2 ],$

$H_2 = H_{\theta}(s),$

$F_1 = \mathbb{E}_{p ( \tilde{s} ) }[ F_2 ],$

$F_2 = F_{\theta}( \tilde{s} ),$

$\tilde{s} := s_{\text{flip}}$

Here, the Hessian term $H_1$ clearly depends on the clean distribution $p_D$, while the outlier term $F_1$ depends on the specific outlier point (derived from $x$). Both terms are evaluated at $\theta = \theta^*$, which itself is a function of $p_D$.

We have incorporated the above explanation in Section 2.

Q1.2: What does the notation $|\cdot|$ mean? As $\theta(\epsilon)$ is a vector, it can't be the absolute value.

R1.2: Apologies. This was a typographical error.

You are absolutely right. The IF expression $\left.\frac{\partial \theta^*(\epsilon)}{\partial \epsilon}\right|_{\epsilon = 0}$ is a vector, and the $|\cdot|$ symbol is inappropriate. We have corrected this in our revised draft.

Q2: The robustness framework and the condition in Definition 2 are difficult to understand.

R2: We have now added the following clarification.

We acknowledge that our original explanation of Definition 2 was brief, primarily due to space constraints. In the camera-ready version—where we will have an additional page—we have expanded this section to provide a clearer and more detailed discussion, as outlined below:

Q2.1: The terms $r_{\theta}$ and $\theta$ are never defined.

R2.1: Both are defined in L65 and L72.

$\theta$ denotes the parameters of the policy (LLM) model $\pi_{\theta}$, and $r_{\theta}$ represents the implicit reward function. We acknowledge a source of confusion stemming from our inconsistent use of $\hat{r}{\theta}$ and $r{\theta}$—these refer to the same quantity. We have now unified the notation throughout the paper to consistently use $r_{\theta}$.

Q2.2: How does $r_{\theta}(x, y^{\text{flip}}_{\text{lose}}) \to \infty$ correspond to the worst-case scenario?

R2.2: It represents maximal misalignment to a flipped label.

In model alignment, the worst case occurs when the LLM is misled to strongly update toward a flipped label, effectively learning the wrong preference. The importance of a data point is implicitly governed by the reward function $r_\theta$, which we cannot observe directly. Since humans are generally more reliable at relative comparisons than assigning absolute scores (e.g., ranking a prompt among 10,000 candidates), we estimate $r_\theta$ via pairwise preferences, expressed as:

$g\_{\theta}(s) = r\_{\theta}(x,y\_{\text{win}}) - r\_{\theta}(x,y_{\text{lose}}).$

The model is aligned by updating its parameters $\theta$ based on the loss gradient $\nabla_\theta \mathcal{L}$ (Eq. 3), which is scaled by $g_\theta(s)$. As $g_\theta(s) \to \infty$, the update becomes overly dominated by this data point.

Now, consider the worst-case: if the flipped label $y_{\text{lose}}^{\text{flip}}$ is assigned an infinitely high reward, i.e., $r_\theta(x, y_{\text{lose}}^{\text{flip}}) \to \infty$, then $g_\theta(s) \to -\infty$, and the model is misaligned with maximum confidence. This scenario captures the core robustness failure.

While we focus on $r_\theta(x, y_{\text{lose}}^{\text{flip}})$ in the limit, the interpretation is equivalent to the worst-case $g_\theta(s)$ due to symmetry. This is why Definition 2 formalizes robustness as the requirement that the model nullifies updates under such extreme (and erroneous) conditions.

Q2.3: How does the condition in Definition 2 relate to noise robustness? This relation is not trivial.

R2.3: Satisfying Definition 2 ensures that the LLM’s parameter update is nullified in response to a flipped label—providing robustness to noise.

This is formalized in Appendix C, Eq. (14), which shows that the influence of a flipped point on the optimal parameters $\theta^*(\epsilon)$ vanishes to first order. As noted in Footnote 1 (Page 3), this relies on a first-order Taylor expansion, and thus does not eliminate higher-order effects.

Nevertheless, by nullifying the first-order influence, we ensure that $\theta^(\epsilon) \approx \theta^*$ (see Line 110), capturing a meaningful notion of robustness to label noise.

Q3: In Figure 2, what is "IF weight"? How does the figure relate to Definition 2, particularly the x-axis?

R3: Explanation added

We agree that this part was particularly unfriendly. We have added the explanation as follows:

Q.3.1: What is "IF Weight" and why is it critical for robustness?

R3.1: It is the scalar multiplier that scales the gradient direction.

Eq.~(3) can be decomposed as:

$$\nabla_\theta \mathcal{L}(s, \pi_\theta) := \sigma(- g_{\theta}(s)) g_\text{grad}(s),$$

where the multiplicative factor $\sigma(- g_{\theta}(s))$ represents the IF weight, and $g_\text{grad}(s)$ represents the direction of the gradient. Here, IF weight determines how strongly the model updates its parameters for each data point. If IF weight tends to zero in the worst-case (i.e., $g_\theta(s_{\text{flip}}) \to -\infty$), the method is robust because it ignores high-confidence misaligned (flipped) inputs.

We summarize the worst-case IF weight behavior for various methods in Table 1:

Table 1: IF weights for all alignment loss.

Figure 2 plots this IF weight (y-axis) versus $g_\theta(s_{\text{flip}})$ (x-axis). The limit $g_\theta(s_{\text{flip}}) \to -\infty$ corresponds to Definition 2. Only Hölder-DPO satisfies Definition 2.

Q.3.2: How does Figure 2 relate to Definition 2 and the x-axis?

R.3.2: The y-axis approaching zero as x-axis to minus infinity corresponds directly to Definition 2.

While the symbolic proof in Table 1 is sufficient to establish the redescending property formally, Figure 2 offers an intuitive, visual interpretation of how each method behaves under extreme cases.

The IF weight on the y-axis reflects how sensitively the model updates its parameters in response to each data point. A high IF weight indicates strong influence on the parameter update. For example, vanilla DPO assigns high weight when the base model makes a mistake (i.e., $g_{\theta^*}(s) < 0$, meaning the base model mispredicts human preference as $y_{\text{lose}}$ over $y_{\text{win}}$), which is reasonable for standard alignment.

In contrast, Hölder-DPO emphasizes updates only when the model’s confidence is moderate, i.e., when $| g_{\theta^*}(s) |$ is small. When the human feedback strongly contradicts the model's prior belief (large negative $g_{\theta} (s)$), Hölder-DPO assumes that the feedback itself may be unreliable, and therefore suppresses the update. This reflects a robust alignment strategy that guards against extreme but potentially erroneous feedback.

Q4: On the tradeoffs of the redescending property.

R4: No additional computational cost; sensitivity to $\gamma$ is the primary trade-off.

Hölder-DPO (Eq. 12) incurs no extra computational cost over vanilla DPO. The main trade-off lies in tuning the hyperparameter $\gamma$, which controls the aggressiveness of downweighting potential outliers.

This is a standard feature of robust estimators, as corroborated by Reviewer FZwu. Similar to the role of hyperparameters in R-DPO or Dr. DPO, $\gamma$ must be chosen appropriately. However, we show that a single default setting ($\gamma = 2.0$) performs robustly across a wide range of settings, demonstrating practical usability.

Q5: What is the effect of $\gamma$?

R5: $\gamma$ serves as a sensitivity knob for outlier detection:

• A small $\gamma$ makes the model more tolerant: data points are downweighted only when the model is highly confident they are outliers (i.e., $s$ is strongly negative). The redescending slope is gentle.
• A larger $\gamma$ increases sensitivity: even moderately negative $s$ values lead to aggressive downweighting. The redescending slope is steep.

Thank you again for the helpful suggestions and being constructive. We have incorporated them into our revision and will continue to strengthen the experimental section in the final version. We hope our rebuttal clarifies our points.

Many thanks,

All the authors

Thank you for your detailed follow-up and for engaging with our discussion.

Which equation in the Appendix E1 is the definition of the IF function? I still can't find it.

R.1-1: Lines 1147-1148 in Appendix E.1

Our previous response (R.1.1) corresponds exactly to the equation between lines 1147–1148 in Appendix E.1. Due to a markdown formatting issue on OpenReview, the equation was split across lines, but it remains the same. To clarify, this equation presents the full derivation of the IF. The general definition of the IF is given in Eq. (4), while Appendix E.1 provides a specific derivation tailored to the DPO loss.

In addition, the requirement in Theorem 5 (and possible the same requirement in Theorem 2) on positive definite of the Hessian matrix is highly unlikely to hold because the loss function is non-convex with respect to $\theta$

R.1-2: This is local, not global loss landscape.

This is good point—indeed, the global loss landscape is non-convex. However, our assumption of a positive definite Hessian applies only locally, at the point of convergence ($\theta^{\*}$). We do not assume global positive definiteness. This local assumption is standard practice, following seminal work such as [Koh & Liang, 2017], and has been adopted in recent studies applying influence functions to large language models (e.g., [Choe et al., 2024]).

Citation

• [Koh & Liang, 2017]: Understanding Black-box Predictions via Influence Functions. ICML2017. https://arxiv.org/abs/1703.04730.
• [Choe et al., 2024]: What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions. https://arxiv.org/abs/2405.13954

The requirement in definition 2, that the optimal solution $\theta^*$ as a function of $\epsilon$ have a vanishing gradient at $\epsilon=0$ seems unnatural, served as a technical bypass rather than a physically interpretable assumption, and hence very stringent. It depends simultaneously on multiple factors, the data distribution, the loss function, and the neural network that $\theta$ is used to parameterized.

R.2: We respectfully disagree. IF is well-known as an interpretability tool thanks to its natural definition.

Definition 2 (redescending property) characterizes robustness as the worst-case behavior of the IF. We argue that IF is a natural and principled metric for sensitivity to data contamination, and that its worst-case value offers an intuitive robustness criterion. To motivate this, consider the question: What is a natural way to define sensitivity to data contamination? We aim to quantify how a data contamination $\epsilon$ affects the optimal solution $\theta^{\*}$. In general sensitivity analysis, taking the gradient is a standard approach. The IF formalizes this idea—it is simply the gradient of the optimal solution $\theta^{\*}$ w.r.t. contamination $\epsilon$. A large norm of the IF indicates high sensitivity, meaning the solution is strongly affected by the noise. Moreover, IF provides pointwise sensitivity, allowing us to identify which specific data points most influence the learned parameters. This intuitive interpretation underlies its wide use in model interpretability, as in [Koh & Liang, 2017]. Conversely, if the IF for a given data point is zero, the model is locally insensitive to that point—it has no influence and could, in principle, be removed without affecting the solution. Definition 2 leverages this idea: if even the worst-case contaminated point has zero influence (i.e., zero IF), then the loss is robust in the strongest possible sense—it nullifies the effect of the most adversarial perturbation. This leads to a natural and interpretable definition of robustness.

Formally, this derivation is shown in Appendix C.

The term $r_\theta$ is neither defined in L65 or L72.

R.3: $r_\theta = \hat{r}_\theta$

To be perfectly precise:

• Line 65 defines the model parameters $\theta$.
• Line 72 defines the implicit reward function and gives it the symbol $r_\theta = \hat{r}_{\theta}$.

As we denoted in our answer for your Q.2.1, we will correct this throughout the manuscript to ensure consistency.

Thank you very much for your insightful and highly positive review and the helpful comments. In our revision, we have improved the clarity and readability by incorporating your suggestions. The following are our detailed responses.

Q1: It would be nice to see confidence intervals for win rates.

R1: Confidence intervals have been added.

The error bars in Figures 3 and 4 already represent the standard deviation across 10 independent runs, providing a clear picture of variability and stability. That said, we agree that confidence intervals can offer a more rigorous means of distinguishing methods. We have now added 95% confidence intervals to Figures 3, 4, and 5, as well as to Tables 1 and 2 (see Appendix H).

Q2: Could you elaborate more on the sensitivity of your method to the choice of the hyperparameter $\gamma$? The discussion is currently limited.

We address this concern in three parts:

Q2.1: Elaboration on the $\gamma$ sensitivity

R2.1: We have expanded the explanation as follows.

$\gamma$ serves as a sensitivity knob for outlier detection:

• A small $\gamma$ makes the model more tolerant: data points are downweighted only when the model is highly confident they are outliers (i.e., $s$ is strongly negative). The redescending slope is gentle.
• A larger $\gamma$ increases sensitivity: even moderately negative $s$ values lead to aggressive downweighting. The redescending slope is steep.

This gives practitioners direct control over the trade-off between robustness to gross errors and sensitivity to data variance.

Q2.2: Expanded Ablation Studies & Scaling Concerns

A2.2: Further analysis will be included in the camera-ready version.

Table 1 currently illustrates the core trade-off between robustness and mislabel precision. We are working to expand this ablation in the appendix, but a full sweep across all models and datasets is infeasible within the rebuttal period due to computational limitations (Google Colab, single A100 GPU), where each one data point in one figure takes a day to compute (see Appendix H.2). Since the optimal $\gamma$ is likely task-specific, we believe our existing results already provide useful guidance for practitioners.

Q2.3: Inaccurate description on L259

R2.3: You are correct.

Thank you for catching our imprecise language. We have revised the text to correctly state that increasing $\gamma$ increases the estimated contamination ratio, rather than "improving" it.

Q3: The paper would benefit from a discussion of why the redescending property is not always prioritized in statistics (the trade-off between robustness and efficiency).

R3: Vanilla DPO is indeed preferred over robust DPO if the dataset is perfectly clean, but this condition is rarely satisfied.

While vanilla DPO may outperform robust methods on perfectly clean datasets, this assumption rarely holds in practice. Even curated datasets like Golden HH (Figure 5a) exhibit non-negligible noise. In these scenarios, vanilla DPO is fragile, degrading significantly with even modest contamination. In contrast, Hölder-DPO remains stable. Interestingly, even in the "clean" setting ($\epsilon = 0$, no artificial noise), Hölder-DPO performs comparably to or better than vanilla DPO (Figures 3 and 4), indicating that the "clean" open dataset is actually slightly noisy (see estimated $\hat{\epsilon}$ are non zero at $\epsilon=0$). Moreover, we observe in Figure 5c that vanilla DPO trained on the cleaned dataset (filtered by Hölder-DPO) outperforms robust methods trained on unfiltered data. This highlights an optimal two-step procedure: first use Hölder-DPO for noise identification and filtering, then apply vanilla DPO for final alignment—if resources and dataset size permit this expensive training.

We have integrated this discussion into Section 5 and Section 6 (Limitations).

Q4: I would strongly suggest you preview the definition of "redescending" much earlier in the paper.

R4: Agreed—we have amended the explanation to Line 36:

The redescending property—the idea that the influence of extreme outliers should diminish to zero.

Thank you once again for your valuable feedback. We believe these revisions significantly improve the paper and hope our responses address your concerns clearly.

Many thanks,

All the authors

Thank you very much for the positive feedback and the helpful comments. In our revision, we have improved the clarity and readability by incorporating your suggestions. The following are our detailed responses.

Q1: The method's novelty appears limited, feeling like an incremental transfer of classical M-estimators to DPO, and its theoretical guarantees are confined to i.i.d. label flips.

We address this concern in three parts:

Q1.1: On the assumption of small contamination.

R1.1: We respectfully disagree.

Our method is explicitly designed to handle heavy contamination, as stated in Assumption 1. Unlike classical influence function (IF) theory based on infinitesimal contamination ($\epsilon \approx 0$), we adopt a tail contamination model, allowing substantial noise (e.g., $\epsilon = 0.4$). Our experiments confirm that Hölder-DPO remains robust even under 40% label noise.

Q1.2: On the noise model (i.i.d. flips) and failure cases (near 50% or systematic noise).

R1.2: We acknowledge this limitation but clarify its scope.

You are correct that our theoretical framework currently focuses on the i.i.d. label flip model, and that identifying mislabeled data becomes fundamentally challenging near a 50% noise ratio. However, this is not a limitation of our method per se, but a theoretical identifiability barrier: when noise and clean data are equally represented, distinguishing the two becomes statistically impossible without a clean reference set. Our approach assumes a bimodal distribution where one mode (typically the majority) corresponds to clean data. If the noise ratio is extreme (e.g., 90%), Hölder-DPO can still isolate the minority clean subset (10%) as outliers. In such cases, the roles can be reversed—the identified minority is treated as clean, and the rest discarded. This reversal can be verified with minimal human effort by inspecting a few top-ranked samples flagged by our method, if human-in-the-loop workflows are permissible. Thus, Hölder-DPO not only enables principled mislabel detection under reasonable assumptions, but also offers a practical, interpretable, and efficient workflow for data inspection with minimal manual oversight.

Table 1: Noise identification conditions

Notably, prior methods cannot identify mislabeled data in arbitrary noise regimes, even with clean labels or human inspection (see Theorem 3). Our i.i.d. flip model is standard [24, 104], and extending to adversarial/systematic noise is a compelling future direction.

Q1.3: On the novelty and generality of the method.

R1.3: We respectfully disagree.

While the redescending property is indeed a classical concept, theoretical novelty often lies not in the conceptual idea itself, but in the rigor, adaptation, and extension to new domains. If formal IF analysis had already been applied to modern LLM alignment methods like DPO, our contribution might be considered a straightforward reuse. However, to the best of our knowledge, no prior work has done so. Thus, we had to lay the theoretical groundwork. As classical redescending estimators are typically studied in much simpler settings (e.g., linear regression), and adapting them to the DPO presents substantial methodological and analytical challenges. Our contributions go beyond a straightforward transfer of classical ideas. Specifically, we introduce three key innovations:

• The First Formal IF Analysis for the DPO Family: Our work is the first to define the redescending property tailored to the DPO context, and provide a formal IF analysis for DPO-style objectives, rigorously proving that no existing methods satisfy the redescending property.
• A Novel, Compatible Objective Function: We designed Hölder-DPO, a new objective that is compatible with the unique log-ratio structure of DPO while provably satisfying the redescending property, a non-trivial theoretical adaptation.
• Gradient-Free Data Valuation: Our most significant practical contribution is the novel connection between this robust loss and a principled, gradient-free data valuation method, enabling scalable mislabel detection—a new capability in this domain. While the nature of contamination ratio estimation is inherited via Hölder-divergence, it is non-trivial that this merit successfully works in LLM fine tuning. Consequently, we show that some special weighting is needed to construct the valid ratio estimator (see Proposition 1 and Appendix F.3).

By laying the groundwork for integrating classical robust tools into modern machine learning systems, we enable plug-and-play extensions to other domains such as diffusion models. In this sense, our contribution is not merely an application of old ideas, but a significant theoretical advancement that modernizes and operationalizes them in previously unexplored contexts.

In the revision, we will clarify this positioning more explicitly in the introduction and related work. We agree that handling structured or adversarial noise remains an important challenge, and we have added a discussion of this as future work in Section 6.

Q2: The alignment performance on real data with unknown noise was not directly evaluated without synthetic corruption. The robustness is primarily demonstrated on simulated noise.

R2: We have already done this.

In Section 5.3, we evaluate Hölder-DPO on the original, unmodified Anthropic HH training set, which is widely recognized to contain substantial real-world label noise. As shown in Figure 5, Hölder-DPO achieves a GPT-4 win rate of ~0.62, outperforming Dr. DPO (~0.59), R-DPO (~0.58), and vanilla DPO (~0.57). This result provides direct empirical evidence that Hölder-DPO’s robustness extends beyond synthetic corruption, yielding improved alignment performance on a real-world benchmark with naturally occurring noise.

We acknowledge that this important result was not sufficiently emphasized. In the revision, we will revise Section 5.3 to explicitly highlight this as a real-world robustness evaluation, underscoring the practical effectiveness of our approach.

Q3: Additional experiments on a more well-established benchmark like Arena-Hard

R3: Yes, it is indeed nice to have, but our experiments are already large as a theoretical paper.

We agree that evaluating on a more challenging benchmark like Arena-Hard would further strengthen our empirical validation. We attempted this, but encountered practical limitations due to computational constraints—specifically, our setup (Google Colaboratory with a single A100 40GB GPU) could not support the larger models and parallel runs required to complete the benchmark within the tight rebuttal period. We plan to include additional benchmarks in the camera-ready version.

That said, we would like to clarify that our current experimental evaluation is comparatively extensive for a theory-oriented paper. We summarize the scope of our setup below for context:

Table 1: Comparison of experimental validataion setups with accepted papers and ours

While our setup may be smaller than that of purely experimental works, our primary contribution is theoretical—we provide the first formal guarantees for redescending and noise identifiability properties in DPO. Our empirical results serve to verify and complement these theoretical findings, and we believe they are appropriate in scope for a theory-driven submission.

Thank you again for the helpful suggestions. We have incorporated them into our revision and will continue to strengthen the experimental section in the final version. We hope our rebuttal clarifies our points.

Many thanks,

All the authors