ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization

Thank you for taking the time to review our paper and for offering detailed feedback. We address them below, following the order in which they appear across the review’s sections.

---

Claims and Evidence (CE)

[CE-Q1] Theoretical motivation.

[CE-A1] In our response to reviewer RzSL under question ID [RS-Q1], we provide a detailed theoretical motivation,including mathematical derivations and sensitivity analysis demonstrating why low-confidence tokens ($p_i(\theta)$) yield higher gradient norms ($||\nabla_\theta \log p_i(\theta)||$). Due to space constraints, we kindly refer the reviewer to that section for complete details.

[CE-Q2] KL budget efficiency at lower end.

[CE-A2] We added additional points at lower KL divergence levels in Figure E to enable a fairer comparison. These additional results clearly show that ConfPO consistently achieves higher alignment scores than SimPO across both low and high KL values.

Figure E: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureE.md

---

Experimental Designs or Analyses (EA)

[EA-Q1 & Q2] Correlations between tokens' confidence and their position in the sequences.

[EA-A1 & A2] In Figure C below, we visualize examples of selected tokens. In general, tokens acting as crucial inflection points within sentences (e.g., tokens initiating lists or new phrases) are selected, while tokens that are merely continuations of preceding tokens are not selected (e.g., in the word "Slack," "Sl" is selected, whereas "ack" is not). This shows that our confidence-based selection identifies important tokens that shape the overall sentence, leading to more efficient KL-budget usage and reduced overoptimization (Figure 5 in the main text).

Since inflection points of sentences are not really located towards certain position in a sentence, we see they are approximately uniformly distributed in the sentence as in Figure D.

Figure C: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureC.md

Figure D: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureD.md

[EA-Q3 & 4] Exploration of soft weighting ConfPO and curriculum learning.

[EA-A3 & 4] Thank you for suggesting this valuable experiment. We explored a soft-weighting version of ConfPO,$\frac{\sum_{i=1}^{|y|} w(y_i)\log\pi_\theta(y_i|x,y_{<i})}{\sum_{i=1}^{|y|} w(y_i)},$ where $w(y_i) = (1 - p_i(\theta))^T$. In Figure F below, we tested soft weighting with temperatures $T=1$ and $5$. When $T=0$, the equation reduces to standard SimPO; as $T$ increases, it approaches our original (hard-weighted) ConfPO. Our results demonstrate that alignment performance consistently improves as $T$ increases, with the original ConfPO achieving the highest performance.

This validates our key hypothesis: high-confidence tokens can impede preference learning by overshadowing more informative signals. This is consistent with Figure 4-(b) of our main paper, where training exclusively on low-confidence tokens improved alignment scores relative to training on all tokens, while training solely on high-confidence tokens resulted in performance worse than even the SFT baseline.

We also experimented with curriculum setups adjusting T from 0 to 10 and 10 to 0 during training. Results showed no clear advantage over standard ConfPO, which can also be found in Figure F below.

Figure F: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureF.md

---

Other Comments or Suggestions

We address each point below:

• Definition of $D$ (Section 3.1):
  Thank you for noting this inconsistency. We will clearly distinguish the dataset notation used for prompts alone versus prompt-response pairs in the revised manuscript.
• Bradley-Terry Model:
  We included the Bradley–Terry (BT) model in Section 3.1 because it serves as the foundational probabilistic model for preference-based reward modeling, commonly used in RLHF. While traditional RLHF explicitly learns a reward model using BT, DPO (introduced in Section 3.2) leverages this same BT framework implicitly by reparameterizing the reward with the joint log probabilities.
• Figure 4 (model and dataset):
  For Figure2~4, we use the Llama-3-Base (8B) with the UltraFeedback dataset. We will clarify this point in the manuscript.
• Equation Duplication:
  We will simplify the presentation by directly defining as given in Equation 7 with added token selection criteria of ConfPO.
• Hyperparameter for Figure 5:
  In Figure 5, we varied the $\beta$ hyperparameter (with $\gamma$ fixed) to achieve different levels of KL divergence. Also we have added more points in the following figure with fair KL range (Figure E: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureE.md).
• Number of tokens selected:
  We apologize for the typo. The correct initial selection rate is indeed approximately 40% for both Mistral-Base and Llama-3-Base. We will correct this in the main text.

Thank you for taking the time to review our paper and for offering detailed feedback. We address them below, following the order in which they appear across the review’s sections.

---

Methods and Evaluation Criteria (ME)

[ME-Q1] Static token selection baseline.

[ME-A1] We address this question under [QA-Q2] below, where we provide further details on the static token selection baseline.

---

Theoretical Claims (TC)

[TC-Q1] Theoretical Motivation

[TC-A1] In our response to reviewer RzSL under question [SW-Q1], we provide a detailed theoretical motivation, including mathematical derivations and sensitivity analysis demonstrating why low-confidence tokens ($p_i(\theta)$) yield higher gradient norms ($||\nabla_\theta \log p_i(\theta)||$). Due to space constraints, we kindly refer the reviewer to that section for complete details.

---

Experimental Designs or Anlayses (EA)

[EA-Q1] Evaluation on downstream tasks

[EA-A1] We have evaluated our trained models on downstream tasks involving math, truthfulness, reasoning, and coding. We find that ConfPO achieves the highest average performance across all baselines. Due to space constraints, please refer to our response to reviewer M4QG under question [ME-Q1] or at the following link: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableB.md

---

Supplementary Material (SM)

[SM-Q1] Typo in Table 7.

[SM-A1] Thank you for pointing out the typo. We have corrected Table 7, which is available at the following link: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableF.md

---

Questions for Authors (QA)

[QA-Q1] Concern about non-continuous update on coherency and performance of downstream task.

[QA-A1] Because each token is generated under a conditional distribution that considers all previously generated tokens, selectively updating some tokens while skipping others does not fragment the model’s coherence. Despite directing the preference objective toward challenging (low-confidence) tokens, the model still maintains a valid joint distribution $p(y)$ at inference time, ensuring coherent token-by-token generation. Empirically, we do not observe drops in fluency or logical consistency on bechmarks like AlpacaEval 2, Arena-Hard, and downstream tasks shown in question [EA-Q1] above. A small-scale human assessment of responses also further confirms that selectively omitting updates on high-confidence tokens does not undermine text coherency.

[QA-Q2] Static token selection baseline.

[QA-A2] We conducted an additional baseline experiment, selecting tokens based on confidence values from a fixed reference policy. Our results (see Table G) indicate that approximately 80% of tokens selected dynamically by the evolving policy model overlapped with tokens chosen by the fixed reference policy; however, the remaining 20% differed, reflecting critical tokens that changed dynamically as training progressed. While token selection via the fixed reference policy slightly improved performance over SimPO, it was noticeably inferior compared to our dynamic ConfPO method. This clearly highlights that token importance evolves as the policy model updates, underscoring the necessity of our dynamic selection strategy for achieving optimal performance.

Table G: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableG.md

[QA-Q3] Does noisy data result in degenerated performance.

[QA-A3] Thank you for raising this important concern. We tested ConfPO with two types of noise:

1. Flipped Preferences: We randomly flipped 20% of the preference labels. Even in this noisy setting, ConfPO outperformed SimPO, confirming our low-confidence token selection does not amplify label noise.

2. Word-Level Noise (EDA): We also introduced word-level noise via data augmentation methods—synonym replacement, random insertion, random swap, and random deletion. Although this aggressive augmentation substantially degraded overall performance, ConfPO still outperformed SimPO.

The detailed results are available at the following link:

Table H: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableH.md

We note that word-level noise is uncommon in recent preference-learning scenarios, which typically use on-policy data collection from advanced LLM-generated text that produces coherent sentences. Thus, randomly flipped preferences represent a more realistic noise scenario.

[QA-Q4] Applicability to on-policy RLHF, such as PPO.

[QA-A4] Our token-selection mechanism indeed applies effectively to both off-policy and partially on-policy settings, as shown. Even with highly on-policy data, our dynamic threshold maintains stable token selection ratios. Extending ConfPO explicitly to PPO-based RLHF, however, involves reward-based signals rather than implicit log-probabilities used by DAAs, and thus lies beyond our current scope but remains promising future work.

Thank you for taking the time to review our paper and for offering detailed feedback. We address them below, following the order in which they appear across the review’s sections.

---

Experimental Designs or Analyses (EA)

[CE-Q1] Experiment on newer models.

[CE-A1] We focused on the four models used by SimPO, as these models had thoroughly tuned hyperparameters for nine existing DAAs, ensuring a fair comparison.

However, following the reviewer’s suggestion, we provide additional experimental results using Qwen2.5 7B, shown in Table E below, where we see ConfPO has higher score than SimPO.

Table E: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableE.md

[CE-Q2] Evaluation on diverse tasks

[CE-A2] We have evaluated our trained models on various downstream tasks, including math and coding, and find that ConfPO achieves the highest average performance than baselines. Due to space constraints, please refer to our response to reviewer M4QG under question [ME-Q1] or at the following link: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableB.md

---

Strengths and Weaknesses (SW)

[SW-Q1] Theoretical Motivation.

[SW-A1] For theoretical clarity, we provide a first-principles analysis showing why low-confidence tokens naturally yield higher gradient norms.

Let $\pi_\theta$ be our policy model, parameterized by $\theta$. Given a prompt $x$ and a response $y =\{y_1,\dots,y_n\}$, let $p_i(\theta)=\pi_\theta(y_i|x,\,y_{<i})$be the probability of the $i$-th token. Our goal is to show that the gradient norm of the log-probability of the $i$-th token, $||\nabla_\theta\log p_i(\theta)||$, is negatively correlated with the confidence of the policy model for that token, $p_i(\theta)$.

Chain Rule and Gradient Norm

Using the chain rule, we have: $\nabla_\theta\log p_i(\theta)=\frac{\nabla_\theta p_i(\theta)}{p_i(\theta)}\implies||\nabla_\theta\log p_i(\theta)||=\frac{||\nabla_\theta p_i(\theta)||}{p_i(\theta)}.$ Thus, if token confidence $p_i(\theta)$ dominates this ratio, the gradient norm of the log probability $\nabla_\theta\log p_i(\theta)$ decreases as confidence increases, establishing the observed negative correlation.

Sensitivity Analysis

To formalize the dominance of $p_i(\theta)$ on $\nabla_\theta \log p_i(\theta)$, consider the ratio $r(\theta)=\frac{||\nabla_\theta p_i(\theta)||}{p_i(\theta)}=\frac{b(\theta)}{c(\theta)},$where $b(\theta)=||\nabla_\theta p_i(\theta)||$ and $c(\theta)=p_i(\theta)$. A local sensitivity test examines partial derivatives:$\frac{\partial r}{\partial b}=\frac{1}{c},\quad\frac{\partial r}{\partial c}=-\frac{b}{c^2}.$To determine which component more strongly influences the ratio, one can sample many tokens $i$, each yielding pairs $(b_i, c_i)$. Then, examine the main effects: $S_b(i)=\frac{1}{c_i},\quad S_c(i) =-\frac{b_i}{c_i^2}.$ Comparing $\mathbb{E}[|S_b|]$ and $\mathbb{E}[|S_c|]$ offers a practical measure of which component dominates in typical scenarios. If $\mathbb{E}[|S_c|-|S_b|]>0$, it shows that variations in $p_i(\theta)$ (the token’s confidence) have the stronger effect on $||\nabla_\theta \log p_i(\theta)||$. Using Monte Carlo estimate with 1000 samples, we indeed find that $\mathbb{E}[|S_c|-|S_b|]>0$ (The exact value can be found in https://anonymous.4open.science/r/ConfPO-B7C6/L/TableI.md). Thus, this describes our observed negative correlation between the confidence and log-probability gradient norm.

[SW-Q2] Visualization of selected tokens.

[SW-A2] In Figure C below, we visualize examples of selected tokens. In general, tokens acting as crucial inflection points within sentences (e.g., tokens initiating new phrases) are selected, while tokens that are merely continuations of preceding tokens are not selected (e.g., in the word "Slack," "Sl" is selected, whereas "ack" is not). This shows that our confidence-based selection identifies important tokens that shape the overall sentence, leading to more efficient KL-budget usage and reduced overoptimization (Figure 5 in the main text).

Figure C: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureC.md

[SW-Q3] Extension to other DAAs.

[SW-A3] In our main text, we showed how ConfPO applies to SimPO (Table 1) and DPO (Table 3). Additionally, we successfully extended ConfPO to IPO with enhanced performance (see response to reviewer M4QG, [RS-Q2]). Due to the space limit, we kindly direct the reviewer to that section or to the following link: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableD.md

[SW-Q4] Theoretical Motivation and Performance.

[SW-A4] Please refer to [SW-Q1] for the theoretical motivation. In terms of the performance, our method consistently improves on existing DAAs (SimPO, DPO, IPO) across multiple benchmarks. For instance, on Llama-3-Base(8B) for AlpacaEval2, ConfPO achieves a +7% win rate over SimPO, showing that our modified loss leads to meaningful alignment gains.

Thank you for taking the time to review our paper and for offering detailed feedback. We address them below, following the order in which they appear across the review’s sections.

---

Claims and Evidence (CE)

[CE-Q1] Computational cost comparison.

[CE-A1] Table A1 compares runtime and GPU memory usage of ConfPO with baseline DAAs (DPO, SimPO). As ConfPO requires no extra forward/backward passes or auxiliary models, its computational cost is nearly identical to the baselines.

Table A1. Runtime and peak GPU memory usage for each method under identical settings.

[CE-Q2] Robustness to hyperparameter choices.

[CE-A2] We want to clarify that we do not claim our method is “robust” to hyperparameters. Rather, in Figure 5 (where we vary the KL-divergence of resulting models by adjusting the hyperparameters), it indicates our method remains more stable when increasing the KL budget which is a key concern in reward-hacking (overoptimization) scenarios. We will revise the manuscript to avoid any confusion about hyperparameter robustness and emphasize that these experiments were specifically designed to demonstrate reduced overoptimization.

However, with regards to this, we have attached Figure A below, comparing ConfPO and SimPO across various hyperparameter settings.

Figure A: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureA.md

---

Methods and Evaluation Criteria (ME)

[ME-Q1] Evaluation on downstream tasks

[ME-A1] We have evaluated our trained models on downstream tasks involving math, QA, reasoning, and coding. We find that ConfPO achieves the highest average performance across all baselines. Due to space constraints, we summarize key results below (Table A2) and provide the full table (Table B) in the following link:

Table B: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableB.md

Table A2. Downstream task results on Llama-3-Base (8B).

---

---

Experimental Designs or Analyses (EA)

[EA-Q1] Comparison with existing token-level DAAs

[EA-A1] We include comparisons with TDPO and SparsePO in Table C below. While TDPO outperforms SimPO by leveraging token-level signals, our ConfPO still achieves higher alignment performance. Notably, TDPO requires a reference model to obtain token-level signals, incurring higher memory and compute overhead compared to ConfPO, which can identify critical tokens without a reference model. Meanwhile, SparsePO demonstrates performance comparable to SimPO.

Table C: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableC.md

---

Relation to Broader Scientific Literature (RS)

[RS-Q1] Theoretical Motivation

[RS-A1] In our response to reviewer RzSL under question [SW-Q1], we provide a detailed theoretical motivation,including mathematical derivations and sensitivity analysis demonstrating why low-confidence tokens ($p_i(\theta)$) yield higher gradient norms ($||\nabla_\theta \log p_i(\theta)||$). Due to space constraints, we kindly refer the reviewer to that section for complete details.

Furthermore, we emphasize that various DAAs utilize the joint log probability as an implicit reward, causing their policy to fundamentally depend on $\nabla_\theta \log p_i(\theta)$. Consequently, our theoretical insights naturally generalize across various DAAs.

[RS-Q2] Extension to more DAAs

[RS-A2] In our main text, we have shown ConfPO to work on both the DPO and SimPO. We have now also applied ConfPO to IPO in Table D below, supporting the general applicability of ConfPO on various DAAs.

Table D: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableD.md

---

Questions for Authors (QA)

[QA-Q1] Analysis of different thresholding strategies.

[QA-A1] In Appendix B (Table 4), we analyze various thresholding strategies (fixed, geometric, arithmetic averages) and find that the arithmetic average, as used in our main experiments, provides the best performance.

[QA-Q2] How sensitive is ConfPO to variations in other hyperparameters such as learning rate?

[QA-A2] We have included Figure B below comparing ConfPO with SimPO across different learning rates (lr). Due to SimPO's design (no explicit KL regularization), it tends to become unstable at higher learning rates. In contrast, ConfPO, by selectively updating only critical tokens, demonstrates increased stability, even at higher lr.

Figure B: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureB.md

[QA-Q3] Computational cost comparison.

[QA-A3] We have provided a detailed comparison under [CE-Q1] above.