VPO: Reasoning Preferences Optimization Based on $\mathcal{V}$-Usable Information

We sincerely appreciate your time and valuable feedback on our research.

Question 1:

Thank you for your detailed feedback.

A series of studies [1-4] have confirmed that in reasoning tasks, the log probabilities of preference samples tend to decline, which may even degrade model performance. Specifically, in complex reasoning scenarios, correctness-based preference labeling fails to fully capture the fine-grained quality of model outputs—particularly for questions with definitive answers, where preference labeling often relies solely on answer correctness. Moreover, reasoning chains contain critical tokens that significantly influence the model's reasoning trajectory, yet current labeling paradigms overlook these key elements [2].

These limitations in preference labeling for reasoning tasks may cause DPO to excessively penalize rejected samples containing partially correct reasoning chains during training, leading to probability declines in preference data with similar reasoning chains. [3] also demonstrated that when reasoning preference data is similar, preference optimization failure and probability decline can occur.

On the other hand, longer outputs in reasoning tasks make the distribution of preference sample pairs more un-uniform and harder to analyze in the output space [5], potentially leading to more pronounced "squeezing effect" during model optimization.

Question 2:

Thank you for your valuable feedback.

The "valley" region indicates that the model's outputs are trapped in low-probability areas. As analyzed in [6], DPO places greater emphasis on avoiding the generation of non-preferred samples, which causes its log probabilities to decline more rapidly and consequently makes it more prone to falling into the "valley" region.

[1] Pang, Richard Yuanzhe, et al. "Iterative reasoning preference optimization." Advances in Neural Information Processing Systems 37, NIPS 2024.

[2] Lin, Zicheng, et al. "Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability."

[3] Pal, Arka, et al. "Smaug: Fixing failure modes of preference optimisation with dpo-positive."

[4] Liu, Zhihan, et al. "Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer." Advances in Neural Information Processing Systems 37,NIPS 2024.

[5] Ren, Yi, and Danica J. Sutherland. "Learning dynamics of llm finetuning." The Thirteenth International Conference on Learning Representations. ICLR 2025

[6] Feng, Duanyu, et al. "Towards analyzing and understanding the limitations of dpo: A theoretical perspective, 2024."

Weaknesses 1:

We sincerely appreciate your constructive suggestions regarding the weighting scheme, which will further refine our experiments.

In Section 2.3, we investigate the selection principles of different weighting methods, specifically examining how to measure the correlation between Chosen/Rejected pairs in reasoning tasks from both token and information perspectives. Reasoning tasks exhibit prefix similarity and solution path diversity. Since token-based similarity metrics (such as Cosine similarity, Levenshtein edit distance, and Jaccard similarity) are sensitive to variations like word overlap, formal similarity, and surface semantics, they may introduce bias in evaluating the relevance of sample pairs. This makes it difficult for them to accurately assess the similarity between sample pairs in reasoning tasks (particularly when certain reasoning paths contain reflective terms or involve multiple thought processes).

Therefore, we employ V-usable information [1-2] to measure the informational correlation between chosen and rejected responses, as it captures deeper semantic relationships and remains unaffected by surface-level variations. To compare with other similarity-based weighting schemes, we replace the weighting scheme v with the cosine similarity between Chosen/Rejected pairs. We evaluate the performance of different preference optimization algorithms on Qwen3-14B-Base, keeping the experimental setup consistent with Section 3. The experimental results are presented in the table below.

The experimental results and log-probability variation curves demonstrate that V-usable information can more effectively mitigate the "squeezing effect". We will conduct more comprehensive experiments and analysis on the weighting scheme in the revised version.

Weaknesses 2:

Thank you for your constructive feedback.

To better understand the differences between these algorithms, we will update the KL curves of various models under different preference optimization methods in the revised version and provide a more detailed analysis.

Questions 1:

Thank you for your note regarding term usage.

In preference optimization, the terms "preference/non-preference" and "chosen/rejected" carry equivalent meanings. We will elaborate on and update this terminology in the revised version to ensure clarity.

Questions 2:

Thank you for your meaningful feedback.

There is a certain positive correlation between the improvement of model performance and maintaining the probability of chosen examples. Our analysis in Section 2.2 and the discussion of the "squeezing effect" in [3] demonstrate that "squeezing effect" causes both chosen and rejected probabilities to decrease simultaneously during DPO training, while the probabilities of certain tokens unrelated to the chosen/rejected pairs become elevated due to this "squeezing". This phenomenon ultimately leads to unstable model outputs and performance degradation.

Maintaining or even increasing the probability of chosen samples can help alleviate the issue of probability mass shifting to irrelevant tokens during DPO training. Our study achieves this by dynamically constraining negative gradients, which effectively preserves chosen sample probabilities, reduces the generation probability of irrelevant tokens, while maintaining the discriminative power between chosen and rejected samples.

Questions 3:

Thank you for your feedback regarding the hyperparameters.

Both IPO and RPO introduce an additional hyperparameter. In the table below, we present the performance variations of IPO and RPO under different hyperparameter settings.

Question 4:

Thank you for your valuable feedback on the analysis of our experimental results, which will make our experimental analysis more comprehensive. We will update the log-probability comparison curves with other baselines in the revised version.

[1] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information, The Thirty-Ninth International Conference on Machine Learning, ICML 2022.

[2] Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. The Eighth International Conference on Learning Representations ICLR 2020.

[3] Ren, Yi, and Danica J. Sutherland. "Learning dynamics of llm finetuning." The Thirteenth International Conference on Learning Representations. ICLR 2025

We are very pleased that our rebuttal has addressed your concerns. We also greatly appreciate your constructive feedback.

V-usable information, as an extension of mutual information oriented toward the effectiveness of prediction tasks, has been proven to be more effective than mutual information in representation learning [1]. A series of extended metrics for mutual information (such as MINE [2] and Mixed-KSG [3]) exhibit a positive correlation with V-usable information, as shown in Figure 1 of [1].

In addition to these mutual information extension methods, we believe that information-theoretic distribution similarity measures closely related to mutual information—particularly KL divergence and cross-entropy, may also exhibit strong positive correlations with V-usable information. However, their application is limited by computational challenges arising from length mismatches when comparing variable-length sequences.

Model-based similarity metrics using internal states—such as those based on hidden states or attention scores—can capture similarity through learned representations, though this requires significant computational overhead. Additionally, we note several similarity optimization methods based on LLM outputs [4-6], which transform semantic textual similarity into a text generation problem. In future work, we will explore the correlation between these methods and V-usable information, as well as their performance when used as factors for preference optimization.

[1] Xu, Yilun, et al. "A theory of usable information under computational constraints." ICLR 2020.

[2] Ishmael Belghazi, Mohamed, et al. "MINE: mutual information neural estimation." ICML 2018.

[3] Gao, Weihao, et al. "Estimating mutual information for discrete-continuous mixtures." NIPS 2017.

[4] Ravfogel, Shauli, et al. "Description-based text similarity." COLM 2024.

[5] Gatto, Joseph, et al. "Text encoders lack knowledge: Leveraging generative llms for domain-specific semantic textual similarity." EMNLP2023.

[6] Xu, Shaochen, et al. "Reasoning before comparison: LLM-enhanced semantic similarity metrics for domain specialized text analysis."

Weaknesses:

We thank you for the constructive feedback about this paper.

In Figure 3, we present a performance comparison between VPO and other preference optimization algorithms on ARC-C, which includes: Definition, Basic Facts & Properties, Structure, Processes & Causal, Teleology / Purpose, among others. Some of these domains involve less structured intermediate steps in reasoning (Problems in Definition-type tasks, such as "What is a worldwide increase in temperature called?"). The results in Figure 3 demonstrate that VPO maintains a strong performance improvement even on commonsense tasks beyond mathematical tasks.

Additionally, we perform preference optimization on the Qwen3-14B model using the ScienceQA training set. ScienceQA spans 12 grade levels and features diverse input sources, covering multi-domain data such as History, Writing Strategies, Vocabulary, Biology, Civics, Physics, and Geography. For each question, we sample 5 responses, constructing a total of 7K preference sample pairs for Qwen3-14B. The experimental results on the ScienceQA test set are shown in the table below.

The above results demonstrate that VPO can still achieve significant performance improvements across multi-domain tasks—even those with less structured intermediate steps. We will include additional experiments and analyses on these tasks in the revised version.

Questions:

Thank you for your valuable feedback regarding the computational cost of V-usable information.

In fact, computing V-usable information only requires a single forward pass on the rejected sample $\theta(y|x,c_l)$ and $\theta(y|x)$ . To further reduce computational overhead, we can concatenate the Reject sample $(x,c_l, y)$ with the Question-Answer sample $(x,y)$ for joint processing. All V-usable information calculations in the paper can be completed within 30 minutes on a single H20 GPU. Additionally, several proxy methods [1] (capable of achieving up to 30x speedup) have been proposed and could be extended to compute V-usable information, enabling more efficient large-scale applications. We will continue to explore optimizations for reducing the computational overhead of V-usable information in future work.

Limitations:

Thank you for your constructive feedback on scalability.

Our experiments on ARC-C and ScienceQA investigate VPO's performance across multi-domain tasks with less structured reasoning chains. Experimental results validate VPO's scalability on general tasks. We will further investigate VPO's performance on less structured data in future research.

[1] Liu, Fengyuan, Nikhil Kandpal, and Colin Raffel. "AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution." The Thirteenth International Conference on Learning Representations. ICLR 2025

Weaknesses 1:

We thank you for the insightful comments and constructive feedback about this paper.

The study in [1] performed off-policy DPO training on the Pythia/Qwen1.5 model using the UltraFeedback and Anthropic-HH datasets. These datasets were specifically curated to enhance the model's performance in instruction following, truthfulness, honesty, and helpfulness, encompassing several relevant task categories including: World Knowledge Question Answering, Creative Writing and Content Generation,Document-based Assistance. Experimental results (Figure 4) of [1] demonstrate significant "squeezing effect" across these tasks.

Our new experiments with Qwen3-14B using UltraFeedback data revealed consistent behavioral patterns. Overall, the "squeezing effect" of DPO on UltraFeedback tasks show no significant difference compared to reasoning tasks (we will provide further discussion in the revised version). Due to the absence of ground-truth annotations in the UltraFeedback dataset, calculation of V-usable information remains infeasible. In future work, we will conduct an in-depth investigation of mitigation strategies for "squeezing effect" in non-reasoning scenarios.

Weaknesses 2:

Thank you for your valuable feedback.

We have conducted a comprehensive comparison of various preference optimization methods on the Qwen3-14B-Base model, maintaining identical experimental settings to those described in Section 3 of our paper. The results are presented in the following table.

The experimental results demonstrate that VPO can maintain good performance improvements on larger models.

Questions 1:

Thank you for your valuable feedback.

We did not evaluate Qwen 2.5 and LLaMA-3.1-8B on the AIME benchmark due to the limited problem set in AIME24 and the fact that the baseline LLM, LLaMA-3.1-8B model without additional fine-tuning struggles to achieve meaningful performance improvements on AIME tasks. The experimental results from our Qwen3-14B evaluation on AIME (presented in the preceding table) confirm that VPO outperforms other preference optimization approaches.

[1] Ren, Yi, and Danica J. Sutherland. "Learning dynamics of llm finetuning." The Thirteenth International Conference on Learning Representations. ICLR 2025