In-context Ranking Preference Optimization

We sincerely thank Reviewer SMDr for their highly positive feedback and insightful evaluation of our work. We greatly appreciate the recognition of our paper's organization, clarity, and theoretical contributions. We're particularly encouraged by the reviewer's acknowledgment of our strong empirical results and the generalizability of the proposed approach.

Response to Weakness 2 and Question 2

We have evaluated IRPO on standard LLM tasks beyond recommendation, including generative retrieval [1–3] and question answering [4–6] (Sections 5.3 and 5.4), where ARC-Challenge also appears in AlpacaEval. These results demonstrate IRPO’s applicability to general LLM alignment benchmarks.

Response to Weakness 3 and Question 3

As mentioned in Line 37–38, unlike prior methods, IRPO aligns LLMs that generate an entire ranked list in one shot (Line 88 in Related Work). As a result, IRPO needs only a single forward pass per sample, whereas DPO, SDPO, and LiPO perform multiple passes to compute pairwise margins, yielding a substantial speedup. We report Run Time (per sample), which is averaged under consistent evaluation settings, to ensure a comprehensive and representative comparison. We show the following complexity comparison results.

Since IRPO’s only a single foward pass, its per-sample runtime is roughly 4× faster.

[1] Shen, Tao, et al. "Retrieval-augmented retrieval: Large language models are strong zero-shot retriever." Findings of the Association for Computational Linguistics ACL 2024. 2024.

[2] Li, Xiaoxi, et al. "From matching to generation: A survey on generative information retrieval." ACM Transactions on Information Systems 43.3 (2025): 1-62.

[3] Xia, Yu, et al. "Knowledge-Aware Query Expansion with Large Language Models for Textual and Relational Retrieval." Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. 2025.

[4] Li, Muzhi, et al. "Retrieval, Reasoning, Re-ranking: A Context-Enriched Framework for Knowledge Graph Completion." Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. 2025.

[5] Chen, Shijie, Bernal Jiménez Gutiérrez, and Yu Su. "Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers." arXiv preprint arXiv:2410.02642 (2024).

[6] Yue, Murong. "A survey of large language model agents for question answering." arXiv preprint arXiv:2503.19213 (2025).

[7] He, Zhankui, et al. "Large language models as zero-shot conversational recommenders." Proceedings of the 32nd ACM international conference on information and knowledge management. 2023.

[8] Hou, Yupeng, et al. "Large language models are zero-shot rankers for recommender systems." European Conference on Information Retrieval. Cham: Springer Nature Switzerland, 2024.

Response to Weakness 1 and Question 1

While prior works, such as OPO, DRPO, and LiPO, combine DPO with differentiable NDCG-style objectives, IRPO is designed for a fundamentally different and practically motivated new setting: aligning LLMs based on sparse, in-context listwise feedback, where users select only a few relevant items from LLM-generated rankings. This sparse supervision setting, distinct from explicit pairwise or fully labeled listwise feedback, arises naturally in applications such as conversational recommendation [7-8], generative retrieval [1-3], and in-context QA [4-6] (see Lines 37–38 and 88). To address this new setting, IRPO introduces several novel contributions:

• Feedback modeling: Unlike prior methods that require external supervision (for example, human-labeled rewards or pre-assigned scores), IRPO directly models partial, implicit preferences from ranked lists without relying on external reward signals.
• Objective design: IRPO integrates both item-level relevance and positional importance using a Plackett-Luce-inspired model with DCG-based discounting, resulting in a fully differentiable surrogate objective tailored for sparse, in-context feedback (Lines 38–40).
• Theoretical contributions: We provide a detailed gradient analysis showing that IRPO's optimization induces importance-weighted gradients that are unbiased and have reduced variance. This theoretical grounding, based on importance sampling, is not present in OPO or DRPO, which focus primarily on empirical performance without such guarantees.

In summary, IRPO is not just a variant of DPO. It identifies and addresses a realistic and previously underexplored setting where in-context ranked list feedback is naturally sparse and non-pairwise. To support robust optimization in this setting, IRPO introduces both modeling innovations, such as positional aggregation of pairwise preferences weighted by graded relevance, and theoretical insights, including unbiased gradient estimators linked to importance sampling with reduced variance.

In addition to responding to the reviewer's question, we also added the suggested baseline LiPO and provided the comparative results. We chose LiPO because it is a recent and representative listwise preference optimization method that explicitly extends DPO using general learning-to-rank principles. Among comparable approaches, LiPO has a more generic design that allows straightforward integration into our setup. Moreover, it serves as a meaningful contrast to IRPO: while both methods handle ranked list feedback, LiPO emphasizes listwise LTR objectives, whereas IRPO is specifically designed to model sparse in-context rankings and positional importance. This comparison helps clarify IRPO’s contributions under realistic, feedback-constrained alignment settings.

As seen above, IRPO consistently outperforms the other baselines across all benchmarks.

Response to Question 2

In our paper, we have provided some qualitative comparisons in Appendix F. To further illustrate the strengths and distinct behavior of IRPO compared to baselines, we now provide additional qualitative examples showcasing ranked lists generated by IRPO and baseline methods. Specifically, we present three representative examples demonstrating how IRPO effectively prioritizes more contextually relevant and coherent responses compared to the baselines:

Example1

Prompt: Pretend you are a question-answering model. I will give you a Question. Based on the Question, please rerank the provided answers in a list from the best to the worst without any extra words.Question: Sam wasn't lying, but he left out important details. He was being what? 1. dishonesty, 2. deceitful, 3. lying, 4. deceptive, 5. imagination, 6. more lies, 7. lie to himself, 8. deception, 9. telling lies, 10. poker face

SFT: 1. dishonesty 2. deceitful 3. lying 4. deceptive 5. imagination

DPO: 1. deceptive 2. lying 3. deceitful 4. deceptive 5. imagination

IRPO: 1. deceitful 2. dishonesty 3. more lies 4. deceptive 5. imagination 6. lie to himself 7. deception 8. telling lies 9. poker face 10. lying

Example2

Prompt: Pretend you are a question-answering model. I will give you a Question. Based on the Question, please rerank the provided answers in a list from the best to the worst without any extra words. Question: At the end of the day as he began to eat he paused and thanked her, it wasn't often she would what? 1. cook dinner, 2. get food, 3. make food, 4. did chores, 5. stretch out, 6. wife asked to, 7. when holding her hand, 8. a meal, 9. were hungry, 10. she cannot do that when all are for her

SFT: 1. wife asked to 2. when holding her hand 3. she cannot do that when all are for her 4. she would 5. cook dinner 6. get food 7. make food 8. did chores 9. stretch out 10. were hungry

DPO: 1. stretch out 2. cook dinner 3. get food 4. make food 5. did chores 6. wife asked to 7. when holding her hand 8. were hungry 9. she cannot do that when all are for her

IRPO: 1. cook dinner 2. she cannot do that when all are for her 3. wife asked to 4. get food 5. a meal 6. stretch out 7. when holding her hand 8. make food 9. were hungry 10. did chores

Example 3:

Prompt: Pretend you are a question-answering model. I will give you a Question. Based on the Question, please rerank the provided answers in a list from the best to the worst without any extra words. Question: No matter what date you put on it, we all know the universe to be what? 1. very old, 2. infinite, 3. infiniverse, 4. real, 5. universe, 6. getting younger, 7. miss universe, 8. the universe, 9. astronomy, 10. date

DPO: 1. infinite 2. astronomy 3. the universe 4. real 5. getting younger 6. universe 7. miss universe 8. date 9. infiniverse 10. very old

SFT: 1. infinite 2. astronomy 3. the universe 4. real 5. getting younger 6. miss universe 7. date 8. infiniverse 9. very old

IRPO: 1. very old 2. infinite 3. infiniverse 4. real 5. universe 6. getting younger 7. miss universe 8. the universe 9. astronomy 10. date

[1] Shen, Tao, et al. "Retrieval-augmented retrieval: Large language models are strong zero-shot retriever." Findings of the Association for Computational Linguistics ACL 2024. 2024.

[2] Li, Xiaoxi, et al. "From matching to generation: A survey on generative information retrieval." ACM Transactions on Information Systems 43.3 (2025): 1-62.

[3] Xia, Yu, et al. "Knowledge-Aware Query Expansion with Large Language Models for Textual and Relational Retrieval." Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. 2025.

[4] Li, Muzhi, et al. "Retrieval, Reasoning, Re-ranking: A Context-Enriched Framework for Knowledge Graph Completion." Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. 2025.

[5] Chen, Shijie, Bernal Jiménez Gutiérrez, and Yu Su. "Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers." arXiv preprint arXiv:2410.02642 (2024).

[6] Yue, Murong. "A survey of large language model agents for question answering." arXiv preprint arXiv:2503.19213 (2025).

[7] He, Zhankui, et al. "Large language models as zero-shot conversational recommenders." Proceedings of the 32nd ACM international conference on information and knowledge management. 2023.

[8] Hou, Yupeng, et al. "Large language models are zero-shot rankers for recommender systems." European Conference on Information Retrieval. Cham: Springer Nature Switzerland, 2024.

Response to Weakness 1,2, and Question 1

While prior works, such as OPO, DRPO, and LiPO, combine DPO with differentiable NDCG-style objectives, IRPO is designed for a fundamentally different and practically motivated new setting: aligning LLMs based on sparse, in-context listwise feedback, where users select only a few relevant items from the LLM-generated rankings. This sparse supervision setting, distinct from explicit pairwise or fully labeled listwise feedback, arises naturally in applications such as conversational recommendation [7-8], generative retrieval [1-3], and in-context QA [4-6] (see Lines 37–38 and 88).

To address this new setting, IRPO introduces several novel contributions:

• Feedback modeling: Unlike prior methods that require external supervision (for example, human-labeled rewards or pre-assigned scores), IRPO directly models partial, implicit preferences from ranked lists without relying on external reward signals.
• Objective design: IRPO integrates both item-level relevance and positional importance using a Plackett-Luce-inspired model with DCG-based discounting, resulting in a fully differentiable surrogate objective tailored for sparse, in-context feedback (Lines 38–40).
• Theoretical contributions: We provide a detailed gradient analysis showing that IRPO's optimization induces importance-weighted gradients that are unbiased and have reduced variance. This theoretical grounding, based on importance sampling, is not present in OPO or DRPO, which focus primarily on empirical performance without such guarantees.

In summary, IRPO is not just a variant of DPO. It identifies and addresses a realistic and previously underexplored setting where in-context ranked list feedback is naturally sparse and non-pairwise. To support robust optimization in this setting, IRPO introduces both modeling innovations, such as positional aggregation of pairwise preferences weighted by graded relevance, and theoretical insights, including unbiased gradient estimators linked to importance sampling with reduced variance.

In addition to responding to the reviewer's question, we also added the suggested baseline LiPO and provided the comparative results. We chose LiPO because it is a recent and representative listwise preference optimization method that explicitly extends DPO using general learning-to-rank principles. Among comparable approaches, LiPO has a more generic design that allows straightforward integration into our setup. Moreover, it serves as a meaningful contrast to IRPO: while both methods handle ranked list feedback, LiPO emphasizes listwise LTR objectives, whereas IRPO is specifically designed to model sparse in-context rankings and positional importance. This comparison helps clarify IRPO’s contributions under realistic, feedback-constrained alignment settings.

As seen above, IRPO consistently outperforms the other baselines across all benchmarks.

Response to Weakness 1 and Question 2

In our paper, we have provided some qualitative comparisons in Appendix F. To further illustrate the strengths and distinct behavior of IRPO compared to baselines, we now provide additional qualitative examples showcasing ranked lists generated by IRPO and baseline methods. Specifically, we present three representative examples demonstrating how IRPO effectively prioritizes more contextually relevant and coherent responses compared to the baselines:

Example 1

Prompt: Pretend you are a question-answering model. I will give you a Question. Based on the Question, please rerank the provided answers in a list from the best to the worst without any extra words.Question: Sam wasn't lying, but he left out important details. He was being what? 1. dishonesty, 2. deceitful, 3. lying, 4. deceptive, 5. imagination, 6. more lies, 7. lie to himself, 8. deception, 9. telling lies, 10. poker face

SFT: 1. dishonesty 2. deceitful 3. lying 4. deceptive 5. imagination

DPO: 1. deceptive 2. lying 3. deceitful 4. deceptive 5. imagination

IRPO: 1. deceitful 2. dishonesty 3. more lies 4. deceptive 5. imagination 6. lie to himself 7. deception 8. telling lies 9. poker face 10. lying

Example 2

Prompt: Pretend you are a question-answering model. I will give you a Question. Based on the Question, please rerank the provided answers in a list from the best to the worst without any extra words. Question: At the end of the day as he began to eat he paused and thanked her, it wasn't often she would what? 1. cook dinner, 2. get food, 3. make food, 4. did chores, 5. stretch out, 6. wife asked to, 7. when holding her hand, 8. a meal, 9. were hungry, 10. she cannot do that when all are for her

SFT: 1. wife asked to 2. when holding her hand 3. she cannot do that when all are for her 4. she would 5. cook dinner 6. get food 7. make food 8. did chores 9. stretch out 10. were hungry

DPO: 1. stretch out 2. cook dinner 3. get food 4. make food 5. did chores 6. wife asked to 7. when holding her hand 8. were hungry 9. she cannot do that when all are for her

IRPO: 1. cook dinner 2. she cannot do that when all are for her 3. wife asked to 4. get food 5. a meal 6. stretch out 7. when holding her hand 8. make food 9. were hungry 10. did chores

Example 3

Prompt: Pretend you are a question-answering model. I will give you a Question. Based on the Question, please rerank the provided answers in a list from the best to the worst without any extra words. Question: No matter what date you put on it, we all know the universe to be what? 1. very old, 2. infinite, 3. infiniverse, 4. real, 5. universe, 6. getting younger, 7. miss universe, 8. the universe, 9. astronomy, 10. date

DPO: 1. infinite 2. astronomy 3. the universe 4. real 5. getting younger 6. universe 7. miss universe 8. date 9. infiniverse 10. very old

SFT: 1. infinite 2. astronomy 3. the universe 4. real 5. getting younger 6. miss universe 7. date 8. infiniverse 9. very old

IRPO: 1. very old 2. infinite 3. infiniverse 4. real 5. universe 6. getting younger 7. miss universe 8. the universe 9. astronomy 10. date

Response to Weakness 2

As mentioned in Lines 37–38, unlike prior methods, IRPO aligns LLMs that generate an entire ranked list in one shot (Line 88 in Related Work). As a result, IRPO needs only a single forward pass per sample, whereas DPO, SDPO, and LiPO perform multiple passes to compute pairwise margins, yielding a substantial speedup. We report Run Time (per sample), which is averaged under consistent evaluation settings, to ensure a comprehensive and representative comparison. We show the following complexity comparison results.

Since IRPO’s only a single forward pass, its per-sample runtime is roughly 4× faster.

Response to Weakness 3 and Question 3

IRPO inherently prioritizes relative comparisons over absolute weight magnitudes, making it less sensitive to specific weighting schemes. To validate this, we performed an additional ablation study evaluating two alternative positional weighting methods:

• abl1: $w(i) = 1/\log(1 + i)$

• abl2: $w(i) = 2^{y_i} - 1/ i$

We compared these variants using the Llama3 backbone across three benchmarks (one from each category).

Inspired

MusiQue

ARC

These ablation results also demonstrate the importance of incorporating relevance into the positional weights: when the relevance term is removed in abl1, we observe a significant performance drop across all benchmarks.

Response to Question 1

As discussed in Section 5.2, we have already considered the scenario where multiple relevance labels exist per item. Regarding the design of IRPO, in Lines 38-40, we emphasized that modeling natural and flexible ranking feedback effectively requires capturing both item relevance and positional importance, of which conventional DPO methods and their underlying preference models are limited in modeling directly. IRPO integrates both item-level relevance and positional importance (via DCG-based discounting) into a unified, differentiable surrogate objective.