We appreciate your insightful feedback and believe it has significantly strengthened our manuscript. We have carefully addressed each of your comments as detailed below:

• Use of SimPO Datasets. We exclude MTBench and Arena-Hard because (1) SimPO shows minimal differences among methods on MTBench, and (2) Arena-Hard evaluation is costly ($30+ per run). In this rebuttal, we include both below using LLaMA-3 (8B) and Mistral (7B):

Llama-3-8B models

Mistral-7B models

These results show LarPO performs comparable or better than competitive baselines on these benchmarks. We will include these results in the revised manuscript.

• Computational Complexity. (1) Pairwise Ranking: Computes preference between a single pair of responses per prompt, with complexity O(1). (2) Contrastive Ranking: Involves a softmax over a candidate list of size k, with complexity O(k). In practice, good performance is achieved with a small k (e.g., with k = 4, contrastive ranking performs better alpaca_eval2 win rate than pairwise ranking), making it effective. (3) LambdaRank: Based on pairwise comparisons with position-aware weighting. Its worst-case complexity is O(k²) but can be reduced to O(k) via subsampling. (4) ListMLE: Computes the likelihood of a full ranking using sequential softmax, with O(k²) complexity. This can also be lowered to O(k) using subsampling strategies.

• Comparison to LiPO[1], DRPO[2], MPPO[3], and reward margin[4][5] papers. We would like to first emphasize that our goal is to connect IR and alignment, not to propose new listwise or hard negative methods. However, we are happy to compare our method to the five mentioned papers and add discussion accordingly in our revision. (1) LiPO [1] proposes LLM alignment as listwise learning to rank and proposes a LambdaRank-based solution. LiPO is a special case of LarPO with LambdaRank. However, we provide extensive theoretical background and demonstrate that many other ranking assumptions can be adopted in LarPO in addition to LambdaRank. (2) We compare LarPO with [2], [4], and [5] in the table below, where LarPO consistently outperforms all methods. Specifically: [2] performs well on the HH dataset (as shown in the original paper) but shows limited effectiveness on Alpaca Eval 2 and MixEval. [4] introduces a reward margin coefficient, which can misguide the LLM if the reward scores lack sufficient granularity. [5] relies on optimizing Eq. 6, but we find its training to be unstable and less robust in our experiments. (3) MPPO [3] is concurrent (posted one month before the ICML deadline), and comparison is left to future work.

• Table 2 Dataset Consistency. All baselines in Table 2 use the same UltraFeedback prompt set and PairRM reward model if needed (as in SimPO); thus, the experiments are in fair condition. To further isolate objective effectiveness, we have added results with advanced reward models (i.e., FsfairX) under the same setup, shown in the response 4 to Reviewer r2Em.

• Temperature Study (Fig 4b). We previously started from T=0.8 due to response concerns that lower temperatures might reduce response diversity, potentially leading to lower-quality preference data. In this rebuttal, we add these results, as shown in the table below:

Surprisingly, low temperatures yield strong performance—possibly because the lower the temperature is, the harder the negatives are—suggesting an interesting direction on in-depth temperature analysis for future investigation. We will add this discussion to the revised manuscript.

• Ablation studies. Section 6 includes ablations on objectives, hard negatives, and candidate construction. Our work is the first to jointly explore all these components in connecting retrieval and LLM alignment which we would think are enough contributions to the main conference.

• Reproducibility. SimPO paper checkpoints: https://huggingface.co/collections/princeton-nlp/simpo-66500741a5a066eb7d445889. We commit to releasing all code, data, and checkpoints upon acceptance.

We thank the reviewer for the helpful feedback and address the points below:

• Baselines with Strong Reward Models. We include iterative DPO with a strong reward model for fair comparison:

LarPO consistently outperforms iterative DPO under the strong reward model, demonstrating the strength of our proposed frameworks.

• Model Size Limitation. Training larger models is resource-intensive and cannot be easily handled by our current computational devices. Our current results on 2B and 7B models show LarPO’s consistent effectiveness across these model sizes and families. We leave scaling to larger models for future work.

• LLaMA-3 Results. We include LLaMA-3-8B model results below. LarPO achieves the best performance across metrics, outperforming all baselines including SimPO:

• Comparison to LiPO. LiPO proposes a LambdaRank-based listwise objective, which can be viewed as a special case of LarPO. However, their objective designs are based on learning-to-rank heuristics without a grounded theoretical basis. In contrast, LarPO is supported by theoretical foundations and generalizes to multiple ranking objectives beyond LambdaRank. We will add this distinction in the revised manuscript.

• W1. Notations in section 2 are not self-contained. We acknowledge this and will revise to ensure all notations are self-contained and clearly defined.

We sincerely thank the reviewer for the constructive feedback, which has strengthened our work. Please find our point-by-point responses below:

• Fig 2: Single dataset? Thank you for the suggestion. We plan to incorporate this in the revised manuscript (e.g., using NQ for both 2(a) and 2(b)). However, note that Figure 2 is intended to highlight the analogy between retriever and LLM scaling behaviors, not to compare them directly, so using different datasets seems acceptable. We adopt NQ and GSM8K since NQ is a standard dataset for retrieval; GSM8K captures LLM reasoning capabilities.

• Fig 2: Metrics are monotonic. We agree that Recall@N and Pass@N obviously increase with N. Our goal was to visualize similar trends in scaling behavior rather than make a strong claim. We will tone down the language in the revised manuscript.

• Fig 2: "Suboptimal" word choice. We agree and will revise to replace "suboptimal."

• Table 2: DPO baseline with strong RM. Thank you. We have now included iterative DPO with a strong reward model:

LarPO consistently outperforms iterative DPO, validating its effectiveness with the strong reward model as well.

• Sec 6.1 vs. Table 2. Section 6.1 isolates the effect of objectives under a fixed setup (e.g., hard negatives, candidate list construction) [1]. Table 2 reflects the joint effects of all factors. We’ll clarify this distinction in the revision.

• Fig 4a: Why GSM/Mathstral/iterative DPO? We use GSM8K and Mathstral because such math problems have ground-truth labels, enabling clear negative type classification. Iterative DPO ensures a fixed candidate list to isolate the impact of negative difficulty.

• Fig 4b: Temperature vs. win rate. We adjust the temperature to induce response hardness but agree other latent factors may play a role. We'll revise the explanation to focus on the importance of choosing an appropriate temperature rather than strong claims (“lower temperatures generate harder negatives”).

• Eq. 1–3: Missing document in set notation. Thank you - we’ll update the notation accordingly with your suggestion (b).

• Fig 1: Analogy only partial. Agreed. We’ll revise to clarify this is a similarity, not an identity. For later tokens, the prompt + decoded tokens act like a dynamic query in bi-encoder retrieval—but they differ fundamentally.

• L131–136: Embedding assumption. Yes, we assume vocab embeddings and LLM hidden states share the same dimension, as in most LLMs. We’ll clarify this in the revision.

• r used for both re-ranker and rule. Thank you - we’ll change the "rule" notation from r() to c() for clarity.

• L255: Fig 2b does not support the statement. Thank you for the comment. The intended intuition is that lower temperatures yield more similar generated responses, increasing overlap between positive and negative samples. This effectively makes the negatives harder. We will revise the explanation to clarify this point in the revision.

• Correlation: reranker vs. alignment. Great point. We believe a correlation may exist, but detailed exploration is beyond this paper’s scope and is an exciting direction for future work.

[1] RLHF Workflow: From Reward Modeling to Online RLHF. TMLR 2024.

We appreciate your insightful feedback, which has significantly strengthened our manuscript. We address each comment below.

• Without hard negatives, how's the performance? Thank you for the question. As shown in Section 6.2 and Figure 4(a), removing hard negatives (i.e., using only easy/easiest ones) leads to significantly worse performance. This highlights the importance of hard negatives in enhancing LLM capabilities.

• Without inclusiveness/memorization, how's the performance? Thank you for the question. We have conducted a detailed study of the inclusiveness and memorization in Figure 4(c) and Table 4. From Figure 4(c), an increase in the candidate list size can contribute to improved performance (inclusiveness). From Table 4, incorporating previous iteration responses can further enhance the LLM’s capability (memorization).

• How does the number of generated responses affect the performance? The study of the number of generated responses can be found in Figure 4(c) and also shown below. We find that increasing the number of responses improves the win rate, indicating that our method can utilize and benefit from more diverse candidates:

• How does the candidate list affect training time? We evaluate training time per iteration of LarPO (contrastive variant) with varying candidate list sizes:

We observe that larger candidate lists incur linearly higher costs but yield substantial performance gains, allowing users to balance efficiency and effectiveness trade-off.

• Algorithm 1 line 13. Thank you for pointing this out. The operation merges all responses from previous iterations. We will revise accordingly.