Learning to Rank for In-Context Example Retrieval

Thank you very much for your time and effort in reviewing our work. We would like to address some of the comments that may stem from misunderstandings. If you find our response helpful, we would sincerely appreciate it if you could consider raising your score :)

W1/W2. Clarification of our contributions:
As described in both the introduction and Figure 1, our key contribution lies in modeling the ranking among ICEs—for example, comparing and ranking different K-shot prompts. In contrast, the Active Example Selection (AES) method (citation [31]) is not closely-related to our work. AES focuses on sequentially selecting examples to construct a K-shot prompt, emphasizing the internal ordering within ICEs. Our baseline, Se2, is the most advanced follow-up to AES, following the same paradigm for intra-ICE ordering but without the need for active learning. The following shows the results of AES on related tasks (taken from Se2 [13] in paper).

Considering both relevance and performance, AES is not suitable as our baseline. We will highlight the evidence in the related work section.

Q. Clarification on inference results: We have provided inference experiments with $K$ ranging from 1 to 15, using various LLMs, as shown in Table 4. A detailed analysis of these results can be found in lines 268–276 of the paper. Our empirical study (Table 3) demonstrates that SEDPO can generalize to a larger number of examples.

Thank you for taking the time to review our paper and offer constructive feedback!

What is AWS method in the table in your response? Is that the result for AES?

We apologize for this typo, AWS method in the table is the result for AES (taken from Se2 [13] in paper).

Does this work suggest that ranking relationship between ICEs is easier to learn than predicting their absolute scores?

Yes. As we discussed in lines 22 to 28, existing methods train retrievers as classifiers using absolute scores, but inference relies on ranking, creating a goal mismatch. Learning to rank is more consistent with the test scenario, and we also provide empirical evidence in lines 223 to 230 of the paper.

Thanks for the additional results and the clarifications. I see that Se2 is a strong baseline to compare SE-DPO with.

Some additional questions:

What is AWS method in the table in your response? Is that the result for AES?

More broadly, does this work suggest that ranking relationship between ICEs is easier to learn than predicting their absolute scores?

Thanks for the clarifications. I have updated the score to 4 and added justification above.

We are grateful for your thoughtful suggestions and will incorporate additional details into the paper accordingly. If you find our responses helpful, we would sincerely appreciate it if you could consider raising your score :)

Q1. On general output and the size of $\mathcal{Y}$: If the general output refers to open-ended QA, the set $\mathcal{Y}$ becomes impractically large, making Equation (2) intractable to compute. We have experimented with sampling $y$ values for $\mathcal{Y}$ directly from the model; however, the results were unsatisfactory. Unlike MCQs, open-ended QA lacks a clear way to quantify and normalize the quality gap between good and bad answers, making reliable supervision difficult.

Q2. Proof of transitivity:
By the definition of $\succ$ (lines 117-118):

• If $e^a \succ e^b|x$, then for all $y \in {\mathcal{Y}}\_{\text{gt}}$ , $\mathrm{S}\_\mathrm{MCQ}(e^a \oplus x,y) > \mathrm{S}\_\mathrm{MCQ}(e^b \oplus x,y)$. (1)
• If $e^b \succ e^c|x$, then for all $y \in \mathcal{Y}\_{\text{gt}}$, $\mathrm{S}\_\mathrm{MCQ}(e^b \oplus x,y) > \mathrm{S}\_\mathrm{MCQ}(e^c \oplus x,y)$. (2)

Note that $\mathrm{S}\_\mathrm{MCQ}(\cdot)$ is a scalar-valued function, and its outputs are real numbers. The ">" relation on the real numbers is transitive: for any real numbers $a, b, c$, if $a > b$ and $b > c$, then $a > c$.

Applying this transitivity to (1) and (2) for each $y \in \mathcal{Y}_{\text{gt}}$:
For all $y \in \mathcal{Y}\_{\text{gt}}$, $\mathrm{S}\_\mathrm{MCQ}(e^a \oplus x,y) > \mathrm{S}\_\mathrm{MCQ}(e^b \oplus x,y)$ and $\mathrm{S}\_\mathrm{MCQ}(e^b \oplus x,y) > \mathrm{S}\_\mathrm{MCQ}(e^c \oplus x,y)$ implies $\mathrm{S}\_\mathrm{MCQ}(e^a \oplus x,y) > \mathrm{S}\_\mathrm{MCQ}(e^c \oplus x,y)$. (3)

By the definition of $\succ$ again, (3) implies $e^a \succ e^c|x$.
Thus, the relation $\succ$ is transitive.
Q.E.D.

Q3. Cost of constructing training data: We provide below the average cost of constructing preference data for all tasks. Experiments were conducted on 8 V100-32GB GPUs with a scoring batch size of 10, using GPT-Neo-2.7B as the ICL model. For each $x$, we only sample $T = 20$ preference pairs. This means the number of processed preference pairs is less than the number of scored entries. For instance, on NLI, the total time for constructing scored data is 794× greater than that for preference data:

Q4. We provide below the extra ablation results on Paraphrase for different $T$, following the setup of Table 2.

As T increases, the performance of SEDPO improves because a broader preference ranking is learned. However, when T is too large, the performance gain decreases due to more low-confidence LLM rankings. As we have discussed this in lines 131-136 and lines 241-243, low-confidence LLM rankings can negatively affect learning --- there is a trade-off.

Minor/Typos. Thank you for your careful reading. We will thoroughly correct all typos and minor issues in the revised version.

Thank you very much for your time and effort in reviewing our work. We sincerely appreciate your thoughtful comments and would like to address the concerns raised. If you find our responses helpful, we would be truly grateful if you could consider adjusting your score accordingly :)

W1. Clarifying our contributions. We believe there may be some misunderstandings regarding the uniqueness of our work. As described in both the introduction and Figure 1, our study focuses on the ranking formulation among ICEs (i.e., different $K$-shot prompts), which is a central contribution highlighted in both the introduction and Figure 1. In contrast, prior works primarily adopt classification-based approaches:

• PromptPG focuses on classification-based retrieval and does not explicitly investigate the internal or external ordering of ICEs. While it notes that ordering can affect model performance, this is mentioned only in the analysis section and not as a core research objective.
• LLM-R similarly does not study the preference order among ICEs. The “ranking” mentioned in its analysis refers to a screening mechanism within a classification formulation.
• UDR introduces a local ranking regularizer into the classification formulation, but still fundamentally operates in a classification framework. We have provided a detailed analysis of UDR’s limitations in lines 68, 203–205, and 218–220, along with supporting experiments in Tables 1 and 2.

W2. Clarification on Table 4 and additional experiments. Table 4 in our paper aims to demonstrate that the performance improvement of our proposed method becomes more pronounced as the model size decreases (lines 274–276). This experiment is designed to explore the boundary of transferability and to help identify suitable application scenarios for SEDPO. We would like to respectfully note that smaller models remain a highly active research area. To further support this point, we provide below an additional evaluation on GPT-2 XL (1.5B), following the same setup as in Table 3:

SEDPO outperforms the second best by up to 5.46%, which demonstrates better transferability on smaller models. We hope this additional evidence further supports the generalizability and effectiveness of our method. If space allows, we’ll add more small-LLM results to Table 4.

W3. Kindly refer to our responses to Q1 and Q2 for clarifications.

W4. Transferability on shot number and intra-ICE ordering. Table 4 presents the transferability of different methods across 1 to 15-shot settings. While our primary focus is on the ordering between ICEs, we also analyze the effect of intra-ICE ordering through additional experiments. This analysis is provided in Table 3 and discussed in lines 259–262.

Q1. Greedy sampling strategy (Section 3.2) We appreciate your attention to the selection process for constructing $K$-shot prompts $e_{[1:K]}$ for training. As described in Section 3.2, considering all non-repetitive combinations is computationally infeasible (line 122). Therefore, we use a greedy sampling approach guided by Eq. (3) to select each example $e_c$ within $e_{[1:K]}$. When selecting $e_1$, we randomly sample $L$ examples (Eq. (2)), rank them using the LLM score, and then sample from these candidates based on the probabilities given in Eq. (3). This approach, which takes into account the importance and diversity of examples, is expected to be more effective than uniform sampling, as cited in line 123.

Q2. Clarifying $Z(x)$ in Eq. (5). Thank you for raising this point. The partition function $Z(x)$ is introduced in Eq. (5), and reference of DPO [4]—which provides a detailed explanation—is cited in the paragraph preceding the formula. To enhance clarity for readers, we will consider adding more explicit references near the $Z(x)$ itself.

Thank you for taking the time to read our manuscript and for your feedback. We’re glad that several of our clarifications addressed your concerns. Below, we respond in detail to your remaining points.

Reviewer’s concern: The contribution related to the ranking component seems limited compared to prior work.

Our contribution lies in modeling retriever training as a ranking problem rather than a classification problem, which is consistent with the test scenario (lines 22-28). Prior work models retriever training as a classification problem, and the training objectives of the two are completely different.

Moreover, the prior ranking within prompts and our ranking between prompts are two distinct concepts:

• Between prompts aims to determine which information is more useful, focusing on the different utility provided by different prompts.
• Within prompts adjusts the input order between selected examples, with more focus on the correct dependencies of the same information.

We sincerely hope that the reviewer can point out the specific limitations of our ranking component compared to prior work. We have carefully examined the papers you provided, and PromptPG and LLM-R are not ranking-based ICL methods; UDR is already our baseline, and we noted in the paper that it trains the retriever by learning classification rather than directly learning to rank.

Reviewer’s concern: With large LLMs already achieving strong zero-shot results, it is more important to focus on improvements for larger LLMs

We acknowledge the zero-shot capabilities of larger LLMs, but few-shot learning can also benefits large models compared to the zeroshot. The decline in gains we are discussing is relative to the second-performed retriever, not the zero-shot (lines 274 - 276). We present below a comparison between our method and the zero-shot results on a larger model, where the gains remain significant. | Model | 0-shot | SE-DPO (1-shot) | SE-DPO (3-shot) | SE-DPO (6-shot) | SE-DPO (9-shot) | SE-DPO (12-shot) | SE-DPO (15-shot) | |---|---|---|---|---|---|---|---| | Llama3-8B-Instruct | 56.4 | 71.9 (+15.5%) | 77.4 (+21%) | 78.5 (+22.1%) | 79.3 (+22.9%) | 80.2 (+23.8%) | 80.3 (+23.9%) | | Llama3.3-70B | 67.6 | 78.6 (+11.0%) | 81.0 (+13.4%) | 82.3 (+14.7%) | 82.9 (+15.3%) | 83.2 (+15.6%) | 83.2 (+15.6%) |

In discussion with Reviewer Cqhk, we also explored retraining our retriever under the supervision of rankings from the larger LLM to further make our method outperform the baseline retriever by a large margin.

Finally, the field of LLMs has not yet reached a consensus on focusing solely on improving zero-shot performance by scaling model size, given that the deployment and inference costs of large LLMs are high, and agent-based features such as deep research also rely on RAG systems. While different scholars adhere to different propositions, we believe these debates should not undermine the validity of our current contributions.

Please let us know if any phrasing still seems unclear or if you’d like additional evidence or discussion on any of these points. Thank you again for your feedback!

Thank you very much for your constructive feedback. We sincerely appreciate your time and effort in reviewing our work and would like to address all of your concerns. If you find our responses helpful, we would be truly grateful if you could consider adjusting your rating accordingly :)

W1/W2. Fairness, overhead, and discussion clarity. We have discussed several limitations of our method—including fairness—in the appendix. Since our algorithm does not introduce inference-time overhead, the additional computational cost remains affordable. We will revise the conclusion section to better highlight these points.

Q1: Text encoder fairness and ablation results (lines 183–188). As described in the paper, both SEDPO and all trainable baselines use BERT-base-uncased to ensure fair comparison (lines 183–188, Table 2). To further show the influence of the embedding model, we conducted additional ablation experiments with RoBERTa-base:

RoBERTa-base is known to outperform BERT-base-uncased across a range of NLP tasks, and as shown above, SEDPO benefits from stronger embeddings as expected. We will include these results in Table 2.

Q2: Robustness to noisy preferences (lines 134 and 242, Table 2). As shown in Table 2 and discussed in lines 134 and 242, replacing our original preference data with randomly sampled preference pairs of the same size significantly degrades performance. This effect is more pronounced than the degradation seen in cases of close-score (low-confidence) preferences. These results suggest that SEDPO is sensitive to preference noise, and that low-confidence LLM rankings can negatively affect learning. We will revise the relevant sections to make this insight clearer and more prominent.