Beyond Content Relevance: Evaluating Instruction Following in Retrieval Models

The proposed data is incremental compared to existing data. For example, InstructIR includes the dimension of Audience, and this paper essentially reuses their data generation pipeline, adding more dimensions.

While there are some similarities between our data generation pipeline and that of InstructIR, there are also significant differences. InstructIR relies heavily on GPT-generated queries and documents. In contrast, our approach emphasizes preserving the naturalness of the dataset by selecting suitable Q-A pairs from existing datasets and then rewriting the documents to better align with our objectives. Additionally, our instructions are carefully designed to reflect real-world user scenarios, ensuring the dataset is more practical and applicable. These differences make our pipeline distinct and tailored to address specific needs.

Similar to InstructIR, both the instruction and the positive examples are generated by GPT-4. Although the authors mention they manually reviewed the quality, I am still concerned about the data’s reliability: the positive document is rewritten by GPT-4 to align with the instructions. However, GPT-4 itself serves as a baseline model with only moderate performance in determining whether a document follows an instruction.

Thank you for raising this important point regarding the reliability of GPT-4-generated data and its impact on our experiments. While it is true that both the instructions and positive documents are rewritten by GPT-4, our pipeline incorporates additional steps to enhance the dataset's robustness. Specifically, beyond generating positive documents, we also create corresponding hard negatives using GPT-4, ensuring a challenging setup for downstream models. These hard negatives are designed to closely resemble the positive examples while deliberately introducing inconsistencies or irrelevant content, making the task of instruction-following evaluation more rigorous.

Furthermore, to mitigate potential biases or limitations in GPT-4, we manually reviewed and curated the generated data to ensure alignment with the instructions and naturalness of the text. This process provides an additional layer of quality assurance.

The proposed metric assumes there are n positive documents under the original mode, with only one remaining positive under the instruction mode. However, this assumption does not always hold in real-world tasks, which limits this metric’s applicability to their dataset alone. For instance, we might have multiple positives under the instruction mode or find that some positives are not strictly positive under the default mode.

We acknowledge that the proposed metric assumes a specific setup where there is one positive document under the instruction-following mode, which may not always align with real-world tasks where multiple positives or varying definitions of positivity could exist. However, it is important to note that our dataset and metric are designed for offline evaluation, focusing specifically on instruction-following scenarios.

Existing datasets and metrics primarily target semantic relevance, but there is currently no standardized benchmark for evaluating instruction-following capabilities. The goal of our work is to bridge this gap by introducing a dataset and metric tailored to this underexplored aspect.

That said, we recognize the importance of extending our approach to better align with real-world tasks where instruction-following systems might operate in more dynamic settings. As part of our future work, we plan to explore how our metric and dataset can be adapted to accommodate such scenarios, including cases with multiple positives or nuanced definitions of relevance.

SICR uses relevance scores, but listwise models do not output relevance scores. How do you define the score for these models?

Yes, the listwise models do not output relevance scores. So we assign relevance scores to each document based on the rank (e.g., rank 1: relevance score = 1/1; rank 2: relevance score = 1/2; rank 3 relevance score = 1/3, and so on). However, it does not matter how to assign different relevance scores to documents of different rank, as long as the larger rank, the smaller the relevance score. In essence, SICR is designed based on the rank.

Dear Reviewer,

Happy Thanksgiving! We sincerely appreciate your constructive feedback and the time you took to review our paper. We hope our responses to your comments have adequately addressed your concerns and questions. If you have any further feedback or additional points that you would like us to clarify, it would be extremely valuable for improving this study. We look forward to hearing from you.

We look forward to your further feedback.

Best regards,

Author

Thank you for your response! We appreciate your insightful comments and will address each of your concerns in detail below.

“the data reflects real-world user scenarios.” While the new data improves upon InstructIR by using more data sources, it is still artificially constructed and does not include instructions from real users.

We sourced our data from several well-established and diverse datasets to ensure broad coverage.The queries in MSMARCO are sourced from real user searches on Bing, while datasets like Stack Overflow and Stack Exchange also consist of questions posed by real users in real-world scenarios. Here are existing datasets we used.

• Audience: BioASQ, scifact
• Keyword: MSMARCO
• Format: Stackoverflow, Stack Exchange
• Language: publichealth-qa
• Length: medical qa
• Source: CNN-english-news

While we recognize the overlap in intent, we want to clarify the distinct purpose and methodology behind InfoSearch. Unlike InstructIR, which pools heterogeneous queries together, our approach is deliberately structured around six distinct dimensions: Audience, Keyword, Format, Language, Length, and Source. This structured design allows us to systematically evaluate retrieval models by analyzing not only the dimensions where they excel but also their ability to follow specific instructions accurately and consistently.

The most significant distinction lies in InfoSearch's focus on structured, multi-faceted evaluation. Each dimension represents a concrete document attribute, and we assess models across original, instructed, and reversely instructed modes, simulating real-world user preferences. This approach enables us to explore instruction-following capabilities beyond content relevance by incorporating document-level attributes like format and language, which are often overlooked in prior work.

This paper attempts to use LLMs to label more (instructions, positive samples, and negative samples), but it does not provide sufficient validation.

Document Construction

• Audience: The Q-A pairs we collected were originally designed for [expert]. Using LLMs, we rewrote these documents to align with the [layman] condition, simplifying technical jargon while maintaining content relevance.
• Keyword: Over 90% of the documents were sourced directly from MSMARCO. For certain documents, we used LLMs to refine and rewrite them to better match the instructions.
• Format: All positive documents were extracted from StackOverflow posts and linked resources. No LLM generation was involved here.
• Language: Documents were directly sourced from publichealth-qa.
• Length: [paragraph] documents were derived from medical_qa. LLMs were employed to rewrite these documents to fit [sentence] and [article] conditions.
• Source: [news] documents were obtained from CNN-english-news, and LLMs were used to transform them into [blog] or [forum] formats.

Hard Negative Documents

All hard negative documents were generated by LLMs to appear contextually similar to the query but deliberately irrelevant.

Instruction Construction

LLMs were used to expand predefined conditions (e.g., [sentence], [paragraph], [article]) into diverse, user-friendly instructions. To validate the quality of the dataset, we implemented the following multi-step process:

1. Model-Assisted Coarse Validation

We used 12 retrieval models (e.g., bge, e5-mistral, NV-Embed) to perform a coarse-grained validation of the generated positive documents. Documents were flagged for review if they met either of the following criteria:

• Ranked below 50 by six or more models.
• Received a relevance score below 0.5 from six or more models.
• These flagged documents underwent a further manual review to confirm whether they were appropriate as positive samples for the associated queries.

We applied the same model-assisted and manual review process to hard negative documents. This ensured that the documents appeared contextually similar to the query while being definitively irrelevant, maintaining the robustness of our dataset.

2. Annotator Validation

The annotations were reviewed by graduate students specializing in computer science, all of whom passed an English proficiency test to ensure accurate task comprehension. We achieved an inter-annotator agreement score of 92%, reflecting high consistency among annotators.

3. Instruction Check

• For five dimensions (excluding Keyword), instructions were generated based on predefined conditions and underwent manual review. Due to their straightforward expansion process, the accuracy of these instructions after review was 99%.
• For the Keyword dimension, instructions were generated by extracting document keywords, with an error rate of 3% in a sample of 20 queries. Errors were iteratively corrected using LLMs, and a recent re-check confirmed that all positive documents contain the specified keywords after adding instructions.

It would be great if the authors could introduce more on the motivation and application scenario for instruction following retrieval.

We sincerely thank the reviewer for their insightful comments and suggestions.

1.Motivation and Application Scenario for Instruction-Following Retrieval

Our proposed dimensions address the complexity of user instructions by influencing not just specific parts of the retrieved content but its overall structure and style. This requires retrieval models to achieve deep semantic understanding and contextual adaptation, going beyond traditional relevance-based approaches to better align with nuanced user needs.

For instance, when a user specifies "Source" as news articles, they are interested in more than just neutral facts—they seek to understand how the media frames or evaluates a particular topic. This involves the biases, perspectives, and storytelling styles unique to news reporting. The user's assessment of the retrieved content will be influenced by the logical structure, language style, and headline formatting typical of journalism. By accurately interpreting and prioritizing the "Source" dimension, the retrieval model can provide more authentic and need-specific information that closely aligns with the user's intent. This not only enhances the effectiveness of the retrieval process but also pushes the model beyond semantic understanding toward a deeper comprehension of user needs.

2.Necessity of Advanced Retrieval Models

Although query rewriting with filters can achieve some level of customization, it often falls short when faced with complex, multidimensional instructions. For example: When length constraints are specified (e.g., "a brief summary of the history of machine learning" vs. "a detailed timeline"), the model needs to understand the granularity of the content within the retrieval process itself, beyond what query rewriting can achieve.

This example highlights the necessity of advanced retrieval models capable of embedding instruction-following behavior directly into their architecture, ensuring the seamless handling of diverse and nuanced requirements. In summary, our response highlights the importance of addressing multidimensional instructions in retrieval tasks and the limitations of existing approaches. By proposing advanced retrieval models and focusing on nuanced user needs, we aim to bridge the gap between query understanding and truly instruction-aware retrieval systems.

Focusing only on the document-level instructions is not enough for evaluating instruction-followed retrieval systems. Even for document-level instructions, some additional dimensions such as temporal and location, should also be considered.

Thank you for pointing out this important aspect. We agree that focusing solely on document-level instructions is not sufficient for a comprehensive evaluation of instruction-following retrieval systems. While our current benchmark serves as a foundational step, we acknowledge the value of incorporating additional dimensions, such as temporal ,location-specific and other aspects. Expanding the benchmark to include these and potentially other dimensions is a key part of our future work, and we are committed to continuously improving its coverage and utility.

For the experiments, I would expect to add some customized filters (such as language requirement) on existing retrieval models.

We acknowledge the importance of incorporating customized filters (such as language requirements) into existing retrieval models. Such extensions would enhance the applicability and practicality of the models. As our current work primarily focuses on validating the core model and evaluating its performance, these customized features were not included in the experiments. However, we will seriously consider integrating these filters into our experimental design in future work.

It is unfair to compare retrieval models with reranking models. I think the most challenging part is the retrieval part, which may fail to retrieve relevant results without good instruction understanding.

Thank you for your valuable feedback. We would like to clarify that this paper focuses on offline evaluation rather than online retrieval. Specifically, we address the challenge by setting carefully designed instructions and reverse instructions for each query, ensuring that the evaluation is conducted in a controlled and consistent manner. This setup allows us to isolate and analyze the effectiveness of the methods in reranking without the variability introduced by online retrieval performance.

What is the underlying retrieval model for the reranking models in Table 3? (How the initial results are prepared?)

For reranking models, we use the top 100 results from E5-mistral for reranking.

Thank you for your thoughtful feedback. We appreciate your continued consideration of our work. Regarding your concern about the dataset limitations, we would like to offer the following responses:

1. This study primarily aims to fill the gap in benchmarks for evaluating instruction-following. Document-level instructions are particularly effective due to their clarity and minimal ambiguity, making them ideal for this purpose.
2. While we acknowledge the absence of non-document-level instructions, we believe this does not diminish the completeness or value of this study. Our benchmark fills an important need by offering a unified platform for instruction-following evaluation, which can be extended in future iterations. That said, we would greatly appreciate any specific suggestions or examples of non-document-level instructions you have in mind. Broadening the scope to include such instructions is a natural direction for future development.
3. We also value your suggestion to include dimensions like temporal or location-specific attributes. These attributes are general and less ambiguous, and align well with our goal of creating a practical and robust benchmark. We will explore these ideas in future updates to enhance the framework.

Thank you again for your insightful comments and guidance, which will undoubtedly help us improve this work further.

Sec 2.3 Evaluation Metric, the motivation for designing WISe and SICR is not clear. In L181 - L183, weakness of Robustness@k is not clearly explained.

In our revised version paper, we explain it. The Robustness@k metric oversimplifies the evaluation of a model's ability to follow instructions.

1. Even if a model demonstrates strong performance across the majority of queries, a single anomalously low score can reduce the overall robustness score.

2. Furthermore, focusing solely on the lowest score disregards variations in the model's responses to different instructions, thus failing to capture the overall performance trend. For instance, the nDCG@k scores for group A are {0.8, 0.5, 0.3, 0.2}, while those for group B are {0.9, 0.9, 0.9, 0.2}. Although group B exhibits a significantly better overall performance, Robustness@k assigns the same score to both groups.

Sec 2.3 Evaluation Metric, since you are comparing with your metrics with pMRR and Robustness@k, it's good to have a comparison table showing your metric is better in handling cases that pMRR and Robustness@k fail, in order to convince your audience that your metrics are effective.

In our revised version paper, we explain it. P-MRR fails to distinguish the importance of ranking, neglecting to highlight the critical role that top K results play in retrieval.

1. Moreover, the linear discount mechanism of P-MRR is insufficiently sensitive to changes in higher rankings, making it ineffective in capturing subtle movements at the top.

2. Lastly, P-MRR demonstrates limitations when addressing special cases and extreme performances. For example, the performance of model 1 is R_{og} = 10 and R_{new} = 5, yielding a P-MRR of -0.5, while model 2’s performance is R_{og} = 100 and R_{new} = 50, resulting in a P-MRR of -0.5. Although both models receive the same score, it is evident that model 1 has a greater impact on the retrieval results. And in the appendix, we We provide a heatmap.

In eq (1), F(q)=fpenalty(Rori,Rins) if ¬Rins<Rori<Rrev; whereas, eq (3) only gives Rori<Rins. More clarifications will be helpful.

In our revised version of the paper, we have added further clarifications to address this concern. Additionally, we have addressed other issues in the paper, such as corrections to Table 2.2 for clarity and accuracy, and resolving citation formatting errors (e.g., \cite) to align with standard practices.

How you select web pages?

To select web pages, we started with the original Q-A pairs from existing datasets. For example, in the "Source" dimension, we used the CNN-English-News dataset, where the documents only satisfied the [news article] condition. To create the [forum post] condition, we searched for related answers on platforms such as Reddit and Quora using the same query. For the [blog] condition, we performed targeted searches on Google to find relevant blog posts.

How can you check the quality?

For GPT-generated documents, we follow a clear and structured guideline to ensure quality:

1. Positive Documents: We verify whether the generated document can effectively answer the query. Additionally, we check if it also satisfies the given instruction while maintaining relevance to the query.

2. Hard Negative Documents: We assess whether the generated document has a high semantic relevance to the query but fails to provide an answer. This ensures that the document poses a meaningful challenge for retrieval models.

Pi is unclear, what i refers to?

Pi refers to the specific gold document within the set of positive documents P associated with a query q. This document becomes the focus in the instructed mode, where a more specific instruction highlights it as the only relevant document. In contrast, the other documents in P are considered irrelevant or negative under this mode.

for reranking model, can you explain more how it works?

For reranking models, we use the top 100 results from E5-mistral for reranking. For point-wise setting, the model evaluates each query-document pair independently. Both the query and a single document are input into the model, and it outputs the probability of the document being relevant to the query. These probabilities are then used as similarity scores for ranking. For example, the model might determine the likelihood of "True" or "False" for each pair, which serves as the basis for re-ranking. For list-wise setting, this approach involves considering the entire list of candidate documents simultaneously. The list is provided as a prompt, often including the query and all candidate documents. The model processes the list and directly outputs the ranked order of document IDs.

Do you consider using general-purpose LLM for doing point-wise reranking? Like llama-3.2, mixtral, qwen2, etc.?

We use LLama-3.1 for doing point-wise reranking and in our future work, we plan to explore incorporating general-purpose large language models (LLMs) such as Llama-3.2, Mixtral, and Qwen2.

Did you change the default submission format, i.e., remove the anonymous author lines? I understand you want to save more spaces but I'm not sure if this is acceptable.

Thank you for explaining my questions regarding Sec 2.3. However, my main concern is the quality check (also pointed out by Reviewer 7Ty8). Who are the annotators? Any annotator biographies? Any inter-annotator agreement scores? Relying solely on GPT-4 might not be reliable.

After reading reviews from other reviewers, as pointed out by Reviewer S77t, it would be beneficial to explain more on:

• The overall motivation of the work -- instruction-following information retrieval and possible applications
• Document-level instructions might overlook some "detailed" features within documents

Who are the annotators? Any annotator biographies? Any inter-annotator agreement scores?

The annotators are graduate students specializing in computer science who were carefully selected and trained for this task. Each annotator underwent an English proficiency test to ensure they could accurately understand and evaluate the documents. The inter-annotator agreement score, calculated based on their judgments of whether a document is a positive or negative match for a given query, is 92%. This high agreement reflects the consistency and reliability of the annotations.

The overall motivation of the work -- instruction-following information retrieval and possible applications

The primary motivation of this work is to address the gap between user expectations for instruction-following capabilities in retrieval models and their current limitations. While large language models (LLMs) excel at generating content based on detailed instructions, retrieval systems have lagged behind, focusing primarily on content relevance without adequately considering user-specific instructions. This limits their applicability in scenarios where nuanced preferences, such as document format or audience, are critical.

Instruction-following retrieval improves LLM outputs by retrieving relevant knowledge documents, enhancing the accuracy and relevance of generated content in tasks like retrieval-augmented generation. For instance:

• If a user requires information aimed at experts, the retrieval system can pre-filter documents to surface content aligned with the expertise level, such as scientific papers or technical manuals.
• This reduces the need for the LLM to adapt generic content to an expert audience, improving both the efficiency and the quality of the generated output.
• Similarly, retrieving documents in specific formats (e.g., official manuals) ensures that the LLM generates text consistent with the preferred style or structure. By enabling retrieval systems to seamlessly align with user-defined instructions, such as audience type or document format, we aim to create a unified framework that optimizes both retrieval and subsequent LLM-based generation. This synergy is particularly valuable in applications like customer support, where domain-specific accuracy and relevance are paramount.

Document-level instructions might overlook some "detailed" features within documents

In our work, document-level instructions are not meant to replace content-level analysis but to complement it by refining retrieval based on user-defined preferences, such as audience, format, and language. However, as pointed out, document-level instructions may overlook more detailed features within the document itself, such as specific paragraphs or sections. To address this, future enhancements could introduce more fine-grained control over retrieval, such as focusing on particular document sections (e.g., paragraphs or tables), which would allow models to retrieve content at a higher level of detail. This could be achieved by fine-tuning models with hierarchical instructions that capture both document-level attributes and finer content-specific features.

In practice, however, users often prioritize broader document-level features over more granular content. In such cases, the combination of document-level retrieval models with supplementary systems, such as paragraph-level or section-based retrievers, could provide a more comprehensive approach to addressing finer granularity. By refining how instructions are aligned with document granularity, future work could make retrieval systems more robust, accurate, and tailored to user needs, offering a seamless integration of both broad and detailed search capabilities.

Could you provide an example of the type of "detailed feature" you have in mind? This would help us better understand your concern to answer it.

Thank you for your rebuttal. I haven't had time to check the updated manuscript will now. I cannot access the version before revision but I can see you avoid using some "magic" numbers in the updated manuscript. I think my questions are well addressed in authors' response and the updated manuscript. I've increased my score accordingly.

Some minor suggestions:

1. I like appendix B, can you somehow manage to put (some of) the analysis in appendix B into the main text?
2. It would be great to include the detailed annotator biographies/information in the appendix

The benchmark’s focus on instruction-following may overlook other critical aspects of retrieval quality, such as semantic understanding and contextual relevance. Overemphasizing instruction compliance may risk sidelining these equally important factors.

Existing benchmarks predominantly focus on semantic relevance and contextual understanding, which are indeed critical aspects of retrieval quality. However, there is currently a lack of datasets and evaluations specifically designed to assess instruction-following capabilities. The primary goal of our work is to bridge this gap by focusing on instruction compliance rather than semantic relevance alone. That said, our queries and documents are grounded in strong semantic foundations—derived from original datasets where queries and their corresponding positive documents are clearly semantically relevant. Building on this, we have augmented the data with carefully designed instructions to enable a finer level of control and analysis. This approach ensures that semantic understanding remains a core aspect while adding a new dimension for evaluating instruction compliance.

The InfoSearch benchmark relies heavily on synthetic data generation, primarily using LLMs like GPT-4 to create instruction-query pairs and rephrased documents. This reliance may introduce artificial biases in the data that do not accurately reflect real-world queries or instruction styles, potentially limiting the benchmark’s validity for evaluating real-world retrieval tasks.

For queries, we curated original Q-A pairs from existing datasets, categorized as follows:

• Audience: BioASQ, scifact
• Keyword: MSMARCO
• Format: Stackoverflow, Stack Exchange
• Language: publichealth-qa
• Length: medical_qa
• Source: CNN-english-news

The queries in MSMARCO are sourced from real user searches on Bing, while datasets like Stack Overflow and Stack Exchange also consist of questions posed by real users in real-world scenarios.

For instruction styles, the InfoSearch benchmark generates instructions that reflect real-world user interactions, such as specifying language preferences. For example, when users require information in a specific language, the instructions explicitly direct the retrieval system to provide responses in that language. A typical instruction for the Language dimension might be: "Please provide the answer in English." or its reversed form, "Please do not provide the answer in English." These instructions mirror scenarios where users might explicitly want content in a particular language, such as when a non-English speaker requests information in their native language or when a user prefers documentation in a global language like English for consistency.

While the benchmark tests models on reversely instructed queries (negative instructions), the analysis does not deeply explore how models interpret and handle negation or exclusion conditions. Understanding these nuances would be valuable for practical applications where users often specify what not to include in results.

We will illustrate with a specific example of e5-mistral search results.

Q: how many calories are in a martini

Adding instructions like [sentence], [non-sentence] and so on, generally lowered the overall semantic similarity between the query and document. However, documents with originally high similarity scores maintained relatively higher scores, indicating that the model did not fully understand the instruction. The instruction merely reduced the similarity across the board without meaningfully improving relevance for the specified condition.

What is the computational feasibility of using WISe and SICR metrics in large-scale production environments?

We would like to clarify that this paper focuses on offline evaluation rather than online retrieval. Specifically, we address the challenge by setting carefully designed instructions and reverse instructions for each query, ensuring that the evaluation is conducted in a controlled and consistent manner. This setup allows us to isolate and analyze the effectiveness of the methods in reranking without the variability introduced by online retrieval performance.

Dear Reviewer,

Happy Thanksgiving! Thank you for your thoughtful and detailed feedback on our paper. We believe we have addressed the key points you raised, but we are eager to know if our responses meet your expectations and if there are any remaining questions. Your continued insights would be greatly appreciated and instrumental in refining this work further.

Kind regards,

Author

Dear Reviewer,

As the deadline for the discussion period approaches, we would like to follow up to ensure that our latest results and clarifications address the concerns and questions you raised.

Regarding your concern about the "focus on instruction-following potentially overlooking other critical aspects of retrieval quality," we’d like to clarify the following:

Semantic relevance is the foundation of our benchmark. We ensure this by using high-quality query-document pairs sourced from real-world datasets such as MSMARCO and StackOverflow. This ensures that the relevance between queries and documents is well-established. Building on these high-quality Q-D pairs, we further design and annotate instructions, with strict quality assurance through manual validation and re-annotation. These instructions enable us to evaluate how well retrieval models comply with task-specific instructions, going beyond mere semantic relevance. We believe that InfoSearch is the first benchmark of its kind to evaluate instruction compliance in retrieval models. This includes embedding-based models, reranking models, and zero-shot LLMs for ranking tasks, providing a more comprehensive perspective compared to benchmarks that solely assess semantic relevance.

Regarding your concern on the quality of synthetic data, we implemented several strict validation measures. For Model-Assisted Coarse Validation, we used 12 retrieval models (e.g., bge, e5-mistral, NV-Embed) to evaluate generated positive documents. Documents were flagged if ranked below 50 or scored below 0.5 by six or more models, then manually reviewed to confirm their validity. Hard negative documents underwent the same process to ensure contextual similarity to queries while being definitively irrelevant. For Annotator Validation, graduate students in computer science with proven English proficiency reviewed the annotations, achieving an inter-annotator agreement score of 92%. Instruction Checks were conducted for five dimensions (excluding Keyword), with manual reviews yielding 99% accuracy. For the Keyword dimension, instructions were generated by extracting keywords, with an initial 3% error rate in a sample of 20 queries. Errors were corrected iteratively using LLMs, and a re-check confirmed that all positive documents contained the specified keywords.

If these updates resolve your concerns, would you mind adjusting your rating accordingly? Please let us know if there are any remaining issues we can address.

Thank you for your time and feedback!

Best regards,

Author

How were the number of instructions decided

In our dataset, except for "Keyword" dimension, the other five dimensions have fixed conditions and we create instructions based on varied conditions(see more details in Figure 1). For example, in the "Language" dimension, we have conditions of [English] and [Chinese], so we create instructions like "please give me English/Chinese answers." in instructed mode. For the special "Keyword" dimension, we create about 4-6 instructions for a core query. In the future, we will extend more dimensions and more conditions.

How was the number of document in the corpus decided

In our dataset, each dimension has 100 core queries (i.e., query with no instruction) and about 1000 documents (see more details in Table 1). For each core query, it has 2-5 positive documents and multiple hard/easy negative documents.

It seems like all documents are merged together from all subsets to create the corpus, how are we sure there are not false negatives in the other sets? A qualitative analysis or GPT-based analysis would be very valuable here.

Instead of merging all document subsets into a single corpus, we created a separate corpus for each dimension, as detailed in Table 1. Since some documents were rewritten using GPT, we conducted a quality check during the preparation phase to ensure reliability. During this test, we found some problems. 1. GPT returns invalid format; 2. GPT sometimes generates false negatives. So we carefully designed the prompt (see appendix) and manually checked the documents generated by GPT to avoid these problems.

How were these documents gathered, just any that were from the original pairs (but then where did the remainder come from?)

To ensure the naturalness of the queries and reduce production costs, we integrated conditions when filtering Q-A pairs from existing datasets or web pages. For the "Keyword" dimension, the MSMARCO dataset was utilized, as it provides multiple positive and negative documents for each query. For the other five dimensions, existing datasets often satisfy only one condition within a dimension. To create documents meeting other conditions, we leveraged GPT to rewrite documents.

Why is this true? I don't necessarily understand why the query is more relevant with the instruction, weren't these gathered from already relevant query and doc pairs? So I see no specific reason why the rank should increase with the instruction as compared to no-instruction.

For queries without instructions, all corresponding documents are treated as positive, as they are relevant to the query in a general sense. However, when instructions are added, they introduce specific requirements, making the relevance criteria more precise.

Here is an example:

• Core Q: how many calories are in a martini?
• Q+ins1: Q + Please give me a short answer.
• Q+ins2: Q + I'd like a paragraph explaining it.
• Q+ins3: Q + Please provide a detailed article.
• Doc1(sentence), Doc2(pargraph), Doc3(article)

Without instructions, all documents (Doc1, Doc2, Doc3) are relevant to the query "Q." When "ins1" is added, Doc1 should be ranked first because the instruction specifies a short answer. Similarly, with "ins2," Doc2 (a paragraph) should take priority, and with "ins3," Doc3 (an article) should be ranked highest. These examples highlight how instructions refine the relevance criteria and demand that the model adapt its ranking accordingly.

why does WISe not go to 0? Why use 20 as the number for the conditional?

When R_{ori} = R_{ins}, the penalty part of WISE is equal to 0, indicating perfect alignment between original and instruction-followed retrieval results. The value 20 is chosen based on the assumption that the top 20 results offer the most utility to users, balancing practical relevance and computational efficiency.

Doesn't WISe/SICR have the same problem as p-MRR in terms of rank of 1 for the original and instructed query?

We address the case where R_{ori} = R_{ins} = 1 by assigning a WISe/SICR score of 1. Additionally, WISe considers both the relative distance (R_{ori} - R_{ins}) and the absolute rank (R_{ins}) in its calculation, making it more sensitive to top-ranked changes and mitigating the limitations of p-MRR.

Working through some examples for WISe, that I found unusual:

• When R_{ori} = R_{ins} = 1, the condition R_{ori}< N and R_{ins} = 1 is satisfied, resulting in a WISE value of 1.
• The rapid decay of WISE values, as observed in questions 2 to 4, is caused by the term \frac{R_{ori} - R_{ins}}{K}. To address this, we revised the formula to \frac{\sqrt{R_{ori} - R_{ins}}}{K}, which ensures smoother score transitions. Figure 5 illustrates this improvement.

Thank you very much for your support and positive feedback! We sincerely apologize for the confusion caused earlier. We truly appreciate your valuable input and will ensure that the details you pointed out are clarified thoroughly in the revised version.

Regarding the modifications to the formula, the addition of the square root was introduced not only to ensure that the WISE score decreases monotonically as the positive rank (R_ori/R_ins) increases but also to make the score variation smoother. This adjustment addresses the issue you pointed out regarding the presence of a pretty large gap, improving the overall interpretability and consistency of the scoring mechanism.

Regarding the metric design, the historical development of our evaluation approach was as follows: initially, we did not pair the instructed mode and reversely instructed mode in our evaluation framework. However, during the course of our analysis, we encountered a significant issue where the ranking of positive documents moved in the same direction across both modes, leading to false positive responses. This observation prompted us to propose a paired evaluation metric, which ensures a more accurate assessment by considering the relationship between the two modes and accounting for shifts in rankings more effectively. We believe this refinement addresses key shortcomings and enhances the reliability of the evaluation process.