Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awareness

Thank you for your valuable and detailed feedback. We would like to address your questions as follows.

Q1: Notations

Thank you for the detailed suggestions on the notations. We will update the final version accordingly.

1. Neutral describes the root queries in the PIR task, which is contextually used to describe the non-perspective part of the user query.

2. q = p+r is q = (q,r). We will unify the notation and add an explanation.

3. S is a typo of the corpus C.

4. The knowledge part refers to the prompt in Appendix Section A.8 used for the experiments in Section 3.3, as mentioned in the last sentence of the second paragraph in Section 3.3.

Q2: Generative Perspectives for Retrieval

Thanks for the suggestions. We extend the experiment with the generative perspectives (50 queries per task, full corpus) on retrieval tasks to test the performance consistency, with Recall@5 as the metric. For each retriever (e.g., BM25), A - B denotes the retrieval performance with annotated perspectives (A) and generative perspectives (B)

From the table above we can observe that the retrieval performances from annotated and generative perspective extraction are similar across tasks and retrievers, which validates our claims in Section 3.4.

Q3: Existing research on cosine similarity

The authors indeed read the referred paper and found it interesting. It was put on ArXiv on March 8 and would be considered concurrent work under COLM policy, but we still intend to add a discussion in the final paper: We use cosine similarity for fair comparison since current retrievers such as DPR and Contriever are tuned on similarity from the cosine similarity function.

Q4: Few-shot IR

We speculate that the proposed PAP method can benefit from the few-shot IR setting. As in our discussion with Reviewer 8C7L, we can design a tunable parameter to decide how much perspective awareness to be involved per query, which can be tuned with the shots given to perform task-specific PAP.

Another potential benefit is to find the task-specific perspectives to improve computational efficiency, as described in Section 3.1.

Thank you for your valuable and detailed feedback and for liking our paper. We would like to address your questions as follows.

Q1: Downstream Tasks

We sincerely thank you for the insights into the choice of downstream tasks. In this paper, we focus on introducing the problem and designing the benchmark. We start with zero-shot PAP to improve current retrievers on the retrieval and downstream tasks that are currently within our included datasets (i.e., AmbigQA and Perspectrum).

We believe continuous effort shall be made to improve retrievers’ perspective awareness (e.g., add tunable parameters and mine automatic signals). We will apply the retrievers with high perspective awareness (which PIR can reveal) to other tasks. We will add a section describing the potential application to other tasks as a roadmap for the authors and other researchers, for example, TheoremQA [1]. We will also work on including these tasks in the PIR benchmark.

Q2: Writing details

Thanks for pointing out the typos and writing comments. We will update them in the final paper. For the intuition on other questions: One potential reason is that the LLMs can understand “two components” (where one is the “intent”) in the prompt in Appendix A.8. As shown in examples in Section A.1, the user meta-description (e.g., “Find a news article”) and general information (e.g., the source news content) can be easy to distinguish. We are experimenting with a zero-shot setting. We believe demonstrations can further stabilize the performance. We are also open to all prompting suggestions.

The large improvement over the no-PAP retrieval baseline is also interesting to the authors. We believe this is rooted in the decoupled relation between the retrieval and downstream task performance. The retrievers may find relevant but incorrect documents, which may hurt the instruction-following LLM performance compared to no-retrieval. We introduce some related readings in the last paragraph of Section 3.3. Together with these related papers, this improvement mismatch suggests our takeaway: beyond relevance that makes a retriever successful on retrieval tasks, we should be careful of the correctness of responding to the perspectives of user queries to make the whole RAG pipeline successful.

[1] Wenhu Chen, Ming Yin, Max Ku, Elaine Wan, Xueguang Ma, Jianyu Xu, Tony Xia, Xinyi Wang, and Pan Lu. 2023. Theoremqa: A theorem-driven question answering dataset. In EMNLP 2023.

Thank you for your valuable and detailed feedback and for liking our paper. We would like to address your questions as follows.

Q1: Beyond geometric approach

Thank you for the valuable insights on the potential methods. In this paper, we focus on introducing the problem and designing the benchmark. We believe continuous effort shall be made to improve retrievers’ perspective awareness. As also mentioned in our response to Reviewer 8C7L, one direction is introducing a tunable parameter to decide how much perspective awareness should be involved per query.

With the motivation from retrieval diversity work [1][2], we also work on how to automatically mine and generate diverse and contrastive retrieval signals to help train retrievers with high perspective awareness. We will extend the future work discussion in current Section 5 in the final paper.

Q2: Writing details

For the heuristics in creating the datasets, we introduce how we leverage and extend the design of source datasets to create the query-document pairs in Section A.2 in the appendix, with examples in Table 7. We will include a detailed introduction of the key reasons and unpack the mention in the final version.

The dependency issue is from the difference in the cases in the wild (general case) and in our dataset design (PIR). In the general case, retrievers only observe query (q), external tools can be applied to sample perspectives (p) and root queries (r) from q, such as the generative perspectives we describe in Sec 3.4

In PIR, to test the model perspective awareness, we design the dataset to have multiple p attached to each r. so that in the metric related to PIR (i.e., in Section 2.3), we write q = p+r, specifically for the PIR task. We will add an explanation of the setting differences in the final paper.

𝑞=𝑝+𝑟 describes the construction of our PIR dataset: To ensure that the perspectives are explicit and have contrastive pairs, we first sample perspectives (p) from existing documents (such that we know the gold document to retrieve for each p), then add them to a root query (r), such that r+p forms the final query (q) submitted to the retriever. In real-world scenarios, p and r tend to be implicit and can only be decomposed from q (the only observable data point) – a case we discussed such decomposition in Sec 3.4. We will explain the equation in more detail.

[1] https://plg.uwaterloo.ca/~gvcormac/novelty.pdf

[2] https://ieeexplore.ieee.org/document/7987038

Thank you for your valuable and detailed feedback and for liking our paper. We would like to address your questions as follows.

Q1: Details about the PAP method

Thank you for your valuable insights on the methodology side. By projection to the p space, we mean the hyperplane that is vertical to the perspective (p) vector. Reducing the query vector component on the perspective vector is equivalent to moving it toward the hyperplane (in the sense of cosine similarity). We will update the introduction of PAP in Section 3.1 with your suggestions.

We agree that a tunable coefficient is interesting and can help reveal the essence of the problem (relations between perspectives and the rest of the query). In the scope of the current draft, we focus on introducing the task and building zero-shot methods. However, we are indeed exploring the coefficient tuning and will update the resources in our code base to be publicized upon acceptance.

Q2: Notation and Baseline

Thanks for the suggestion on the notation. We will improve the final version as suggested. We design dual-sum and tri-sum for the comprehensiveness of possible operations on the similarity score level, while others, such as add and concatenation, are operations on the vector/embedding level.

Q3: Clarification on SimCSE

SimCSE is a sentence embedding model with two versions: the unsupervised one is fine-tuned with zero-shot contrastive signals from dropouts; the supervised one is further fine-tuned on NLI tasks. We mean that SimCSE is not specifically fine-tuned on retrieval tasks (In contrast, the Contriever model used in our paper is further tuned on MS-MARCO, a retrieval task). We will add a detailed explanation in the final version.