Retrieving Texts by Abstract Descriptions

We appreciate your detailed evaluation of our submission and the constructive feedback you provided. The recognition of the improved results we find in both human and automatic evaluations motivates the need to tailor models to our notion of similarity. We agree that this work underscores the potential of utilizing large language models for crafting tailored training datasets. Regarding the issue you raised with the baselines, we appreciate the suggestion to evaluate on additional existing encoders. Please see our general response (“Baselines”) for additional results on 4 existing SOTA models.

We appreciate your engagement with our work and would like to address the claim that our proposed approach merely addresses a specific instance of text-to-text similarity rather than introducing a fundamentally new form of similarity-based search.

The retrieval in this paper appears specialized in a specific domain and not universally applicable. Indeed, we do not aim to propose a “universally applicable” solution, and doubt that such a solution exists. Which current similarity approach would you consider universal in the sense that you mention? The very core of this paper is pointing out major limitations in the corpus-based, “universal” notion of similarity, which is ill-defined. We empirically show that models trained without an explicit notion of similarity fail to generalize to the very natural use case of retrieving text by abstract description. As you noted, “the description-based similarity also belongs to the similarity between texts”. We did not claim otherwise. This work is focused on similarity between texts, and our contribution is in introducing and formally defining a unique notion of similarity which is as useful as it is not addressed by current approaches.

The retriever itself lacks novelty: We clarify that our primary contribution lies not in the intricacies of modeling but in drawing attention to a critical limitation in existing methodologies. True innovation, in our view, at times does not come from improving conventional modeling techniques. Rather, it involves reevaluation of implicit, unexplored blind spots within existing frameworks. Our work addresses the inadequacy of the current understanding of similarity, which we posit is a substantial and crucial contribution, equal in importance to advancements on the modeling side. We respectfully disagree with the assumption that innovation must come in the form of a new model. Please also refer to the general response for additional explanation about the innovation in this work.

Different encoders: Please note that Colbert is a late-interaction model, whose application is orthogonal to our approach. Please see our general response for results on 4 additional SOTA encoders.

Hallucination in GPT-generated descriptions: The concern is justified. However, our dataset is not a goal by itself. Rather, it is used to train an encoder, which we then thoroughly evaluate in both human and automatic evaluation. As such, even if the dataset contains biases of all sorts (which it probably does, like any other dataset), it is still useful for tackling the problem of enabling search by abstract descriptions. At the same time, please note that as we describe in the paper, we did conduct an human evaluation study of the generated descriptions, focusing on their faithfulness to the original text. As we report in the paper this human evaluation indicated that the descriptions are largely of high quality and are faithful to the text. Could you please explain if there are unconvincing aspects in our evaluation?

We appreciate your engagement with our submission and the constructive feedback provided. The recognition of our paper's fresh perspective, novel description-based matching approach, rigor, and comprehensiveness is encouraging. We look forward to addressing the concerns you raised:

Risks of Using Large Language Models: The concern you raise is justified. However, our dataset is not a goal by itself. Rather, it is used to train an encoder, which we then thoroughly evaluate in both human and automatic evaluation. As such, even if the dataset contains biases of all sorts (which it probably does, like any other dataset, human-generated or not), it is still useful for tackling the problem of enabling search by abstract descriptions. Please note that as we describe in the paper, we did conduct a human evaluation study of the generated descriptions, focusing on their faithfulness to the original text. This human evaluation indicated that the descriptions are largely of high quality and are faithful to the text. Furthermore, human evaluation is also performed to evaluate the results retrieved by the final model trained on the GPT-generated data, and this model is significantly better than any of the baselines (figure 2).

Fairness in Comparison with Baselines: We emphasize that our modification of the dataset is a deliberate choice to highlight the limitations of existing frameworks and address the specific challenges of abstract semantic information retrieval. We see it essential to point out to the fact that the SOTA models fail to capture this notion of similarity, despite them being trained on orders of magnitude more data. In other words, we empirically show the limitations of existing SOTA models in capturing the exact sense of similarity we are arguing for. Please see also “innovation” under the general response.

Theoretical Support for Description-Based Relevance and Practical Need for Abstract Descriptions: Our emphasis is on the conceptual innovation rather than the modeling side: we explicitly state in our introduction that our goal is to introduce a unique form of similarity—description-based similarity. This entails defining a specific notion of retrieval by similarity, and describing its usefulness in certain information-seeking scenarios. Please also refer to the sections “innovation” and “Use Cases of Description-Based Search” in our general response.

Inclusion of Strong Dense Retrieval Baselines: We added 4 SOTA text encoders to the automatic evaluation - please refer to the general response under “Baselines”.

How to make sure abstract description is what we actually need in practical search? The idea to perform a user study that empirically evaluates the kind of queries end-users are interested in is highly interesting. Such study has the potential to uncover specific use cases where the ability to search by abstract descriptions is particularly beneficial. We believe, however, that it is beyond the scope of this work: we do discuss in detail several naturally occurring scenarios where such ability is warranted, and the primary focus of our work is to showcase the limitations of existing methods in fulfilling this requirement and to propose a language model-assisted technique to address it. While a user study could offer supplementary insights, we view it as a distinct contribution, separate from the specific goals outlined in our present work.

We appreciate your review of our submission and the constructive feedback provided. We highly value the recognition of our proposed novel approach to retrieval by description, which is distinct from regular search methods. Additionally, we appreciate the acknowledgment of our comprehensive evaluation strategy, which include both human and automatic evaluation.

Grammatical Errors: Thanks for pointing out grammatical errors. We apologize for any oversight in our proofreading. In our revised manuscript, we addressed the issues you raised, and we will revise the manuscript again for the CR version.

Use Cases of Description-Based Search: Please refer to the general response under “Use Cases of Description-Based Search” and "Innovation" for a detailed response.

We appreciate the insightful feedback you provided, particularly the acknowledgment of the description-based similarity as an interesting type of similarity and its potential value to the information retrieval field. We refer to the specific issues you raised:

Exploration of Applications: We appreciate your acknowledgment of the interesting nature of our proposed task but recognize the need for a more detailed exploration of its potential applications. Please refer to the general response under “Use Cases of Description-Based Search” and under “Innovation” for a detailed response.

Expected Outperformance of the Proposed Method: Our emphasis is on drawing attention to the need to retrieve by description, and showcasing the limitations of existing encoders in this task. We find it essential to test models on our data to showcase the limitations of SOTA general-purpose encoders in capturing the notion of similarity we argue is useful. Their failure is a nontrivial finding, as these models were trained on orders of magnitude more data. It is not possible to highlight this inherent limitation of existing models without evaluating them on data tailored for our notion of similarity. Please refer also to our explanation under “innovation” in the general response.

How is precision @k decreases when k is increasing? Indeed,as we note in the paper, the improvement over the baselines is smaller when k increases. Our hypothesis is that this is a by-product of the standard contrastive learning objectives we employ, which focus on the top result, and do not directly optimize the tail of the distribution (in other words, the loss takes into account a single positive at a time). We agree this is an interesting phenomenon to study in future work (particularly - can we design contrastive learning methods that take the tail of the distribution into account?).